Optimal Load Parameters

#1
by FelipeCODX - opened

One thing I never quite know when using these LLMs is what the optimal load parameters are. Could someone lend me a hand with this or point me in the right direction to learn what those mean?

image.png

I've settled on these parameters, which seem to work well for a RTX 4090 paired with an i9-13900k. Any suggestions?

image.png

mradermacher pinned discussion

Nope, I'm just quantizing the model. But in most cases, there are no prescribed "optimal" settings because these settings are also a matter of taste.

Sign up or log in to comment