Optimal Load Parameters
#1
pinned
by
FelipeCODX
- opened
One thing I never quite know when using these LLMs is what the optimal load parameters are. Could someone lend me a hand with this or point me in the right direction to learn what those mean?
I think I found the documentation: https://github.com/oobabooga/text-generation-webui/wiki/04-%E2%80%90-Model-Tab
I've settled on these parameters, which seem to work well for a RTX 4090 paired with an i9-13900k. Any suggestions?
mradermacher
pinned discussion
Nope, I'm just quantizing the model. But in most cases, there are no prescribed "optimal" settings because these settings are also a matter of taste.