For a context of at least 32K tokens which version on a 2x16GB Gpu Config?
#3
by
Kalemnor
- opened
Question as in the title. And what version for at least 16K Tokens?
@Kalemnor Hi there, take a look at my response here, there is no one answer as it depends on a few things but hopefully this will give you some good hints. You may be able to even run IQ4_XS if you don't mind taking a heavy hit on performance and you tweak a few options, otherwise you'll have to use smaller quants.