What’s the context size?
Howdy! I’m very interested in testing this model out. Would you mind letting me know what’s the context size for this it? Is it 32k? Thank you in advance for your answer!
It should be 32k yes!
I need to remake it tho, there was one dataset badly formatted, so it sometime spit out "*" character without reason.
But I think it's the only problem, you should be good, I will retrain later as it cost a little haha
Awesome, thank you so much for the answer and for such a quick one too! I’m gonna do a 4.65 exl2 quant for it quickly then and jump into testing! Thanks for the heads up!
Thank you, so in oobabooga we should set n_ctx to 32768, but what about rope_freq_base ? By default n_ctx is 200 000 and rope_freq_base is 5 000 000 (which I guess is high because of the hight n_ctx)