Is it only 3 tokens per second?
#2
by
mcian91
- opened
Or am I doing something wrong?
I usually get something like 20+ tokens per second on 13B. Remember your speed is going to depend on an assortment of factors. Would you mind telling us how your running the model? Perhaps what CPU/GPU you have?