it just keeps generating <pad> tokens
#1
by
NePe
- opened
I have no idea why but whatever i try it only generates pad tokens :(
Same thing here and same with every else's 4 bit quants too, even with dev branch of transformers
I don't think anybody's testing their quants before posting them
Checked by downloading the full version and loading it with bnb 4bit, it produces the same results. It seems like its a hf/model issue not this specific upload.
Checked by downloading the full version and loading it with bnb 4bit, it produces the same results. It seems like its a hf/model issue not this specific upload.
use torch_dtype=torch.bfloat16
https://huggingface.co/google/gemma-2-27b-it/discussions/16