google/gemma-7b · Weird completions

Feb 23

Is anyone also having weird completions with the model?

For example, I gave a simple few-shot example of a sentiment classification task and it is generating completions such as:
"786786786786786786786786786786786786786786786786786786786786786786786786786786786786786786786786786786786786786786786786786786786786786786786786786786"

Also, accuracy on Winogrande is close to random (53%).

EarthWorm001

Feb 23

I encountered similar issues: https://huggingface.co/google/gemma-7b/discussions/33

suryabhupa

Google org Feb 24

Can you send us the particular prompts you're using (I imagine this is with the pretrained model)?

rodrigo-nogueira

Feb 26

I was accidentally using add_special_tokens=False, which does not properly prepend a <bos> token (token_id=2) to the sequence:

tokenizer("I like", return_tensors="pt", add_special_tokens=False)
>>> {'input_ids': tensor([[235285,   1154]]), 'attention_mask': tensor([[1, 1]])}

tokenizer("I like", return_tensors="pt", add_special_tokens=True)
{'input_ids': tensor([[     2, 235285,   1154]]), 'attention_mask': tensor([[1, 1, 1]])}

Just to be clear, using add_special_tokens=True solved the problem.

rodrigo-nogueira changed discussion status to closed Feb 26