`rope_scaling` must be a dictionary with two fields
I switched from 3-8B to the new 3.1-8B and I got this error.
I upgraded transformers model and also upgraded trl and it resolved it.
But what is rope_scaling, how do I learn about it?
I am new to all this and want to grow so please be gentle as my silly ask.
pip install --upgrade transformers trl
worked for me
Thank you for the note. I was stuck with the rope_scaling issue when I executed the following code from HF. Your tip helped me to resolve the issues. Thanks again.
import transformers
import torch
model_id = "meta-llama/Meta-Llama-3.1-8B"
pipeline = transformers.pipeline(
"text-generation", model=model_id, model_kwargs={"torch_dtype": torch.bfloat16}, device_map="auto"
)
pipeline("Hey how are you doing today?") #******
Now, the command is running at the last line (where it has *****) but it looks like it has been running for a while (more than 5 mins). Is that expected?
Thanks for your help.
Suresh
I ran your code on colab, it took 13.5 seconds to generate answer
@Chan-Y - Wow. Thanks for doing it Chan. I was running it in Google Colab. Does this cause this problem in Colab?
It was giving this msg when it was stuck at the pipeline() command above.
Setting pad_token_id to eos_token_id:128001 for open-end generation.
/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py:1259: UserWarning: Using the model-agnostic default max_length (=20) to control the generation length. We recommend setting max_new_tokens to control the maximum length of the generation.
warnings.warn(
Sorry, it was this msg that appeared yesterday which I failed to mention it here.
Thanks for your help.