tiiuae

/

falcon-7b-instruct

like 908

Can you share the line of code where you load the model?

from transformers import AutoTokenizer
import transformers
import torch
model = "tiiuae/falcon-7b-instruct"
tokenizer = AutoTokenizer.from_pretrained(model)

pipeline = transformers.pipeline(
"text-generation",
model=model,
tokenizer=tokenizer,
torch_dtype=torch.bfloat16,
trust_remote_code=True,
device_map="auto",
)

sequences = pipeline(
"Write a poem about Valencia",
max_length=200,
do_sample=True,
top_k=10,
num_return_sequences=1,
eos_token_id=tokenizer.eos_token_id,
)

for seq in sequences:
print(f"Result: {seq['generated_text']}")

Strange! it works in my environment, try loading the model as

model = AutoModelForCasualLM("tiiuae/falcon-7b-instruct", torch_dtype=torch.bfloat16, trust_remote_code=True, device_map="auto", )

What is the size of RAM your device has?

And can you please share the full code which worked.

Only 16 GB, if you feel the issue is with RAM, try increasing the size of swap file (page file in windows).

However if RAM is the case, usually python process get terminated. Error you posted is related to identify the model type while loading it. just a wild guess, try removing the downloaded model from the cache and run the script again

And can you please share the full code which worked.

Sorry, I may not able to share the code. :(
However I could able to run the code snippet you've shared

I also have 16 gb ram, the above code which you told, should have AutoModelForCausalLM.from_pretrained right?
And I have already written the below in pipeline:
torch_dtype=torch.bfloat16, trust_remote_code=True, device_map="auto"

Could you run the code snippet shared or you had to make some modifications

I made the modification you suggested and I got the following error.

ValueError: The current device_map had weights offloaded to the disk. Please provide an offload_folder for them. Alternatively, make sure you have safetensors installed if the model you are using offers the weights in this format.

Could you run the code snippet shared or you had to make some modifications

I ran the one you have shared. Well I had to fix indentations (in the last line), other than that I did not do any change.

Ok, did it give the expected output of a poem?

Yep. It gave a nice poem

Which environment you used, I am using vs code

Well mine was on a cloud instance, access though the terminal. I do not think issue is related to the environment.

let's see if some one else who has faced the same issue replies to this

Okay Thanks

I made the modification you suggested and I got the following error.

ValueError: The current device_map had weights offloaded to the disk. Please provide an offload_folder for them. Alternatively, make sure you have safetensors installed if the model you are using offers the weights in this format.

I am getting the same error

The model is over 16GB so you will need to figure out where the remainder can be stored. If you dont have GPU, then it must be stored on hard drive, so the error you are getting is saying that you need to identify some folder for it store the spillover from the model.