dunzhang/stella_en_400M_v5 · Error in running the model in CPU

Sep 27

while running the same code below for bge-large-en-v1.55 and gte-large-en-v1.5. I didn't faced any error. But for this model I am facing the error

from sentence_transformers import SentenceTransformer
from sentence_transformers.util import cos_sim
from tqdm.autonotebook import tqdm, trange
import torch 
device = torch.device("cpu")
sentences = ['That is a happy person', 'That is a very happy person']

model = SentenceTransformer('dunzhang/stella_en_400M_v5', 
                            trust_remote_code=True,
                           device ="cpu"
                            )
embeddings = model.encode(sentences)
print(cos_sim(embeddings[0], embeddings[1]))

Navanit-AI

Sep 27

The error is

    "name": "ValueError",
    "message": "Attention bias and Query/Key/Value should be on the same device
  query.device: cpu
  attn_bias   : cuda:0

@infgrad Kindly help please.

ASKowa

29 days ago

Did you find a solution?

jdchmiel

20 days ago

in config.json change both to false:
"unpad_inputs": true,
"use_memory_efficient_attention": true,
and then you also do not need xformers package.
It then runs on CPU