michaelfeil

/

ct2fast-mpt-7b-chat

like 1

There is no easy fix for this, a typical problem in any library.
As in https://github.com/michaelfeil/hf-hub-ctranslate2/blob/e236f006593fb00633f9874fe414c87bd9735813/hf_hub_ctranslate2/translate.py#LL309C1-L334C70
You might want to set end_token =["User“, „user“] or so.

thanks for the reply @michaelfeil .
i noticed that you can get perfect answers in the mpt-7b-chat space : https://huggingface.co/spaces/mosaicml/mpt-7b-chat
could you have a look at the app.py file.

maybe it can suggest a fix to the issue.

how would you re-write the inference example you mentioned in the readme to sort of resolve the issue even a bit.could you paste the updated inference script in your next reply?

say for eg , in the below script

from hf_hub_ctranslate2 import TranslatorCT2fromHfHub, GeneratorCT2fromHfHub
from transformers import AutoTokenizer

model_name = "michaelfeil/ct2fast-mpt-7b-chat"
model = GeneratorCT2fromHfHub(
# load in int8 on CUDA
model_name_or_path=model_name,
device="cuda",
compute_type="int8_float16",
)
outputs = model.generate(
text=["User: what is the largest mountain on mars? Bot:"],
max_length=256
)
print(outputs)

it would be great if the output was :
Bot : The largest mountain on mars in Olympus Mons.

It actually gives the right answer , but it then continues with some other text. I want it to stop at Olympus Mons. @michaelfeil

also you can check this notebook for eg @michaelfeil
https://colab.research.google.com/drive/1s2beg55SEV8ICHufhwcflYtUNskTX8kO

it ends the sentence perfectly!

You might set end_token=[„User“], as keyword to model.generate

@michaelfeil , i updated the code as follows

from hf_hub_ctranslate2 import TranslatorCT2fromHfHub, GeneratorCT2fromHfHub
from transformers import AutoTokenizer

model_name = "michaelfeil/ct2fast-mpt-7b-chat"
model = GeneratorCT2fromHfHub(
# load in int8 on CUDA
model_name_or_path=model_name,
device="cuda",
compute_type="int8_float16",
)
outputs = model.generate(
text=["User: what is the largest mountain on mars? Bot:"],
end_token=["User"],
max_length=256
)

print(outputs)

but i am still getting same continues output :

User and Bot are also no special tokens, they are just dummies I used for a bunch of other models.
Not sure why the stop tokens do not work in this case.

Can’t support, if you found a good solution, feel free to share it here.

Okay, stop tokens are ["<|im_end|>", "<|endoftext|>"])
And messages should be formatted in the style

f"<|im_start|>user\n{item[0]}<|im_end|>",
f"<|im_start|>assistant\n{item[1]}<|im_end|>",
]

See the app.py in mpt chat space.

@michaelfeil could you paste the whole script here if possible?

Sorry, can’t provide further help.