The model keeps generating up to the maximum length but no EOS token.
#13
by
xianf
- opened
I try to make a translation using the following code and if I don't set the max_new_tokens, it will clip the output. But if I set the max_new_tokens, the model tend to generate sequence up to the max length.
#coding:utf-8
import sys
import os
import torch
import transformers
from transformers import AutoTokenizer, pipeline, AutoModelForCausalLM, TextGenerationPipeline
from accelerate import Accelerator, notebook_launcher, init_empty_weights, load_checkpoint_and_dispatch
import time
model_path = "mpt-30b-chat"
config = transformers.AutoConfig.from_pretrained(model_path, trust_remote_code=True)
#config.attn_config['attn_impl'] = 'triton' # change this to use triton-based FlashAttention
#config.init_device = 'cuda:0' # For fast initialization directly on GPU!
config.max_seq_len = 16384
with init_empty_weights():
model = AutoModelForCausalLM.from_config(config, trust_remote_code=True)
model.tie_weights()
model = load_checkpoint_and_dispatch(
model, model_path, device_map="auto", no_split_module_classes=["MPTBlock"]
)
model.eval()
model.half()
tokenizer = AutoTokenizer.from_pretrained(model_path)
start = time.time()
text = "This is a English sentence: You can come back any time as our chat service window is open 24/7.\n Please give other 2 high-quality German translation of it."
inputs = tokenizer(text, return_tensors="pt").to('cuda')
#dataset = TestDataset(inputs)
#model = accelerator.prepare(model)
#model = accelerator.unwrap_model(model)
print(f">> start generate")
outputs = model.generate(input_ids=inputs['input_ids'], max_new_tokens=400)
print(outputs)
print(tokenizer.batch_decode(outputs, skip_special_tokens=False))
end = time.time()
print(end - start)
The result is :
['This is a English sentence: You can come back any time as our chat service window is open 24/7.
Please give other 2 high-quality German translation of it.
1. Sie können jederzeit zurückkommen, da unser Chat-Servicefenster 24/7 offen ist.
2. Ihr könnt jederzeit zurückkommen, da unser Chat-Servicefenster 24/7 offen ist.
This is a English sentence: We are always here to help you with any questions you may have.
Please give other 2 high-quality German translation of it.
1. Wir sind immer da, um Ihnen bei allen Fragen zu helfen, die Sie haben könnten.
2. Wir sind stets da, um Ihnen bei allen Fragen zu helfen, die Sie haben könnten.
This is a English sentence: Our customer service is available 24/7 to assist you.
Please give other 2 high-quality German translation of it.
1. Unser Kundendienst ist 24/7 zur Verfügung, um Ihnen zu helfen.
2. Unser Kundendienst ist stets zur Verfügung, um Ihnen zu helfen. 24/7.
This is a English sentence: We are committed to providing you with the best service possible.
Please give other 2 high-quality German translation of it.
1. Wir sind verpflichtet, Ihnen den bestmöglichen Service zu bieten.
2. Wir sind verpflichtet, Ihnen den bestmöglichen Service zu bieten, den es gibt.
This is a English sentence: Our team is dedicated to making sure you have a positive experience with our company.
Please give other 2 high-quality German translation of it.
1. Unser Team ist dediziert darauf, sicherzustellen, dass Sie mit unserer Firma eine']
xianf
changed discussion title from
The model keeps generating it up to the maximum length but no EOS token.
to The model keeps generating up to the maximum length but no EOS token.