Missing Output problem

#190
by chaydaroglu - opened

When I process 12k document OCR output through for loop to LLM, some of the responses are not completed. But when I input individually, they return the correct value. What could be the reason for this problem (ps. max token size is 1024) should i increase the max token size or decrease it?

Same here! Any updates ?

Mistral AI_ org

Hi, Im not sure if Im understanding, the responses are being cut ? Thats most likely because of the max new token yeah.

Same problem

With max_new_tokens, response in just empty string
With min_new_tokens, response is cut and doesn't follow instructions

Mistral AI_ org

Same problem

With max_new_tokens, response in just empty string
With min_new_tokens, response is cut and doesn't follow instructions

What value do u put for max new tokens?

Sign up or log in to comment