How to make the model only output the "response" part?
#1
by
buchylx
- opened
Hi mate, when I use the example code, I realized that the output of the model is [prompt + response], how can I force the model only output its response?
Thank you
And also, is it possible to get stream output?
Answer to Q1: we can add a parameter "return_full_text=False" when we call it.
Yeah return_full_text is what I'd recommend.
For streaming output, that's a bit more complex. Have a look at how text-generation-webui does it, in files modules/text_generation.py. starting line 281 in the current file. It's not too complex.