grounded_generation_template support?
#3
by
aw49cj2xf
- opened
In the original model card they mention grounded_generation_template. Does this exl2 version support it? If so, how to employ grounded_generation_template in ExllamaV2?
The grounded generation stuff is just a particular prompt format that the model is trained to work with. This will work with the EXL2 model as long as you format the prompt accordingly. You can use a Transformers AutoTokenizer as in the example in the readme, calling apply_grounded_generation_template to get the prompt which you then forward to ExLlamaV2 for inference.
Thank you very much. It works
aw49cj2xf
changed discussion status to
closed