Float16 and Int8 Produce Wrong Results
I was not able to get correct results by following the README.MD. Even the provided example of language translation does not work correctly for float16 and int8 use cases. Did anyone manage to get it working?
Could you provide the outputs that you obtained? (and the full reproduction script)
check out this PR https://huggingface.co/google/flan-t5-xxl/discussions/5
Please use the main branch of transformers - see: https://github.com/huggingface/transformers/pull/20760
pip install git+https://github.com/huggingface/transformers.git@main
@ybelkada has this made it into a release yet or do I still need to install from source?
Hello
@deathcrush
,
I think so yes, can you try pip uninstall transformers && pip install --upgrade transformers and let me know?
@ybelkada , the latest release generates "Wie alt sind Sie?" when prompted "translate English to German: How old are you?", so works correctly for this example. I loaded google/flan-t5-xxl in 8 bit mode.
Awesome! Thanks for double checking!