TGI Compatible
#5
by
MrAiran
- opened
Whenever I try to load it in TGI I end up with this error, llama2 managed to load without problems
https://github.com/huggingface/text-generation-inference
text-generation-inference | File "/opt/conda/lib/python3.9/site-packages/text_generation_server/utils/weights.py", line 129, in <listcomp>
text-generation-inference | w = [self.get_tensor(f"{p}.g_idx") for p in prefixes]
text-generation-inference | File "/opt/conda/lib/python3.9/site-packages/text_generation_server/utils/weights.py", line 66, in get_tensor
text-generation-inference | filename, tensor_name = self.get_filename(tensor_name)
text-generation-inference | File "/opt/conda/lib/python3.9/site-packages/text_generation_server/utils/weights.py", line 53, in get_filename
text-generation-inference | raise RuntimeError(f"weight {tensor_name} does not exist")
text-generation-inference | RuntimeError: weight model.layers.0.self_attn.q_proj.g_idx does not exist
text-generation-inference |