Bug when deploying to Inference Endpoints

#14
by gpantalos - opened

I get this bug when deploying as is:

2023/10/19 20:07:32 ~ INFO | Using device GPU 2023/10/19 20:07:32 ~ INFO | No custom pipeline found at /repository/handler.py 2023/10/19 20:07:32 ~ 2023-10-19 18:07:32,283 | INFO | Initializing model from directory:/repository 2023/10/19 20:07:32 ~ KeyError: 'fuyu' 2023/10/19 20:07:32 ~ File "/app/huggingface_inference_toolkit/utils.py", line 261, in get_pipeline 2023/10/19 20:07:32 ~ self.pipeline = get_pipeline(model_dir=model_dir, task=task) 2023/10/19 20:07:32 ~ File "/opt/conda/lib/python3.9/site-packages/starlette/routing.py", line 682, in startup 2023/10/19 20:07:32 ~ Application startup failed. Exiting. 2023/10/19 20:07:32 ~ raise KeyError(key) 2023/10/19 20:07:32 ~ File "/opt/conda/lib/python3.9/site-packages/transformers/models/auto/configuration_auto.py", line 710, in __getitem__ 2023/10/19 20:07:32 ~ config_class = CONFIG_MAPPING[config_dict["model_type"]] 2023/10/19 20:07:32 ~ File "/opt/conda/lib/python3.9/site-packages/transformers/models/auto/configuration_auto.py", line 998, in from_pretrained 2023/10/19 20:07:32 ~ config = AutoConfig.from_pretrained(model, _from_pipeline=task, **hub_kwargs, **model_kwargs) 2023/10/19 20:07:32 ~ File "/opt/conda/lib/python3.9/site-packages/transformers/pipelines/__init__.py", line 705, in pipeline 2023/10/19 20:07:32 ~ File "/app/huggingface_inference_toolkit/handler.py", line 45, in get_inference_handler_either_custom_or_default_handler 2023/10/19 20:07:32 ~ File "/app/webservice_starlette.py", line 57, in some_startup_task 2023/10/19 20:07:32 ~ await handler() 2023/10/19 20:07:32 ~ await self._router.startup() 2023/10/19 20:07:32 ~ async with self.lifespan_context(app) as maybe_state: 2023/10/19 20:07:32 ~ File "/opt/conda/lib/python3.9/site-packages/starlette/routing.py", line 705, in lifespan 2023/10/19 20:07:32 ~ Traceback (most recent call last): 2023/10/19 20:07:32 ~ return HuggingFaceHandler(model_dir=model_dir, task=task) 2023/10/19 20:07:32 ~ File "/opt/conda/lib/python3.9/site-packages/starlette/routing.py", line 584, in __aenter__ pqshc 2023-10-19T18:07:32.285Z  2023/10/19 20:07:32 ~ hf_pipeline = pipeline(task=task, model=model_dir, device=device, **kwargs) 2023/10/19 20:07:32 ~ File "/app/huggingface_inference_toolkit/handler.py", line 17, in __init__ 2023/10/19 20:07:32 ~ inference_handler = get_inference_handler_either_custom_or_default_handler(HF_MODEL_DIR, task=HF_TASK)

could you upload a step by step tutorial for deploying to inference endpoints?
Thanks

Hey! That is expected it's not supported in inference endpoint yet

Ok thanks, please let me know when they are.

Sign up or log in to comment