| tags: | |
| - image-to-text | |
| - image-captioning | |
| - endpoints-template | |
| license: bsd-3-clause | |
| library_name: generic | |
| # Fork of [salesforce/BLIP](https://github.com/salesforce/BLIP) for a `image-captioning` task on 🤗Inference endpoint. | |
| This repository implements a `custom` task for `image-captioning` for 🤗 Inference Endpoints. The code for the customized pipeline is in the [pipeline.py](https://huggingface.co/florentgbelidji/blip_captioning/blob/main/pipeline.py). | |
| To use deploy this model a an Inference Endpoint you have to select `Custom` as task to use the `pipeline.py` file. -> _double check if it is selected_ | |
| ### expected Request payload | |
| ```json | |
| { | |
| "image": "/9j/4AAQSkZJRgABAQEBLAEsAAD/2wBDAAMCAgICAgMC....", // base64 image as bytes | |
| } | |
| ``` | |
| below is an example on how to run a request using Python and `requests`. | |
| ## Run Request | |
| 1. prepare an image. | |
| ```bash | |
| !wget https://huggingface.co/datasets/mishig/sample_images/resolve/main/palace.jpg | |
| ``` | |
| 2.run request | |
| ```python | |
| import json | |
| from typing import List | |
| import requests as r | |
| import base64 | |
| ENDPOINT_URL = "" | |
| HF_TOKEN = "" | |
| def predict(path_to_image: str = None): | |
| with open(path_to_image, "rb") as i: | |
| image = i.read() | |
| payload = { | |
| "inputs": [image], | |
| "parameters": { | |
| "do_sample": True, | |
| "top_p":0.9, | |
| "min_length":5, | |
| "max_length":20 | |
| } | |
| } | |
| response = r.post( | |
| ENDPOINT_URL, headers={"Authorization": f"Bearer {HF_TOKEN}"}, json=payload | |
| ) | |
| return response.json() | |
| prediction = predict( | |
| path_to_image="palace.jpg" | |
| ) | |
| ``` | |
| Example parameters depending on the decoding strategy: | |
| 1. Beam search | |
| ``` | |
| "parameters": { | |
| "num_beams":5, | |
| "max_length":20 | |
| } | |
| ``` | |
| 2. Nucleus sampling | |
| ``` | |
| "parameters": { | |
| "num_beams":1, | |
| "max_length":20, | |
| "do_sample": True, | |
| "top_k":50, | |
| "top_p":0.95 | |
| } | |
| ``` | |
| 3. Contrastive search | |
| ``` | |
| "parameters": { | |
| "penalty_alpha":0.6, | |
| "top_k":4 | |
| "max_length":512 | |
| } | |
| ``` | |
| See [generate()](https://huggingface.co/docs/transformers/v4.25.1/en/main_classes/text_generation#transformers.GenerationMixin.generate) doc for additional detail | |
| expected output | |
| ```python | |
| ['buckingham palace with flower beds and red flowers'] | |
| ``` | |