README.md

---
license: apache-2.0
language:
- en
tags:
- visual_bert
- vqa
- easy_vqa
---
# Visual BERT finetuned on easy_vqa
This model is a finetuned version of the VisualBERT model on the easy_vqa dataset. The dataset is available at the following [github repo](https://github.com/vzhou842/easy-VQA/tree/master/easy_vqa)

## VisualBERT
VisualBERT is a multi-modal vision and language model. It can be used for tasks such as visual question answering, multiple choice and visual reasoning.
For more info on VisualBERT, please refer to the [documentation](https://huggingface.co/docs/transformers/model_doc/visual_bert#overview)

## Dataset
The dataset easy_vqa, with which the model was fine-tuned, can be easily installed via the package easy_vqa:
```python
pip install easy_vqa
```

An instance of the dataset is composed of a question, the answer of the question (a label) and the id of the image related to the question.
Each image is 64x64 and contains a shape (rectangle, triangle or circle) filled with a single color (blue, red, green, yellow, black, gray, brown or teal) 
in a random position.

The questions of the dataset inquire about the shape (e.g. What is the blue shape?), the color of the shape (e.g. What color is the triangle?) 
and the presence of a particular shape/color in both affermative and negative form (e.g. Is there a red shape?).
Therefore, the possible answers to a question are: the three possible shapes, the eight possible colors, yes and no.

More information about the package functions which allow to load the images and the questions can be found in the dataset's [repo](https://github.com/vzhou842/easy-VQA/tree/master/easy_vqa)
as well an utility script to generate new instances of the dataset in case Data Augmentation is needed.

## How to Use
Load the image processor and the model with the following code:
```python
processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-finetuned-vqa")

model = VisualBertForQuestionAnswering.from_pretrained("daki97/visualbert_finetuned_easy_vqa")
```

## COLAB Demo
An example of the usage of the model with the easy_vqa dataset is available [here](https://colab.research.google.com/drive/1yQfmz6wiSasRl6z-DmP-X403r3lZFqQS#scrollTo=HeVnH8BKkYCI)