Spaces:
Running
Running
adding readme.md
Browse files
app.py
CHANGED
@@ -1,6 +1,7 @@
|
|
1 |
import streamlit as st
|
2 |
import os
|
3 |
import torch
|
|
|
4 |
from transformers import AutoTokenizer
|
5 |
from jax import numpy as jnp
|
6 |
import json
|
@@ -89,3 +90,10 @@ if query:
|
|
89 |
)
|
90 |
|
91 |
st.image(image_paths)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
import streamlit as st
|
2 |
import os
|
3 |
import torch
|
4 |
+
from pathlib import Path
|
5 |
from transformers import AutoTokenizer
|
6 |
from jax import numpy as jnp
|
7 |
import json
|
|
|
90 |
)
|
91 |
|
92 |
st.image(image_paths)
|
93 |
+
|
94 |
+
|
95 |
+
def read_markdown_file(markdown_file):
|
96 |
+
return Path(markdown_file).read_text()
|
97 |
+
|
98 |
+
intro_markdown = read_markdown_file("readme.md")
|
99 |
+
st.markdown(intro_markdown, unsafe_allow_html=True)
|
readme.md
ADDED
@@ -0,0 +1,27 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
# Italian CLIP
|
2 |
+
|
3 |
+
....
|
4 |
+
|
5 |
+
# Novel Contributions
|
6 |
+
The original CLIP model was trained on 400millions text-image pairs; this amount of data is not available for Italian and the only datasets for captioning in the literature are MSCOCO-IT (translated version of MSCOCO) and WIT. To get competitive results we follewed three directions: 1) more data 2) better augmentation and 3) better training.
|
7 |
+
|
8 |
+
## More Data
|
9 |
+
|
10 |
+
## Better Augmentations
|
11 |
+
|
12 |
+
## Better Training
|
13 |
+
|
14 |
+
different optimizer and backbone freezing
|
15 |
+
|
16 |
+
# Scientific Validity
|
17 |
+
To better understand how well our clip-italian model works we run an experimental evaluation. Since this is the first clip-based model in Italian, we used the multilingual CLIP model as a comparison baseline.
|
18 |
+
|
19 |
+
We selected two different tasks:
|
20 |
+
+ image-retrieval
|
21 |
+
+ zero-shot classification
|
22 |
+
|
23 |
+
## Image Retrieval
|
24 |
+
|
25 |
+
## Zero-shot classification
|
26 |
+
|
27 |
+
# Broader Outlook
|