Fine-tuning Florence-2 - Microsoft's Cutting-edge Vision Language Models
β’
103
Merve Noyan PRO
merve
AI & ML interests
VLMs, vision & co
Articles
Organizations
Posts
41
Post
2258
Just shipped: introduction to vision language models (aka image-text-to-text) https://huggingface.co/tasks/image-text-to-text
Learn about more machine learning tasks at https://huggingface.co/tasks
Learn about more machine learning tasks at https://huggingface.co/tasks
Post
5374
Fine-tune Florence-2 on any task π₯
Today we release a notebook and a walkthrough blog on fine-tuning Florence-2 on DocVQA dataset @andito @SkalskiP
Blog: https://huggingface.co/blog π
Notebook: https://colab.research.google.com/drive/1hKDrJ5AH_o7I95PtZ9__VlCTNAo1Gjpf?usp=sharing π
Florence-2 is a great vision-language model thanks to it's massive dataset and small size!
This model requires conditioning through task prefixes and it's not as generalist, requiring fine-tuning on a new task, such as DocVQA π
We have fine-tuned the model on A100 (and one can also use a smaller GPU with smaller batch size) and saw that model picks up new tasks π₯Ή
See below how it looks like before and after FT π€©
Play with the demo here andito/Florence-2-DocVQA πββοΈ
Today we release a notebook and a walkthrough blog on fine-tuning Florence-2 on DocVQA dataset @andito @SkalskiP
Blog: https://huggingface.co/blog π
Notebook: https://colab.research.google.com/drive/1hKDrJ5AH_o7I95PtZ9__VlCTNAo1Gjpf?usp=sharing π
Florence-2 is a great vision-language model thanks to it's massive dataset and small size!
This model requires conditioning through task prefixes and it's not as generalist, requiring fine-tuning on a new task, such as DocVQA π
We have fine-tuned the model on A100 (and one can also use a smaller GPU with smaller batch size) and saw that model picks up new tasks π₯Ή
See below how it looks like before and after FT π€©
Play with the demo here andito/Florence-2-DocVQA πββοΈ
Collections
22
spaces
98
No application file
π
Florence 2 DocVQA
Running
on
Zero
1.04k
π¨π»βπ€
ChatGPT Prompt Generator
Running
on
Zero
35
π¨
Paligemma Tracking
Running
on
Zero
61
β‘
Owl Tracking
Powerful foundation model for zero-shot object tracking
Running
on
Zero
46
π
Paligemma Doc
Try PaliGemma on document understanding tasks
Running
on
Zero
36
π
BLIP2 with transformers
BLIP2 (cutting edge image captioning) in π€transformers
models
83
datasets
22
merve/YouCook2
Viewer
β’
Updated
β’
2k
β’
2
merve/faiss_embeddings
Updated
merve/pokemon-ds-embeddings
Viewer
β’
Updated
β’
833
β’
3
β’
4
merve/tr-h4-norobots
Updated
β’
2
β’
10
merve/lego_sets_latest
Viewer
β’
Updated
β’
61
β’
4
β’
1
merve/ai-tube-dummy
Updated
β’
1
merve/my-blog-images
Viewer
β’
Updated
β’
1
β’
1
β’
1
merve/turkish_instructions
Viewer
β’
Updated
β’
51.6k
β’
106
β’
33
merve/ner-flags
Updated
merve/xlm-roberta-large-df
Viewer
β’
Updated
β’
30
β’
2