About Fine Tuning the Florence 2 model on naver-clova-ix/cord-v2 dataset.
I'm completely new to fine tuning VLM models. but want to learn how to perform fine tuning on VLMs. i want to fine tune this model on naver-clova-ix/cord-v2 data set for Image to JSON purposes. Need Guidance.
@Kaustubh-kapare94 this blog article on fine-tuning florence-2 might help you: https://huggingface.co/blog/finetune-florence2 From there you probably only need to adapt the dataset implementation. If you want to generate the bounding box labels from the dataset make sure to convert the coordinates into the bins Florence-2 is using to describe locations and revert those back to coordinates after the generation.
I've read this article before but the naver-clova-ix/cord-v2 dataset is completely different...I've fine-tuned this model obejct detection purpose as well...but this dataset is completely different from what I've tackled before
From what I see it has object detection data in it (boxes in x1y1x2y2x3y3x4y4) format that you could translate into text for Florence fine-tuning e.g. <target>title text<loc_142><loc_142><loc_142><loc_142><loc_142><loc_142><loc_142><loc_142><target>...or if you want to predict the format structure in the dataset you could do sth like <s_menu><s_nm> Nasi Campur Bali</s_nm><s_cnt> 1 x</s_cnt><s_price> 75,000</s_price> <sep/><s_nm> Bbk Bengil Nasi</s_nm><s_cnt> 1 x</s_cnt><s_price> 125,000</s_price>.... Maybe this Notebook from
@nielsr
is helpful https://github.com/NielsRogge/Transformers-Tutorials/blob/master/Pix2Struct/Fine_tune_Pix2Struct_on_key_value_pair_dataset_(PyTorch_Lightning).ipynb. There is code where he translates the json structure of naver-clova-ix/cord-v2 into tokens.