nuua

/

ko-deplot

like 5

Hello,
I'm not sure if I understood the question correctly, but if you already have the korean based chart images, you would also have to prepare the labels corresponding to those images in the format deplot uses (markdown table format).
For the actual fine-tuning phase, I used this notebook as a reference.
https://github.com/huggingface/notebooks/blob/main/examples/image_captioning_pix2struct.ipynb

You would likely be using your chart images as the model's input and your table labels as output.

First of all, thanks for the comment. Yes i made corresponding labels as well.
I was wondering how to make own datasets with image and label?
do I have to put them in same directory and then load_dataset ?

Hey, sorry for the late reply.
If you are referring to the load_dataset function from the reference code, you will have to reimplement the dataset class using the reference code and the collator function. They are both almost identical to the original code. We then used pytorch's DataLoader to load the data. You don't have to put the images and the labels in the same directory, you just have to specify their paths to the dataset class.

Note that we are not able to provide the exact code due to unforseen circumstances at the moment.

thanks for the reply. Have you tried evaluation? As I checked from the paper they used RNSS and RMS for model evaluation. But I am planning to evaluate with BLEU and BLEURT. Because I'm planning to generate text summarization from outputs (tables).

Unfortunately we haven't tried evaluation using RNSS and RMS like the paper but we used our internal metrics using levenshtein distance.

thanks for your explanation.
I have tried to run the model but i got this error below:

Exception Traceback (most recent call last)
in <cell line: 4>()
2 from PIL import Image
3
----> 4 processor = Pix2StructProcessor.from_pretrained('nuua/ko-deplot')
5 model = Pix2StructForConditionalGeneration.from_pretrained('nuua/ko-deplot')
6

5 frames
/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_fast.py in init(self, *args, **kwargs)
109 elif fast_tokenizer_file is not None and not from_slow:
110 # We have a serialization from tokenizers which let us directly build the backend
--> 111 fast_tokenizer = TokenizerFast.from_file(fast_tokenizer_file)
112 elif slow_tokenizer is not None:
113 # We need to convert a slow tokenizer to build the backend

Exception: data did not match any variant of untagged enum PyPreTokenizerTypeWrapper at line 137678 column 3

how can I solve this problem?