ilu000 commited on
Commit
bcb9d7b
1 Parent(s): a7363ba

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +348 -347
README.md CHANGED
@@ -1,348 +1,349 @@
1
- ---
2
- language:
3
- - en
4
- library_name: transformers
5
- license: apache-2.0
6
- tags:
7
- - gpt
8
- - llm
9
- - multimodal large language model
10
- thumbnail: >-
11
- https://h2o.ai/etc.clientlibs/h2o/clientlibs/clientlib-site/resources/images/favicon.ico
12
- pipeline_tag: text-generation
13
- ---
14
- # Model Card
15
- The H2OVL-Mississippi-800M is a compact yet powerful vision-language model from H2O.ai, featuring 0.8 billion parameters. Despite its small size, it delivers state-of-the-art performance in text recognition, excelling in the Text Recognition segment of OCRBench and outperforming much larger models in this domain. Built upon the robust architecture of our H2O-Danube language models, the Mississippi-800M extends their capabilities by seamlessly integrating vision and language tasks.
16
-
17
- <div align="center">
18
- <img src="./assets/text_recognition.png" alt="Mississippi-2B Benchmarks" width="600"/>
19
- </div>
20
-
21
- ## Key Features:
22
-
23
- - 0.8 Billion Parameters: Balance between performance and efficiency, making it suitable for OCR and document processing.
24
- - Trained on 19 million image-text pairs, with a focus on OCR, document comprehension, and chart, figure, and table interpretation, the model is optimized for superior OCR performance.
25
-
26
-
27
- <div align="center">
28
- <img src="./assets/perf_size.png" alt="Mississippi-2B Benchmarks" width="600"/>
29
- </div>
30
-
31
-
32
- ## Usage
33
-
34
- ### Install dependencies:
35
- ```bash
36
- pip install transformers torch torchvision einops timm peft sentencepiece flash_attn
37
- ```
38
-
39
- ### Sample demo:
40
-
41
- ```python
42
- import torch
43
- from transformers import AutoConfig, AutoModel, AutoTokenizer
44
-
45
-
46
- # Set up the model and tokenizer
47
- model_path = 'h2oai/h2ovl-mississippi-800m'
48
- config = AutoConfig.from_pretrained(model_path, trust_remote_code=True)
49
- config.llm_config._attn_implementation = 'flash_attention_2'
50
- model = AutoModel.from_pretrained(
51
- model_path,
52
- torch_dtype=torch.bfloat16,
53
- config=config,
54
- low_cpu_mem_usage=True,
55
- trust_remote_code=True).eval().cuda()
56
- tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True, use_fast=False)
57
- generation_config = dict(max_new_tokens=2048, do_sample=True)
58
-
59
- # pure-text conversation
60
- question = 'Hello, how are you?'
61
- response, history = model.chat(tokenizer, None, question, generation_config, history=None, return_history=True)
62
- print(f'User: {question}\nAssistant: {response}')
63
-
64
-
65
- # Example for single image
66
- image_file = './examples/image.jpg'
67
- question = '<image>\nRead the text in the image.'
68
- response, history = model.chat(tokenizer, image_file, question, generation_config, history=None, return_history=True)
69
- print(f'User: {question}\nAssistant: {response}')
70
-
71
-
72
- ```
73
-
74
-
75
- ## Benchmarks
76
-
77
- ### Performance Comparison of Similar Sized Models Across Multiple Benchmarks - OpenVLM Leaderboard
78
-
79
- | **Models** | **Params (B)** | **Avg. Score** | **MMBench** | **MMStar** | **MMMU<sub>VAL</sub>** | **Math Vista** | **Hallusion** | **AI2D<sub>TEST</sub>** | **OCRBench** | **MMVet** |
80
- |----------------------------|----------------|----------------|-------------|------------|-----------------------|----------------|---------------|-------------------------|--------------|-----------|
81
- | Qwen2-VL-2B | 2.1 | **57.2** | **72.2** | 47.5 | 42.2 | 47.8 | **42.4** | 74.7 | **797** | **51.5** |
82
- | **H2OVL-Mississippi-2B** | 2.1 | 54.4 | 64.8 | 49.6 | 35.2 | **56.8** | 36.4 | 69.9 | 782 | 44.7 |
83
- | InternVL2-2B | 2.1 | 53.9 | 69.6 | **49.8** | 36.3 | 46.0 | 38.0 | 74.1 | 781 | 39.7 |
84
- | Phi-3-Vision | 4.2 | 53.6 | 65.2 | 47.7 | **46.1** | 44.6 | 39.0 | **78.4** | 637 | 44.1 |
85
- | MiniMonkey | 2.2 | 52.7 | 68.9 | 48.1 | 35.7 | 45.3 | 30.9 | 73.7 | **794** | 39.8 |
86
- | MiniCPM-V-2 | 2.8 | 47.9 | 65.8 | 39.1 | 38.2 | 39.8 | 36.1 | 62.9 | 605 | 41.0 |
87
- | InternVL2-1B | 0.8 | 48.3 | 59.7 | 45.6 | 36.7 | 39.4 | 34.3 | 63.8 | 755 | 31.5 |
88
- | PaliGemma-3B-mix-448 | 2.9 | 46.5 | 65.6 | 48.3 | 34.9 | 28.7 | 32.2 | 68.3 | 614 | 33.1 |
89
- | **H2OVL-Mississippi-0.8B** | 0.8 | 43.5 | 47.7 | 39.1 | 34.0 | 39.0 | 29.6 | 53.6 | 751 | 30.0 |
90
- | DeepSeek-VL-1.3B | 2.0 | 39.6 | 63.8 | 39.9 | 33.8 | 29.8 | 27.6 | 51.5 | 413 | 29.2 |
91
-
92
-
93
- ## Prompt Engineering for JSON Extraction
94
-
95
- ### Overview
96
-
97
- This guide demonstrates how to create prompts for extracting information and converting it into structured JSON outputs. It starts with basic examples and progresses to more complex JSON structures, including handling data from images of tables and charts. The objective is to help users design effective prompts that can be used in various applications, such as natural language processing, chatbots, or data extraction from visual inputs.
98
-
99
- ### Table of Contents
100
-
101
- 1. [Getting Started](#getting-started)
102
- 2. [Extracting Simple Information](#example-1-extracting-simple-information-from-an-image)
103
- 3. [Extracting Nested Information](#example-2-extracting-nested-information-from-an-image)
104
- 4. [Extracting Lists and Arrays](#example-3-extracting-lists-and-arrays-from-an-image)
105
- 5. [Extracting Tables](#example-4-extracting-table-data-from-an-image)
106
- 6. [Extracting Charts](#example-5-extracting-chart-data-from-an-image)
107
- 7. [Best Practices](#best-practices)
108
-
109
- ---
110
-
111
- ### Getting Started
112
-
113
- To get started with JSON extraction from images, it's essential to have a clear understanding of the visual content you want to extract and the structure of the desired JSON output. The following examples will guide you through crafting prompts to achieve this.
114
-
115
-
116
- #### Example 1: Extracting Simple Information from an Image
117
-
118
- **Hypothetical Scenario:**
119
- You have an image of a form that contains basic details like "Name," "Date of Birth," and "Address."
120
-
121
- **Prompt:**
122
- ```
123
- Extract the details from the form image and structure them into JSON format:
124
- {
125
- "name": "",
126
- "date_of_birth": "",
127
- "address": ""
128
- }
129
- ```
130
-
131
- **Expected Output:**
132
- ```json
133
- {
134
- "name": "John Doe",
135
- "date_of_birth": "1990-01-01",
136
- "address": "1234 Elm Street, Springfield"
137
- }
138
- ```
139
-
140
- #### Example 2: Extracting Nested Information from an Image
141
-
142
- **Hypothetical Scenario:**
143
- You have an image of a form that contains detailed personal information, including contact details and emergency contacts.
144
-
145
- **Prompt:**
146
- ```
147
- Extract the information from the form and format it as follows:
148
- {
149
- "personal_details": {
150
- "name": "",
151
- "age": 0,
152
- "gender": ""
153
- },
154
- "contact": {
155
- "phone": "",
156
- "email": ""
157
- },
158
- "emergency_contact": {
159
- "name": "",
160
- "relation": "",
161
- "phone": ""
162
- }
163
- }
164
- ```
165
-
166
- **Expected Output:**
167
- ```json
168
- {
169
- "personal_details": {
170
- "name": "Sarah Connor",
171
- "age": 35,
172
- "gender": "Female"
173
- },
174
- "contact": {
175
- "phone": "555-1234",
176
- "email": "[email protected]"
177
- },
178
- "emergency_contact": {
179
- "name": "Kyle Reese",
180
- "relation": "Friend",
181
- "phone": "555-5678"
182
- }
183
- }
184
- ```
185
-
186
-
187
- #### Example 3: Extracting Lists and Arrays from an Image
188
-
189
- **Hypothetical Scenario:**
190
- You have an image of a schedule that lists several events, their times, and locations.
191
-
192
- **Prompt:**
193
- ```
194
- Extract the event details from the schedule image and structure them into JSON:
195
- {
196
- "events": [
197
- {
198
- "name": "",
199
- "time": "",
200
- "location": ""
201
- }
202
- ]
203
- }
204
- ```
205
-
206
- **Expected Output:**
207
- ```json
208
- {
209
- "events": [
210
- {
211
- "name": "Morning Meeting",
212
- "time": "09:00 AM",
213
- "location": "Conference Room 1"
214
- },
215
- {
216
- "name": "Lunch Break",
217
- "time": "12:00 PM",
218
- "location": "Cafeteria"
219
- },
220
- {
221
- "name": "Project Update",
222
- "time": "02:00 PM",
223
- "location": "Conference Room 2"
224
- }
225
- ]
226
- }
227
- ```
228
-
229
-
230
- #### Example 4: Extracting Table Data from an Image
231
-
232
- Images of tables often contain structured data that needs to be parsed and converted to JSON. The following example demonstrates how to handle tabular data extraction.
233
-
234
- **Hypothetical Scenario:**
235
- You have an image of a table listing product names, prices, and quantities.
236
-
237
- **Prompt:**
238
- ```
239
- Extract the data from the table image and format it as JSON:
240
- {
241
- "products": [
242
- {
243
- "product_name": "",
244
- "price": "",
245
- "quantity": 0
246
- }
247
- ]
248
- }
249
- ```
250
-
251
- **Expected Output:**
252
- ```json
253
- {
254
- "products": [
255
- {
256
- "product_name": "Apples",
257
- "price": "$2",
258
- "quantity": 10
259
- },
260
- {
261
- "product_name": "Bananas",
262
- "price": "$1",
263
- "quantity": 20
264
- },
265
- {
266
- "product_name": "Oranges",
267
- "price": "$3",
268
- "quantity": 15
269
- }
270
- ]
271
- }
272
- ```
273
-
274
-
275
- #### Example 5: Extracting Chart Data from an Image
276
-
277
- Charts include metadata and data points that need to be accurately extracted. Here's how to structure prompts to extract chart data from images.
278
-
279
- **Hypothetical Scenario:**
280
- You have an image of a bar chart that shows monthly sales figures.
281
-
282
- **Prompt:**
283
- ```
284
- Extract the details of the bar chart from the image, including the title, axis labels, and data points and format it as JSON:
285
- {
286
- "chart": {
287
- "title": "",
288
- "x_axis": "",
289
- "y_axis": "",
290
- "data_points": [
291
- {
292
- "label": "",
293
- "value": 0
294
- }
295
- ]
296
- }
297
- }
298
- ```
299
-
300
- **Expected Output:**
301
- ```json
302
- {
303
- "chart": {
304
- "title": "Monthly Sales Report",
305
- "x_axis": "Months",
306
- "y_axis": "Sales (in $)",
307
- "data_points": [
308
- {
309
- "label": "January",
310
- "value": 500
311
- },
312
- {
313
- "label": "February",
314
- "value": 600
315
- },
316
- {
317
- "label": "March",
318
- "value": 700
319
- }
320
- ]
321
- }
322
- }
323
- ```
324
-
325
- ## Best Practices
326
-
327
- 1. **Be Explicit**: Clearly define the desired keys and structure in your prompt to avoid ambiguity.
328
- 2. **Use Examples**: Provide sample outputs so that the system can understand the expected format.
329
- 3. **Anticipate Variations**: Consider possible variations in the visual data and ensure the prompt can accommodate them.
330
- 4. **Start Simple**: Begin with simple structures, and progressively increase complexity as needed.
331
- 5. **Test and Iterate**: Refine your prompts through testing to ensure accuracy and consistency in outputs.
332
-
333
- ## Acknowledgments
334
-
335
- We would like to express our gratitude to the [InternVL team at OpenGVLab](https://github.com/OpenGVLab/InternVL) for their research and codebases, upon which we have built and expanded. We also acknowledge the work of the [LLaVA team](https://github.com/haotian-liu/LLaVA) and the [Monkey team](https://github.com/Yuliang-Liu/Monkey/tree/main/project/mini_monkey) for their insights and techniques used in improving multimodal models.
336
-
337
- ## Disclaimer
338
-
339
- Please read this disclaimer carefully before using the large language model provided in this repository. Your use of the model signifies your agreement to the following terms and conditions.
340
-
341
- - Biases and Offensiveness: The large language model is trained on a diverse range of internet text data, which may contain biased, racist, offensive, or otherwise inappropriate content. By using this model, you acknowledge and accept that the generated content may sometimes exhibit biases or produce content that is offensive or inappropriate. The developers of this repository do not endorse, support, or promote any such content or viewpoints.
342
- - Limitations: The large language model is an AI-based tool and not a human. It may produce incorrect, nonsensical, or irrelevant responses. It is the user's responsibility to critically evaluate the generated content and use it at their discretion.
343
- - Use at Your Own Risk: Users of this large language model must assume full responsibility for any consequences that may arise from their use of the tool. The developers and contributors of this repository shall not be held liable for any damages, losses, or harm resulting from the use or misuse of the provided model.
344
- - Ethical Considerations: Users are encouraged to use the large language model responsibly and ethically. By using this model, you agree not to use it for purposes that promote hate speech, discrimination, harassment, or any form of illegal or harmful activities.
345
- - Reporting Issues: If you encounter any biased, offensive, or otherwise inappropriate content generated by the large language model, please report it to the repository maintainers through the provided channels. Your feedback will help improve the model and mitigate potential issues.
346
- - Changes to this Disclaimer: The developers of this repository reserve the right to modify or update this disclaimer at any time without prior notice. It is the user's responsibility to periodically review the disclaimer to stay informed about any changes.
347
-
 
348
  By using the large language model provided in this repository, you agree to accept and comply with the terms and conditions outlined in this disclaimer. If you do not agree with any part of this disclaimer, you should refrain from using the model and any content generated by it.
 
1
+ ---
2
+ language:
3
+ - en
4
+ library_name: transformers
5
+ license: apache-2.0
6
+ tags:
7
+ - gpt
8
+ - llm
9
+ - multimodal large language model
10
+ - ocr
11
+ thumbnail: >-
12
+ https://h2o.ai/etc.clientlibs/h2o/clientlibs/clientlib-site/resources/images/favicon.ico
13
+ pipeline_tag: text-generation
14
+ ---
15
+ # Model Card
16
+ The H2OVL-Mississippi-800M is a compact yet powerful vision-language model from H2O.ai, featuring 0.8 billion parameters. Despite its small size, it delivers state-of-the-art performance in text recognition, excelling in the Text Recognition segment of OCRBench and outperforming much larger models in this domain. Built upon the robust architecture of our H2O-Danube language models, the Mississippi-800M extends their capabilities by seamlessly integrating vision and language tasks.
17
+
18
+ <div align="center">
19
+ <img src="./assets/text_recognition.png" alt="Mississippi-2B Benchmarks" width="600"/>
20
+ </div>
21
+
22
+ ## Key Features:
23
+
24
+ - 0.8 Billion Parameters: Balance between performance and efficiency, making it suitable for OCR and document processing.
25
+ - Trained on 19 million image-text pairs, with a focus on OCR, document comprehension, and chart, figure, and table interpretation, the model is optimized for superior OCR performance.
26
+
27
+
28
+ <div align="center">
29
+ <img src="./assets/perf_size.png" alt="Mississippi-2B Benchmarks" width="600"/>
30
+ </div>
31
+
32
+
33
+ ## Usage
34
+
35
+ ### Install dependencies:
36
+ ```bash
37
+ pip install transformers torch torchvision einops timm peft sentencepiece flash_attn
38
+ ```
39
+
40
+ ### Sample demo:
41
+
42
+ ```python
43
+ import torch
44
+ from transformers import AutoConfig, AutoModel, AutoTokenizer
45
+
46
+
47
+ # Set up the model and tokenizer
48
+ model_path = 'h2oai/h2ovl-mississippi-800m'
49
+ config = AutoConfig.from_pretrained(model_path, trust_remote_code=True)
50
+ config.llm_config._attn_implementation = 'flash_attention_2'
51
+ model = AutoModel.from_pretrained(
52
+ model_path,
53
+ torch_dtype=torch.bfloat16,
54
+ config=config,
55
+ low_cpu_mem_usage=True,
56
+ trust_remote_code=True).eval().cuda()
57
+ tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True, use_fast=False)
58
+ generation_config = dict(max_new_tokens=2048, do_sample=True)
59
+
60
+ # pure-text conversation
61
+ question = 'Hello, how are you?'
62
+ response, history = model.chat(tokenizer, None, question, generation_config, history=None, return_history=True)
63
+ print(f'User: {question}\nAssistant: {response}')
64
+
65
+
66
+ # Example for single image
67
+ image_file = './examples/image.jpg'
68
+ question = '<image>\nRead the text in the image.'
69
+ response, history = model.chat(tokenizer, image_file, question, generation_config, history=None, return_history=True)
70
+ print(f'User: {question}\nAssistant: {response}')
71
+
72
+
73
+ ```
74
+
75
+
76
+ ## Benchmarks
77
+
78
+ ### Performance Comparison of Similar Sized Models Across Multiple Benchmarks - OpenVLM Leaderboard
79
+
80
+ | **Models** | **Params (B)** | **Avg. Score** | **MMBench** | **MMStar** | **MMMU<sub>VAL</sub>** | **Math Vista** | **Hallusion** | **AI2D<sub>TEST</sub>** | **OCRBench** | **MMVet** |
81
+ |----------------------------|----------------|----------------|-------------|------------|-----------------------|----------------|---------------|-------------------------|--------------|-----------|
82
+ | Qwen2-VL-2B | 2.1 | **57.2** | **72.2** | 47.5 | 42.2 | 47.8 | **42.4** | 74.7 | **797** | **51.5** |
83
+ | **H2OVL-Mississippi-2B** | 2.1 | 54.4 | 64.8 | 49.6 | 35.2 | **56.8** | 36.4 | 69.9 | 782 | 44.7 |
84
+ | InternVL2-2B | 2.1 | 53.9 | 69.6 | **49.8** | 36.3 | 46.0 | 38.0 | 74.1 | 781 | 39.7 |
85
+ | Phi-3-Vision | 4.2 | 53.6 | 65.2 | 47.7 | **46.1** | 44.6 | 39.0 | **78.4** | 637 | 44.1 |
86
+ | MiniMonkey | 2.2 | 52.7 | 68.9 | 48.1 | 35.7 | 45.3 | 30.9 | 73.7 | **794** | 39.8 |
87
+ | MiniCPM-V-2 | 2.8 | 47.9 | 65.8 | 39.1 | 38.2 | 39.8 | 36.1 | 62.9 | 605 | 41.0 |
88
+ | InternVL2-1B | 0.8 | 48.3 | 59.7 | 45.6 | 36.7 | 39.4 | 34.3 | 63.8 | 755 | 31.5 |
89
+ | PaliGemma-3B-mix-448 | 2.9 | 46.5 | 65.6 | 48.3 | 34.9 | 28.7 | 32.2 | 68.3 | 614 | 33.1 |
90
+ | **H2OVL-Mississippi-0.8B** | 0.8 | 43.5 | 47.7 | 39.1 | 34.0 | 39.0 | 29.6 | 53.6 | 751 | 30.0 |
91
+ | DeepSeek-VL-1.3B | 2.0 | 39.6 | 63.8 | 39.9 | 33.8 | 29.8 | 27.6 | 51.5 | 413 | 29.2 |
92
+
93
+
94
+ ## Prompt Engineering for JSON Extraction
95
+
96
+ ### Overview
97
+
98
+ This guide demonstrates how to create prompts for extracting information and converting it into structured JSON outputs. It starts with basic examples and progresses to more complex JSON structures, including handling data from images of tables and charts. The objective is to help users design effective prompts that can be used in various applications, such as natural language processing, chatbots, or data extraction from visual inputs.
99
+
100
+ ### Table of Contents
101
+
102
+ 1. [Getting Started](#getting-started)
103
+ 2. [Extracting Simple Information](#example-1-extracting-simple-information-from-an-image)
104
+ 3. [Extracting Nested Information](#example-2-extracting-nested-information-from-an-image)
105
+ 4. [Extracting Lists and Arrays](#example-3-extracting-lists-and-arrays-from-an-image)
106
+ 5. [Extracting Tables](#example-4-extracting-table-data-from-an-image)
107
+ 6. [Extracting Charts](#example-5-extracting-chart-data-from-an-image)
108
+ 7. [Best Practices](#best-practices)
109
+
110
+ ---
111
+
112
+ ### Getting Started
113
+
114
+ To get started with JSON extraction from images, it's essential to have a clear understanding of the visual content you want to extract and the structure of the desired JSON output. The following examples will guide you through crafting prompts to achieve this.
115
+
116
+
117
+ #### Example 1: Extracting Simple Information from an Image
118
+
119
+ **Hypothetical Scenario:**
120
+ You have an image of a form that contains basic details like "Name," "Date of Birth," and "Address."
121
+
122
+ **Prompt:**
123
+ ```
124
+ Extract the details from the form image and structure them into JSON format:
125
+ {
126
+ "name": "",
127
+ "date_of_birth": "",
128
+ "address": ""
129
+ }
130
+ ```
131
+
132
+ **Expected Output:**
133
+ ```json
134
+ {
135
+ "name": "John Doe",
136
+ "date_of_birth": "1990-01-01",
137
+ "address": "1234 Elm Street, Springfield"
138
+ }
139
+ ```
140
+
141
+ #### Example 2: Extracting Nested Information from an Image
142
+
143
+ **Hypothetical Scenario:**
144
+ You have an image of a form that contains detailed personal information, including contact details and emergency contacts.
145
+
146
+ **Prompt:**
147
+ ```
148
+ Extract the information from the form and format it as follows:
149
+ {
150
+ "personal_details": {
151
+ "name": "",
152
+ "age": 0,
153
+ "gender": ""
154
+ },
155
+ "contact": {
156
+ "phone": "",
157
+ "email": ""
158
+ },
159
+ "emergency_contact": {
160
+ "name": "",
161
+ "relation": "",
162
+ "phone": ""
163
+ }
164
+ }
165
+ ```
166
+
167
+ **Expected Output:**
168
+ ```json
169
+ {
170
+ "personal_details": {
171
+ "name": "Sarah Connor",
172
+ "age": 35,
173
+ "gender": "Female"
174
+ },
175
+ "contact": {
176
+ "phone": "555-1234",
177
+ "email": "[email protected]"
178
+ },
179
+ "emergency_contact": {
180
+ "name": "Kyle Reese",
181
+ "relation": "Friend",
182
+ "phone": "555-5678"
183
+ }
184
+ }
185
+ ```
186
+
187
+
188
+ #### Example 3: Extracting Lists and Arrays from an Image
189
+
190
+ **Hypothetical Scenario:**
191
+ You have an image of a schedule that lists several events, their times, and locations.
192
+
193
+ **Prompt:**
194
+ ```
195
+ Extract the event details from the schedule image and structure them into JSON:
196
+ {
197
+ "events": [
198
+ {
199
+ "name": "",
200
+ "time": "",
201
+ "location": ""
202
+ }
203
+ ]
204
+ }
205
+ ```
206
+
207
+ **Expected Output:**
208
+ ```json
209
+ {
210
+ "events": [
211
+ {
212
+ "name": "Morning Meeting",
213
+ "time": "09:00 AM",
214
+ "location": "Conference Room 1"
215
+ },
216
+ {
217
+ "name": "Lunch Break",
218
+ "time": "12:00 PM",
219
+ "location": "Cafeteria"
220
+ },
221
+ {
222
+ "name": "Project Update",
223
+ "time": "02:00 PM",
224
+ "location": "Conference Room 2"
225
+ }
226
+ ]
227
+ }
228
+ ```
229
+
230
+
231
+ #### Example 4: Extracting Table Data from an Image
232
+
233
+ Images of tables often contain structured data that needs to be parsed and converted to JSON. The following example demonstrates how to handle tabular data extraction.
234
+
235
+ **Hypothetical Scenario:**
236
+ You have an image of a table listing product names, prices, and quantities.
237
+
238
+ **Prompt:**
239
+ ```
240
+ Extract the data from the table image and format it as JSON:
241
+ {
242
+ "products": [
243
+ {
244
+ "product_name": "",
245
+ "price": "",
246
+ "quantity": 0
247
+ }
248
+ ]
249
+ }
250
+ ```
251
+
252
+ **Expected Output:**
253
+ ```json
254
+ {
255
+ "products": [
256
+ {
257
+ "product_name": "Apples",
258
+ "price": "$2",
259
+ "quantity": 10
260
+ },
261
+ {
262
+ "product_name": "Bananas",
263
+ "price": "$1",
264
+ "quantity": 20
265
+ },
266
+ {
267
+ "product_name": "Oranges",
268
+ "price": "$3",
269
+ "quantity": 15
270
+ }
271
+ ]
272
+ }
273
+ ```
274
+
275
+
276
+ #### Example 5: Extracting Chart Data from an Image
277
+
278
+ Charts include metadata and data points that need to be accurately extracted. Here's how to structure prompts to extract chart data from images.
279
+
280
+ **Hypothetical Scenario:**
281
+ You have an image of a bar chart that shows monthly sales figures.
282
+
283
+ **Prompt:**
284
+ ```
285
+ Extract the details of the bar chart from the image, including the title, axis labels, and data points and format it as JSON:
286
+ {
287
+ "chart": {
288
+ "title": "",
289
+ "x_axis": "",
290
+ "y_axis": "",
291
+ "data_points": [
292
+ {
293
+ "label": "",
294
+ "value": 0
295
+ }
296
+ ]
297
+ }
298
+ }
299
+ ```
300
+
301
+ **Expected Output:**
302
+ ```json
303
+ {
304
+ "chart": {
305
+ "title": "Monthly Sales Report",
306
+ "x_axis": "Months",
307
+ "y_axis": "Sales (in $)",
308
+ "data_points": [
309
+ {
310
+ "label": "January",
311
+ "value": 500
312
+ },
313
+ {
314
+ "label": "February",
315
+ "value": 600
316
+ },
317
+ {
318
+ "label": "March",
319
+ "value": 700
320
+ }
321
+ ]
322
+ }
323
+ }
324
+ ```
325
+
326
+ ## Best Practices
327
+
328
+ 1. **Be Explicit**: Clearly define the desired keys and structure in your prompt to avoid ambiguity.
329
+ 2. **Use Examples**: Provide sample outputs so that the system can understand the expected format.
330
+ 3. **Anticipate Variations**: Consider possible variations in the visual data and ensure the prompt can accommodate them.
331
+ 4. **Start Simple**: Begin with simple structures, and progressively increase complexity as needed.
332
+ 5. **Test and Iterate**: Refine your prompts through testing to ensure accuracy and consistency in outputs.
333
+
334
+ ## Acknowledgments
335
+
336
+ We would like to express our gratitude to the [InternVL team at OpenGVLab](https://github.com/OpenGVLab/InternVL) for their research and codebases, upon which we have built and expanded. We also acknowledge the work of the [LLaVA team](https://github.com/haotian-liu/LLaVA) and the [Monkey team](https://github.com/Yuliang-Liu/Monkey/tree/main/project/mini_monkey) for their insights and techniques used in improving multimodal models.
337
+
338
+ ## Disclaimer
339
+
340
+ Please read this disclaimer carefully before using the large language model provided in this repository. Your use of the model signifies your agreement to the following terms and conditions.
341
+
342
+ - Biases and Offensiveness: The large language model is trained on a diverse range of internet text data, which may contain biased, racist, offensive, or otherwise inappropriate content. By using this model, you acknowledge and accept that the generated content may sometimes exhibit biases or produce content that is offensive or inappropriate. The developers of this repository do not endorse, support, or promote any such content or viewpoints.
343
+ - Limitations: The large language model is an AI-based tool and not a human. It may produce incorrect, nonsensical, or irrelevant responses. It is the user's responsibility to critically evaluate the generated content and use it at their discretion.
344
+ - Use at Your Own Risk: Users of this large language model must assume full responsibility for any consequences that may arise from their use of the tool. The developers and contributors of this repository shall not be held liable for any damages, losses, or harm resulting from the use or misuse of the provided model.
345
+ - Ethical Considerations: Users are encouraged to use the large language model responsibly and ethically. By using this model, you agree not to use it for purposes that promote hate speech, discrimination, harassment, or any form of illegal or harmful activities.
346
+ - Reporting Issues: If you encounter any biased, offensive, or otherwise inappropriate content generated by the large language model, please report it to the repository maintainers through the provided channels. Your feedback will help improve the model and mitigate potential issues.
347
+ - Changes to this Disclaimer: The developers of this repository reserve the right to modify or update this disclaimer at any time without prior notice. It is the user's responsibility to periodically review the disclaimer to stay informed about any changes.
348
+
349
  By using the large language model provided in this repository, you agree to accept and comply with the terms and conditions outlined in this disclaimer. If you do not agree with any part of this disclaimer, you should refrain from using the model and any content generated by it.