Upload folder using huggingface_hub

README.md

@@ -11,7 +11,7 @@ pipeline_tag: image-text-to-text
 
 ## Introduction
 
-We are excited to announce the release of InternVL 2.0, the latest addition to the InternVL series of multimodal large language models. InternVL 2.0 features a variety of **instruction-tuned models**, ranging from 2 billion to 108 billion parameters. This repository contains the instruction-tuned InternVL2-Llama3-76B model.
+We are excited to announce the release of InternVL 2.0, the latest addition to the InternVL series of multimodal large language models. InternVL 2.0 features a variety of **instruction-tuned models**, ranging from 1 billion to 108 billion parameters. This repository contains the instruction-tuned InternVL2-Llama3-76B model.
 
 Compared to the state-of-the-art open-source multimodal large language models, InternVL 2.0 surpasses most open-source models. It demonstrates competitive performance on par with proprietary commercial models across various capabilities, including document and chart comprehension, infographics QA, scene text understanding and OCR tasks, scientific and mathematical problem solving, as well as cultural understanding and integrated multimodal capabilities.
 
@@ -29,23 +29,23 @@ InternVL 2.0 is a multimodal large language model series, featuring models of va
 | :--------------------------: | :-------------: | :------------: | :-----------: | :------------------: |
 |          Model Size          |        -        |       -        |      40B      |         76B          |
 |                              |                 |                |               |                      |
-|    DocVQA<sub>test</sub>     |      87.2       |      86.5      |     93.9      |                      |
-|    ChartQA<sub>test</sub>    |      78.1       |      81.3      |     86.2      |                      |
-|    InfoVQA<sub>test</sub>    |        -        |      72.7      |     78.7      |                      |
-|    TextVQA<sub>val</sub>     |        -        |      73.5      |     83.0      |                      |
-|           OCRBench           |       678       |      754       |      837      |                      |
-|      MME<sub>sum</sub>       |     2070.2      |     2110.6     |    2315.0     |                      |
-|         RealWorldQA          |      68.0       |      67.5      |     71.8      |                      |
-|     AI2D<sub>test</sub>      |      89.4       |      80.3      |     87.1      |                      |
-|      MMMU<sub>val</sub>      |      63.1       |      58.5      |     53.9      |                      |
-|  MMBench-EN<sub>test</sub>   |      81.0       |      73.9      |     86.8      |                      |
-|  MMBench-CN<sub>test</sub>   |      80.2       |      73.8      |     86.5      |                      |
-|    CCBench<sub>dev</sub>     |      57.3       |      28.4      |     80.6      |                      |
-|  MMVet<sub>GPT-4-0613</sub>  |        -        |       -        |     68.5      |                      |
-| MMVet<sub>GPT-4-Turbo</sub>  |      67.5       |      64.0      |     65.5      |                      |
-|          SEED-Image          |        -        |       -        |     78.2      |                      |
-|   HallBench<sub>avg</sub>    |      43.9       |      45.6      |     56.9      |                      |
-| MathVista<sub>testmini</sub> |      58.1       |      57.7      |     63.7      |                      |
+|    DocVQA<sub>test</sub>     |      87.2       |      86.5      |     93.9      |         TODO         |
+|    ChartQA<sub>test</sub>    |      78.1       |      81.3      |     86.2      |         88.4         |
+|    InfoVQA<sub>test</sub>    |        -        |      72.7      |     78.7      |         82.0         |
+|    TextVQA<sub>val</sub>     |        -        |      73.5      |     83.0      |         84.4         |
+|           OCRBench           |       678       |      754       |      837      |         TODO         |
+|      MME<sub>sum</sub>       |     2070.2      |     2110.6     |    2315.0     |        2414.7        |
+|         RealWorldQA          |      68.0       |      67.5      |     71.8      |         TODO         |
+|     AI2D<sub>test</sub>      |      89.4       |      80.3      |     87.1      |         87.6         |
+|      MMMU<sub>val</sub>      |      63.1       |      58.5      |     53.9      |         55.2         |
+|  MMBench-EN<sub>test</sub>   |      81.0       |      73.9      |     86.8      |         86.5         |
+|  MMBench-CN<sub>test</sub>   |      80.2       |      73.8      |     86.5      |         86.3         |
+|    CCBench<sub>dev</sub>     |      57.3       |      28.4      |     80.6      |         81.0         |
+|  MMVet<sub>GPT-4-0613</sub>  |        -        |       -        |     68.5      |         69.8         |
+| MMVet<sub>GPT-4-Turbo</sub>  |      67.5       |      64.0      |     65.5      |         TODO         |
+|          SEED-Image          |        -        |       -        |     78.2      |         78.2         |
+|   HallBench<sub>avg</sub>    |      43.9       |      45.6      |     56.9      |         TODO         |
+| MathVista<sub>testmini</sub> |      58.1       |      57.7      |     63.7      |         TODO         |
 
 - We simultaneously use InternVL and VLMEvalKit repositories for model evaluation. Specifically, the results reported for DocVQA, ChartQA, InfoVQA, TextVQA, MME, AI2D, MMBench, CCBench, MMVet, and SEED-Image were tested using the InternVL repository. MMMU, OCRBench, RealWorldQA, HallBench, and MathVista were evaluated using the VLMEvalKit.
 
@@ -59,9 +59,9 @@ InternVL 2.0 is a multimodal large language model series, featuring models of va
 | :------------------: | :----: | :------: | :--------------: | :-----------: | :------------------: |
 |      Model Size      |   -    |   34B    |       34B        |      40B      |         76B          |
 |                      |        |          |                  |               |                      |
-|       MVBench        |   -    |    -     |        -         |     72.5      |                      |
-| Video-MME<br>wo subs |  59.9  |   59.0   |       52.0       |      TBD      |         TBD          |
-| Video-MME<br>w/ subs |  63.3  |   59.4   |       54.9       |      TBD      |         TBD          |
+|       MVBench        |   -    |    -     |        -         |     72.5      |         TODO         |
+| Video-MME<br>wo subs |  59.9  |   59.0   |       52.0       |     TODO      |         TODO         |
+| Video-MME<br>w/ subs |  63.3  |   59.4   |       54.9       |     TODO      |         TODO         |
 
 - We evaluate our models on MVBench by extracting 16 frames from each video, and each frame was resized to a 448x448 image.
 
@@ -71,6 +71,8 @@ Limitations: Although we have made efforts to ensure the safety of the model dur
 
 We provide an example code to run InternVL2-Llama3-76B using `transformers`.
 
+We also welcome you to experience the InternVL2 series models in our [online demo](https://internvl.opengvlab.com/). Currently, due to the limited GPU resources with public IP addresses, we can only deploy models up to a maximum of 26B. We will expand soon and deploy larger models to the online demo.
+
 > Please use transformers==4.37.2 to ensure the model works normally.
 
 ```python
@@ -317,6 +319,16 @@ print(f'Assistant: {response}')
 
 ## Deployment
 
+### LMDeploy
+
+TODO
+
+### vLLM
+
+TODO
+
+### Ollama
+
 TODO
 
 ## License
@@ -344,7 +356,7 @@ If you find this project useful in your research, please consider citing:
 
 ## 简介
 
-我们很高兴宣布 InternVL 2.0 的发布，这是 InternVL 系列多模态大语言模型的最新版本。InternVL 2.0 提供了多种**指令微调**的模型，参数从 20 亿到 1080 亿不等。此仓库包含经过指令微调的 InternVL2-Llama3-76B 模型。
+我们很高兴宣布 InternVL 2.0 的发布，这是 InternVL 系列多模态大语言模型的最新版本。InternVL 2.0 提供了多种**指令微调**的模型，参数从 10 亿到 1080 亿不等。此仓库包含经过指令微调的 InternVL2-Llama3-76B 模型。
 
 与最先进的开源多模态大语言模型相比，InternVL 2.0 超越了大多数开源模型。它在各种能力上表现出与闭源商业模型相媲美的竞争力，包括文档和图表理解、信息图表问答、场景文本理解和 OCR 任务、科学和数学问题解决，以及文化理解和综合多模态能力。
 
@@ -393,8 +405,8 @@ InternVL 2.0 是一个多模态大语言模型系列，包含各种规模的模
 |       模型大小       |   -    |   34B    |       34B        |      40B      |         76B          |
 |                      |        |          |                  |               |                      |
 |       MVBench        |   -    |    -     |        -         |     72.5      |                      |
-| Video-MME<br>wo subs |  59.9  |   59.0   |       52.0       |      TBD      |         TBD          |
-| Video-MME<br>w/ subs |  63.3  |   59.4   |       54.9       |      TBD      |         TBD          |
+| Video-MME<br>wo subs |  59.9  |   59.0   |       52.0       |     TODO      |         TODO         |
+| Video-MME<br>w/ subs |  63.3  |   59.4   |       54.9       |     TODO      |         TODO         |
 
 - 我们通过从每个视频中提取16帧来评估我们的模型在MVBench上的性能，每个视频帧被调整为448x448的图像。
 
@@ -404,6 +416,8 @@ InternVL 2.0 是一个多模态大语言模型系列，包含各种规模的模
 
 我们提供了一个示例代码，用于使用 `transformers` 运行 InternVL2-Llama3-76B。
 
+我们也欢迎你在我们的[在线demo](https://internvl.opengvlab.com/)中体验InternVL2的系列模型。目前，由于具备公网IP地址的GPU资源有限，我们目前只能部署最大到26B的模型。我们会在不久之后进行扩容，把更大的模型部署到在线demo上，敬请期待。
+
 > 请使用 transformers==4.37.2 以确保模型正常运行。
 
 示例代码请[点击这里](#quick-start)。
@@ -414,6 +428,14 @@ InternVL 2.0 是一个多模态大语言模型系列，包含各种规模的模
 
 TODO
 
+### vLLM
+
+TODO
+
+### Ollama
+
+TODO
+
 ## 开源许可证
 
 该项目采用 MIT 许可证发布，而 LLama3 则采用 Llama 3 Community License 许可证。

configuration_intern_vit.py

@@ -1,6 +1,6 @@
 # --------------------------------------------------------
 # InternVL
-# Copyright (c) 2023 OpenGVLab
+# Copyright (c) 2024 OpenGVLab
 # Licensed under The MIT License [see LICENSE for details]
 # --------------------------------------------------------
 import os

configuration_internvl_chat.py

@@ -1,6 +1,6 @@
 # --------------------------------------------------------
 # InternVL
-# Copyright (c) 2023 OpenGVLab
+# Copyright (c) 2024 OpenGVLab
 # Licensed under The MIT License [see LICENSE for details]
 # --------------------------------------------------------
 

conversation.py

@@ -330,13 +330,16 @@ def get_conv_template(name: str) -> Conversation:
     return conv_templates[name].copy()
 
 
-# Note that for inference, using the Hermes-2 and internlm2-chat templates is equivalent.
+# Both Hermes-2 and internlm2-chat are chatml-format conversation templates. The difference
+# is that during training, the preprocessing function for the Hermes-2 template doesn't add
+# <s> at the beginning of the tokenized sequence, while the internlm2-chat template does.
+# Therefore, they are completely equivalent during inference.
 register_conv_template(
     Conversation(
         name='Hermes-2',
         system_template='<|im_start|>system\n{system_message}',
         # note: The new system prompt was not used here to avoid changes in benchmark performance.
-        # system_message='我是书生·万象，英文名是InternVL，是由上海人工智能实验室及多家合作单位联合开发的多模态大语言模型。人工智能实验室致力于原始技术创新，开源开放，共享共创，推动科技进步和产业发展。',
+        # system_message='我是书生·万象，英文名是InternVL，是由上海人工智能实验室及多家合作单位联合开发的多模态大语言模型。',
         system_message='你是由上海人工智能实验室联合商汤科技开发的书生多模态大模型，英文名叫InternVL, 是一个有用无害的人工智能助手。',
         roles=('<|im_start|>user\n', '<|im_start|>assistant\n'),
         sep_style=SeparatorStyle.MPT,
@@ -357,7 +360,7 @@ register_conv_template(
         name='internlm2-chat',
         system_template='<|im_start|>system\n{system_message}',
         # note: The new system prompt was not used here to avoid changes in benchmark performance.
-        # system_message='我是书生·万象，英文名是InternVL，是由上海人工智能实验室及多家合作单位联合开发的多模态大语言模型。人工智能实验室致力于原始技术创新，开源开放，共享共创，推动科技进步和产业发展。',
+        # system_message='我是书生·万象，英文名是InternVL，是由上海人工智能实验室及多家合作单位联合开发的多模态大语言模型。',
         system_message='你是由上海人工智能实验室联合商汤科技开发的书生多模态大模型，英文名叫InternVL, 是一个有用无害的人工智能助手。',
         roles=('<|im_start|>user\n', '<|im_start|>assistant\n'),
         sep_style=SeparatorStyle.MPT,
@@ -376,7 +379,7 @@ register_conv_template(
         name='phi3-chat',
         system_template='<|system|>\n{system_message}',
         # note: The new system prompt was not used here to avoid changes in benchmark performance.
-        # system_message='我是书生·万象，英文名是InternVL，是由上海人工智能实验室及多家合作单位联合开发的多模态大语言模型。人工智能实验室致力于原始技术创新，开源开放，共享共创，推动科技进步和产业发展。',
+        # system_message='我是书生·万象，英文名是InternVL，是由上海人工智能实验室及多家合作单位联合开发的多模态大语言模型。',
         system_message='你是由上海人工智能实验室联合商汤科技开发的书生多模态大模型，英文名叫InternVL, 是一个有用无害的人工智能助手。',
         roles=('<|user|>\n', '<|assistant|>\n'),
         sep_style=SeparatorStyle.MPT,

modeling_intern_vit.py

@@ -1,6 +1,6 @@
 # --------------------------------------------------------
 # InternVL
-# Copyright (c) 2023 OpenGVLab
+# Copyright (c) 2024 OpenGVLab
 # Licensed under The MIT License [see LICENSE for details]
 # --------------------------------------------------------
 from typing import Optional, Tuple, Union

preprocessor_config.json

@@ -0,0 +1,19 @@
+{
+  "crop_size": 448,
+  "do_center_crop": true,
+  "do_normalize": true,
+  "do_resize": true,
+  "feature_extractor_type": "CLIPFeatureExtractor",
+  "image_mean": [
+    0.485,
+    0.456,
+    0.406
+  ],
+  "image_std": [
+    0.229,
+    0.224,
+    0.225
+  ],
+  "resample": 3,
+  "size": 448
+}