|
--- |
|
license: apache-2.0 |
|
language: |
|
- en |
|
base_model: |
|
- meta-llama/Llama-3.2-11B-Vision-Instruct |
|
pipeline_tag: visual-question-answering |
|
|
|
tags: |
|
|
|
- indox |
|
- phoenix |
|
- osllm.ai |
|
- language |
|
--- |
|
# Model Card for Model ID |
|
|
|
<!-- Provide a quick summary of what the model is/does. --> |
|
|
|
Llama-3.2V-11B-cot is the first version of [LLaVA-o1](https://github.com/PKU-YuanGroup/LLaVA-o1), which is a visual language model capable of spontaneous, systematic reasoning. |
|
|
|
## Model Details |
|
|
|
<!-- Provide a longer summary of what this model is. --> |
|
|
|
- **License:** apache-2.0 |
|
- **Finetuned from model:** meta-llama/Llama-3.2-11B-Vision-Instruct |
|
|
|
## Benchmark Results |
|
|
|
| MMStar | MMBench | MMVet | MathVista | AI2D | Hallusion | Average | |
|
|--------|---------|-------|-----------|------|-----------|---------| |
|
| 57.6 | 75.0 | 60.3 | 54.8 | 85.7 | 47.8 | 63.5 | |
|
|
|
## Reproduction |
|
|
|
<!-- This section describes the evaluation protocols and provides the results. --> |
|
|
|
To reproduce our results, you should use [VLMEvalKit](https://github.com/open-compass/VLMEvalKit) and the following settings. |
|
|
|
| Parameter | Value | |
|
|-------------------|---------| |
|
| do_sample | True | |
|
| temperature | 0.6 | |
|
| top_p | 0.9 | |
|
| max_new_tokens | 2048 | |
|
|
|
You may change them in [this file](https://github.com/open-compass/VLMEvalKit/blob/main/vlmeval/vlm/llama_vision.py), line 80-83, and modify the max_new_tokens throughout the file. |
|
|
|
Note: We follow the same settings as Llama-3.2-11B-Vision-Instruct, except that we extend the max_new_tokens to 2048. |
|
|
|
After you get the results, you should filter the model output and only **keep the outputs between \<CONCLUSION\> and \</CONCLUSION\>**. |
|
|
|
This shouldn't have any difference in theory, but empirically we observe some performance difference because the jugder GPT-4o can be inaccurate sometimes. |
|
|
|
By keeping the outputs between \<CONCLUSION\> and \</CONCLUSION\>, most answers can be direclty extracted using VLMEvalKit system, which can be much less biased. |
|
|
|
## How to Get Started with the Model |
|
|
|
You can use the inference code for Llama-3.2-11B-Vision-Instruct. |
|
|
|
## Training Details |
|
|
|
### Training Data |
|
|
|
<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. --> |
|
|
|
The model is trained on the LLaVA-o1-100k dataset (to be released). |
|
|
|
### Training Procedure |
|
|
|
<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. --> |
|
|
|
The model is finetuned on [llama-recipes](https://github.com/Meta-Llama/llama-recipes) with the following settings. |
|
Using the same setting should accurately reproduce our results. |
|
|
|
| Parameter | Value | |
|
|-------------------------------|---------------------------------------------------| |
|
| FSDP | enabled | |
|
| lr | 1e-5 | |
|
| num_epochs | 3 | |
|
| batch_size_training | 4 | |
|
| use_fast_kernels | True | |
|
| run_validation | False | |
|
| batching_strategy | padding | |
|
| context_length | 4096 | |
|
| gradient_accumulation_steps | 1 | |
|
| gradient_clipping | False | |
|
| gradient_clipping_threshold | 1.0 | |
|
| weight_decay | 0.0 | |
|
| gamma | 0.85 | |
|
| seed | 42 | |
|
| use_fp16 | False | |
|
| mixed_precision | True | |
|
|
|
|
|
## Bias, Risks, and Limitations |
|
|
|
<!-- This section is meant to convey both technical and sociotechnical limitations. --> |
|
|
|
The model may generate biased or offensive content, similar to other VLMs, due to limitations in the training data. |
|
Technically, the model's performance in aspects like instruction following still falls short of leading industry models. |
|
|
|
|
|
|
|
|
|
**About [osllm.ai](https://osllm.ai)**: |
|
|
|
[osllm.ai](https://osllm.ai) is a community-driven platform that provides access to a wide range of open-source language models. |
|
|
|
1. **[IndoxJudge](https://github.com/indoxJudge)**: A free, open-source tool for evaluating large language models (LLMs). |
|
It provides key metrics to assess performance, reliability, and risks like bias and toxicity, helping ensure model safety. |
|
|
|
1. **[inDox](https://github.com/inDox)**: An open-source retrieval augmentation tool for extracting data from various |
|
document formats (text, PDFs, HTML, Markdown, LaTeX). It handles structured and unstructured data and supports both |
|
online and offline LLMs. |
|
|
|
1. **[IndoxGen](https://github.com/IndoxGen)**: A framework for generating high-fidelity synthetic data using LLMs and |
|
human feedback, designed for enterprise use with high flexibility and precision. |
|
|
|
1. **[Phoenix](https://github.com/Phoenix)**: A multi-platform, open-source chatbot that interacts with documents |
|
locally, without internet or GPU. It integrates inDox and IndoxJudge to improve accuracy and prevent hallucinations, |
|
ideal for sensitive fields like healthcare. |
|
|
|
1. **[Phoenix_cli](https://github.com/Phoenix_cli)**: A multi-platform command-line tool that runs LLaMA models locally, |
|
supporting up to eight concurrent tasks through multithreading, eliminating the need for cloud-based services. |
|
|
|
|
|
|
|
**Disclaimers** |
|
|
|
[osllm.ai](https://osllm.ai) is not the creator, originator, or owner of any Model featured in the Community Model Program. |
|
Each Community Model is created and provided by third parties. osllm.ai does not endorse, support, represent, |
|
or guarantee the completeness, truthfulness, accuracy, or reliability of any Community Model. You understand |
|
that Community Models can produce content that might be offensive, harmful, inaccurate, or otherwise |
|
inappropriate, or deceptive. Each Community Model is the sole responsibility of the person or entity who |
|
originated such Model. osllm.ai may not monitor or control the Community Models and cannot, and does not, take |
|
responsibility for any such Model. osllm.ai disclaims all warranties or guarantees about the accuracy, |
|
reliability, or benefits of the Community Models. osllm.ai further disclaims any warranty that the Community |
|
Model will meet your requirements, be secure, uninterrupted, or available at any time or location, or |
|
error-free, virus-free, or that any errors will be corrected, or otherwise. You will be solely responsible for |
|
any damage resulting from your use of or access to the Community Models, your downloading of any Community |
|
Model, or use of any other Community Model provided by or through [osllm.ai](https://osllm.ai). |
|
|
|
|
|
|
