LLaVA++ (LLaMA-3 and Phi-3-Mini)
Collection
Extending Visual Capabilities of LLaVA with LLaMA-3 and Phi-3
•
11 items
•
Updated
•
23
LLaMA-3-V: Extending the Visual Capabilities of LLaVA with Meta-Llama-3-8B-Instruct
Repository Overview
This repository features LLaVA v1.5 trained with the Meta-Llama-3-8B-Instruct LLM. This integration aims to leverage the strengths of both models to offer advanced vision-language understanding.
Training Strategy
- Pretraining: Only Vision-to-Language projector is trained. The rest of the model is frozen.
- Fine-tuning: LLM is LoRA fine-tuned. Only the vision-backbone (CLIP) is kept frozen.
- Note: The repository contains projector and LoRA weights.
Key Components
- Base Large Language Model (LLM): Meta-Llama-3-8B-Instruct
- Base Large Multimodal Model (LMM): LLaVA-v1.5
Training Data
- Pretraining Dataset: LCS-558K
- Fine-tuning Dataset: LLaVA-Instruct-665K
Download It As
git lfs install
git clone https://huggingface.co/MBZUAI/LLaVA-Meta-Llama-3-8B-Instruct-lora
Contributions
Contributions are welcome! Please 🌟 our repository LLaVA++ if you find this model useful.
- Downloads last month
- 9
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social
visibility and check back later, or deploy to Inference Endpoints (dedicated)
instead.