It's raining vision language models ☔️ CuMo is a new vision language model that has MoE in every step of the VLM (image encoder, MLP and text decoder) and uses Mistral-7B for the decoder part 🤓  

![image_1](image_1.jpg)

The authors firstly did pre-training of MLP with the by freezing the image encoder and text decoder, then they warmup the whole network by unfreezing and finetuning which they state to stabilize the visual instruction tuning when bringing in the experts.  

![image_2](image_2.jpg)

The mixture of experts MLP blocks above are simply the same MLP blocks initialized from the single MLP that was trained during pre-training and fine-tuned in pre-finetuning 👇  

![image_3](image_3.jpg)

It works very well (also tested myself) that it outperforms the previous sota of it's size LLaVA NeXt! 😍 I wonder how it would compare to IDEFICS2-8B You can try it yourself [here](https://t.co/MLIYKVh5Ee).  

![image_4](image_4.jpg)

> [!TIP]
Ressources:  
[CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts](https://arxiv.org/abs/2405.05949) 
by Jiachen Li, Xinyao Wang, Sijie Zhu, Chia-Wen Kuo, Lu Xu, Fan Chen, Jitesh Jain, Humphrey Shi, Longyin Wen (2024) 
[GitHub](https://github.com/SHI-Labs/CuMo)  

> [!NOTE]
[Original tweet](https://twitter.com/mervenoyann/status/1790665706205307191) (May 15, 2024)