<story><title>MM1: Methods, Analysis and Insights from Multimodal LLM Pre-training</title><url>https://arxiv.org/abs/2403.09611</url></story><parent_chain></parent_chain><comment><author>nl</author><text>This is an awesome paper, and the somewhat negative sentiment in the discussion here is surprising.&lt;p&gt;The ablation studies are well done, comprehensive and expensive to do. People will be using the conclusions from this for years, and that is much more impactful than if an upcoming Siri product ourperforms the GPT model at that same point in time.&lt;p&gt;A few really interesting points:&lt;p&gt;Synthetic datasets substantially (1%+) increase performance for Image Encoder Pre-training&lt;p&gt;Architecture of the Visual&amp;lt;-&amp;gt;Language model connector doesn&amp;#x27;t seem to matter.&lt;p&gt;Interleaving text and image data improves few shot performance, but image captioning data improves zero-shot numbers.&lt;p&gt;The ideal mix of data types is 5:5:1 for Interleaved:Captions:Plain Text (!)&lt;p&gt;Synthetic captioning data helps substantially at this point too (up to 4% gain)&lt;p&gt;The appendices are amazing: lots of details about learning rates tried, batch sizes.&lt;p&gt;The &amp;quot;explain these figures&amp;quot; are really really good. See page 37.</text></comment>
<story><title>MM1: Methods, Analysis and Insights from Multimodal LLM Pre-training</title><url>https://arxiv.org/abs/2403.09611</url></story><parent_chain></parent_chain><comment><author>reaperman</author><text>This looks competitive against CLIP, and surprisingly great at VQA style prompts, but it doesn&amp;#x27;t seem like the paper supports comparing it to GPT-4. We don&amp;#x27;t see any tests for coding performance, math homework, legal document review, or any of the myriad other things that people use GPT-4 for on a daily basis.</text></comment>