<story><title>How Meta trains large language models at scale</title><url>https://engineering.fb.com/2024/06/12/data-infrastructure/training-large-language-models-at-scale-meta/</url></story><parent_chain></parent_chain><comment><author>dudus</author><text>&amp;gt; Since we did not have time to change the cooling infrastructure, we had to remain in an air-cooled environment. The mechanical and thermal designs had to change to accommodate this, and that triggered a validation cycle to support a large-scale deployment.&lt;p&gt;&amp;gt; All of these hardware-related changes were challenging because we had to find a solution that fit within the existing resource constraints, with a very small degree of freedom to change and meet a tight schedule.&lt;p&gt;Seems like the time constraints put into the team impacted the overall quality of the model.</text></comment>
<story><title>How Meta trains large language models at scale</title><url>https://engineering.fb.com/2024/06/12/data-infrastructure/training-large-language-models-at-scale-meta/</url></story><parent_chain></parent_chain><comment><author>xvector</author><text>&amp;gt; So we decided to build both: two 24k clusters, one with RoCE and another with InfiniBand. Our intent was to build and learn from the operational experience.&lt;p&gt;I love how they built two completely insane clusters just to learn. That&amp;#x27;s badass.</text></comment>