<story><title>Practical Tips for Finetuning LLMs Using LoRA (Low-Rank Adaptation)</title><url>https://magazine.sebastianraschka.com/p/practical-tips-for-finetuning-llms</url></story><parent_chain></parent_chain><comment><author>kromem</author><text>I&amp;#x27;ve been increasingly wondering if the field considering LLMs as a continuum as opposed to a set of distinct thresholds is leading to erroneous &amp;quot;rules of thumb&amp;quot; as most research on methodology is concentrated in smaller and more accessible model experimentation right now.&lt;p&gt;We generally recognize (nearly ad nauseum) that mouse models of medical research don&amp;#x27;t necessarily translate to humans.&lt;p&gt;Similarly, I&amp;#x27;d imagine most would laugh at the idea that a neurology researcher who found the best way to get a fruit fly&amp;#x27;s brain to navigate a maze should extrapolate that methodology to a dolphin or a chimp&amp;#x27;s brain.&lt;p&gt;Maybe we should be defining &amp;quot;weight classes&amp;quot; for LLMs and grouping research based on those classes. Like &amp;quot;these are the techniques that work best for lightweight models&amp;quot; but not necessarily assuming those as a general rule of thumb for &amp;quot;heavyweight models.&amp;quot;&lt;p&gt;Even something like the discussion of synthetic data on model collapse is a good example of where there might be a very significant difference in the effect on model quality for a cheaper and less sophisticated model generating synthetic data to feed back into itself and a much more complex and sophisticated model. Maybe the lesson is actually &amp;quot;recursive training on synthetic data leads to model collapse &lt;i&gt;in lightweight and medium weight models&lt;/i&gt;.&amp;quot;&lt;p&gt;So while the writeup is a great one on fine tuning 7B models with LoRA, I would be curious just what % of the recommendations hold true in replication for even just a 65B model.</text></comment>
<story><title>Practical Tips for Finetuning LLMs Using LoRA (Low-Rank Adaptation)</title><url>https://magazine.sebastianraschka.com/p/practical-tips-for-finetuning-llms</url></story><parent_chain></parent_chain><comment><author>hkonsti</author><text>LoRA blew me away the first time I looked into it. Especially since you can host many LoRA adapters at once for a fraction of the cost of hosting an entire model by sharing the base between the adapters. I built a little tool to make LoRA fine-tuning easier. The adapters export to Huggingface. You can check it out here: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;app.haven.run&quot;&gt;https:&amp;#x2F;&amp;#x2F;app.haven.run&lt;/a&gt;</text></comment>