<story><title>Meta AI Unleashes Megabyte, a Scalable Model Architecture</title><url>https://www.artisana.ai/articles/meta-ai-unleashes-megabyte-a-revolutionary-scalable-model-architecture</url></story><parent_chain></parent_chain><comment><author>mkaic</author><text>Ugh. I just spent the past few days reading this exact paper and preparing a detailed presentation on it for my job as an AI researcher, and headlines like this make me roll my eyes very hard.&lt;p&gt;MEGABYTE is indeed a cool new architecture, but it is still very much just a proof of concept at the moment. The paper shows that the model can compete with (but not decimate) vanilla Transformers on the scale of ~1B parameters in long-sequence prediction tasks. They did not &amp;quot;unleash&amp;quot; anything, and scalability to very large parameter counts and datasets still has not been tested.&lt;p&gt;I&amp;#x27;m certainly very excited to see where this architecture goes as the community gets ahold of it and starts developing it, but to call it &amp;quot;revolutionary&amp;quot; this early on is disingenuous. I personally have a few experiments I want to run with it, but I put the probability of it being a true GPT-killer at &amp;lt;20%. I would love to be wrong, though!</text></comment>
<story><title>Meta AI Unleashes Megabyte, a Scalable Model Architecture</title><url>https://www.artisana.ai/articles/meta-ai-unleashes-megabyte-a-revolutionary-scalable-model-architecture</url></story><parent_chain></parent_chain><comment><author>jacobn</author><text>My main argument against the AI doomsayers has so far been that the current scaling laws simply make runaway singularity style scenarios algorithmically impossible (if for each step of improvement you need 10x parameters and 100x training, you quickly run into a brick wall).&lt;p&gt;This is part of why I’m not worried about the current crop of generative AI. I am however both curious and concerned about what the tsunami of talent and $$$ chasing the current trend will achieve.&lt;p&gt;If this n^(4&amp;#x2F;3) alt transformer compute scaling is real (and there’s been many a pretender, so it’s too early to tell), then that could fundamentally change the overall AI scaling law, substantially lowering the brick wall.&lt;p&gt;And that could be a game changer.</text></comment>