<story><title>Meta AI Unleashes Megabyte, a Scalable Model Architecture</title><url>https://www.artisana.ai/articles/meta-ai-unleashes-megabyte-a-revolutionary-scalable-model-architecture</url></story><parent_chain><item><author>p-e-w</author><text>I don&amp;#x27;t buy the idea (with either architecture) that &amp;quot;10x&amp;quot;-type scaling is required for another breakthrough.&lt;p&gt;Think of a human with below average intelligence. Then think of a human genius. Now consider how incredibly similar their brains are, despite the massive performance gap. It&amp;#x27;s not like one has 10x the number of neurons&amp;#x2F;synapses&amp;#x2F;connections etc. of the other. They&amp;#x27;re both healthy human brains, and you need powerful technology to even distinguish them structurally.&lt;p&gt;Considering this, it seems perfectly possible that a model like GPT-4 is just a hair&amp;#x27;s breadth away from vastly superhuman performance. It certainly beats the average human at many tasks already. The gap between a moron and a genius is a lot larger than that between GPT-4 and a superhuman.</text></item><item><author>jacobn</author><text>My main argument against the AI doomsayers has so far been that the current scaling laws simply make runaway singularity style scenarios algorithmically impossible (if for each step of improvement you need 10x parameters and 100x training, you quickly run into a brick wall).&lt;p&gt;This is part of why I’m not worried about the current crop of generative AI. I am however both curious and concerned about what the tsunami of talent and $$$ chasing the current trend will achieve.&lt;p&gt;If this n^(4&amp;#x2F;3) alt transformer compute scaling is real (and there’s been many a pretender, so it’s too early to tell), then that could fundamentally change the overall AI scaling law, substantially lowering the brick wall.&lt;p&gt;And that could be a game changer.</text></item></parent_chain><comment><author>wudangmonk</author><text>You mean the average brain with about 100 billion neurons with about 1000 connections each bringing it to around 100 trillion connections. With an estimated 1000 &amp;quot;AI&amp;quot; neurons required per biologial neuron.&lt;p&gt;I don&amp;#x27;t think you are givin these &amp;quot;below average&amp;quot; intelligence individuals enough credit. What we consider a genius is the equivalent of a dog show obstacle course. We measure intelligence&amp;#x2F;genius as whatever is hard for humans and completely ignore what is easy because we fail to see the complexity behind the easy stuff.</text></comment>
<story><title>Meta AI Unleashes Megabyte, a Scalable Model Architecture</title><url>https://www.artisana.ai/articles/meta-ai-unleashes-megabyte-a-revolutionary-scalable-model-architecture</url></story><parent_chain><item><author>p-e-w</author><text>I don&amp;#x27;t buy the idea (with either architecture) that &amp;quot;10x&amp;quot;-type scaling is required for another breakthrough.&lt;p&gt;Think of a human with below average intelligence. Then think of a human genius. Now consider how incredibly similar their brains are, despite the massive performance gap. It&amp;#x27;s not like one has 10x the number of neurons&amp;#x2F;synapses&amp;#x2F;connections etc. of the other. They&amp;#x27;re both healthy human brains, and you need powerful technology to even distinguish them structurally.&lt;p&gt;Considering this, it seems perfectly possible that a model like GPT-4 is just a hair&amp;#x27;s breadth away from vastly superhuman performance. It certainly beats the average human at many tasks already. The gap between a moron and a genius is a lot larger than that between GPT-4 and a superhuman.</text></item><item><author>jacobn</author><text>My main argument against the AI doomsayers has so far been that the current scaling laws simply make runaway singularity style scenarios algorithmically impossible (if for each step of improvement you need 10x parameters and 100x training, you quickly run into a brick wall).&lt;p&gt;This is part of why I’m not worried about the current crop of generative AI. I am however both curious and concerned about what the tsunami of talent and $$$ chasing the current trend will achieve.&lt;p&gt;If this n^(4&amp;#x2F;3) alt transformer compute scaling is real (and there’s been many a pretender, so it’s too early to tell), then that could fundamentally change the overall AI scaling law, substantially lowering the brick wall.&lt;p&gt;And that could be a game changer.</text></item></parent_chain><comment><author>nemothekid</author><text>&amp;gt;&lt;i&gt;Think of a human with below average intelligence. Then think of a human genius.&lt;/i&gt;&lt;p&gt;LLMs are not AGI. A human with below average intelligence is still a league above a chimpanzee. A chimpanzee will never be able to read, not because &amp;quot;it&amp;#x27;s too dumb&amp;quot;, but because a chimp&amp;#x27;s brain lacks the actual hardware for reading. The LLM is the chimpanzee. The gap between an LLM and a &amp;quot;human with below average intelligence&amp;quot; is far more than 10x.</text></comment>