<story><title>Llama3 implemented from scratch</title><url>https://github.com/naklecha/llama3-from-scratch</url></story><parent_chain><item><author>brcmthrowaway</author><text>Wait, are you saying SoTA NN research hasnt evolved from hardcoding a bunch of layer structures and sizes?&lt;p&gt;I&amp;#x27;m kind of shocked. I thought there would be more dynamism by now and I stopped dabbling in like 2018.</text></item></parent_chain><comment><author>curious_cat_163</author><text>There is a tick-tock between searching the dominant NN architectures (tick) and optimizing for accuracy, compute and inference latency and throughput (tock).&lt;p&gt;This particular (tock) is still playing out. The next (tick) does not feel imminent and will likely depend on when we discover the limits of the transformers when it comes to solving for long tail of use-cases.&lt;p&gt;My $0.02.</text></comment>
<story><title>Llama3 implemented from scratch</title><url>https://github.com/naklecha/llama3-from-scratch</url></story><parent_chain><item><author>brcmthrowaway</author><text>Wait, are you saying SoTA NN research hasnt evolved from hardcoding a bunch of layer structures and sizes?&lt;p&gt;I&amp;#x27;m kind of shocked. I thought there would be more dynamism by now and I stopped dabbling in like 2018.</text></item></parent_chain><comment><author>delusional</author><text>The innovation is the amount of resources people are willing to spend right now. From looking at the research code it&amp;#x27;s clear that the whole field is basically doing a (somewhat) guided search in the entire space of possible layer permutations.&lt;p&gt;There seems to be no rhyme or reason, no scientific insight, no analysis. They just try a million different permutations, and whatever scores the highest on the benchmarks gets published.</text></comment>