<story><title>Non-determinism in GPT-4 is caused by Sparse MoE</title><url>https://152334H.github.io/blog/non-determinism-in-gpt-4/</url></story><parent_chain></parent_chain><comment><author>jiggawatts</author><text>Floating point inaccuracies are generally deterministic - running the same calculations twice ought to yield the same results, down to the bit.&lt;p&gt;You only get divergent results if there is some other source of state or entropy: not zeroing buffers correctly, race conditions, not setting rounding mode flags consistently, etc…&lt;p&gt;From the quality of the code I’ve seen being cobbled together in the AI&amp;#x2F;ML ecosystem I would assume all three of those issues going on, and maybe more.</text></comment>
<story><title>Non-determinism in GPT-4 is caused by Sparse MoE</title><url>https://152334H.github.io/blog/non-determinism-in-gpt-4/</url></story><parent_chain></parent_chain><comment><author>alpark3</author><text>_If_ 3.5 is a MoE model, doesn&amp;#x27;t that give a lot of hope to open source movements? Once a good open source MoE model comes out, maybe even some type of variation of the decoder models available(I don&amp;#x27;t know whether MoE models have to be trained from scratch), that implies a lot more can be done with a lot less.</text></comment>