<story><title>A neural network solves and generates mathematics problems by program synthesis</title><url>https://arxiv.org/abs/2112.15594</url></story><parent_chain></parent_chain><comment><author>bcaine</author><text>Not to pour too much cold water on this, but the claim of 100% accuracy has a huge caveat. In the paper (Page 4) they state:&lt;p&gt;&lt;i&gt;Interaction. The original question may not be a prompt that synthesizes a program whose execution results in the correct answer. In addition, the answer may require multiple steps with clear plots or other modalities. We therefore may interactively prompt Codex until reaching the correct answer or visualizations, making the minimum necessary changes from the original question&lt;/i&gt;&lt;p&gt;Which to me basically sounds like they had a human in the loop (that knows how to solve these math problems) that kept changing the question until it gave the correct answer. They do measure the distance (using a sentence embedding model) of the original question to the one that yielded the correct answer, but that feels a bit contrived to me.&lt;p&gt;Nevertheless, its still really cool that the correct answer is indeed inside the model.</text></comment>
<story><title>A neural network solves and generates mathematics problems by program synthesis</title><url>https://arxiv.org/abs/2112.15594</url></story><parent_chain></parent_chain><comment><author>vanusa</author><text>&lt;i&gt;This represents a milestone for higher education.&lt;/i&gt;&lt;p&gt;Just from that sentence, you know the paper is suspect.&lt;p&gt;Whatever the merits of the algorithm -- there&amp;#x27;s no reason to be believe that what it produces (a set of random auto-generated questions) will be useful for actual &lt;i&gt;education&lt;/i&gt; in any way.&lt;p&gt;There are more red flags -- see autogen question U9, for example:&lt;p&gt;&lt;pre&gt;&lt;code&gt;    If f(x) = (x^2 - x) &amp;#x2F; (x^2 + x), what is f(2)?
&lt;/code&gt;&lt;/pre&gt;
When was the last time your human teacher (at university level) handed you a rational polynomial that was not in reduced terms?&lt;p&gt;And my favorite, X7:&lt;p&gt;&lt;pre&gt;&lt;code&gt;    Find the smallest prime number that is divisible by 7 and 13.&lt;/code&gt;&lt;/pre&gt;</text></comment>