<story><title>How will AI learn next?</title><url>https://www.newyorker.com/science/annals-of-artificial-intelligence/how-will-ai-learn-next</url></story><parent_chain><item><author>rented_mule</author><text>Anyone who has iterated on trained models for long enough knows that feedback loops can be a serious problem. If your models are influencing the generation of data that they are later retrained on, it gets harder and harder to even maintain model performance. The article mentions one experiment in this direction: &amp;quot;With each generation, the quality of the model actually degraded.&amp;quot; This happens whenever there aren&amp;#x27;t solid strategies to avoid feedback loop issues.&lt;p&gt;Given this, the problem isn&amp;#x27;t just that there&amp;#x27;s not enough new content. It&amp;#x27;s that an ever-increasing fraction of the content in the public sphere will be generated by these models. And can the models detect that they are ingesting their own output? If they get good enough, they probably can&amp;#x27;t. And then they&amp;#x27;ll get worse.&lt;p&gt;This could have a strange impact on human language &amp;#x2F; communication as well. As these models are increasingly trained on their own output, they&amp;#x27;ll start emulating their own mistakes and more of the content we consume will have these mistakes consistently used. You can imagine people, sometimes intentionally and sometimes not, starting to emulate these patterns and causing shifts in human languages. Interesting times ahead...</text></item></parent_chain><comment><author>Nevermark</author><text>Humans have that to, but the reason civilization doesn&amp;#x27;t go full crazy is our use of language and concepts are tied to doing objective things, which keeps things (mostly) grounded.&lt;p&gt;Where it isn’t grounded, as in endless online conversations with like minded people (closed loop feedback) about informally abstracted (poorly specified constraints) and emotion invoking (high reinforcement) topics, people go batshit too.&lt;p&gt;So the more AI models actually practice what they know in objective environments, the more likely that output-&amp;gt;input feedback will inform introspection toward self-improvement, and less like an iterative calculation of the architecture’s resonant  frequencies or eigenvalues.</text></comment>
<story><title>How will AI learn next?</title><url>https://www.newyorker.com/science/annals-of-artificial-intelligence/how-will-ai-learn-next</url></story><parent_chain><item><author>rented_mule</author><text>Anyone who has iterated on trained models for long enough knows that feedback loops can be a serious problem. If your models are influencing the generation of data that they are later retrained on, it gets harder and harder to even maintain model performance. The article mentions one experiment in this direction: &amp;quot;With each generation, the quality of the model actually degraded.&amp;quot; This happens whenever there aren&amp;#x27;t solid strategies to avoid feedback loop issues.&lt;p&gt;Given this, the problem isn&amp;#x27;t just that there&amp;#x27;s not enough new content. It&amp;#x27;s that an ever-increasing fraction of the content in the public sphere will be generated by these models. And can the models detect that they are ingesting their own output? If they get good enough, they probably can&amp;#x27;t. And then they&amp;#x27;ll get worse.&lt;p&gt;This could have a strange impact on human language &amp;#x2F; communication as well. As these models are increasingly trained on their own output, they&amp;#x27;ll start emulating their own mistakes and more of the content we consume will have these mistakes consistently used. You can imagine people, sometimes intentionally and sometimes not, starting to emulate these patterns and causing shifts in human languages. Interesting times ahead...</text></item></parent_chain><comment><author>HDThoreaun</author><text>I think it&amp;#x27;s reasonable to say that this was actually the point of releasing LLMs publicly. The companies that created them wanted a moat and figured the data they had could be it if they poisoned anyones attempt to collect the same data in the future.</text></comment>