<story><title>Lee Sedol Beats AlphaGo in Game 4</title><url>https://gogameguru.com/alphago-4/</url></story><parent_chain><item><author>argonaut</author><text>If it&amp;#x27;s true that AlphaGo started making a series of bad moves after its mistake on move 79, this might tie into a classic problem with agents trained using reinforcement learning, which is that after making an initial mistake (whether by accident or due to noise, etc.), the agent gets taken into a state it&amp;#x27;s not familiar with, so it makes another mistake, digging an even deeper hole for itself - the mistakes then continue to compound. This is one of the &lt;i&gt;biggest&lt;/i&gt; challenges with RL agents in the real, physical world, where you have noise and imperfect information to confront.&lt;p&gt;Of course, a plausible alternate explanation is that AlphaGo felt like it needed to make risky moves to catch up.</text></item></parent_chain><comment><author>jacquesm</author><text>The same happens to people, especially people that study theory. You can totally throw them off their game by making a non-standard move, even a relatively bad one as long as it breaks their existing pre-conceived notions about how the game should progress.&lt;p&gt;Of course against a really strong player you&amp;#x27;re going to get beaten after that but a weak player strong on theory will have a harder time.</text></comment>
<story><title>Lee Sedol Beats AlphaGo in Game 4</title><url>https://gogameguru.com/alphago-4/</url></story><parent_chain><item><author>argonaut</author><text>If it&amp;#x27;s true that AlphaGo started making a series of bad moves after its mistake on move 79, this might tie into a classic problem with agents trained using reinforcement learning, which is that after making an initial mistake (whether by accident or due to noise, etc.), the agent gets taken into a state it&amp;#x27;s not familiar with, so it makes another mistake, digging an even deeper hole for itself - the mistakes then continue to compound. This is one of the &lt;i&gt;biggest&lt;/i&gt; challenges with RL agents in the real, physical world, where you have noise and imperfect information to confront.&lt;p&gt;Of course, a plausible alternate explanation is that AlphaGo felt like it needed to make risky moves to catch up.</text></item></parent_chain><comment><author>kotach</author><text>What you are talking about here is called &amp;quot;label bias&amp;quot;. [2] It is present only if training is done badly.&lt;p&gt;When you have a game of Go, or Super Mario level. You don&amp;#x27;t want to make your decisions by just checking the local features and doing them, because it can be the case that by compounding errors you end up in a state you never saw, and all of the future decisions won&amp;#x27;t be good.&lt;p&gt;One can avoid these situations by training jointly over the whole game.&lt;p&gt;For example, maximum entropy models can work for decision making problems but their training leaves them in a &amp;quot;label bias&amp;quot; state because the training is trying to minimize loss of local decisions, instead of trying to minimize the future regret of current local decision.&lt;p&gt;The solution to these label bias problems are Conditional Random Fields, or Hidden Markov Models. You could accomplish the same with Recursive Neural Networks if you trained them properly. For example, there is no search part (monte carlo tree search, or dynamic programming [viterbi] like it is in CRFs or HMMs) in RNNs but they are completely adequate for decision based problems (sequence labeling etc.). Why is that the case? Because search results are present in the data, there&amp;#x27;s no need to search if you can just learn to search from the data.&lt;p&gt;If DeepMind open-sourced the hundreds of millions of games that AlphaGo played, it is quite possible to train a model that wouldn&amp;#x27;t need a Monte Carlo search and would work quite well, because you would learn the model to make local decisions to minimize future regret, not to minimize its local loss. [1]&lt;p&gt;The only reason why reinforcement learning is used is because there are too few human games of Go available for the model to generalize well. Reinforcement learning can be used in the setting of joint learning because you play out the whole game before you do the learning. This means that you can try to learn a classifier that will minimize the regret by making a proper local decision. Although, as far as I know, and can see from the paper, they didn&amp;#x27;t train AlphaGo jointly over the game sequence.&lt;p&gt;But! Now they have a lot of data and they can repeat the process.&lt;p&gt;[1]: &lt;a href=&quot;http:&amp;#x2F;&amp;#x2F;arxiv.org&amp;#x2F;abs&amp;#x2F;1502.02206&quot; rel=&quot;nofollow&quot;&gt;http:&amp;#x2F;&amp;#x2F;arxiv.org&amp;#x2F;abs&amp;#x2F;1502.02206&lt;/a&gt;&lt;p&gt;[2]: &lt;a href=&quot;http:&amp;#x2F;&amp;#x2F;repository.upenn.edu&amp;#x2F;cgi&amp;#x2F;viewcontent.cgi?article=1162&amp;amp;context=cis_papers&quot; rel=&quot;nofollow&quot;&gt;http:&amp;#x2F;&amp;#x2F;repository.upenn.edu&amp;#x2F;cgi&amp;#x2F;viewcontent.cgi?article=1162...&lt;/a&gt;</text></comment>