<story><title>Alpha Go Zero: How and Why It Works</title><url>http://tim.hibal.org/blog/alpha-zero-how-and-why-it-works/</url></story><parent_chain><item><author>shghs</author><text>The main reason AlphaGo Zero learns so much faster than its predecessors is because it uses temporal-difference learning.[1] This effectively removes a &lt;i&gt;huge&lt;/i&gt; amount of the value network&amp;#x27;s state space for the learning algorithm to search through, since it bakes in the assumption that a move&amp;#x27;s value ought to equal that of the best available move in the following board position, which is exactly what you&amp;#x27;d expect for a game like Go.&lt;p&gt;A secondary reason for AlphaGo Zero&amp;#x27;s performance is that it combines both value and policy networks into a single network, since it&amp;#x27;s redundant to have two networks for move selection.&lt;p&gt;These are the two biggest distinguishing characteristics of AlphaGo Zero compared to previous AlphaGos, and the OP doesn&amp;#x27;t discuss either of them.&lt;p&gt;[1] &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;en.wikipedia.org&amp;#x2F;wiki&amp;#x2F;Temporal_difference_learning&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;en.wikipedia.org&amp;#x2F;wiki&amp;#x2F;Temporal_difference_learning&lt;/a&gt;</text></item></parent_chain><comment><author>smallnamespace</author><text>Interestingly, the idea behind temporal difference learning is more or less the intuition behind how people price derivatives in finance.&lt;p&gt;The expected value of a contract at time T, estimated at some time t &amp;lt; T, is assumed to be equal (up to discounting) for all t -- e.g. if today we think the contract will be worth $100 a year later, then we also think that the &lt;i&gt;expected estimate, made n months from now, of the value [12-n] months later, will also be $100&lt;/i&gt;. This allows you to shrink the state space considerably.&lt;p&gt;You can usually work out the payoff of a derivatives in different scenarios given rational exercise decisions by all contract participants. The calculation assumes that every market participant makes the best possible decision given the information they had available at the time by either explicitly or implicitly building a tree and working backwards, back-propagating the &amp;#x27;future&amp;#x27; value back to the root.&lt;p&gt;This closely resembles the modeling of a discrete adversarial game, except the payoffs need to make reference to random market variables like the stock price, so the tree nodes are not just indexed by participant action, but also by variables.&lt;p&gt;There&amp;#x27;s actually a nice resemblance between the Longstaff-Schwarz method of pricing American options and MCTS + Alphago, except that the former is using kernel regressions instead of deep neural nets and we sample from a continuous space with an assumed probability distribution instead of a discrete space guided by a policy network [1].&lt;p&gt;[1] &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;people.math.ethz.ch&amp;#x2F;~hjfurrer&amp;#x2F;teaching&amp;#x2F;LongstaffSchwartzAmericanOptionsLeastSquareMonteCarlo.pdf&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;people.math.ethz.ch&amp;#x2F;~hjfurrer&amp;#x2F;teaching&amp;#x2F;LongstaffSchw...&lt;/a&gt;</text></comment>
<story><title>Alpha Go Zero: How and Why It Works</title><url>http://tim.hibal.org/blog/alpha-zero-how-and-why-it-works/</url></story><parent_chain><item><author>shghs</author><text>The main reason AlphaGo Zero learns so much faster than its predecessors is because it uses temporal-difference learning.[1] This effectively removes a &lt;i&gt;huge&lt;/i&gt; amount of the value network&amp;#x27;s state space for the learning algorithm to search through, since it bakes in the assumption that a move&amp;#x27;s value ought to equal that of the best available move in the following board position, which is exactly what you&amp;#x27;d expect for a game like Go.&lt;p&gt;A secondary reason for AlphaGo Zero&amp;#x27;s performance is that it combines both value and policy networks into a single network, since it&amp;#x27;s redundant to have two networks for move selection.&lt;p&gt;These are the two biggest distinguishing characteristics of AlphaGo Zero compared to previous AlphaGos, and the OP doesn&amp;#x27;t discuss either of them.&lt;p&gt;[1] &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;en.wikipedia.org&amp;#x2F;wiki&amp;#x2F;Temporal_difference_learning&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;en.wikipedia.org&amp;#x2F;wiki&amp;#x2F;Temporal_difference_learning&lt;/a&gt;</text></item></parent_chain><comment><author>gwern</author><text>&amp;gt; These are the two biggest distinguishing characteristics of AlphaGo Zero compared to previous AlphaGos, and the OP doesn&amp;#x27;t discuss either of them.&lt;p&gt;David Silver disagrees. The most critical distinguishing characteristic is the expert&amp;#x2F;tree iteration which makes stable self-play possible at all.</text></comment>