
By assumption, the current efficiency is \(\pi _X\left(\mathbf {p},\mathbf {q}\right)\!+\!\pi _Y\left(\mathbf {p},\mathbf {q}\right)\!=\!2R\) . After any deviation of player \(X\)  to some other strategy \(\mathbf {p^{\ast }}\) , this efficiency takes the form
\(\pi _X\left(\mathbf {p^{\ast }},\mathbf {q}\right)\!+\!\pi _Y\left(\mathbf {p^{\ast }},\mathbf {q}\right) = a_1\,\left(2R\right) + a_2\, \left(T\!+\!S\right) + a_3\, \left(2P\right) , where a_{1},a_{2},a_{3}\geqslant 0 and a_{1}+a_{2}+a_{3}=1. Because R\! \geqslant \!\left(T\!+\!S\right) /2 and R\!>\!P, this efficiency is at most 2R.\item By assumption, the current efficiency is \pi _X\left(\mathbf {p},\mathbf {q}\right)\!+\!\pi _Y\left(\mathbf {p},\mathbf {q}\right)\!=\!T\!+\!S. After a deviation, the efficiency again takes the form of Eq.~\ref {eq:efficiency_deviation}. Because now \left(T\!+\!S\right) /2 \!\ge \! R \!\ge \! P, again it follows that this efficiency is at most T\!+\!S.\item First, assume that \left(T\!+\!S\right)/2 \! > \!P, and suppose player X deviates by choosing a strategy \mathbf {p^{\ast }}\!=\!\mathbf {p}\!+\!\varepsilon \, \mathbf {e}_{DD}, with \varepsilon \!>\!0 being arbitrarily small. Again, the efficiency after the deviation takes the form of Eq.~\ref {eq:efficiency_deviation}, with either a_1 or a_2 strictly positive. Because R\!>\!P and \left(T\!+\!S\right) /2 \! >\!P, this efficiency is larger than the current efficiency of 2P. Hence, player X is able to locally improve its objective function.\) 
For the other direction, assume now that \(\left(T\!+\!S\right)/2 \! \leqslant \!P\)  and that player \(Y\)  adopts the strategy ALLD. Then any deviation \(\mathbf {p^{\ast }}\)  by player \(X\)  leads to an efficiency of the form
\(\pi _X\left(\mathbf {p^{\ast }},\mathbf {q}\right)\!+\!\pi _Y\left(\mathbf {p^{\ast }},\mathbf {q}\right) = a\, \left(T\!+\!S\right) +\left(1\!-\!a\right)\, \left(2P\right) ,which is at most 2P. Thus player X is unable to locally (or globally) improve its objective function.\) 
In particular, this result implies that in donation games, players cannot settle at mutual defection if at least one of them applies FMTL. On the other hand, in the special case of a prisoner's dilemma with \((T\!+\!S)/2 \! \leqslant \!P\) , mutual defection can be stable even if both players value efficiency. This effect arises because in our framework (and in evolutionary game theory more generally), learners adapt their strategies independently from each other. In games in which one player alone is unable to escape from inefficient strategy profiles, players may thus end up in a rest point that they both wish to avoid.
Evolutionary dynamics of learning rules
Here, we briefly outline several qualitative observations about the evolutionary dynamics of learning rules. Recall that the dynamics we consider depend on mean payoffs only. Suppose that \(\mathcal {R}\)  and \(\mathcal {S}\)  are learning rules. For \(\mathcal {X},\mathcal {Y}\in \left\lbrace \mathcal {R},\mathcal {S}\right\rbrace \) , we let \(a_{\mathcal {X}\mathcal {Y}}\left(n\right)\)  denote the mean payoff to \(\mathcal {X}\)  against \(\mathcal {Y}\)  after \(n\)  learning steps, averaged over initial conditions and learning trajectories.
These mean payoffs motivate a qualitative comparison among learning rules based on how they perform against one another, on average. We consider three broad conditions:

\(\mathcal {R}\)  exploits \(\mathcal {S}\)  after \(n\)  steps if \(a_{\mathcal {R}\mathcal {S}}\left(n\right) >a_{\mathcal {S}\mathcal {R}}\left(n\right)\) ;

\(\mathcal {R}\)  is fair relative to \(\mathcal {S}\)  after \(n\)  steps if \(a_{\mathcal {R}\mathcal {S}}\left(n\right) =a_{\mathcal {S}\mathcal {R}}\left(n\right)\) ;

\(\mathcal {R}\)  is exploited by \(\mathcal {S}\)  after \(n\)  steps if \(a_{\mathcal {R}\mathcal {S}}\left(n\right) <a_{\mathcal {S}\mathcal {R}}\left(n\right)\) .

Note that each of these conditions disregards the performance of a learning rule against itself, i.e. \(a_{\mathcal {R}\mathcal {R}}\left(n\right)\)  and \(a_{\mathcal {S}\mathcal {S}}\left(n\right)\) . As we will see below, these conditions do not alone allow one to conclude anything about evolutionary stability, but they do provide reasonable classifications for a first comparison of two learning rules.
Example 1 In Fig. REF b, team-based learning (maximizing \(\pi _{X}+\pi _{Y}\) ) is exploited by selfish learning, and selfish learning exploits team-based learning (after sufficiently many learning steps). In all of our examples, FMTL is fair relative to selfish learning after long enough learning horizons, but it can be exploited by selfish learning on shorter timescales (e.g. Fig. REF a). However, as the example of Fig. REF a shows, FMTL can still be evolutionary stable relative to selfish learning even when it is being exploited, due to the superior performance of FMTL against itself relative to selfish learning against itself. By symmetry, any learning rule is fair relative to itself.

Example 2 Selfish learning, i.e. caring about one's own payoff only, can actually still be exploited by some learning rules. To see why, we can formulate a learning rule that takes advantage of selfish learners. Suppose that \(\chi >1\) . In the donation game with benefit \(b\)  and cost \(c\) , consider the following learning rule: if \(\pi _{X}\geqslant \chi \pi _{Y}\) , then maximize \(\pi _{X}+\pi _{Y}\) ; otherwise, minimize \(\left|\pi _{X}-\chi \pi _{Y}\right|\) . When \(b=2\) , \(c=1\) , \(\chi =2\) , and this learning rule is paired with a selfish learner, the mean payoffs to the two learners over \(10^{5}\)  runs are \(1.1788\)  and \(0.5894\) , respectively. We do not wish to dwell on this particular learning rule; rather, this example is intended simply to illustrate that even selfish learners can be exploited by learners with sufficiently coercive preferences.

The following result says that if \(\mathcal {S}\)  does not exploit an opposing learning rule, \(\mathcal {R}\) , then \(\mathcal {S}\)  cannot be evolutionary stable relative to \(\mathcal {R}\)  whenever \(\mathcal {R}\)  is optimal against itself. (To simplify the notation, we suppress the number of learning steps, \(n\) .)
Proposition 4 
Suppose that \(\mathcal {R}\)  either exploits \(\mathcal {S}\)  or is fair relative to \(\mathcal {S}\) , and suppose furthermore that \(a_{\mathcal {R}\mathcal {R}}\)  is maximal (which can be determined by looking at the feasible region for the repeated game). Then, either (i) \(\mathcal {R}\)  is neutral relative to \(\mathcal {S}\)  or (ii) \(\mathcal {R}\)  is evolutionarily stable relative to \(\mathcal {S}\) .

Since \(\mathcal {R}\)  exploits \(\mathcal {S}\)  or is fair relative to \(\mathcal {S}\) , we have \(a_{\mathcal {R}\mathcal {S}}\geqslant a_{\mathcal {S}\mathcal {R}}\) . Moreover, since \(a_{\mathcal {R}\mathcal {R}}\)  is maximal and the feasible region is convex, we must have \(a_{\mathcal {R}\mathcal {R}}\geqslant a_{\mathcal {S}\mathcal {R}}\) . If \(a_{\mathcal {R}\mathcal {R}}>a_{\mathcal {S}\mathcal {R}}\) , then \(\mathcal {R}\)  is automatically evolutionarily stable relative to \(\mathcal {S}\) . But if \(a_{\mathcal {R}\mathcal {R}}=a_{\mathcal {S}\mathcal {R}}\) , then \(a_{\mathcal {R}\mathcal {S}}\geqslant a_{\mathcal {S}\mathcal {R}}=a_{\mathcal {R}\mathcal {R}}\geqslant a_{\mathcal {S}\mathcal {S}}\) . If these inequalities are all equalities, then \(\mathcal {R}\)  is neutral relative to \(\mathcal {S}\) ; otherwise, \(\mathcal {R}\)  satisfies Maynard Smith's second condition [1]} and is thus evolutionarily stable relative to \(\mathcal {S}\) .

For example, in the battle of the sexes game, we know that FMTL is both fair and optimal against itself after sufficiently many learning steps (Fig. REF b). Thus, selfish learning cannot be evolutionary stable relative to FMTL. In this case, since selfish learning is also optimal against itself, FMTL is neutral relative to selfish learning. Of course, one can always deduce evolutionary stability by looking at the mean payoffs \(a_{\mathcal {X}\mathcal {Y}}\)  themselves. Proposition  REF simply says that a learner who performs optimally against itself is at least neutral against any nonexploitative opponent.
