Alejandro Rosas

Evolutionary game theory meets social science: Is there a unifying rule for human cooperation?

Evolutionary game theory has shown that human cooperation thrives in different types of social interactions with a PD structure. Models treat the cooperative strategies within the different frameworks as discrete entities and sometimes even as contenders. Whereas strong reciprocity was acclaimed as superior to classic reciprocity for its ability to defeat defectors in public goods games, recent experiments and simulations show that costly punishment fails to promote cooperation in the IR and DR games, where classic reciprocity succeeds. My aim is to show that cooperative strategies across frameworks are governed by a common underlying rule or norm. An analysis of the reputation and action rules that govern some representative cooperative strategies both in models and in economic experiments confirms that different frameworks share a conditional action rule and several reputation rules. The common conditional rule contains an option between costly punishment and withholding benefits that provides alternative enforcement methods against defectors. Depending on the framework, individuals can switch to the appropriate strategy and method of enforcement. The stability of human cooperation looks more promising if one mechanism controls successful strategies across frameworks. 12

INTRODUCTION

For several decades, evolutionary game theory has investigated the evolutionary dynamics of cooperative strategies with computer simulations and analytical methods. Social scientists studying human cooperation with the tools of evolutionary game theory usually understand strategies as standing for such human rules or norms as they encounter in their everyday scientific practice. Taken strictly, talk of strategies and rules in evolutionary game theory makes no reference to the causes of behaviour, neither psychological nor social. Rules stand in the theory for the action patterns displayed towards all possible situations in the game. For example, TIT FOR TAT is the rule: "cooperate in the first move, thereafter copy in move t+1 whatever your opponent did in move t" [START_REF] Axelrod | The evolution of cooperation[END_REF]. However, when it comes specifically to models of human cooperation, rules can be taken to refer to the social and/or psychological causes of behaviour. Human cognition and action involves rules in this causal sense.

The viewpoint of social scientists suggests a question that seems of some consequence for the current debate about the evolution of human cooperation. This paper is devoted to posing 1 KLI for Evolution and Cognition Research, Altenberg, Austria. Email: alejandro.rosas@kli.ac.at 2 Dept. of Philosophy, Univ. Nacional de Colombia, Bogotá, Colombia.

Email: arosasl@unal.edu. co and trying to answer it. It is the question whether the cooperative strategies that succeed in different interaction frameworks and are separately investigated in evolutionary game theory could in fact result from one and the same psychological mechanism in a rule-like format, adjustable to the variety of frameworks. Cooperation thrives against defectors in various PD frameworks, namely in two-person repeated games with the same person, in two-person repeated games with different persons, and in nperson (n>>2) repeated games with the same or different persons. In each of them cooperative strategies are studied as discrete entities, but perhaps a common rule underlies them all. Adopting a unifying perspective based on the concept of a strategy as a rule, I argue that cooperation across frameworks can be viewed as sharing an underlying common rule prescribing retaliation or punishment against defectors. I also review some experiments that support this same claim. A caveat: though cooperative strategies can in principle occur in any social organism, the claim about a common normative mechanism is raised here only in relation to human cooperation.

COOPERATIVE STRATEGIES WITHIN AND ACROSS FRAMEWORKS

Since Trivers' (1971) classic paper and [START_REF] Axelrod | The evolution of cooperation[END_REF] formal treatment of reciprocity, several cooperative strategies have proven successful at the population level against uncooperative strategies such as ALLD or other selfish strategies in repeated PD games. Cooperative strategies exploit mutual rewards through reciprocity in direct and indirect two-person games; or they cooperate in n-person public goods games (PGG) and punish in a linked dyadic game to enhance cooperation in the PGG. Especially in the two-person games, strategies are legion. However, I will introduce common labels for the successful cooperative strategies within each type of game. Within two-person repeated games, strategies like TIT FOR TAT, CONTRITE TIT FOR TAT, TIT FOR TWO TATS and others that exploit reciprocity against ALLD or against conditionally uncooperative strategies, are here labelled direct reciprocity (DR) strategies. Their common element is to cooperate in repeated interactions with the same player, when the player cooperates and avoids cheating behaviour. Analogously, cooperative strategies that succeed through reciprocity in the evolutionary dynamics of repeated two-person games where players never meet twice in exchanged roles are labelled indirect reciprocity (IR) strategies. They cooperate with opponents if these have a good reputation; otherwise they defect. IR strategies are sometimes called after the reputation rule they follow, e.g., SCORING, STANDING and JUDGING; more recently, some theorists refer to reputation rules as social norms and reserve 'strategy' for the action rule that prescribes an action

A c c e p t e d m a n u s c r i p t

for each possible game situation. Proponents of strong reciprocity (SR) regard it as a single strategy. The various possible combinations of reputation rules with punishment have not been an issue for theorists until very recently (see Ohtsuki et al. 2009).

The existence of multiple strategies within the first two frameworks may seem, at first sight, to contradict the aim of this paper. The common labels may seem an arbitrary device to artificially create unity where there is none. However, this paper is not concerned with reducing multiplicity of cooperative strategies within game-types; it is concerned with reducing it across game-types. Unity across game-types is compatible with variation within them. Suppose that there are two irreducible cooperative strategies within the direct reciprocity framework. As long as these two strategies have analogues within the indirect and the strong reciprocity frameworks, rules will be common across frameworks.

The merits of the cooperative strategies across frameworks are often discussed in a competitive attitude: for example, when direct or indirect reciprocity are compared to strong reciprocity and the superiority of the latter is defended or contested [START_REF] Gintis | Explaining altruistic behavior in humans[END_REF][START_REF] Burnham | The Biological and Evolutionary Logic of Human Cooperation[END_REF][START_REF] Ohtsuki | Indirect reciprocity provides only a narrow margin of efficiency for costly punishment[END_REF]). Typically, proponents of SR have claimed its unique success against free riders in public goods games. Though PGGs remain an important justification for the existence of SR strategies, this claim needs revision in the face of some experiments [START_REF] Milinski | Reputation helps solve the 'tragedy of the commons[END_REF]; see 8 below). The claim that SR stands out as a biologically and psychologically altruistic strategy [START_REF] Gintis | Explaining altruistic behavior in humans[END_REF] against the selfish character of IR and DR strategies often accentuates the rivalry. But the altruistic character of SR has been justly questioned (Haley and Fessler 2004;[START_REF] Quervain | The neural basis of altruistic punishment[END_REF]Rosas 2008a). Moreover, discovering a common rule to all cooperative strategies across the different frameworks would surely deflate the rivalry. These strategies share a common concern with deterring free riders, which is, e.g., irrelevant in cooperation through by-product mutualism or kin selection.

STRATEGIES AS RULES OF TWO TYPES

Taken cognitively, cooperative action rules establish in propositional format whether to give or not, depending on the reputation of the recipient and/or the actor. Taking both reputations into account, there are four possible game situations to consider and a rule consists in a string of four values that establish whether to cooperate or defect in each of these four situations. There are 16 (2 4 ) such strings or action rules. Not all strategies take both reputations into account. CO is the typical cooperative action rule for IR that looks only at the recipient's reputation and prescribes to give to the good, and not to give to the bad. But if the actor's reputation depends on its action, it is in the actor's interest to look at its own reputation before acting [START_REF] Leimar | Evolution of cooperation through indirect reciprocation[END_REF][START_REF] Brandt | The logic of reprobation: assessment and action rules for indirect reciprocation[END_REF]. SELF is the uncooperative action rule that dictates cooperation only when the actor's reputation is bad. Action rules can also take account of both the actor's and the recipient's score. AND is the rule that prescribes to give only when the recipient is good and the actor is bad. OR prescribes to give when either the recipient is good or the actor is bad. Simulations in [START_REF] Brandt | The logic of reprobation: assessment and action rules for indirect reciprocation[END_REF] and analyses in [START_REF] Ohtsuki | How should we define goodness?reputation dynamics in indirect reciprocity[END_REF] show that SELF and AND are not evolutionary successful, whereas CO and OR achieve high fitness. CO and OR may be irreducible cooperative rules; yet our goal here is not to eliminate variation within a game's framework, but to find out if there are common rules shared by cooperative strategies across frameworks.

A conditional action rule like CO will operate together with a reputation module, which assesses the reputation of potential recipients according to a rule. In most models reputations are simplified to binary scores-either good (G) or bad (B) -though in real life reputation is assigned in a continuous scale. For the discussion here I will assume the simplified binary model. The rule must take as input the previous action of the potential recipient, and possibly the reputation of its respective recipient and/or its reputation when it moved as donor. Taking all three factors into account there are eight possible actions to assess. A reputation rule is a string of eight values giving the reputations, G or B, for these eight actions. In total there are 256 (2 8 ) "third order assessment" rules. Multiplying 256 times the 16 possible actions rules results in 4096 possible strategies just for the IR game type [START_REF] Ohtsuki | How should we define goodness?reputation dynamics in indirect reciprocity[END_REF]. Fortunately for our purpose, the number of relevant strategies is much smaller. Not all action rules are cooperative, and not all possible reputation rules promote cooperative action rules. For example, the cooperative action rule known as CO, which prescribes action on the basis of the reputation of the recipient only ("give to the good and do not give to the bad") will undermine itself if it uses SCORING, a reputation rule that assigns scores only on the basis of past actions. SCORING contradicts CO because it assesses a potential recipient as bad if it previously refused to give to the bad, and as good if it gave to the bad.

Because of this problem Ohtsuki and Iwasa (2004) carried out an exhaustive search of evolutionary stable combinations of reputation rules with cooperative action rules. They found eight such ESS combinations that exclusively use cooperative action rules CO (six) and OR (two) (under OR, a bad actor will cooperate with a bad recipient to become good) and reputation rules like STANDING and JUDGING. In simulations of the dynamics and the ability to invade populations of defectors, [START_REF] Brandt | The logic of reprobation: assessment and action rules for indirect reciprocation[END_REF] found that three reputation rules suggested by common moral intuition (SCORING, STANDING and JUDGING), support the success of cooperative action rules CO and OR. Their success was twice as high with JUDGING than with SCORING. Later, Ohtsuki and Iwasa (2007) investigated the dynamics of three action rules: ALLC, ALLD and CO (the latter labelled DISC in their paper) under 16 second-order reputation rules and found two reputation rules that promote CO: STERN JUDGING (KANDORI in their paper) and SIMPLE STANDING. Since ALLD and ALLC have no use for a reputation module, their study investigates the ability of the combination of CO with these reputation rules to invade ALLD and ALLC. A previous study by [START_REF] Pacheco | Sternjudging: a simple, successful norm which promotes cooperation under indirect reciprocity[END_REF] had investigated the dynamics of all 16 possible action rules under individual and group selection and found that the same reputation rules as in Ohtsuki and Iwasa (2007) promote cooperation. In a different study that investigates the emergence of cooperative strategies out of non-cooperative ones through group selection and probabilistic rather than deterministic rules, [START_REF] Scheuring | Evolution of generous cooperative norms by cultural group selection[END_REF] found that the dominant cooperative norm is GENEROUS JUDGING, a norm where defection against the bad is rewarded less frequently than in STERN JUDGING, and cooperation with the bad is as likely to be punished as to be rewarded.

A c c e p t e d m a n u s c r i p t

Cooperation was promoted in these studies mostly in the form of CO. CO is the conditionally cooperative strategy that can be stated in English as "Cooperate with the good, defect against the bad, independently of your own reputation." The reputation rules that promote CO are second-order rules because they only need to take into account the action of the actor and the reputation of the recipient. STERN JUDGING fits exactly to CO, for it assigns a good reputation to those and only to those that follow CO. In English, STERN JUDGING says: "Cooperate with the good and defect against the bad and you will be good; otherwise you will be bad".

Given these convergent results and our interest in unity of rules across frameworks, I shall focus on CO and ask whether it is shared across frameworks. CO takes account of the recipient's reputation only. There are 4 possible actions of this sort: C-G; D-B; C-B; D-G (cooperate with the good; defect against the bad; etc). Four strings, each combining two of these four actions, constitute the possible action rules. The only cooperative rule that can beat defectors is CO: the string C-G, D-B. Further, I shall ask whether three second-order reputation rules are shared. Table 1 gives the three reputation rules and their assessment of actions and rules. ALLC and ALLD are compared to CO as part of its usual environment of competition.

Table 1. Three action rules and their assessment by three second-order reputation rules.

In table 1, CO is the only action rule that cooperates conditionally on the reputation of the recipient. The middle column shows the assessment of the actions and action rules on the left according to three second-order reputation rules listed on the right. These rules only take into account the action of the actor and the reputation of the recipient, except SCORING that only cares for the action. The farthest right column describes the differences between the reputation rules, highlighted in grey. SIMPLE STANDING is the most generous reputation rule because it totally approves of ALLC and CO; STERN JUDGING is more generous than SCORING and, crucially, more consistent in the promotion of conditional cooperation.

ACTION RULES

The question then is whether the cooperative strategies belonging to direct, indirect and strong reciprocity frameworks are constructed with CO and the reputation rules mentioned above. CO underlies most DR strategies, namely, those that prescribe giving to good opponents and not giving to bad ones, e.g., TFT and Contrite TFT. As we shall see, these differ in the criteria that mark an opponent as bad (the reputation rules), but the same difference appears within IR as well, namely between SCORING and SIMPLE STANDING. If DR and IR cooperative strategies are both based on the action rule CO, and if they share similar reputation rules, how do DR and IR strategies differ? DR strategies apply only when the interaction between the same two players is repeated; IR strategies apply only when repeated dyadic encounters are excluded. In this way, no overlaps occur between DR and IR strategies and evolutionary game theory can assess their evolutionary dynamics separately.

In fact, this is the only difference between DR and IR cooperative strategies; and the question arises whether the underlying rules should be different only for this reason, in particular since pure IR interaction domains are suspiciously artificial. Traffic in the streets may adequately reproduce the typical IR one-shot structure; but it is hopeless to try to keep updated about the reputations of traffic participants. In domains with a smaller circle of participants you cannot exclude repeated interaction. In this case, simulations show that IR strategies are not always stable against DR strategies. DR/TFT beats IR/SCORING (see 6 and 7 below) when pairs meet repeatedly (40 times) both in small [START_REF] Dreber | Winners don't punish[END_REF] and in large (100) groups [START_REF] Roberts | Evolution of direct and indirect reciprocity[END_REF]). The reason is that IR/SCORING punishes co-operators that justifiably defect on defectors, whereas DR identifies cooperators better through direct interaction. In contrast, IR/STANDING avoids punishing punishers (see sect. 6) and beats DR/CTFT (contrite TFT, see sect. 7) in the same environment. Interestingly, though it may seem that IR and DR strategies are competing in those simulations, the alleged IR players in that model are in fact switching between IR and DR as required. Switching seems unavoidable in natural contexts and raises the question whether common rules facilitate it.

In Roberts' simulations players interact in pairs and any two of them, A and B, will meet about 40 times. In this model any player's memory extends only to one period. DR players remember only the opponents' last move towards them. But IR players remember the last move, no matter whether it was towards them or towards others. Thus, if A is an IR player, it will switch in this model between DR and IR, depending on the target of B's last move. Switching is easier if the player applies the same action and reputation rules in both frameworks, but it is still possible under different rules if the inputs univocally trigger their corresponding rules. However, if we add long-term memory, the chances that A's memories of any given agent's actions are either only towards A or only towards others will be very low. If the actions and the memories are mixed, rules for DR and IR will be triggered and, if different, a potential conflict will ensue. Consistency is maintained if the same reputation rules and the same action rules are shared across DR and IR contexts. Sharing rules means that reputation rules can process inputs (memories) of both types and output a reputation that is treated by a common action rule indistinctly of origin. This sharing of rules across frameworks must explain in part the evolutionary scope and stability of human cooperation.

A GENERAL ACTION RULE

I shall now formulate a general rule that also encompasses SR strategies. Cooperative rules need to cope with defectors if they are to succeed [START_REF] Trivers | The evolution of reciprocal altruism[END_REF]). The action rule CO in DR and IR prescribes to withhold benefits from them, and succeeds at the To punish is to respond to a defection with spite in a separate dyadic interaction, where the cost to the punisher is usually less than the cost inflicted on the punished [START_REF] Fehr | Altruistic punishment in humans[END_REF]. This form of punishment is more severe than withholding benefits, but both costly punishment and withholding benefits legitimately count as forms of punishment. To avoid confusions, I will use 'enforcement' as the umbrella concept to cover both withholding benefits and costly punishment.

When the structure of the game does not allow targeting defectors specifically, a dyadic interaction must be added for this purpose [START_REF] Boyd | Punishment allows the evolution of cooperation (or anything else) in sizable groups[END_REF]. When SR is invoked in connection to public goods, ultimatum and third-party punishment games (Fehr and Fischbacher 2004a,b) the underlying structure consists in two games linked in two stages. The SR strategy comes into play as spite in a dyadic second stage, directed to a player perceived as violating a norm of cooperation in the first stage. Strictly speaking, the domain of SR is a dyadic interaction linked as second stage to a social dilemma interaction -either dyadic or n-person, n>>2 -in a first stage. This two-stage structure can be repeated. Repeated PGGs with punishment are sometimes played in the lab reshuffling the groups after each period, so as to imitate one-shot encounters [START_REF] Fehr | Cooperation and punishment in public goods experiments[END_REF]. But this is instrumental to proving that punishment is biologically altruistic. In real life, SR strategies are more often played in stable groups. I here define SR, contrary to the will of its initial proponents, as involving costly punishment, but not necessarily altruistic punishment, because the punisher may receive compensation either through the future cooperation of the punished or from third parties interested in promoting cooperation. The only difference between SR strategies and DR/IR strategies concerns the part of the cooperative action rule that instructs how to enforce against defectors. The common action rule instructs to cooperate with co-operators and to enforce against defectors, allowing a choice between spite (SR) or withholding cooperation (DR/IR). The question arises, naturally, why these two methods of enforcement and whether both are really needed. The answer is that they are. Costly punishment should be engaged only when withholding benefits is unable to enforce cooperation; but then it must be engaged. Experiments suggest that this happens more often in connection with public goods (Rockenbach and Milinksi 2006).

REPUTATION RULES

Reputation rules are often discussed as if they were exclusive to IR. But if they are the criteria that mark players as good or bad conditional on their behaviour -rules taking behaviour as input and delivering a reputation as output -all conditional strategies must use them. Whether the behavioural data come from direct interaction as in DR and in public goods games, or from observation and hearsay as in IR, the rules for processing input data can be the same. Here, again, variety within frameworks is to be distinguished from variety across frameworks. If reputations rules are diverse, but the same diversity is relevant in all frameworks, it is likely that any individual using a particular rule within a framework will see reason to use the same rule across frameworks.

Reputation rules have been developed extensively in relation to IR [START_REF] Brandt | The logic of reprobation: assessment and action rules for indirect reciprocation[END_REF][START_REF] Ohtsuki | How should we define goodness?reputation dynamics in indirect reciprocity[END_REF]2006;[START_REF] Ohtsuki | Global analyses of evolutionary dynamics and exhaustive search for social norms that maintain cooperation by reputation[END_REF]). An actor A meets a potential recipient B. A first evokes a memory of B's last move as actor towards a third party C. Reputation is then allocated as a binary score, i.e., good or bad, according to a rule. The simplest rule considers only whether B helped C or not; this rule is called SCORING. Other rules take into account the reputation of the recipient C (secondorder) and yet others take also into account the reputation of the observed actor B (third-order). SIMPLE STANDING and STERN JUDGING are two second-order rules. They assign a good reputation to B for refusing to help C if C was bad. STERN JUDGING additionally gives a bad score to cooperation with a bad recipient. In the former case, defection against a bad recipient is viewed as justified defection or punishment. In the latter, cooperation with a bad recipient is seen as the omission of due punishment, i.e., as treason.

REPUTATION RULES ACROSS FRAMEWORKS

Important here is that these rules are applied across the different frameworks. Reputation rules for DR can be seen as special instances of rules for IR: when A evokes a memory of a past interaction of B with C such that C=A. TFT uses SCORING: it only considers whether B defected or cooperated. With this simple rule, two TFT players can lock into mutual punishment in a sequential two-person game. This happens when A makes a mistake and defects, provoking B's defection, which A does not recognize as justified. This motivated CTFT (contrite tit for tat), a strategy that recognizes its errors of implementation, so that in the previous case A returns to cooperation [START_REF] Sugden | The Economics of Rights, Cooperation and Welfare[END_REF][START_REF] Boerlijst | The Logic of Contrition[END_REF]). CTFT applies the same rule as SIMPLE STANDING: A will not mark B as having a bad reputation for refusing to help A if at that moment A was bad. A interprets B's defection as justified punishment. It may seem harder to make sense of equivalents for STERN JUDGING in DR, which would mark an opponent as bad for cooperating with A when A had defected in the previous round. This form of "insistent cooperation" matches practices that we know as flattery or obsequiousness. Though this behaviour is sometimes accepted by the flattered, it is sometimes rejected. Sometimes, the targets of flattery or obsequiousness react with contempt and anger because those behaviours betray either weakness or the intention to manipulate. In this case their reputation rule assesses such "co-operators" as bad.

SIMPLE STANDING and STERN JUDGING have hardly played any role in the modelling of SR in repeated public good games with punishment. A punisher in the second stage of these games only needs SCORING to assess reputations of players in the first stage, because defecting in the PGG stage is a sure sign of being a defector. If a player wants to punish defectors, it should punish them in the second two-person stage, not by defecting in the first n-person stage. Similarly, cooperating in the PGG stage cannot be interpreted as treason, for treason is omitting punishment of defectors in the second punishment stage. In one of the first simulations of altruistic punishment, [START_REF] Axelrod | An evolutionary approach to norms[END_REF] modelled the punishment of non-punishers in a third stage, which treats the omission of punishment in the second stage as treason and uses the same reputation rule as STERN JUDGING. Another way to introduce these second-

A c c e p t e d m a n u s c r i p t

order rules in SR would be to model an IR game where players have the option of costly punishing defectors (SR) instead of withholding benefits from them. In this case, SR players would not want to punish punishers (STANDING); and perhaps they would want to punish traitors (STERN JUDGING). This last possibility has been explored recently in some experiments and simulations, where punishment is stabilized by norms that dictate cooperation with those who punish the bad (SIMPLE STANDING) and punishment for those who either cooperate with or withhold benefits (defect) from the bad (similar to STERN JUDGING). The overall conclusion of these studies, however, is that costly punishment does not promote cooperation in DR/IR environments better than withholding benefits [START_REF] Ohtsuki | Indirect reciprocity provides only a narrow margin of efficiency for costly punishment[END_REF]Dreber et al. 2008;[START_REF] Rand | Direct reciprocity with costly punishment: Generous tit-for-tat prevails[END_REF]). This is not surprising, for it is intuitively not advisable to use costly punishment where withholding benefits is otherwise sufficient to enforce cooperation. The use of costly punishment must be justified by peculiarities of the interaction that entail that withholding benefits is not open or not efficient to achieve enforcement. Public goods games are such cases. Surely, withholding cooperation has a similar effect to costly punishment when pair-wise DR or IR interactions are linked to PGGs [START_REF] Milinski | Reputation helps solve the 'tragedy of the commons[END_REF][START_REF] Barclay | Trustworthiness and competitive altruism can also solve the "tragedy of the commons[END_REF]. But this arrangement is not always feasible in real life: if defectors can intimidate others into transferring benefits, the few that can resist intimidation will not achieve deterrence without costly punishment. Also, contribution to public goods in modern societies (paying taxes) is not publicly observable. Only institutional agents using costly punishment are here able to enforce. These special circumstances have not been taken into account in those recent studies: they have arranged a competition between withholding benefits and costly punishment in DR and IR environments without any special circumstance that would justify the introduction of costly punishment. Costly punishment is appropriate when withholding benefits is not effective; and when it is appropriate, it is important to remember that the same reputation rules that promote cooperation in IR games will promote it too when costly punishment is advisable.

SCORING, SIMPLE STANDING and STERN JUDGING do not reflect any difference between the frameworks. They reflect, rather, differences in moral assessments that are relevant within any framework (see table 2). In conclusion, the fundamental challenge for unifying strategies across frameworks is the distinction between DR/IR on one hand, and SR on the other. I have suggested that this distinction rests on a common ground: a general rule that prescribes enforcing cooperation with two enforcement options.

Table 2. Reputation and action rules across frameworks

In table 2, actor A assigns a reputation (G=Good; B=Bad) to a recipient B according to three possible reputation rules based on two sorts of data: 1) B's last move (C or D) in whichever game B was previously involved, e.g., towards a recipient C, or towards A itself when A was B's recipient in an iterated two-person PD, or towards both in an n-person PD; and 2) the reputation of C or A. SCORING only registers B's move; SIMPLE STANDING and STERN JUDGING also pay attention to the reputation of C or A as B moves. The action rule tells A to give (C) when B is good and to withhold a benefit (D) or costly punish (P) when B is bad. Differences between reputation rules concern the assessment of actions to the bad, and they affect all frameworks in the same way (highlighted in grey). Only the action rule determines a difference for frameworks between enforcing through P or through D. The circumstance dictates whether P is justified to achieve effective deterrence.

ENFORCEMENT AND REPUTATION IN EXPERIMENTAL GAMES

In this section I review economic experiments that suggest that a common normative mechanism underlies cooperative strategies across the three frameworks under consideration.

SR was introduced upon the discovery, both experimental and theoretical, that in the n-person PD with n >> 2 defecting in response to defectors causes the breakdown of cooperation, whereas responding spitefully in a linked game increases cooperation levels [START_REF] Kim | The free rider problem: experimental evidence[END_REF][START_REF] Yamagishi | The Provision of a Sanctioning System as a Public Good[END_REF]Boyd andRicherson 1988, 1992). It rose to popularity when experimental economists specifically excluded in the lab -in anonymous one-shot interactions -all benefits that could have been caused to punishers through inducing the punished to cooperate in subsequent periods. Results showed that subjects still punished under these conditions, when no benefit could be expected, and achieved high levels of cooperation [START_REF] Fehr | Cooperation and punishment in public goods experiments[END_REF]2002). Given the importance of public goods for human large-scale cooperation, the success of 'altruistic' punishment arose enthusiasm. But this success was soon rivalled by the performance of an IR strategy in a similar game.

An IR strategy succeeded in promoting the provision of public goods in an experiment by [START_REF] Milinski | Reputation helps solve the 'tragedy of the commons[END_REF]. In their design, withholding benefits in dyadic, one-shot encounters figured instead of costly punishment in a second stage after the PGG. The experiment revealed that the superiority claimed for SR rested partially on a misunderstanding. Supporters of SR said that DR was expressed in an iterated PGG when co-operators responded with defection to defection, which caused cooperation to unravel. But, just as enforcement of cooperation through costly punishment requires the addition of dyadic game in a second stage after the PGG, a fair comparison with DR should keep the same design. In the added second stage, co-operators can punish defectors dyadically -the game-theoretic context for a DR strategy -by withholding benefits from them or by not choosing them for interaction at all (Rosas 2008b). [START_REF] Milinski | Reputation helps solve the 'tragedy of the commons[END_REF] experimented with an IR game in the second stage; an experiment with DR in the second stage has also been conducted, with the same effect [START_REF] Barclay | Trustworthiness and competitive altruism can also solve the "tragedy of the commons[END_REF]). [START_REF] Panchanathan | Indirect reciprocity can stabilize cooperation without the second-order free rider problem[END_REF] produced the model corresponding to Milinski's experiment and found that IR can enforce cooperation "without the second-order free rider problem", i.e., without biological altruism. The fundamental insight, arguably, is that The moral to this story is that cooperative strategies share common grounds across the DR, IR and SR frameworks. A cooperative strategy cannot survive without retaliating against defectors [START_REF] Trivers | The evolution of reciprocal altruism[END_REF]). Across the frameworks, all strategies do this, differing only in their method. This justifies introducing the term 'norm' for strategies in IR frameworks [START_REF] Nowak | The evolution of indirect reciprocity[END_REF], though advocates of SR sometimes write as if norms were exclusive to SR (Fehr andFischbacher 2003, Gintis et al. 2003). Moreover, in third-party punishment experiments third parties punish defectors in an observed one-shot twoperson PD. Advocates of strong reciprocity interpret this behaviour as enforcing a norm (Fehr and Fischbacher 2004a). So, obviously, these third parties view the two-person PD as governed by a norm and the defectors as norm-violators. There is no reason to believe that the participants in the two-person PD view their interaction differently. Cooperation in all social dilemmas is norm driven, not only in cases where SR is required. However, the enforcing power of DR and IR should not suggest that they could in every case replace SR. It remains true that in some cases it is necessary to costly punish to succeed in enforcing cooperation.

An experiment by [START_REF] Rockenbach | The efficient interaction of indirect reciprocity and costly punishment[END_REF] shows that people understand the enforcing power of both IR and SR strategies. It also suggests that they do not readily renounce to costly punishment in the context of public good provisioning. The experiment studies, in two treatments, enforcement of contribution to public goods. Subjects can either contribute all or contribute nothing. One treatment has two stages: subjects participate in a public goods game (first stage) having chosen beforehand whether punishment will be possible in a second stage. The other treatment has three stages. The first two stages are as just described. Then both groups (with and without punishment) play an IR game where each participant is once an actor and once a recipient. In this way, the experiment obtains data for three different methods to discipline those who defected in the public goods game: 1. combining SR with IR; 2. only through IR; 3. only through SR. Results show that subjects confronted with the choice between 1 and 2 prefer 1. Combining SR with IR enforces a higher number of full contributions than either SR or IR alone and it happens at a lower cost in punishment points than in SR alone. Thus, the combination of SR with IR boosts the efficiency of cooperation for the whole group. This combination builds "a low cost solution to social dilemmas" (ibid.), providing an apt target for natural selection. The experiment suggests not only that SR and IR express and enforce the same norm for conditional cooperation, but that in connection with public goods games people feel the need to apply costly punishment. The norm: "Cooperate with others in your group; enforce against defectors", leaves open whether enforcement should happen as costly punishment or as withholding benefits. But, in connection to public goods games, a combination of both is apparently best to enforce the maximum contribution at the lowest possible cost.

Experiments provide general evidence to confirm that also reputation rules are shared. When subjects play public goods games linked to two-person games, where they can either act on SR or IR or both [START_REF] Rockenbach | The efficient interaction of indirect reciprocity and costly punishment[END_REF]; or when third parties observe a two-person DR game and punish defectors (Fehr and Fischbacher 2004a), they in fact treat reputations as shared by strategies across different frameworks. Players acquire bad reputations in an n-person PGG or in a two-person DR game; and then this reputation triggers responses in subjects playing either SR or IR strategies. Subjects use the same criteria for reputation across game types and across strategies. SCORING, STANDING and JUDGING reflect differences in moral assessment; but the differences apply to any game type and a player will probably have reason to stick to one criterion when moving across game-types (see table 2).

In models, SIMPLE STANDING and STERN JUDGING promote cooperation better than SCORING. But an experiment by [START_REF] Milinski | Cooperation though Indirect Reciprocity: Image Scoring or Standing Strategy[END_REF] showed that subjects use SCORING even when second-order information suitable for the use of STANDING was available. A possible explanation is that the experiment hindered the use of STANDING by the way the information about the actions was presented. The information did not come nor was easily transformable into the format C-G; D-B (cooperate with the good; defect against the bad; etc), which is the format of the input for second-order assessment strategies. The information that gives the reputation (G or B) of the receiver of the potential receiver's action being assessed was not given. It had to be inferred from histories of cooperative or defective actions taken from a network created as seven players interacted randomly and continuously in pairs for 16 rounds. This represented an impossible task for working memory, as the authors acknowledge. The question is whether this impossible task represents what is normally required in real life to assess reputations. The authors seem to believe that it does, but I would cautiously look forward to further experimentation with input information presented in a more appropriate format.

CONCLUSIONS

Analyses of strategies as rules in both models and economic experiments suggest that a common rule prescribing cooperation with the good and enforcement against the bad underlies all strategies. The common rule takes information about individual reputations as input and outputs decisions either to cooperate or to enforce. This mechanism is restricted to norms of cooperation and is less abstract than the universal normative mechanism advocated in [START_REF] Sripada | A framework for the psychology of norms[END_REF]. Yet it is sufficiently abstract to encompass all frameworks of interaction (see fig. 1). Theoretically, this hypothesis could be modelled by constructing a type of agent that withholds cooperation from defectors and resorts to costly punishment only when it is not possible to enforce through withholding benefits, for example, in public goods games that cannot be linked naturally to DR or IR games in a second stage. Punishment would also be necessary if defectors were capable of intimidating most players and purloining benefits in two-person games. The few agents that could resist intimidation would have to inflict costly punishment on the defectors; withholding benefits would not effectively enforce. The fact that the norm relies heavily on the ability to see the similarities across frameworks probably restricts its domain to humans. An important consequence for the evolutionary game theory of cooperation is that the question of stability can be posed in relation to the common norm rather than to each strategy individually. The stability of human cooperation looks more promising from this perspective than from the viewpoint of any one strategy considered in isolation.

  dyadic games. But if defectors cannot be specifically targeted with defection, as happens in public goods games, investment in costly punishment (SR) is a viable option.

  assigned to B by three reputation rules (St and J are 2 nd order rules)Action rule CO across frameworks, dictates AÕ s move towards B given BÕ s reputation BÕ s move to C(withhold benefits or costly punish to enforce cooperation in the first-stage PGG, provided both methods appear in the second dyadic stage of a two-stage design.

A c c e p t e d m a n u s c r i p tFigure 1 .

 1 Figure 1. The basic cognitive architecture underlying cooperative strategies. A module for assigning reputations (here STERN JUDGING) embedded in a cooperative action module is fed with beliefs about B's past actions and outputs a reputation for B. The action module is fed with beliefs about reputation, interaction types and efficient enforcement and outputs the appropriate action (last row). G = good; B = bad; D = withhold benefits; P = costly punishment; C = cooperate. DR = direct reciprocity; IR = indirect reciprocity; SR = strong reciprocity.

Acknowledgments: I thank the KLI for Evolution and Cognition Research and the Universidad Nacional de Colombia for funding. For helpful comments on previous drafts I thank Karthik Panchanathan, Francesco Guala and two anonymous referees for the JTB.


Ido Filin
email: ido.filin@helsinki.fi

Target size and optimal life history when individual growth and energy budget are stochastic

Keywords: diffusion process, energy-predation tradeoff, hazard function, state-dependent life history

I extend my previous work on life history optimization when body mass is divided into reserves and structure components. Two important innovations are: (1) effect of finite target size on optimal structural growth; (2) incorporating reproduction in the optimization objective. I derive optimal growth trajectories and life histories, given that the individual is subject to both starvation mortality and exogenous hazards (e.g., predation). Because of overhead costs in building structural mass, it is optimal to stop structural growth close to the target size, and to proceed only by accumulating reserves. Higher overhead costs cause earlier cessation of structural growth and smaller final structures. Semelparous reproduction also promotes early cessation of structural growth, compared to when only survival to target size is maximized. In contrast, iteroparous reproduction can prolong structural growth, resulting in larger final structures than in either the survival or the semelparous scenarios.

Increasing the noise in individual growth lowers final structural mass at small target sizes, but the effect is reversed for large target sizes. My results provide predictions for comparative studies. I outline important consequences of my results to additional important evolutionary questions: evolution of sexual dimorphism, optimization of clutch size and evolution of progeny and adult sizes.

A c c e p t e d m a n u s c r i p t 1. Introduction

Life history and developmental transitions often require the attainment of some target size. For example, "in a variety of animals and plant species adult size appears to be determined by a size threshold for maturation" (Roff, 2002, p. 201). Developmental transitions in insects and amphibians are dependent upon reaching some critical size [START_REF] Day | Developmental thresholds and the evolution of reaction norms for age and size at life-history transitions[END_REF][START_REF] Nijhout | The control of body size in insects[END_REF][START_REF] Mirth | Size assessment and growth control: how adult size is determined in insects[END_REF]. More generally, such thresholds play a key role in recent approaches to developmental plasticity and phenotypic evolution (West-Eberhard, 2003).

Thus, target size is an important life history variable, and should affect optimal life history decisions during growth and development of an individual (e.g., [START_REF] Day | Developmental thresholds and the evolution of reaction norms for age and size at life-history transitions[END_REF]. In this paper, I study the effect of a given finite target size on optimal life history, when individual growth is stochastic. I employ a dynamic optimization approach (e.g., Perrin and Sibly, 1993;[START_REF] Iwasa | Dynamic optimization of plant growth[END_REF][START_REF] Irie | Optimal growth pattern of defensive organs: the diversity of shell growth among mollusks[END_REF], and extend my previous work [START_REF] Filin | A diffusion-based approach to stochastic individual growth and energy budget, with consequences to life-history optimization and population dynamics[END_REF], by considering not only survival, but also reproduction, as part of the optimization objective.

Following much recent work on size-structured populations and individual growth (e.g., Persson et al., 1998;[START_REF] Kooijman | Dynamic energy and mass budgets in biological systems[END_REF][START_REF] Gurney | Resource allocation, hyperphagia and compensatory growth[END_REF][START_REF] Filin | A diffusion-based approach to stochastic individual growth and energy budget, with consequences to life-history optimization and population dynamics[END_REF], I divide the total body mass of an individual into a reversibly growing component (hereafter, reserves) and an irreversible component (hereafter, structure). The mass of reserves varies stochastically in time, for example, because of fluctuations in consumption, assimilation and metabolic maintenance. I explore how costs of structural growth, noise level in reserves dynamics, and mode of reproduction (e.g., semelparous or iteroparous), all affect optimal investment in structure versus reserves, when the individual is subject to both starvation risk and exogenous mortality. Finally, I discuss how my model and results can be easily applied to a wealth of additional evolutionary problems: evolution of sexual dimorphism, optimal clutch size, and optimal progeny and adult sizes.

Effect of target size on life history optimization

Basic formulations

I denote structural mass by z(t) and reserves mass by y(t). Total body mass is then z(t)

+ y(t). I define the time-varying maximum of total body mass

(1) [ ] ) ( ) ( max ) ( 0 τ τ θ τ z y t t t + = ≤ ≤
i.e., the maximal total mass reached up to time t. (Below, I interchangeably refer to θ as total body mass, total mass, or body mass.) Both θ(t) and z(t) grow irreversibly, i.e., can only increase through time, while y(t) is free to either increase or decrease. Because y represents reserves mass, when y(t) drops to a level a (the starvation boundary), the individual dies of starvation. Throughout the rest of this paper I set the starvation boundary to zero (i.e., a = 0), to avoid unnecessarily cumbersome notation, and because a nonzero starvation boundary results in only minor modifications (if any) to the expressions presented below (see Appendix B), and does not qualitatively change the general conclusions. Therefore, an individual dies of starvation only after exhausting all of its reserves (i.e., when y[t] hits zero).

Following [START_REF] Filin | A diffusion-based approach to stochastic individual growth and energy budget, with consequences to life-history optimization and population dynamics[END_REF], structural growth occurs only each time a new body mass threshold is reached. In mathematical guise, the irreversibly growing structural mass z is a non-decreasing function of θ, i.e., z(t) = z[θ(t)]. Whenever the organism reaches a new maximum of total mass (i.e., a new, higher value of θ), there can be structural growth associated with this crossing of a new body mass threshold. In addition, structural growth always occurs on the expense of reserves y. A unit of structural mass is built by consuming 1+α units of reserves mass. The dimensionless parameter α represents overhead costs of building structural mass. In addition to starvation mortality (if y[t] hits the starvation boundary), the individual is subject to various exogenous hazards (e.g., predation and disease). These are captured by the mortality or hazard rate μ(y, z). Survival probability from initial reserves mass y 1 to final reserves mass y 2 (≥ y 1 ), while keeping z fixed, is given by (2) where η(y, z) is the hazard density, the hazard function for survivorship through transitions in the value of reserves mass (i.e., along the y-axis, from a lower value of reserves y 1 to a higher value y 2 ). The hazard density η describes mortality per unit of gain in mass (analogously, mortality rate μ, describes mortality per unit of time, i.e., along the time axis), and encapsulates within it both starvation mortality and exogenous mortality.

( ) ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ - = ∫ 2 1 , exp ) , ( 2
In the following, initial body mass of an individual will be denoted by θ 0 and target body mass by θ 2 (θ 1 will denote the size at which structural growth ceases; see below).

Survival probability from initial total mass θ 0 to final total mass θ 2 , when structural mass grows according to z = z(θ), is given by (3)

( )( ) ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ + - - = + + - - = ∫
i.e., fitness is given by F = S θ R, namely, survival probability to target size multiplied by a terminal reward obtained at that size. The terminal reward R (e.g., Houston and McNamara, 1999, ch. 3; also known as final function; Perrin and Sibly, 1993, appendix therein) depends on target size θ 2 and on structural mass z 2 obtained at this target size.

The state variable in this optimization problem is z, with which a costate variable is associated, denoted by λ. The value of the costate variable at each body mass θ between θ 0 and θ 2 (i.e. λ[θ]), quantifies the benefit of investing in structural growth, compared to just accumulating reserves. For analyzing the effect of target size on the optimal structural growth curve, z * (θ), I require the boundary condition for the value of λ at the target size θ 2

(5) 2 2 log ) ( z R ∂ ∂ = θ λ (Appendix A).
In the following sections I derive the optimal structural growth curve for several forms of the objective (fitness) function (Eq. [4]; more precisely, for several forms of the terminal reward R). First, I consider maximization of survival probability to target size (Eq. ). This is also the optimization objective in [START_REF] Filin | A diffusion-based approach to stochastic individual growth and energy budget, with consequences to life-history optimization and population dynamics[END_REF]. Later, I extend the analysis to consider objective functions that include reproduction.

Maximizing survival probability to target size: optimal cessation of structural growth due to overhead costs

When maximizing survival probability to final size the terminal reward function in Eq. ( 4) is R = 1. The boundary condition for λ (Eq. [5]) then becomes λ(θ 2 ) = 0. I have previously studied optimal structural growth for this case [START_REF] Filin | A diffusion-based approach to stochastic individual growth and energy budget, with consequences to life-history optimization and population dynamics[END_REF]Eq. [12] therein).

However, in that study I did not consider the possible effect of a finite target size on the optimal structural growth curve z * (θ) (in effect, θ 2 is taken to be infinite in Filin, 2009). [START_REF] Filin | A diffusion-based approach to stochastic individual growth and energy budget, with consequences to life-history optimization and population dynamics[END_REF] showed that when nonzero structural growth occurs (i.e., ż * > 0), z * (θ) follows a singular arc (a term borrowed from dynamic optimization theory; e.g., Perrin and Sibly, 1993). Along singular arcs the costate variable λ and the structural mass z satisfy the condition λ(θ) = αη(θ -z, z) (Appendix A). However, because λ(θ 2 ) = 0, this condition can never be satisfied at the target size, unless there are no overhead costs of building structural mass (i.e., if α = 0; η is always positive, unless growth is completely deterministic, i.e., σ 2 = 0, and there are no exogenous hazards, i.e., μ = 0). Therefore, the pair (θ 2 , z 2 ) (where

z 2 = z * [θ 2 ]
) never lies on a singular arc, unless α = 0. I conclude that, for finite target size θ 2 and nonzero overhead costs of structural growth (α > 0), the optimal structural growth curve z * (θ) always ends with a plateau, along which structural mass remains constant (z * [θ] = z 2 ), and only reserves grow.

As we consider increasingly lower body masses below θ 2 , λ(θ) may gradually increase from its boundary value λ(θ 2 ) = 0 until the condition λ(θ) = αη(θz, z) is finally satisfied at some body mass, denoted by θ 1 (≤ θ 2 ; the equality may hold only if α = 0). At that point, the optimal growth curve connects with a singular arc, and as we proceed backwards toward even lower values of θ (i.e., θ < θ 1 ), the optimal growth curve exhibits positive structural growth (ż * > 0). The overall pattern of optimal structural growth is to grow along the singular arc until θ 1 , and then proceed to the target size θ 2 , by switching to only accumulating reserves. Therefore, θ 1 is termed switching size.

The switching size, in turn, determines the optimal final structural mass achieved by the growing individual, because z 2 = z * (θ 2 ) = z * (θ 1 ) (and the latter is determined by the singular arc). Both θ 1 and z 2 will change as target size θ 2 varies. I can therefore obtain a switching curve z 2 (θ 2 ) that determines the value of the optimal final structural mass as a function of target size. Figure 1a provides specific examples of such switching curves and optimal structural growth curves for different target sizes. It is important to note that, in general, the singular arc is also defined for values of θ above the switching size θ 1 (diamonds in Fig. 1a). However, given a finite target size, it is optimal to abandon the singular arc once structural mass has hit the switching curve (i.e., at the switching size).

When reserves growth and exogenous mortality depend only on structural mass (i.e., g, σ 2 and μ are functions of z only), Filin, (2009) provides the following expression for the hazard density ( 6)

ϕ χ χ η - = ) coth( ) , ( y z y where (7) 2 2 4 2 , 2 σ ϕ σ μ σ χ g g = + =
are both functions of structural mass z (i.e., χ =χ[z] and ϕ =ϕ[z]). The quantity χ combines both starvation risk (decreasing with g and increasing with σ 2 ) and exogenous hazards (increasing with μ), to determine the range of values of reserves mass in which starvation is the predominant cause of mortality (as opposed to exogenous hazards). (This range can be described by the interval [0, χ -1 ].) The quantity ϕ measures how stochastic/deterministic the The approach to calculating the switching curve is to solve for the switching size and target size (θ 1 andθ 2 , respectively), given a value of final structural mass z 2 . For that purpose, I define the following dimensionless variables ( 8)

χ ϕ χ ϕ χ χ χ = ′ ′ = ′ = = Φ B A y Y , , , 2
where ϕ' = dϕ/dz and χ' = dχ/dz (note the change in the definitions of Y and A, compared to Filin, 2009). I then derive (Appendix A) the following expression for the switching curve 14) in [START_REF] Filin | A diffusion-based approach to stochastic individual growth and energy budget, with consequences to life-history optimization and population dynamics[END_REF]; using the revised definitions of Y and A in Eq. [8]). I used these expressions (Eqs. [9] and [10]) to calculate the switching curves and growth trajectories presented in Fig. 1a.

(9) ( ) ( ) ( ) Φ Y A B A Y f B A Y f tar tar - - = 1 1 2 coth , , , , α where the function f tar is defined as f tar (Y 2 , A, B) = (Y -A)cothY -BY. In addition, Y 2 = χ(θ 2 - z 2 ), Y 1 = χ(θ 1 -z 2 ),

Maximizing expected allocation to reproduction: semelparity and iteroparity

Both [START_REF] Filin | A diffusion-based approach to stochastic individual growth and energy budget, with consequences to life-history optimization and population dynamics[END_REF] against starvation during growth (in addition to providing raw material and fuel for structural growth), and as the terminal reward gained once target size is achieved. Therefore, in addition to the overhead costs of structural growth, discussed above, investment in structure entails an additional cost in term of reduced fecundity or yield. The boundary condition in Eq. ( 5) becomes λ(θ 2 ) = -1/(θ 2 -z 2 ).

The expression for the terminal reward from the previous paragraph describes semelparous mode of reproduction. All reserves are utilized in a single burst of reproduction, causing the individual to die of starvation immediately after. However, if the individual retains some reserves after reproduction (denoted by y 3 ) it may survive to reproduce additional times, i.e., iteroparous mode of reproduction. The individual sacrifices immediate reproduction (by not utilizing all of its reserves mass in a single reproduction event), for the sake of surviving to future reproduction events. Thus, the tradeoff between present and future reproduction (e.g., between fecundity and parental survival; Roff, 2002, pp. 126-150, pp. 188-198) is generated mechanistically in this model, mediated by the value of y 3 (i.e., the level of reserves retained after reproduction).

I adopt here the concept of a reproduction buffer [START_REF] Kooijman | Dynamic energy and mass budgets in biological systems[END_REF]p. 115), such that immediately after each reproduction event, reserves mass is y 3 (reproduction buffer emptied), and subsequent reproduction events occur each time the individual regains reserves mass θ 2z 2 (reproduction buffer full). Survival between reproduction events is then given by s = S y (y 3 ,θ 2 -z 2 ) (recall that S y represents survival along reserves mass transitions, while holding structural mass fixed; Eq. [2]). The expected number of reproduction events is 1/(1-s), and the terminal reward is then R = (θ 2 -z 2 -y 3 )/(1-s). The boundary condition in Eq. ( 5) becomes The following analysis again concerns the case with reserves growth and exogenous mortality dependent only on structural mass (i.e., g, σ 2 and μ are functions only of z). For semelparous reproduction, the expression for the switching curve is

(11) ( ) ( ) ( ) Φ Y A B A Y f Y A B A Y f tar tar - - = + 1 1 2 2 coth , , , ,
α
where again Y 1 obeys the singular arc equation (Eq. [10]). Figure 1b demonstrates that, indeed, structural growth ceases earlier for semelparous reproduction (dotted curves; compared with the case of maximizing only survival to target size; previous section; solid curves in Fig. 1b).

For iteroparous reproduction, Appendix A provides the expression for the switching curve. As Fig. 1b demonstrates the switching curve, in this case, has two branches: upper and lower. Only the upper branch is a solution of the switching curve equation (Appendix A).

Thus, such a solution exists only if the value of the final structural mass z 2 is high enough.

The lower branch of the switching curve represents the additional constraint θ 2 ≥ z 2 + y 3 that must be obeyed in the case of iteroparous reproduction (not surprisingly the lower branch intersects the abscissa axis at θθ 2 = -y 3 = -4 in the case of Fig. 1b). When the equality θ 2 = z 2 + y 3 holds, we obtain the limit of continuous reproduction. The amount of reserves spent in each reproduction event goes to zero (θ 2 -z 2 -y 3 = dy → 0), survival probability between reproduction events goes to one (s → [1-η(y 3 , z 2 )dy]; Eq. [2]), and the expected number of reproduction events becomes infinite (the terminal reward is nonetheless finite: R = [θ 2 -z 2 -

y 3 ]/[1-s] → 1/η[y 3 , z 2 ]
). From a biological perspective, however, such continuous reproduction, which consists of an infinite number of infinitesimally small progeny, can only be regarded as an approximation, at best. I further consider this issue below in the discussion.

Discussion

In this study, I extended my previous work [START_REF] Filin | A diffusion-based approach to stochastic individual growth and energy budget, with consequences to life-history optimization and population dynamics[END_REF] and considered optimal stopping conditions for structural growth, when individuals must grow to some given finite target size, and when reproduction is included in the optimization objective. I found that even when the optimization objective is maximizing survival to target size (as in [START_REF] Filin | A diffusion-based approach to stochastic individual growth and energy budget, with consequences to life-history optimization and population dynamics[END_REF]i.e., no reproduction) it is optimal to abandon structural growth altogether close to the target size, and to proceed only by accumulating reserves.

Only when there are no overhead costs of building structural mass (α = 0), is it optimal to keep investing in structure all the way to the target size. The reason is that the individual pays for such overhead losses of reserves in increased mortality, due to the extra time required to regain those lost reserves. Close to the target size, it is optimal to avoid any such losses of reserves mass (and thus instantaneous total body mass z[t] + y[t]). In addition, Fig. 1a demonstrates that the higher the overhead costs (i.e., higher α), the earlier structural growth ceases and the smaller final structural mass is (see also Figs 2 and 3 in the supplementary material).

A c c e p t e d m a n u s c r i p t 13 13

The effect of reducing the level of noise in the dynamics of reserves (i.e., reducing the growth variance σ 2 ) is less straightforward. At small target sizes, a lower noise level increases the final structural mass. However, the cessation of structural growth occurs earlier (i.e., at a smaller total body mass), because lower noise levels also promote faster structural growth. At large target sizes the effect is reversed, lower noise levels cause smaller final structural masses (Fig. 1a; see also Figs 2 and 3 in the supplementary material).

Filin ( 2009) also found that varying the growth variance may either increase or decrease final structural mass, depending on the function of the structural trait (e.g., foragingrelated or defensive). However, the effect of growth variance in this paper is fundamentally different from the one found by [START_REF] Filin | A diffusion-based approach to stochastic individual growth and energy budget, with consequences to life-history optimization and population dynamics[END_REF]. Because target size is, in effect, infinite in Filin (2009), the results described in that work concern the effect of growth variance on the asymptotic structural mass (i.e., at very large total body masses: θ → ∞). Here, by contrast, I consider the effect of the growth variance on optimal cessation of structural growth due to a finite target size, and at final structural masses potentially far from the asymptotic value. As discussed above, the signs of these two different effects may in fact be opposite.

When I incorporate reproduction into the optimization objective, the results vary depending on whether reproduction is semelparous or iteroparous (Fig. 1b). Semelparous reproduction always causes earlier cessation of structural growth and smaller final structural masses, compared to the case with no reproduction (when survival is the optimization objective). Iteroparous reproduction introduces a new parameter into the model, y 3 , the amount of reserves retained by the individual after a reproduction event. As Fig. 1b demonstrates, at very small target sizes (but nonetheless larger than y 3 ) the switching size is the initial size and there is no structural growth at all throughout the entire lifetime of the individual (the region right of the iteroparity switching curve in Fig. 1b). At larger target sizes, cessation of structural growth is delayed, compared to either semelparous reproduction A related question is that of sexual dimorphism (this issue was also briefly addressed in the context of deterministic dynamic optimization models in the discussion of [START_REF] Kozlowski | Optimal age and size at maturity in annuals and perrenials with determinate growth[END_REF]. Because the benefits and costs of body size and structures vary between males and females within a species, the terminal reward function should also depend on sex.

As discussed in the previous paragraph, this would affect the stopping condition, i.e., males and females will cease structural growth at different body mass and attain different final sizes

A c c e p t e d m a n u s c r i p t

15 of structures. However, early growth and development will be identical. This is a pattern of bimaturism, and in the context of heterochrony, may lead to males, for example, being hypermorphic compared to females [START_REF] Mcnamara | Sexual dimorphism: the role of heterochrony[END_REF] i.e., sexual dimorphism due to differences in timing of developmental transitions between males and females). However, if there are additional sex-specific differences in consumption, assimilation, metabolism or mortality, the optimal growth trajectories of males and females may diverge earlier in life, and before final structural size is attained. When such differences are caused by differences in behavior between males and females (e.g., Rennie et al., 2008), the model may be extended to include reversible behavioral transitions (which can then be optimized separately for males and females), in addition to the irreversible structural growth, as presented and discussed in [START_REF] Filin | A diffusion-based approach to stochastic individual growth and energy budget, with consequences to life-history optimization and population dynamics[END_REF].

As discussed in the previous section, when θ 2 = z 2 + y 3 , iteroparous reproduction becomes continuous, consisting of an infinite number of infinitesimally small progeny. There are at least two ways to remedy this biologically questionable result. First, one can define an upper bound s max < 1 for the survival probability between reproduction events (i.e., s). As a result, the expected number of reproduction events can never exceed 1/(1-s max ) (which is finite because s max < 1). The parameter s max may embody mortality factors during the reproduction event itself, e.g., due to increased susceptibility to predation. (See also Fig. 3 in the supplementary material.)

A second more mechanistic way to avoid continuous reproduction is to introduce a minimum nonzero amount of reserves that the individual must expend during each reproduction event. For example, this amount may represent the costs of producing a single egg. Denoting this amount by a egg , the constraint on target size becomes θ 2 ≥ z 2 + y 3 + a egg .

When the equality holds, the individual reproduces in single eggs, i.e., clutch size is one, rather than continuously as before (if a egg = 0). Figure 4 in the supplementary material further as part of the model, using this last modification, optimal clutch size arises as a byproduct of the life history optimization within this model. Thus, the theoretical framework presented in this study also addresses this important ecological and evolutionary problem.

There exists an analogy between the model described in this paper and dynamic optimization models for optimal size at maturity when season length is finite [START_REF] Cohen | Maximizing final yield when growth is limited by time or by limiting resources[END_REF]Vincent and Pulliam, 1980;[START_REF] Kozlowski | Optimal allocation of energy to growth and reproduction[END_REF]. A comparison with such models, demonstrates that body mass θ is analogous to the time-coordinate in those models. Target size is analogous to season length, and θ -θ 2 of Fig. 1 is analogous to the 'time-to-go' until season end, which determines the optimal switch between growth and reproduction in those models. However, growth in those models is deterministic. Therefore, exploring, for example, the effect of noise in growth is not within their scope.

This study explored the optimal way to invest in structural growth, starting from some initial condition and finishing at a given final target size. Target size was taken to be a fixed parameter, and growth and life history were optimized under that constraint. Elsewhere I will additionally explore the simultaneous optimization of the endpoints, i.e., of initial and target sizes (e.g., representing optimal progeny and adult sizes, respectively). [START_REF] Kozlowski | Optimal initial size and adult size of animals: consequences for macroevolution and community structure[END_REF] previously obtained such optimal initial and adult sizes for a deterministic life history model.

The stochastic model in this work enables me to investigate additional questions, such as the effect of noise in the growth of individuals on optimal progeny and adults sizes.

Concerning the above discussion of sexual dimorphism and optimal clutch size, optimizing target size itself will enable to explore sexual size dimorphism in the adult (total) body mass, in addition to sexual dimorphism in the allocation between structure and reserves.

Optimization of initial and adult sizes will also result in simultaneous optimization of egg size and clutch size, as a egg will no longer be a fixed parameter but will depend on the initial I confine the analysis, hereafter, to the special case of reserves growth and exogenous mortality dependent only on structural mass z (i.e., Eqs. [6]-[8]). Applying the conclusion that between θ 1 and θ 2 no structural growth occurs (i.e., the optimal control is u * = 0; thus, z * (θ) = const = z 2 ), Eq. (A.1) becomes

ϕ χ χ χ λ ′ - + ′ - ′ = Y Y Y Y 2 2 2 csch csch coth
where Y = χy = χ(θz) (Eq. [8]), ϕ' = dϕ/dz and χ' = dχ/dz, all evaluated at z = z 2 .

Integrating from θ

1 to θ 2 gives 2 1 coth coth ) ( ) ( 1 2 Y Y Y Y Y ⎥ ⎦ ⎤ ⎢ ⎣ ⎡ ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ ′ - - ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ ′ + = χ ϕ χ χ χ θ λ θ λ
where Y 1 = χ(θ 1 -z 2 ) and Y 2 = χ(θ 2 -z 2 ). Multiplying both hand sides by χ/χ', using the definitions of the dimensionless variables in Eq. ( 8), and recalling that (θ 1 , z 2 ) lies on the singular arc (therefore, λ

[θ 1 ] = αη[θ 1 -z 2 , z 2 ] = αχcothY 1 -αϕ), I finally obtain (A.2) [ ] [ ] ( ) Φ Y A BY Y A Y BY Y A Y - - - - = Λ - - - 1 1 1 1 2 2 2 coth coth ) ( coth ) (
A.3) ) ( 2 θ λ χ χ ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ ′ = Λ
And using the definition of f tar in the main text (under Eq. [9]), Eq. (A.2) can be rewritten as

(A.4) ( ) ( ) ( ) Φ - - = Λ - 1 1 2 coth , , , , Y A B A Y f B A Y f tar tar α
When the objective function is survival probability to final size, Λ = 0, and I obtain Eq. ( 9). When the objective function includes semelparous reproduction λ(θ 2 ) = -1/(θ 2 -z 2 ) = χ/Y 2 . Equation A4 then becomes Eq. ( 11). For iteroparous mode of reproduction, λ(θ 2 ) = -1/(θ 2 -z 2 -y 3 ) + s'/(1-s) (where s' stands for ∂s/∂z 2 ).The survival probability s between reproduction events is given by

(A.5) ( ) [ ] 3 2 2 3 exp sinh sinh Y Y Φ Y Y s - =
(the expression for S y in Eq. (3) of Filin, 2009) where Y 3 = χy 3 (y 3 is taken to be a fixed parameter of the model). Consequently, s'/s= ∂logs/∂z 2 = Y 3 '(cothY

3 -Φ) -Y 2 '(cothY 2 -Φ) + Φ'(Y 2 -Y 3 ). Given that Y 2 = χ(θ 2 -z 2 )
, and Φ = ϕ/χ, I obtain (χ/χ')Y 2 ' = Y 2 -A and (χ/χ')Φ' = B -Φ (using definitions in Eq. [8]). The mass of reserves immediately after a reproduction event is y 3 , and its dimensionless counterpart is Y 3 = χy 3 . Given that y 3 is a fixed parameter, (χ/χ')Y 3 ' = Y 3 , and then Eq. (A.3) becomes [ ]

3 3 3 2 2 2 3 2 coth coth ) ( 1 Y Y BY AΦ BY Y A Y s s Y Y A - + + - - - - - - = Λ
Finally, by additionally utilizing the definition of f tar , I arrive at the expression for the switching curve in the case of iteroparous reproduction the standard specific model, described below (α = 1; solid lines); α is increased to 3 (dotted lines); growth variance σ 2 is uniformly reduced by a factor of 10 (dashed lines). The optimization objective is survival to target size (i.e., no reproduction). In the uppermost sample growth trajectory I marked the points representing (from left to right) the initial size, the switching size and the target size. The other three sample trajectories all emanate from the same initial point. However, because of differences in α or σ 2 both the growth path and the switching size are different. When overhead costs of structural growth (i.e., α) are increased the switching curve becomes lower, but the rate of structural growth is only slightly reduced, causing smaller final structural mass and earlier cessation of structural growth (compare solid with dotted lines). When noise in individual growth is reduced (i.e., σ 2 is reduced), the switching curve is higher for small target sizes, but lower for large target sizes (compare solid and dashed curves). Because a reduced growth variance strongly affects the rate of structural growth, cessation of structural growth occurs earlier, but at higher final structural mass. (2/3)z], σ 2 = 4(0.1z 2/3 + 0.9z), μ = 0.2, and α = 1. The motivation for these specific expressions follows from the work of Kooijman (2000, ch.3), where z here is comparable to Kooijman's structural volume. The mean growth rate g is the difference between assimilation, proportional to surface area, and maintenance, proportional to volume. Time and size were rescaled such that maximum mean growth rate is g = 1, and it occurs at z = 1.

(A.6) ( ) [ ] ( ) ( ) Φ Y A B A Y f Y Y BY AΦ s s Y Y A B A Y f s tar tar - - = - + - + - + - A c c
The growth variance σ 2 is the sum of noise in assimilation (again proportional to surface area) and noise in maintenance (proportional to structural mass; e.g., representing independent fluctuations in metabolism among cells). Mortality rate is taken to be constant,

  of structural growth (while θ[t] and thus z[t] remain unchanged), the dynamics of reserves y(t) is described by a diffusion process on the interval [0, θ -z]. This diffusion process is characterized by g(y, z) (the mean growth rate) and by σ 2 (y, z) (the growth variance) representing, respectively, the mean balance of input (e.g., assimilation) and output (e.g., metabolic maintenance) fluxes of energy and mass, and the random fluctuations around this mean balance. In general, both mean growth rate g and growth variance σ 2 are dependent on both reserves mass y and structural mass z (but see following sections). (The units of g are that of [mass/time], while the units of σ 2 are that of [mass 2 /time].)

  : when ϕ → ±∞, the growth of reserves mass is practically deterministic, while when ϕ → 0 reserves dynamics behaves like Brownian motion. The units of ϕ, χ and η are that of [mass -1 ].

  and χ, A, B and Φ are all evaluated at structural mass z = z 2 . Because (θ 1 , z 2 ) lies on the singular arc, f arc (Y, B) = [2Y -sinh(2Y)] + B[coth(2Y)-1] (the singular arc equation; Eq. (

  and this study, up to this point, have only considered survival probability as the objective (fitness) function. Clearly, however, a full description and optimization of a lifecycle must include reproduction as well. The simplest (yet still realistic) form of introducing reproduction into the optimization objective is choosing the terminal reward function in Eq. [4] to be R = θ 2 -z 2 , i.e., the mass of reserves accumulated at target available for production of progeny. Alternatively, R can describe yield, for example, in agricultural crops. The role of reserves is now dual, serving as both insurance

  2 ) = -1/(θ 2 -z 2 -y 3 ) + (∂s/∂z 2 )/(1-s). Additional structural growth does not occur after the individual begins to reproduce (i.e., after reaching θ 2 for the first time; determinate growth).For semelparous reproduction, λ(θ2 ) is now negative (λ[θ 2 ] = -1/[θ 2 -z 2 ]), comparedwith λ(θ 2 ) = 0 when the objective function was survival probability to target size (previous section). Integrating backwards from target size θ 2 towards lower body masses, λ(θ) would take longer to reach the singular arc λ(θ) = αη(θz, z) (the right-hand-side of this equation is nonnegative). Therefore, for a given target size, θ 2 , I expect earlier cessation of structural growth (smaller switching size, θ 1 ) and smaller final structural mass (z 2 ), when semelparous reproduction is taken into consideration in the objective function. The effect of iteroparous reproduction on the switching curve is less straightforward (see below).

  , causing larger final structural mass for a given initial condition (see growth trajectory examples in Fig.1b). Finally, at even larger target sizes the switching curve for iteroparous reproduction falls below that for no reproduction and asymptotically approaches that of semelparous reproduction. The above qualitative conclusions remain unchanged also when mortality is sizedependent, decreasing with structural mass (see Figs 2b and 3b in the supplementary material). The effect of size-dependent mortality in this case is only quantitative, prolonging structural growth and thus resulting in larger final structural mass. The effects of increased overhead costs or reduced noise level remain qualitatively the same. The growth trajectory examples in Fig.1b demonstrate that, given identical initial conditions, target size and parameter values (including identical functional forms of g, σ 2 and μ), the reproduction mode (i.e., no reproduction, semelparity or iteroparity) does not affect the growth trajectory itself. The reproduction mode only affects the stopping condition, i.e., the body mass and structural mass at which structural growth ceases. (Mathematically, that is because the reproduction mode only affects the terminal reward function R in Eq. [4].). This provides an interesting prediction for comparative studies, where closely related species or populations, or even different individuals within the same population, may exhibit different growth patterns, depending on the reproduction mode they adopt (semelparous or iteroparous).

  . It is important to note that, although not explicitly formulated

  to optimization. Finally, I am also currently working on an extension of the model to indeterminate growersIn conclusion, the theoretical framework, presented in this study, provides a powerful tool for addressing a wide variety of life-history and evolutionary questions, under biologically realistic conditions, including: subjection to both starvation mortality and exogenous hazards, noise in the dynamics of individual state, and distinction between reversible and irreversible components of individual size. These two last aspects of the growth and development of individuals have important implications to evolution of life history (as well as to population dynamics;[START_REF] Filin | A diffusion-based approach to stochastic individual growth and energy budget, with consequences to life-history optimization and population dynamics[END_REF], that have not yet been fully explored and boundary condition given by the derivative of the final function (terminal reward R in Eq.[4]) with respect to the state variable z e., Eq. [5];Intriligator, 1971, p. 348).The switching function for investing in structural mass is given byΣ = ∂H/∂u, or 0, I obtain singular control, i.e., this is the equation satisfied by a singular arc, to which I refer throughout the main text. Singular control additionally requires dΣ/dθ = 0, which results in the general equation for the singular arc presented byFilin,

Fig. 1

 1 Fig. 1 (a) Switching curves (thick lines) and sample growth trajectories (thin lines) given:

  (b)Switching curves and sample growth trajectories for the standard specific model (below), reproduction (solid lines), semelparous reproduction (dotted lines), iteroparous reproduction (dashed lines; y 3 = 4). Because in all three cases the values and functional forms of the model parameters are identical (see below), the growth paths are identical (given the same initial condition). The only variation is that structural growth ceases at different body masses (i.e., switching sizes) according to the mode of reproduction. The examples provided in both panels have been obtained using the following standard specific model: g = 3[z 2/3 -

  independent of structural mass (but see online Figs 2 and 3 for examples with size-dependent mortality).

  Fig 1: 536

  Leonardsson, K., de Roos, A.M., Gyllenberg, M., Christensen, B., 1998. Ontogenetic scaling of foraging rates and the dynamics of a size-structured consumerresource model. Theor. Popul. Biol. 54, 270-293. Rennie, M.D., Purchase, C.F., Lester, N., Collins, N.C., Shuter, B.J., Abrams, P.A., 2008.Lazy males? bioenergetic differences in energy acquisition and metabolism help to explain sexual size dimorphism in percids. J. Anim. Ecol. 77, 916-926. Roff, D.A., 2002. Life history evolution. Sinauer Associates Inc., Sunderland, MA. Vincent, T.L., Pulliam, H.R., 1980. Evolution of life history strategies for an asexual annual plant model. Theor. Popul. Biol. 17, 215-231. West-Eberhard, M.J., 2003. Developmental plasticity and evolution. Oxford University Press, New York.

		22
	Persson, L.,
	e p t e d	m a n u s c r i p t
		22

Acknowledgements

This research was supported by an Academy of Finland funding to the Finnish Centre of Excellence in Analysis and Dynamics Research.

Appendix A: Derivation of the optimal structural growth curve and the switching curve Applying the Pontryagin maximum principle (Intriligator, 1971, pp. 344-348;Perrin and Sibly, 1993, appendix therein;Perrin et al., 1993), I obtain the Hamiltonian

where z is the state variable, λ is the costate variable associated with z, and u is the control variable, defined as z = u (recall that the dot sign stands for derivative with respect to θ). The dynamics of the costate variable is given by ( )

Similarly the hazard density in Eq. ( 6), is given by

Subsequently, the expressions for the switching curves in these cases remain the same (Eqs.

[9] and [10], and Eq. [A.6] in appendix A).

For semelparous reproduction, I define ã as the amount of reserves that cannot be utilized for reproduction, once target size is achieved. It may be possible that the amount of reserves that cannot be utilized for maintenance to avoid starvation, i.e., the starvation boundary a, can nonetheless be utilized for reproduction (in this case ã ≤ a). In contrast, some reserves may be utilized for maintenance, but not mobilized for reproduction (in that case ã ≥ a). Regardless of which case occurs, the terminal reward is R = θ 2 -z 2 -ã, and the boundary condition in Eq. ( 5) takes the form λ(θ 2 ) = -1/(θ 2 -z 2 -ã). The expression for the switching curve is then