A norm knockout method on indirect reciprocity to reveal indispensable norms

Although various norms for reciprocity-based cooperation have been suggested that are evolutionarily stable against invasion from free riders, the process of alternation of norms and the role of diversified norms remain unclear in the evolution of cooperation. We clarify the co-evolutionary dynamics of norms and cooperation in indirect reciprocity and also identify the indispensable norms for the evolution of cooperation. Inspired by the gene knockout method, a genetic engineering technique, we developed the norm knockout method and clarified the norms necessary for the establishment of cooperation. The results of numerical investigations revealed that the majority of norms gradually transitioned to tolerant norms after defectors are eliminated by strict norms. Furthermore, no cooperation emerges when specific norms that are intolerant to defectors are knocked out.

Reciprocity is a fundamental mechanism that underlies all cooperative societies. Theoretically it is well known that direct reciprocity, typified by the ''I'll help you if you help me" attitude, promotes cooperative regimes 1,2 . However, in recent societies that have high relational mobility, indirect reciprocity such as ''I'll help you and somebody else will help me" plays a more important role in promoting cooperation. Indirect reciprocity has therefore been the focus of much research in the interdisciplinary fields in recent decades [3][4][5][6][7] .
Many theoretical studies on indirect reciprocity have explored norms that become evolutionarily stable against defection and the invasion of free riders, and several typical norms have been proposed [8][9][10][11] . These approaches have clarified the robust norms that can maintain the cooperative regime. The norms in the studies on the indirect reciprocity are regarded as assessment rules that label the other's action as either Good or Bad. They include tolerant norms that assess cooperative behaviors toward defectors as good 4 and strict norms that assess such behaviors as bad 12 . Other theoretical studies analysing the global dynamics of norms assume that at most a few robust norms are shared in the population [13][14][15] .
Their approaches have clarified the robustness of the norms against invasion of other norms including free riders when the norms are acceptable in the population. However, little is known about a process by which gradual changes toward cooperation occur as new norms emerge and compete, which is to say, the co-evolutionary process of norm-diversity and cooperation. A study on the indirect reciprocity has dealt with co-existing different norms and has analysed their frequencies in the population as a consequence of a dynamical process 16 . In the study, each individual keeps a private image of everyone else and errors of perception and implementation are included in a limited strategy space. Although they have considered some action rules and assessment rules, all possible norms in indirect reciprocity have not been studied all-together. How cooperation evolves cannot be fully understood unless the evolution of norms is also considered.
It is thus a challenging task to theoretically understand how cooperation can be formed even under a collection of norms in a social system. How is the co-existence of cooperation and diversity possible at all? Are there any indispensable norms needed to facilitate the evolution of cooperation in the melting pot of norms, even though some norms never become dominant? Do norms that could be accepted as a result of the co-evolutionary process have common aspects? These questions can be addressed only if all possible norms are considered, and the combination of norms governing a group can evolve.
Here we explore the dynamics of co-evolution of cooperation by using different social norms. The process of the evolution of norms has a transition from stricter to tolerant norms. Additionally, we find a set of norms that Scientific RepoRts | 7:44146 | DOI: 10.1038/srep44146 seem not to have an impact on promoting cooperation, but are fundamental to allow a transition to a cooperative regime from a defective regime.

Results
Agent-based simulations 17,18 are an optimal tool to tackle the challenge outlined above. See Methods for the details of our agent-based model described by the ODD protocol 19 . Using an evolutionary game theoretical framework and constructing an interaction model based on players' private rules and local information, we model a giving game to elucidate the dynamics of the evolution of cooperation amid the coexistence of diverse norms (Fig. 1). We conducted numerical simulations of all 16 possible norm combinations that could react to the four combinations of assessment criteria to clarify the dynamics of the evolution of cooperation from the melting pot of diverse norms. Figure 2A shows time-series graphs of each norm's population and cooperation ratio. As shown, the majority undergo an alternation from strict to tolerant norms, mostly in the order of SH → SJ → ST. Figure 3A shows the transition in the norm with the greatest population ratio. In many cases, the majority transitioned from the state where strict SH 11 was the majority to SJ 5,12 . Afterwards, the majority norm changed to tolerant ST 4,20,21 and ALLG. In contrast, as shown in Figs 2B and 3B, in an environment with errors, alternation from strict norms to tolerant norms was observed. However, the likelihood of going through SJ decreased. Alternation paths through IS 22,23 , which could not be seen in an environment without errors 24 , increased. It is important to note here that similar paths toward cooperation are observed when only ALLB-individuals are initially assumed. New norms are created during the evolutionary process at the same time cooperation evolves. This indicates that cooperation and diversity of norms jointly evolve in the model.
Why does the alternation of norms emerge? For one thing, in states in which defection is dominant, ALLB (BBBB) and SH (GBBB) coexist and jointly form the majority. However, BGBB and IS (GGBB) continue to exist as the minority. The characteristic of these groups is having the evaluation rule of **BB. Evaluation rule **BB assesses donors that took D as B, regardless of the evaluation of the recipient. In states in which defection is dominant, those who adopt **BB strategies consider many partners as B. As a result, cooperation does not occur for the most part. The ALLB and SH norms thus survive because they do not lower their own cost. On the other hand, after cooperation is achieved, ALLG (GGGG), ST (GGBG), IS (GGBB), and GGGB coexist. The common characteristic of these norms is having the evaluation rule of GG**. Thus, reciprocally cooperating norms survive. Because SJ (GBBG), which becomes the majority temporarily when the cooperation ratio rises in an environment without errors, does not belong to either group, it cannot stably exist. Also, it is rare that SJ makes up the majority temporarily in an environment with errors. Meanwhile, because IS belongs to both norm groups with **BB and GG**, IS can constantly exist.
We discover several norms that are indispensable to the evolution of cooperation. Reputation-based cooperation cannot emerge without indispensable norms. To elucidate indispensable norms for the evolution of cooperation, we propose a novel analysis using the norm knockout method. This method enables us to determine which norms are indispensable for the evolution of cooperation. The norm knockout method is inspired by the targeted gene knockout technique used in genetic engineering 25 . Gene knockout, a genetic technique in which one of an organism's genes is made inoperative, is used to research genes whose sequences are known but whose functions are not well-understood. Researchers infer the gene's function from differences between the knockout animal and a normal animal. For simulating evolution, we utilized a method that removed only one particular norm from the population to understand whether that norm is an indispensable one that plays a critical role in the evolution of cooperation. Figure 4 shows the cooperation ratio when a particular norm is knocked out. Regardless of whether there is an error, if SH or IS is knocked out, cooperation does not evolve at all. We define indispensable norms in the When SH and ALLB coexist, cooperation does not emerge. When ALLB is completely driven out by SH, SJ invades and the cooperation ratio abruptly rises. At the same time, SH is driven out by SJ. After cooperation is completely achieved, SJ permits the invasion of ST, and also coexists with other tolerant norms (IS, ST, GGGB, and ALLG). Finally, strategies whose norm is expressed as GG** (in other words, norms that constantly cooperate if cooperation has been selected in the past by the recipient) coexist. In (B), both errors in perception and implementation were introduced, and simulation similar to a was run (b = 5, c = 1, N = 500, R = 500, G = 1000, p = 0.001, q = 0.001). As in (A), when SH and ALLB coexist, cooperation does not emerge. However, cooperation is achieved without going through SJ. evolution of cooperation as the norms that, when knocked out, have an average cooperation ratio of less than 0.1 after 1,000 generations. In an environment with no errors, SH and IS are indispensable norms. In an environment with errors, SH, IS, and ST are indispensable norms.
When an indispensable norm is knocked out, cooperation does not evolve. When cooperation evolves, alternation from strict norms to tolerant norms was observed, as shown in Figs 2 and 3. To analyse whether alternation also occurs when a norm is knocked out, the population ratio of norms when typical norms are knocked out is displayed as time-series graphs (see Fig. 5). Figure 5 shows the results in the cases where SH or IS were knocked out. We discovered that the first condition for the necessary process when cooperation evolves is whether SH can antagonize ALLB. No norm that resists the invasion of ALLB appears in a society in which SH does not exist. Also, in a society in which IS does not exist, SH cannot antagonize ALLB. We found that IS is a norm indispensable for SH to resist ALLB.

Discussion
Our model offered two major findings on the evolution of cooperation on indirect reciprocity. On the one hand, the most essential contribution is the discovery of indispensable norms by the norm knockout method. By using the norm knockout method, we were able to elucidate the existence of norms indispensable for the evolution of cooperation from a melting pot of norms. Regardless of the existence of errors, SH and IS were indispensable norms. In addition, in an environment with errors, ST is an indispensable norm. Interestingly, SH and IS are reconciled to the minorities after the cooperative regime emerges while they temporarily become major norms in the process of dynamics. We call such minority norms required for the evolution of cooperation "unsung hero norms". The results clearly illustrate the two roles of norms: one to catalyse a cooperative regime and the other to maintain the regime. Norms having the GG** for the evaluation rule play the latter role.
On the other hand, we discovered alternation of norms. Recent analysis of evolutionary stability against the invasion of free riders could identify neither superiority among norms nor the process on the path to cooperation.  Among studies on indirect reciprocity, ours is the first exhaustive theoretical analysis on all possible norms, although several studies have addressed the comparison of two types of reciprocal norms 14,15,26 . Others analyse the alternation of norms in direct reciprocity [27][28][29] . We find the alternation of the norms and also discover the indispensable norms that are required to foster indirect reciprocity.
An empirical study 30 supports the co-existence of various norms in the cooperative regime and indicates that the ST norm plays a more important role in human cooperation than SJ, which is consistent with our simulation. This is because we show that the SJ norm cannot survive in the cooperative regime, while the ST one can. Our approach may provide deep insight on the evolution of cooperation because several norms absolutely play an essential role in order to evolve cooperation even though, on the surface, it seems as though they are not directly leading to the evolution of the cooperation.
The present work considers a single action rule (cooperate with Good, defect with Bad) to stress the role of multiple assessment rules. However, the other papers stress the role of multiple co-existing action rules 13,[31][32][33] . Integrating the multiple assessment and action rules may be a useful extension of this paper. We analyse what happens when one norm is absent from the population; however, we have not analysed all the indispensable combinations of norms yet. Extending the norm knockout method to combinations of norms may also be a useful extension of this paper.

Methods
In this section, we describe the details of our agent-based model that uses the norm knockout method. The following model description follows the ODD protocol 19 . Purpose. The aim of the model is to understand the dynamics of norms during the evolution of cooperation, and to find indispensable norms without which cooperative societies could never emerge. In particular, we reveal the effect of these indispensable norms on indirect reciprocity using a new methodology we call the "norm knockout method". We utilize the giving game framework 34 for simulation.

Entities, state variables, and scales.
The entities in the model are agents who play as donor and recipient in the giving game with no spatial structures. The donor chooses cooperation or defection with a recipient using an image that the donor has to the recipient. An image is either Good or Bad. If a donor's image to a recipient is Good, the donor cooperates with the recipient. If the image is Bad, the donor defects. The group size of the model is N. Each agent has it's own norm and a list of images to other agents. The agent also has a probability of errors and a payoff of the game.
The norm of an agent is denoted as one of four possible "assessment combinations", and there are two possible "alleles (G/B)" at the "locus" for each of the four assessment combinations. The first locus of the gene represents an assessment rule to an agent who cooperates with a Good recipient. The second locus represents an assessment rule to an agent who cooperates with a Bad recipient. The third locus represents an assessment rule to an agent who defects with a Good recipient. The fourth locus represents an assessment rule to an agent who defects with a Bad recipient. Incidentally, all agents evaluate themselves as Good. For instance, ALLG always assesses others as Good, and thus, its "genotype" is GGGG using the above mentioned definition of the four loci. Similarly, ALLB is described as BBBB, IS as GGBB, ST as GGBG, and SJ as GBBG.
Each agent has two different types of errors: one, the probability that the agent's updating of its evaluation of others, Good/Bad, is inverted (errors in perception), described as p, and two, the probability to perform an action differently from the one prescribed by its action rule (errors in implementation), described as q. The evolution process of norms involves adopting a genetic algorithm 35 . State variables and initialization in the simulation are shown in Table 1.
Process overview and scheduling. Our simulation runs throughout G generations. A generation consists of R rounds. The agents play the giving game R times as donor in each generation. At the end of a generation, they evolve their own norms using accumulated payoffs that are obtained in the generation. One round has two phases: (A) a phase to play giving games and (B) a phase to update images. After all agents play giving games in phase (A), all agents update their images in phase (B). In phase (A), each agent becomes a donor and each donor randomly chooses a recipient from N − 1 players excluding itself. The donor chooses whether to give benefits to the recipient or not. At that time, the action of the donor is inverted with the probability q. The donor who cooperates pays cost c and the recipient receives benefit b (b > c > 0). In phase (B), each agent (set to i) evaluates and updates an image to the other agent (set to j). The new image to j from i's viewpoint depends on j's action (C/D) as a donor in the last round and depends on the image to k from the viewpoint of i (G/B), where k was a recipient of j in the last round. At that time, the image to j is inverted with the probability p. In the first round in a generation, j's action (C/D) is regarded as random.
After R rounds of the giving game are played in every generation, agents evolve their norm. The evolution process of norms involves adopting the genetic algorithm 35 . Because each locus of norm has an independent meaning for assessment to others, the adaptive process should contain a combination of these elements rather than a string of norms. We have modeled a process of updating norms not as a string of norms but rather as four different assessment rules, which enables the norms to be interpreted as different situations depending on the norm genotype. The first, second, third, and fourth loci represent the assessment for pro-social behavior, tolerant behavior, anti-social behavior, and justified defection (punishment), respectively. Each agent randomly selects two agents from N agents (including itself) to become its parents. For choosing parents, we adopt a roulette selection method. This roulette selection sets a probability distribution of all agents as Π i = (U i − U min ) 2 /∑ j (U j − U min ) 2 , where U i denotes the agent i's accumulated payoff in a generation given by U i = bW − cV, with W being the number of donations i received in the generation and V the number of donations i gave. U min means a minimum value Scientific RepoRts | 7:44146 | DOI: 10.1038/srep44146 of the accumulated payoffs among all. Finally, each agent updates its norm using a uniform crossover technique. With mutation rate m, each locus is inverted for maintaining the diversity of the norm space.
Design concepts. Basic principles. An agent-based simulation is utilized to study indirect reciprocity. We explore how different combinations of norms interact to produce an evolutionary progression towards cooperation.
Emergence. A cooperative regime in the situation of social dilemma emerges from interactions among agents who have various social norms.
Adaptation. The agents of the model play the giving game using their images to others. The agents update their images to others using their norms every round. They evolve their own norms using accumulated payoffs that are obtained in the generation. A norm that can obtain a higher payoff can increase the population through the generation.
Objectives. The objective of all agents is to maximize their own payoff. To maximize payoff, they change their own norm at the end of each generation.
Learning. The agents change their norms in each generation using a genetic algorithm. The fitness of each agent is calculated from the accumulated payoff in the generation. To select the parents of the agent, the model utilizes a roulette selection method.
Interaction. The interaction between the agents is one to one interaction. The giving game consists of the donor and the recipient. There are no spatial structures in the society.
Stochasticity. The interaction between agents is a stochastic process because interaction partners are chosen randomly from the society. At the start of the simulation, each agent is randomly assigned a norm of all 16 norms.
Observation. Three indexes are used for observation: average cooperation ratio in the society, the transition of norms with the greatest populations, and population ratio of each norm.
Initialization. At the start of simulation, the norm of each agent is chosen randomly from all 16 possible norm combinations. In the first round of each generation, the evaluation of all agents is initialized as Good 11,23,36 and payoff of the agents is initialized as 0.
Input data. After initialization, the model does not include any external inputs, i.e., the number of agents (N), error ratio (p, q), benefit (b), and cost (c) are constant.

Submodels.
The norm knockout method. The norm knockout method is implemented as follows. When we knock out a particular norm, that norm is removed in the first round of each generation. Concretely, if the norm of an agent evolves into a norm that is knocked out as a result of the adopting process, the norm of the agent is changed to one of the other 15 norms randomly. In other words, the norm that is knocked out will never exist at all in the society.