extfieldsofstudy:
Computer Science
Economics

s2fieldsofstudy:
Economics
Computer Science

text:
Neural Payoff Machines: Predicting Fair and Stable Payoff Allocations Among Team Members

In many multi-agent settings, participants can form teams to achieve collective outcomes that may far surpass their individual capabilities. Measuring the relative contributions of agents and allocating them shares of the reward that promote long-lasting cooperation are difficult tasks. Cooperative game theory offers solution concepts identifying distribution schemes, such as the Shapley value, that fairly reflect the contribution of individuals to the performance of the team or the Core, which reduces the incentive of agents to abandon their team. Applications of such methods include identifying influential features and sharing the costs of joint ventures or team formation. Unfortunately, using these solutions requires tackling a computational barrier as they are hard to compute, even in restricted settings. In this work, we show how cooperative game-theoretic solutions can be distilled into a learned model by training neural networks to propose fair and stable payoff allocations. We show that our approach creates models that can generalize to games far from the training distribution and can predict solutions for more players than observed during training. An important application of our framework is Explainable AI: our approach can be used to speed-up Shapley value computations on many instances.

Introduction
The ability of individuals to form teams and collaborate is crucial to their performance in many environments. The success of humans as a species hinges on our capability to cooperate at scale [27]. Similarly, cooperation between learning agents is necessary to achieve high performance in many environments [34,7,54,28] and is a fundamental problem in artificial intelligence [50,13]. Individual agents are often not incentivized by the joint reward achieved by a team but rather by their share of the spoils. Hence, teams are only likely to be formed when the overall gains obtained by the team are appropriately distributed between its members. However, understanding how collective outcomes arise from subsets of locally interacting parts, or measuring the impact of individuals on the team's performance, remain open problems.
Direct applications exist in multiple domains. One example is identifying the most influential features that drive a model to make a certain prediction [15,35,6,52,33,32]; one of the cornerstones of explainable AI [2,40]. Another example is sharing the costs of data acquisition or a joint venture in a fair way between participants [8,1], or sharing gains between cooperating agents [24,49]. In many legislative bodies individual participants have different weights, and passing a decision requires support from a set of participants holding the majority of the weight; different states in the US electoral college have different numbers of electors, and different countries in the EU Council of Ministers vary in their voting weight. Here, would like to quantify the true political power held by each participant, or allocate a common budget between them [38,10].
Cooperative game theory can provide strong theoretical foundations underpinning such applications. The field provides solution concepts that measure the relative impact of individuals on team performance, or the individual rewards agents are entitled to. Power indices such as the Shapley value [48] or Banzhaf index [9] attempt to divide the joint reward in a way that is fair, and have recently been used to compute feature importance [35]. In contrast, other solutions such as the Core [26] attempt to offer a stable allocation of payoffs, where individual agents are incentivised to continue working with their team, rather than attempting to break away from their team in favor of working with other agents. Despite their theoretical appeal, these solution concepts are difficult to apply in practice due to computational constraints. Computing them is typically a hard problem, even in restricted environments [21,12,17].
Our contribution: We construct models that predict fair or stable payoff allocations among team members, combining solution concepts from cooperative game theory with the predictive power of neural networks. These neural "payoff machines" take in a representation of the performance or reward achievable by different subsets of agents, and output a suggested payoff vector allocating the total reward between the agents. By training the neural networks based on different cooperative solution concepts, the model can be tuned to aim for a fair distribution of the payoffs (the Shapley value or Banzhaf index) or to minimize the incentives of agents to abandon the team (the Least-Core [26,39,17]). Figure 1 depicts the two well-studied classes of games on which we evaluate our approach: weighted voting games [38,10,12] and feature importance games in explainable AI [15,35].
Weighted voting games (WVGs) are arguably the most well-studied class of cooperative games. Each agent is endowed with a weight and a team achieves its goal if the sum of the weights of the team members exceeds a certain threshold (quota). We train the neural networks by generating large sets of such games and computing their respective game theoretic solutions. Our empirical evaluation shows that the predictions for the various solutions (the Shapley value, Banzhaf index and Least-Core) accurately reflect the true game theoretic solutions on previously unobserved games. Furthermore, the resulting model can generalize even to games that are very far from the training distribution or with more players than the games in the training set.
Feature importance games are a model for quantifying the relative influence of features on the outcome of a machine learning model [15,35]. Solving these games for the Shapley value (or other game theoretic measure) provides a way to reverse-engineer the key factors that drove a model to reach a specific decision. This approach is model-agnostic, thus can be applied to make any "blackbox" model more interpretable [2]. One drawback of this approach is the computational complexity of calculating the Shapley value, making such analysis slow even when using approximation algorithms. Our approach provides a way to significantly speedup Explainable AI analyses, particularly for datasets with a large number of instances.  [38,10]. Our approach predicts solution concepts which reflect the true political influence of players or stable payoff allocations. Right: Applying the Shapley value in Feature Importance Games enables quantifying the relative impact of features on the decisions of a model. In this example, a model predicts the price of a house based on several features. Our approach predicts the Shapley values φ of features, which are a commonly used metric for their impact. This speeds-up Explainable AI analysis, as each new instance can be analyzed quickly without a full computation of Shapley values.

Preliminaries
We provide a brief overview of cooperative game theory (examined in details in various books [45,12]) and discuss how solution concepts in cooperative game theory have been applied in Explainable AI [15,35,36,14,56].

Cooperative Game Theory
A (transferable utility) cooperative game consists of a set N = {1, 2, . . . , n} of agents, or players, and a characteristic function v : 2 n → R which maps each team of players, or coalition C ⊆ N , to a real number. This number indicates the joint reward the players obtained as a team. Games v where v : 2 n → {0, 1} (binary range) are simple games.
Weighted voting games (WVGs) are a restricted class of simple cooperative games [12], where each agent i has a weight w i and a team of agents C ⊆ N wins if the sum of the weights of its participants n i=1 w i exceeds a quota q. Formally, a WVG is defined as the triple (w, q, v) with weights w = (w 1 , w 2 , . . . , w n ) ∈ R n ≥0 and quota (threshold) q ∈ R ≥0 where for any If v(C) = 0 we say C is a losing coalition, and if v(C) = 1 we say it is a winning coalitions. WVGs have been thoroughly investigated as a model of voting bodies, such the US Electoral College or the EU Council of Ministers [38,10].
The characteristic function defines the joint value of a coalition, but it does not specify how the value should be distributed among the agents. Solution concepts attempt to determine an allocation p = (p 1 , . . . , p n ) of the utility v(N ) achieved by the grand coalition of all the agents; an allocation p is called an imputation if for any player i we have p i ≥ 0 and n i=1 p i = v(N ). 1 This allocation is meant to achieve some desiderata, such as fairly reflecting the contributions of individual agents, or achieving stability in the sense that no subset of agents is incentivized to abandon the team and form a new team. We describe three prominent solution concepts that we use in our analysis.
The Core. Rational players may abandon the grand coalition of all the agents if they can increase their individual utility by doing so. The Core is defined as the set of all payoff vectors where no subset of agents can generate more utility, as measured by the characteristic function, than the total payoff they are currently awarded by the payoff vector. As such, the Core is viewed as the set of stable payoff allocations. Formally, the core [26] is defined as the set of all imputations p such that n i=1 p i = v(N ) and that i∈C p i ≥ v(C) for any coalition C ⊆ N . The ε-Core and Least-Core [39,17]. Some games have empty cores, meaning that no payoff allocations achieves full stability (i.e. for any imputation p there exists at least one coalition C such that v(C) > i∈C p i ). In such cases, researchers have proposed minimizing the instability. A relaxation of the core is the ε-core, consisting of imputations p where for any value coalition C we have p(C) ≥ v(C) − ε. Given an imputation p the difference v(C) − i∈C p i is called the excess of the coalition, and represents the total improvement in utility the members of C can achieve by abandoning the grand coalition and working on their own. For an imputation in the ε-core, no agent subset C can achieve an addition of ε in utility over the current total payoff offered to the team (i.e. no coalition has an excess of more than ε). The minimal ε for which the ε-core is non-empty is called the Least-Core Value (LCV). The Least-Core minimizes the incentive of any agent subset to abandon the grand coalition, and the LCV thus represents the degree of instability (excess) under the imputation that best minimizes this instability. We find the set of payoffs associated with the LVC through linear programming (full details in Appendix C).
We now discuss two power indices, payoff distributions reflecting the true influence a player has on the performance of the team, that fairly allocate the total gains of the teams among the agents in it. [48] measures the average marginal contribution of each player across all permutations of the players. The Shapley value is the unique solution concept that fulfills several natural fairness axioms [19], and has thus found many applications from estimating feature importance [15,35] to pruning neural networks [55,23,25]. Formally, we denote a permutation of the players by π , where π is a bijective mapping of N to itself, and the set of all such permutations by Π. By C π (i) we denote all players appearing before i in the permutation π. The Shapley value 1 In a WVG, the value of a coalition is bounded by 1, so if the grand coalition N indeed has a value of v(N ) = 1 then a solution would be a payoff vector p = (p1, . . . , pn) where n i=1 pi = 1.

The Shapley Value
φ i (v) of player i is defined as: Intuitively, one can consider starting with an empty coalition and adding the players' weights in the order of the permutation; the first player whose addition meets or exceeds the quota is considered the pivotal player in the permutation. The Shapley value then measures the proportion of permutations where a player is the pivotal player.
The Banzhaf index [9] is another method for distributing payoffs according to a players' ability to change the outcome of the game, but it reflects slightly different fairness axioms [51]. The Banzhaf index β i of a player i is defined as the marginal contribution of a player across all subsets not containing that player: In practice, we first compute the set of winning coalitions C win ⊆ C and count, for each player, the number of times it is critical or pivotal, that is, v({C}\i) = 0.

Speeding Up Explainable AI In Large Datasets
An important goal of our work is to speedup Shapley/Core computations of many data instances.
In machine learning, we train a model f θ to learn a mapping from some set of input features to an outcome. This could we can train a model to predict the price of a house based on a number of features, such as the number of rooms, the year it was built, and so on ( Figure 1). In such settings, it is desirable, but challenging, to explain the model outputs in terms of the input features. Explainable AI addresses this issue, and recent years showed several applications of game theoretic metrics for measuring feature importance in the machine learning community [52,2].
The fastest method to approximate Shapley values (also used in the SHAP package) is a Monte-Carlo approach [35]. A number of other methods exist whose runtime and accuracy depend on the number of samples used, usually on the order of several thousands [30,15,37,5]. In a model setting, the characteristic function takes the value of the trained model output for a given instance: f θ * (x). The Shapley value of a feature i in a data instance x, φ x,i is the effect that feature has on the model outcome. Sampling based-methods compute the contributions with respect to a base value, which is the average model output across all instances: Sampling based methods are not ideal for large datasets because they require a large number of re-evaluation samples per computation. We show how our approach can be employed to speedup Shapley or Core computations of many instances by training models to learn representations of feature attribution schemes.

Methods
Our approach uses machine learning to create game-theoretic estimators. We generate synthetic datasets of games to train our models, spending compute up-front to allow for instant solutions afterwards. Our first domain concerns weighted voting games (WVGs) as they provide a generic framework for studying cooperation in multi-agent settings.
Afterwards, we apply our approach to the Shapley-based feature importance setting. The main idea is that we can speed-up the computations of the relative impact of features on the predictions of a machine learning model. We do this by training models to approximate Shapley values of features, and examine their performance on previously unobserved instances.

Weighted Voting Games
We consider two types of models: fixed-size models that always predict a fixed number of outputs for games with a fixed number of players, and variable-size ones that predict a variable number of outcomes and can handle variable numbers of players. We now describe our data generation process, models, training procedure, and evaluation metrics used in weighted voting games.

Data and Models: Fixed-Size
For each n player game, we generate G independent and identically distributed games. The training dataset is given by D n fixed = X ∈ R G×n , P ∈ R G×K where K denotes the number of outputs of interest. The features are obtained in two steps. First, we sample a weight vector w ∼ Beta(α = 1, β = 1) on the interval from 1 to 2n, and a quota q ∼ N (µ = 1 4 (2n + 1)n, σ 2 = 2n). To create games where players are dependent on each other to achieve the task at hand, the quota distribution is parameterized such that the average drawn quota is half of the sum of the players' weights. We then normalize the weights by the quota to get the feature vector x = 1 q · w. In other words, we have n features which are the weights normalized as the respective proportion of the quota. Thus, a value x i > 1 indicates that player i is a winning coalition by itself, and needs other players to meet the quota otherwise ( Figure E.2).
We train models to predict the three solutions: the Least-Core, the Shapley values, and the Banzhaf indices. For the Shapey values and the Banzhaf indices, the model predicts the payoff allocation (p 1 , . . . , p n ) so K ≡ n. For the Least-Core solution, we train the model to not only predict the payoff allocation p 1 , . . . , p n , but also to predict the Least Core Value ε min (so in this case the model has K = n + 1 outputs). For each n player game, we produce a model f θ : R n → R K , where θ are the model parameters. We use deep feedforward networks for f θ . Appendix D.1 contains the full details about the experimental setup.

Data and Models: Variable-size
For the variable-size case, we consider a maximal number of possible players M and pad the inputs with zeros for games with less than M players. Hence, we generate a single dataset D var = X ∈ R G×M , P ∈ R G×K . Similarly to the fixed-size dataset, the feature matrix X contains the normalized weights and P the corresponding solutions with either K = M + 1 or K = M , with the ground truth output vector again padded with zeros when there are fewer than M players. Hence, we allow for the prediction of up to M players, and we shuffle the data so that players are located at random positions. The Least Core Value (LCV) is not shuffled but stored at the last element of each row (Figure E.1).
We produce a single model g η : R n → R K that can be used for different number of players n, where η are the model parameters. The model learns to allocate the joint payoff among at most M players. During prediction time we pad the input with zeros when there are fewer than M players, and we redistribute the payoffs allocated to non-player entries among the players according to their original share of the joint payoff.

Training and Evaluation
During training we minimize the Mean Square Error (MSE) between the true and predicted solutions. For the variable-size models we also include the M − n padded locations so that the model learns not to allocate value to non-player entries.
Evaluation metrics. Given a predicted payoff vectorp = (p 1 ,p 2 , . . . ,p n ), we consider multiple metrics for assessing the models' performance. First, we quantify the models' predictive performance via the Mean Absolute Error (MAE), defined for each game as MAE = 1 where n is the number of players. For the Least-Core there is another natural game theoretic metric. The goal of the Least-Core is to minimize the incentives of any subset to abandon the current team and form its own sub-team. Given a suggested payoff vectorp, the maximal excess v(C)− i∈Cp i over all possible coalitions C measures the incentive to abandon the team, and serves as a good measure for the quality of the model.
Test data. We sample weights w ∼ Beta(α, β) with varying parameters for α and β to assess our models' ability to generalize to previously unobserved instances their ability generalize to games far outside of the training distribution (full details in Table 2, Figure D

Feature Importance Games
We perform an experiment to show that neural networks can provide a faster alternative for measuring feature importance at scale. We select a dataset (details in Appendix D), train a model, and use those to construct a dataset from features to Shapley values using the SHAP KernelExplainer [35] Following, we partition our dataset into a train and test, and incrementally change the proportions between the two. For each increment, we train a model for 100 epochs and test it on the remainder of the unseen instances.

Results
We present experimental results that allow us to assess how well neural models are able to learn a representation of the various solution concepts. We first describe the predictive performance of neural networks in the WVG setting, and consider properties of these solutions that make them hard to learn. We then consider the explainable AI domain, and study the performance and sample complexity of Shapley feature importance prediction.

Weighted Voting Games: Evaluation
For our WVG analysis we train a selection of fixed-size neural networks f N θ * for each number of players n ∈ {4, 5, . . . , 19, 20} 2 on G = 5, 000 games each. We also train a single variable-size model g η * that is trained on one dataset containing G = 17, 500 games in total, consisting of 2, 500 games for each number of players n ∈ {4, 5, . . . , 9, 10} games. The data is padded with zeros to allow for payoff allocation up to M = 20 players (see details in Section 3.1). Table 1 shows that the ability to handle games of variable numbers of players comes at the cost of having a lower accuracy. However, even for variable-size models, and even under a significant distribution shift, the errors in predicting all solution concepts are low. We further note that the error in predicting Least-Core based payoffs are generally larger than for the Shapley and Banzhaf power indices. One possible reason is that for these solutions, there are many cases where a small perturbation in the weights or quota results in a large perturbation of the Least-Core solution [20,58]. We now highlight several conclusions from our analysis of the results.

Predictive performance
Fixed-size models display stable performance across solution concepts. Our first observation is that the fixed-size neural networks are adept at estimating solutions across the three considered concepts. The average error per player ranges from 0.005 to 0.087 (Least Core), 0.004 to 0.069 (Shapley values) and 0.012 to 0.067 (Banzhaf indices). Table 5 provides a complete summary of our models' in-sample performance.
Fixed-size models are robust to shifts in the weight distribution. As shown in Figure 2, the predictive performance (Mean MAE) of the fixed-size models is consistent across test datasets. To account for the natural decrease in the MAE as N increases, we also display the average payoff per player (1/n). As expected, the error scales approximately with the average payoff per player.
Variable-size models are robust to shifts in the weight distribution. Figure 3 shows that the variable-size models are also able to generalize outside the training distribution. Across test sets, we observe a stable performance that decays with the number of players, as is expected. For the Least Core, we see that there is an abrupt decrease in performance for the excess beyond 8 players.
Variable-size models generalize to a larger number of players. Our main objective is to investigate how to leverage machine learning to perform scalable value estimation in large multi-agent settings. To test this, we train the variable-size models on games up N = 10 players, and evaluate on N + 1, N + 2, . . . , N + 10 players (full details in section 3.1.2). there is no significant decrease in performance for games with more than ten players across solution concepts: variable-size models are able to extrapolate beyond the number of players seen during training. This suggests that there is be valuable information in the small player games such that games of larger sizes can be inferred.
Fixed-size models outperform variable-size models. Table 1 contains the Mean MAEs across solution concepts and test sets. Corresponding error distributions are showed in Figure 4. From these, we conclude that in almost all settings the fixed-size networks outperform the variable-size networks. Variable-sized networks tend to have larger errors for all predicted variables and display more variance in their predictions.

Discontinuities in the Solution Space
The function mapping from game parameters to solutions contains discontinuities. Discontinuous jumps emerge from the players' interdependence and the effect of the quota and are difficult for a model to learn. We analyze two examples that demonstrate how our models respond to such transitions.
Solution concepts are step-wise constant functions which are difficult to capture. Consider taking a WVG and changing the weight of a player. This only changes the game theoretic solutions when it restructures the subset of winning coalitions. Thus, the function outputs the same value until the weight reaches a certain threshold where the structures of winning coalitions change, at which point the solution can change drastically. Learning these kind of functions is difficult, as the error around the discontinuity point is often large.
Our analysis examines an n player game with a fixed weight vector w = (w 1 , w 2 , . . . , w n ) and considers an array of quotas q = (q 1 , q 2 , . . . , q K ) = (min(w), min(w) + , . . . , n i=1 w i ), where = 0.1. We solve the game for each combination of the fixed weights and the changing quota in q to obtain a matrix P. Figure 5 shows the fixed-size model predictions for two selected WVGs. The models capture the overall effect of the changes to the quota, but do poorly close to the discontinuity point (where the ground truth solution incurs a large change).

Speeding Up Feature Importance Calculations By Predicting Shapley Values
We examine the sample efficiency and performance of neural networks trained to predict Shapley values on sets of features. We consider two well-known datasets: the classical UCI Bank Marketing dataset [42] (17 features, 11,162 observations) and the Melbourne Housing dataset [47] (21 feature, 34,857 observations). Full details in Appendix D.2). Overall, we find that neural networks achieve high performance on the test set with very few training samples. Figure 6 displays the models' performance (in RMSE) as a function of samples available for training. We observe that the error decays approximately exponentially with the proportion of data used for training on both the Banking and Melbourne datasets. A model trained on just 1 percent of the Melbourne dataset has an RMSE of 19.72 on the resultant test set. Increasing the number of training samples to 3 percent results in a RMSE of less than 0.07, a 99.6 percent decrease in error. We observe a similar trend for the Banking dataset: the RMSE decreases with 93.8 percent (from 4.10 to 0.25) when the available training data increases from 0.5 to 4 percent. These results highlight the strength of this approach: the computational cost of training the Shapley prediction network is very small as compared to the speedup obtained on the vast majority of the data (and any subsequent instances analyzed later). See Appendix I for a more extensive analysis.

Discussion and Conclusion
We considered "neural payoff machines", models that take in a representation of a cooperative game and solve it to produce a distribution of the joint gains of a team amongst its members. Our analysis focused on two concise representations of a characteristic function: Weighted voting games, and feature importance games for explainable AI. Our analysis shows that neural models capture cooperative game solutions well, and can generalize well outside the training distribution, even extrapolating to more players than in the training set.
We saw that the Least core is a harder solution concept to learn than the Shapley value and Banzhaf index. Potentially, the fact that the Least core is complex and can only be computed by solving multiple Linear Programs makes a hard concept to grasp from examples.
The methods we discussed can drive better analysis of decision making bodies and for approximating feature importance, but such methods can also be applied to other classes of cooperative games, so long as one can generate data consisting of game samples and their solutions. For instance, the same approach can be used for other cooperative representations, such as graph-based games [17,3,4] or set-based representations [44]. The direct applications that we have considered in this paper are also intrinsically valuable on their own, as we allow speeding up explainable AI analysis and political influence estimation. Some questions remain open for future work. Can similar techniques be used for non-cooperative games or for other types of cooperative games? Can this approach be applied to other solutions such as the Kernel or Nucleolous [12]? Finally, are there better neural network designs, such as graph neural networks or transformers, that can exploit invariances or equivariances in the game's representation or be better equipped to deal with sharp discontinuities in the solutions?

Appendix A Notation
The following notation is used throughout this paper: • n the number of players in the game • G the number of games • M the maximum sequence length in variable feedforward networks • q the quota, threshold that determines when a coalition can complete the goal or task • w i the weight of a player i • x i = w i /q, the normalized weight of a player i • N the grand coalition • C a coalition (subset of players) • C min the minimal set of winning coalitions: the coalitions where each player is pivotal • X feature matrix • P solution matrix • p i the allocated reward or payoff to a player i • ε the least core value (LCV), i.e. joint payoff to sacrifice such that there exists a feasible payoff vector in the least core

B Motivating The Use Of The Shapley Value To Analyze Influence In Weighted Voting Games
We provide an intuitive example motivating the use of Shapley values for measuring the true influence of participants, in weighted voting games or political power in decision making bodies.
Consider a parliament with 100 seats, and imagine that following the elections we have two large parties of 49 seats each, and one tiny party who has only managed to get 2 seats. However, note that having a majority means controlling a majority of the seats, i.e. more than 50 seats. This means that neither of the two big parties has a majority on its own (and of course the small party hasn't got a majority on its own).
Hence, to achieve a required majority and form a government or pass a bill, the parties have to work in teams. Clearly, a coalition consisting of the two big parties has a majority, as together they have 49+49=98 seats out of the total of 100. However, a coalition consisting of any of the two big parties and the one small party also has 49+2=51 seats, which is also a majority.
Hence, in the setting discussed above, any two parties form a winning coalition that has a majority and is able to form a government or pass a bill. This can be modelled as a weighted voting game, where the weights are (w 1 = 49, w 2 = 49, w 3 = 2) and the quota is q = 50.
Intuitively, as any two parties or more are a winning team and as any single party is a losing team, we might say that the weights are irrelevant. Although the big parties each have almost 25 times the seats as the small party, they all are symmetric in their opportunities to form a winning coalition, and one could claim they actually have equal political power. Clearly, the small party is in a very strong negotiation position, and could demand for instance to control a significant proportion of the joint budget.
In the above weighted voting game, the Shapley value of all agents would be equal (each getting a value of 1 3 ), as it considers the marginal contributions of the parties over all permutations of players. Due to the symmetry between the parties, they have identical marginal contributions.
The above specific example illustrates some of the principles behind the fairness axioms that the Shapley value exhibits, and illustrates its importance for measuring influence in decision making bodies.

C Algorithm For Generating Least Core Solutions
The least core solutions can be generated by considering all possible coalitions. A simple improvement, however, is to use the minimal set of winning coalitions [16] instead. In this section, we describe the idea behind this approach and the algorithm used in this paper.
Let N ≡ {1, 2, . . . , n} denote the set of all N ∈ Z ≥0 players, let q ∈ R ≥0 denote the quota and let w i ∈ R ≥0 denote the weights for player i, for all i ∈ N . The characteristic function is given by v(C) ∈ [0, 1], for coalitions C ⊆ N .
The naive approach to find the least core solutions is by considering all winning coalitions C win defined as and to find a least core solution by solving LP-problem C.1a. In the LP-problem (C.1a), all winning min p1,p2,...,pn,ε ε s.t. coalitions are considered, Inside those winning coalitions there exists a subset, that could branch out and form a smaller coalition. This yields a higher payoff for the players in the sub-coalition and is hence a Pareto improvement for the "decision-makers". Therefore, we can instead only consider the subset of minimal winning coalitions, given by and solve the more efficient LP-problem C.1b.

D Experimental details
In this section, we elaborate on the experimental procedures, model architectures and hyperparameters used in our experiments.
Code and notebook. Following the guidelines, we provide python code to reproduce our results, a pedagogical notebook (called CooperativeGameTheoryPrimer.ipynb) that introduces the most important game theory formalisms and the necessary data in the accompanied zip file.

D.1 Weighted Voting Games
Optimization procedure fixed-size feedforward networks. For each N -player dataset D N fixed we perform R = 100 independent runs with a maximum of E = 6000 epochs and an early-stopping criteria which breaks a run if, after a baseline of 500 epochs, the validation loss does not improve for 75 consecutive epochs. For each run, 70 percent is allocated for training and 30 percent for validation. After training, we select the model with the best validation loss. The selected models are stored and used for evaluation. In all experiments we use the Adam [29] optimizer, ReLU [43] activation functions for the hidden layers, Softmax functions for output layer of the payoffs and a Sigmoid for the output layer of epsilon.
Optimization procedure var-size feedforward network. We perform R = 100 independent runs for one variable dataset D var with a maximum of E = 15000 number of epochs and an early-stopping criteria which breaks a run if, after a baseline of 500 epochs, the validation loss does not improve for 75 consecutive epochs. For each run, 70 percent is allocated for training and 30 percent for validation. After training, we select the model with the best validation loss, which is stored and used for evaluation. In all experiments we use the Adam [29] optimizer, ReLU [43] activation functions for the hidden layers, Softmax functions for output layer of the payoffs and a Sigmoid for the output layer of epsilon.
Hyperparameters. All results are generated with multi-layer perceptrons (feedforward networks) of varying hidden-layer size and width. Further hyperparameters are the dropout rate, weight decay in the Adam optimizer, and the learning rate. Our standard model architectures are depicted in Figure E.1 Right.
Approximated ground-truth Shapley values. To obtain the ground-truth solutions for Shapley values above nine players, we perform Monte-Carlo sampling to obtain a subset of all possible permutations. For each of the R = 10 resamples we randomly sample P = 1000 permutations and average across the resamples to obtain approximations of the ground-truth. To validate that the approximated labels are representative we compute the Mean MAE between the true and approximated Shapley values for n = 5 up to n = 10 players with 1000 games per n-player game. Across all n-player games the mean MAE between the true and approximated Shapley values was 0.0011 and never exceeded 0.0014. Figure D.2 shows the distributions per player together with the true label (red dot) for 8 and 9 player games.
Parameterizations for test datasets. Figure D.1 shows the five standardized test distributions that are generated with the parameters in Table 2.

D.2 Explainable AI
We provide a step-wise explanation of how we performed the experiment in our explainable AI section, that is, train a neural network to predict the Shapley values for a dataset of features: 1. Select a dataset. The first step is to select a dataset of choice. In our example, we end up with a feature matrix X N ×F where each row is an observation and each column a feature, and a target vector y ∈ R N or y ∈ [0, 1] depending on the task.
2. Feature to target. Following, we estimate a model f θ : R F → R that predicts one output for each set of features.
3. Compute Shapley values through sampling. We use the trained model from step 2 to obtain Shapley values with the SHAP package KernelExplainer [35], which is based on LIME [35]. This yiels a matrix of Shapley values, one for each feature in every data instance: Φ ∈ R N ×F .  Table 3) for 100 epochs, and evaluate on the 1 − t i N remainder (unseen) samples.
Datasets and preprocessing. Table 4 summarizes the considered datasets. The UCI Banking dataset contains 15 features and a binary target (whether the client has subscribed a term deposit), whereas the Melbourne housing dataset contains 21 features and a regression target (the price of a house). We exclude the "duration" feature in the Banking dataset to prevent data leakage, as is recommended in the documentation. Moreover, we exclude the "Rooms" feature as there is a lot of overlap in this and the "Bedroom2" feature. Compute Shapley values. We use the trained models to obtain Shapley values for each instance in the feature dataset X using the SHAP kernel explainer object [35].
Feature to Shapley: procedure. Finally, we can train a neural network using the datasets X, Φ.
To investigate the sample complexity required to learn a representative mapping from features to Shapley values, we conduct the following experiment. First, we create an array of T = 999 linearly spaced training splits t = (0.001, 0.002, . . . , 0.998, 0.999). Each element in t i in t indicates the ratio of samples that are used for training the neural network, leaving the remaining (1 − t i )N samples for evaluation. For each train/test partition in t then, train a feed forward neural network on t i N samples through minimizing the MSE between the predicted and actual Shapley values (architecture and hyperparameters specified in Table 3) for 100 epochs, and evaluate on the 1 − t i N samples. Figure 6 depicts the resulting RMSE for both datasets as a function of the training fraction.

F Additional Results
In-sample predictive performance. For completeness, we provide the MAEs for each solution concepts on the in sample dataset up to n = 15 players in Table 5. Capturing step-wise jumps: full description. We provide a more detailed description to Figure  5. In the Figure on the left, each player is a winning coalition by itself when q = 1. Up to q = 3, player 3 is more critical than players 1 and 2 and receives a larger share of the joint payoffs. All players obtain equal payoffs when the only winning coalition is the grand coalition. On the right side: each player is a winning coalition by itself when q = 1, which results in an equal payoff of 0.2 for each player. Increasing the quota continuously alters the set of winning coalitions, thereby the relative importance of each player. At q = 3 players 1, 2 and 3 have become less critical and obtain a payoff of 0.18 each compared to 0.22 for player 4 and 5. Finally, when q = N i=1 w i = 7, the only winning coalition that can be formed is the grand coalition: each player is required and the joint payoff is divided equally.

G Hardware details
We trained our neural networks on an internal compute cluster with 10 Quadro RTX 6000 GPUs and two Intel(R) Xeon(R) Gold 5218 CPUs, each of which has 16 physical cores (32 logical processors).

H Asset licensing
Two main assets were used in this work:

I Evaluating the Speedup Achieved by Our Approach, Depending on the Size of the Dataset
Many domains involve datasets with a large number of features, sometimes with over thousands or millions of data instances. A prominent example is language modelling, where large bodies of text are generated, translated or processed in different manners [46]. In such cases, computing the Shapley value for each data instance using sampling-based methods takes an extremely long time.
Our approach is targeted at speeding up feature importance computations of many instances. We perform a simple experiment to demonstrate the effectiveness of a model-based approach for explaining model decisions. We take the Melbourne Housing dataset and obtain 9000 instances by 13 features after preprocessing steps (encoding the categorical features and standardization). Concretely, we measure the Mean Mean Absolute Error (MMSE) of our model predictions and the SHAP package to the ground-truth dataset. We note that speedup does come at a slightly reduced empirical performance: the model MSE on the test set is, MSE = 1.15 × 10 −3 , compared to the MSE with SHAP: MSE = 7.4 × 10 −5 . However, we expect that hyperparameter tuning and better preprocessing will improve the quality of the model predictions.
All together, it is easy to see how this win in computation time translates to larger datasets. Consider a similar dataset with 100,000 or a million entries. Computing Shapley values on the entire dataset with SHAP will take weeks (40 or 400 hours), while the time for our procedure remains relatively the same: building the train set and training the neural network only takes a fraction of the time required by the sampling-based approach: the analysis using our method would take a few hours.

J Heuristics and Simpler Models
We now consider a common heuristic for payoff allocation in weighted voting games, the weight proportional payoff allocation. We use this as a benchmark to evaluate our models' performance against. We find that our fixed models outperform this benchmark heuristic across solution concepts and test sets.
WeightProportional baseline. The WeightProportional (WP) heuristic divides the payoffs in a way that is directly proportional to the players' weights, that is, the payoff of a player j is: for any n−player game. WP provides meaningful benchmark as it is the most naive method for allocating payoffs. The resulting payoffs (and thus errors) do not take into account the quota, which may have a major impact on the correct solutions by means of the emergent set of winning coalitions. As such, any improvement on this benchmark thus suggests a better understanding of the collaborative structures in the game at hand.
We also compare the predictive performance of our models against the predictive performance of multinomial regression models. These models have just one layer with n inputs and n outputs for each n−player game and a softmax nonlinearity at the end. Across all game sizes, our models outperform the linear models by 45 % to 95 % percent. Results are found in Table 6.

K Hard cases
Across solution concepts (Shapley, Banzhaf and Least core), discontinuous jumps in the solution space yield large errors (see Section 4.1.2 and Figures 5, L.4). In this section, we highlight failure cases for the Least core. In contrast to the power indices, a solution in the least core is obtained by solving a Linear Program under several hard constraints. Thus, a solution to a weighted voting game is only feasible when, for each winning coalition C ∈ C win , we have that where p i is the payoff of player i in coalition C. In words, each winning coalition must obtain joint reward that is at least as good as what they would have obtained on their own. If this criteria is Majority of the model predictions fail to meet hard constraints. The fact that the model is not able to predict many feasible solutions in these large games is unsurprising. The set of winning coalitions increases approximately exponentially in n. As such, the more players in a game, the more hard constraints need to be satisfied in order for a solution to be feasible per game. All hard constraints need to be met in order for a solution to be feasible. We expect that including a penalty term for infeasible predictions during training will help to improve this metric and hope to address this in future work.
When the least core is a set, simplex based methods tend to find corner solutions, which leads to ambiguity in the solutions. Further analysis of individual weighted voting games reveals that the most problematic games are cases where the least core contains multiple correct solutions, but the full joint payoff is allocated to a single player. In other words, payoffs are divided such that one player receives one, and all the others zero. To illustrate what is happening here, consider a simple four player game that produced the largest error ( Figure K.2). We focus our analysis on the game on the left, but our insights apply to a wide range of n-player games in the dataset. This particular game has the following weights and quota (rounded to one decimal place for simplicity) w = (2.8, 1.6, 6.6, 1.5); q = 12.1 where we can see that no player is a winning coalition by itself (all w i 's < q). In fact, the only winning coalition in this game is the grand coalition, that is, The solution to this weighted voting game is: where player one receives the full joint payoff, even while its' weight is smaller than the weight of player three (2.8 < 6.6). The reason why we obtain this solution comes down to two facts. First of all, the least core is a set, and can therefore contain multiple correct solutions. Secondly, simplex based methods return corner solutions. Thus, in cases where several players are required to form a winning coalition, the solver will allocate the joint payoff to an arbitrary player that belongs to this critical subset. Figure  WVG w = (1, 1) with q = 2, so the only winning coalition is the grand coalition. Solving this game results in either of the corner solutions: the payoff vector p = (1, 0) or p = (0, 1) with a LCV = 0. However, if we define α as the solution of a player one, any combination p = (α, 1 − α) is a feasible solution in the least core.
Obtaining the full set of solutions for this game can be done empirically. In principle, we can take the payoff vector and LCV above, generate the set of permutations of length two (player pairs) and check whether the solution is still feasible if we give a small value δ from one player (whose payoff is not zero) to any other player. However, this approach does not scale to larger games. It would be more efficient to derive rules for acquiring the full least core set from a single solution. We limited our analysis above to four-player games. However, we observe the above behavior irrespective of the number of players in the game. Therefore, addressing this issue is critical for improving the predictive performance of our models.

L Applying Our Models to Voting In The European Council
In this section we apply our trained models to predict voting power of member states in the EU Council. First, we take the weights from four countries in the EU voting council [31], see Table 7, and define the quota as the majority vote (50 % of the total weights + 1). Our (weighted) voting game consists of the member states Hungary, Netherlands, Poland and Ireland with the weights w = (12,13,27,7), and quota q = 30, 5.
Following, we divide the weights by the quota and feed the normalized weights (see Section 3.1). We We observe that the contribution measured by the Power indices (Shapley and Banzhaf) is less than the weight, while the Core attributes a larger share of the joint payoff to this member state.
Next, we perform a sensitivity analysis. In such an analysis, we are interested in the interdependence of member states' voting power. Holding all other game parameters constant, we increment the weight of Hungary (w Hungary = 7) with values of one, until it exceeds the quota of 30,5. Figure  L.2 depicts how the member states' voting power (in Shapley value) changes as Hungary gains absolute power (as we increase the weight). The predicted payoff for Hungary is depicted by the red triangles, the model predictions for the other states in blue, and the actual solutions by the green dots. We observe that there are two transition points in the solution space. One transition point occurs when w Hungary = 17.5/18.5. Hungary can now form a winning coalition both with the Netherlands (w Netherlands = 12) and with Poland (w Poland = 13), which raises its power in the voting and accordingly its' share of the joint payoff. We notice that our model is able to predict accurate payoffs overall, with a spike in absolute error per player at this transition point.
The second transition point occurs when w Hungary = 30.5 = q. Hungary is now also a winning coalition by itself, thereby obtaining a larger share of the joint payoff. We observe that its' Shapley value increases to approximately 0.6, obtaining the more than half of the total reward. Our model is able to capture this transition almost perfectly (MMAE < 0.01).
We perform a similar analysis, this time perturbing the quota. Figure L.3 depicts the same game with the original weights fixed, increasing the value of the quota from small to the sum of the weights (notice that this is the same experiment we performed in Section 4.1.2). The value of the quota determines the set of winning coalitions, thereby dictating the payoff of each player. As such, perturbing in this game yields a large number of discontinuous jumps in the member states solutions compared to our previous analysis perturbing the weight of Hungary. Our model is able to capture the overall changes in the solutions but has difficulty with capturing large jumps in payoffs for small quota increments.
Our variable models models have been trained on games up to ten players and can be applied to instantly predict the payoffs of member states in larger councils. To illustrate this, we take all member states in Table 7 and use our variable models to predict the payoffs for each, again setting the quota as the majority vote. We report the prediction quality of our models with respect to the ground truth (in Mean Mean Absolute Error), which is 0.013 for Shapley, 0.004 for Banzhaf, and 0.05 for the least core. Figure  While there is room for improvement in terms of prediction quality, our approach provides a useful tool for obtaining quick estimates of the voting power in medium size games (10 ≤ n < 50). Furthermore, while approximation methods exist for Shapley and Banzhaf, there are currently no effective methods for estimating payoffs in the least core. This solution concept is more challenging as the solutions are obtained by solving a Linear Program with a hard constraints. Thus, solving a game with 15 or even 20 players contains on the order of 2 15 or 2 20 constraints, which slow down the solvers dramatically. Here, we can apply the same technique, solve for the small/medium sizegames, then leverage our neural network approach to tackle a higher numbers of players. In other words, provided a solver that can handle up to M players, we can train a neural network on this number of players, allowing us to extend the number of players even beyond M , possibly up to 2M .

M Real World Applications of Weighted Voting Games
Weighted voting games, and cooperative game theory in general, provide valuable tools for reasoning about the allocation of collective outcomes to individual agents. As such, their applicability span numerous domains. In this section, we highlight several real-world applications of our work. In all examples, our framework can be applied to both speed up and enhance the scalability. Dinar et al. [18] utilize solution concepts from cooperative game theory -including the Core, the Shapley Value, the Nucleolus and Nash -to allocate water resources to competing parties in the presence of scarcity. More recently, Mirzaei-Nodoushan et al. [41] also performed an extensive evaluation of both cooperative and non-cooperative game theoretic approach for water management in trans-boundary river basins.
Moreover, cooperative game theory has proven useful in analyzing the costs of joint activities [57]. Vázquez-Brage et al. [53] combine a model of airport games with the Shapley value to fairly distribute the aircraft landing fees among airlines. When an airport is build, the size of the runway is chosen to match the largest designed airplaine. The authors address how to fairly divide costs between airlines provided the different designs and sizes of their planes.
Bistaffa et al. [11] applied cooperative game theory to analyze the social ridesharing problem. Mobility platforms such as Lyft allow their users to share their trip with nearby users in real-time, providing a more environmentally friendly and cost effective alternative to commuting alone. They study how the costs of the ride should be divided among passengers. Another application is in supply chain management, where Fiestras-Janeiro et al. [22] study how to coordinate actions between suppliers, manufactorers, retailers and customers.


Bibliometric review about eco-cites and urban sustainable development: trend topics

The world is undergoing exponential urbanization, with consequences for society in general, both for cities and people, with serious economic, social and environmental implications. Therefore, it is crucial to reconsider urban models of cities, to ensure global, long-term sustainability. Numerous researchers have argued that one of these models involves eco-city projects, and this study aims to carry out a systematic review of the literature on this topic, through bibliometrics. To do so, the Smarter, Prisma and Vantage Point methods were used, reflecting the originality of the study. The results obtained show that China has aroused researchers’ interest regarding its eco-city projects, with most studies being empirical, and so this study complements them. The main contribution of this research lies in identifying the keywords most used by authors, these being eco-cities, sustainable cities (cluster 1) and eco-cities, China, sustainable urban development and eco-cycle model (cluster 2). Finally, implications for theory and practice are presented, together with suggestions for future research.

Introduction
The world's population is due to growth by 2 billion in the next 30 years, from the current 7,7 billion people to 9,7 billion in 2050, according to the United Nations-World Population Prospects 2019 (UN, 2019). In Europe, urban areas account for around 67% of the population of the European Union, representing approximately 80% of the energy used and contributing 85% to Europe's Gross Domestic Product (UE, 2021). This means that 75% of Europe's population live in urban areas, creating environmental challenges, and it sustainability (Miao & Lang, 2015). This line of thought is clear in an empirical study carried out in China by Xie et al. (2020), which concluded that building an eco-city is a high ambition, although necessary if aiming to manage and sustain urban economic growth while considering environmental questions.
The vast literature on topics involving the eco-city concept in its various aspects contains numerous empirical studies about China (e.g. Caprotti & Romanowicz, 2013;Kang et al., 2019;Liu et al., 2014;Miao & Lang, 2015;Visser, 2019;Xu, 2017) and a shortage of bibliometric studies on scientific production in the field which would allow conclusions to be drawn about the state of the art. This gap is confirmed by a search on the WoS to identify the number of systematic literature reviews, resulting in six documents between 2016 and 2018. Also the arguments that "eco cites can be seen as a new life style" and that "cities often focus on one or two elements of the paradigm and do not really follow an integrated approach. Secondly, some cities wanted to be eco-cities in the 1990s and are trying to become sponge cities now (Zhuanghe), or smart cities (Shenzhen and Beijing), where they were an eco-city before (both). This has to do with the support received from the national level" ( van Dijk & Zhang, 2019, p. 14). This argument confirms the importance of elaborating a systematic review of the literature on the topic, as highlighted by Wachsmuth et al. (2016), who concluded that cities everywhere are facing unprecedented challenges, with new measures and actions being necessary to achieve broader goals of ecological effectiveness, economic growth and equitable social development, given that they have become one of the largest contributors to greenhouse gas emissions and represent 60 to 80% of global energy consumption (UN-Habitat, 2016). In addition, Saiu (2017, p. 3) concludes that "the vast body of literature on the projects analyzed can be subdivided into two general categories: single case studies and comparative research. This second category is often focused on the projects carried out in the same geographical, socio-economic and political context", corroborating the justification for carrying out a systematic review of the literature on the topic. Therefore, this study aims to make a bibliometric analysis of eco-cities and urban sustainable development, so as to identify trend topics, using qualitative (descriptive analysis and Prisma) and quantitative (Smarter and Vantage Point methods) methodology. The study's contribution lies in using an innovative methodology resorting to three methods simultaneously to determine the state of the art of the subject. This will allow the elaboration of a theoretical, conceptual study of which, according to Saiu (2017), there is a shortage in the academic sphere. In addition, the use of the SMARTER method was crucial during the process of deciding the eligibility of the studies, given that its multiple attributes remove the subjectivity inherent in this phase. Furthermore, this method has been little used in research on cities, and this is the innovative character of this study.

Bibliometrics
The systematic literature review requires all the stages followed to be transparent (Briner & Denyer, 2012) that the collection and systematization of data is specific (Hadengue et al., 2017) and that the results obtained are clear (Briner & Denyer, 2012). In this context, Tranfield et al. (2003) define the stages to follow in this type of review, i.e. planning the review, locating studies, evaluating their contributions and a summarized analysis of the 1 3 results obtained, which were corroborated by Xiao and Watson (2019). The review presented here followed the bibliometric analysis for scientific mapping of the topic under discussion (Connor & Voos, 1981;Garfield, 1979;Powell et al., 1996;Quinlan et al., 2008;Wasserman & Faust, 1994;White & Griffith, 1981). The systematic review of the literature carried out through bibliometrics aims to identify the most relevant topics, as defended by several authors (e.g. Connor & Voos, 1981;Howard & Katherine, 1998;Powell et al., 1996;Quinlan et al., 2008;Wasserman & Faust, 1994) which assumes scientific documents (Article or review, Proceedings papers and Book Review) as the unit of analysis for grouping them, according to their objectives and issues central banks (Grácio, 2016). Consequently, the operationalization of a bibliometry requires the definition of rigorous steps to follow, which are: a) identification, assessment, content analysis in specific areas (Rowley & Slack, 2004); b) to assess critical aggregate value by summarizing the literature on the topic studied, identifying gaps and indications for future avenues (Mentzer & Kahn, 1995); c) guarantee the standardized descriptive analysis (e.g. authors, scientific journals, citations and co-citations, keywords, countries, publications by year) (Prasad & Tata, 2005;Treinta et al., 2014) and conceptual contents (Seuring & Müller, 2008); and d) through a methodical and structured research design, defining criteria and keywords and criterion (filtering and eligibility criteria). This means that the bibliometric analysis adopts a mixed methodology to allow assessment of the qualitative and quantitative coverage of a given area of interest (Geaney et al., 2015), as is presented here.
The adoption of the bibliometric methods referred to above is related to the appropriateness of the provision of this systematic review presented here with internal and external validity and, thus, excluding subjectivity. In other words, for the purpose of this study it is crucial to use a mixed approach, by using different qualitative and quantitative methodological procedures, since the qualitative and quantitative methods are complementary to each other and not mutually exclusive (Minayo & Sanches, 1993;Patton, 1991Patton, , 2002. On the other hand, the quantitative method has a strong external validity and a weak external validity, while the qualitative method has a strong internal validity and a little external validity (Serapioni, 2000). In these circumstances, the methods chosen for the elaboration of bibliometrics allow the weak validities to be overcome. This means that SMARTER (quantitative approach), Vantage Point (quantitative approach) and Prisma (qualitative approach-eligibility). In short, this research uses a mixed methodology, given that the eligibility of the documents was obtained by the qualitative PRISMA method and the bibliometry was carried out by the quantitative SMARTER and VANTAGE POINT methods. Additionally, a descriptive and content analysis is presented, for its replicability and valid inferences (Bryman, 2012;Krippendorff, 2012), to complement the results obtained by bibliometrics, endowing this investigation with internal and external validity. This means that the use of different methods of bibliometric analysis allows a holistic combination for the exploration of research trends in a given topic in the literature (Chang et al., 2015). In addition, Pasadeos et al. (1998) advocated that bibliometric methods have become dominant and influential in defining future research on new topics.
Carrying out the bibliometrics involved: (1) the SMARTER method; (2) the PRISMA method; and (3) VANTAGE POINT software. Table 1 summarizes each of these methods.
Therefore, the use of the SMARTER method allowed, objectively, the reduction in the size of the initial database, through multi-criteria attributes (Choulak et al., 2019;Evidence-based et al., 2010;Garza-Reyes, 2015;Rutten-Van Mölken et al., 2018), while the PRISMA method improved the database returned by SMARTER, by applying eligibility criteria. Finally, the use of VANTAGE POINT, developed by Porter and Cunningham (2004), allowed the elaboration of a text-mining, to find patterns and relationships on the basis of final documents selected in the WoS (n = 70) (see Fig. 1) and if approaching who, what, when, and where. This software has been used in previous research in mapping scientific analyses (e.g. Morel et al., 2009;Porter & Youtie, 2009;Shapira et al., 2010).

Data collection and search flowchart
Data were collected online from the WoS database. WoS is widely recognized as a database focused on peer-reviewed articles (Geissdoerfer et al., 2017), mostly used for bibliometric studies due to presenting more standardized and consistent data (Chen et al., 2017a, b). Table 2 shows the criteria used in the search of WoS.
According to the items in Table 2, Fig. 1 presents the sequential procedures of triangulation of the three methods referred to in Table 1.
The data collection process required the definition of the two search strings (search 1 and 2), supported by the previous reading of the literature on eco-cities, without any serial filter, returned 1414 scientific documents, the number of which was reduced to 203 with the application of the serial filters defined in Table 1. On the other hand, the research was carried out on the WoS and the documents taken from the Science Citation Index (SCI), Science Citation Mostly used in studies in the field of medicine (Dolan, 2010), but recently used in the field of management (Marques & Franco, 2020); Supported by the multi-attribute utility theory (MAUT), as an analytical instrument linked to the area of decision analysis (Keeney & Raiffa, 1976); Use of oscillating weights (SMARTs) and exploratory channels, by using distinct weighting methods, aiming to improve and reduce the load and time of the decision-maker (Edwards & Barron, 1994); Use of the multi-criteria method for the decision-making of the American School (MCDM), which postulates a function of multi-attribute utility to represent decision-makers' preferences (Barfod et al., 2016); questions being the comparability of alternatives, transitivity in the relation of preference and indifference; formation of hierarchies and establishing a measure of global merit (Keeney & Raiffa, 1993); This method is applied at the stage of selecting scientific documents (Garza-Reyes, 2015), varying according to the criteria adopted (Choulak et al., 2019;Dolan, 2010; Rutten-Van Mölken et al., 2018) 2. PRISMA method A flow diagram (Donato & Donato, 2019); Allows definition of eligibility criteria (Adiyarta et al., 2020); These criteria guide data selection (Liberati et al., 2009) 3. VANTAGE POINT software Text-mining tool for discovering knowledge in search results from patent and literature databases (Pradhan, 2016, page 29); Can produce various networks, co-occurrence analyses, word clouds, descriptive analyses (Pradhan, 2016) "It allows the user to analyse large volumes of structured text to discover patterns and relationships and quickly address who, what, when, and where" (Cobo et al., 2011(Cobo et al., , p. 1389 Index Expanded (SCI-Expanded), Social Science Citation Index (SSCI) and Social Scientific Citation Index (AHCI) compiled by Thomson / Reuters, in ISI's online database, one of the largest repositories and frequently used in this type of research, which includes numerous scientific publications and all bibliographic information about their authors, citations, journals, among others. Based on the 203 documents obtained, it became crucial to proceed to a scientific cut, using the SMARTER method, which led to the definition of decision criteria shown in Fig. 2. Thus, this method reduced the final basis to be analysed in this study, for 133 articles. However, since this is a quantitative methodology, the application of the PRISMA method was crucial to analyse the 133 articles by means of two eligibility criteria (Fig. 4),  Search 1-("eco cit*" or "eco-cit*" or "ecological cit*" or "green cit*" and "sustainability" or "triple bottom") Search 2-("eco cit*" or "eco-cit*")

Document selection
The SMARTER method was applied to the total of 203 documents selected. This adopts different criteria to assess different alternatives for different lines of arguments (Marques & Franco, 2020) and in this way present a common result (Garza-Reyes, 2015). Figure 2 shows the criteria used for systematization of the 133 articles (see Fig. 1). With this method, it is essential to define the relative importance of the criteria and respective weights, i.e. exploration of the ordering of criteria (Marques & Franco, 2020). According to Olson and Dorai (1992), the centroid method attributes different weights, where W1 is the weight of the most important objective, W2 the weight of the second most important objective and W k for the most important K objective. Consequently, for the K objectives, we have Eqs. 1, 2 and K, the importance of the respective criteria, with the total sum of weights equal to 1 (Olson & Dorai, 1992): Edwards and Barron (1994) presented the stages to be followed rigorously in this method, as summarized in Table 3. Table 4 presents the potential alternatives (A n ) for the 203 documents (S4). The steps defined in S4 led to S5, obtained by creating a spreadsheet with the following Eqs. 1-5 applied to all D203s, as follows:  Table 5 shows the weights of the ROC curve used to analyse assessment of the criteria (S6), as defined by Roberts and Goodwin (2002).
Finally, Table 6 contains the utilities for each attribute (W )).
From the results obtained in the nine stages and through Fig. 3, the cut year can be determined, as graphic representation of the ROC curve shows the cut axis. This means that the ROC curve has the great advantage of allowing a relation to be established between sensitivity and specificity and representing the whole universe of possible events (Azevedo & Pereira, 2010).
In Fig. 3, the cut is the penultimate point on the ROC curve, i.e. a value of 96.13 for sensitivity and 64.52 for specificity, meaning a cut in the ROC weights from 64.52. In this specific case, the cut of the 133 documents is according to time, which defined a cut of approximately 10 years from the date of publication (Cerón-Palma et al., 2012).
After determining the cut year (2012) PRISMA was applied (Fig. 4) to obtain the final sample of documents for the bibliometric analysis using VANTAGE POINT.

Qualitative (descriptive) and quantitative analysis
The typology of the final 70 documents selected presents the distribution shown in Fig. 5.

Table 3 Stages (S)
An Potencial alternatives, Cm Criterion, Wm Roc weights S1 -Objectives and decision-making: ordering of the alternatives for selection of the state of the art by the decision-maker S2 -Hierarchy of the attributes defined to value the alternatives S3 -Alternative articles to be used to replicate the model proposed, through a search of WoS with the keywords and filters indicated in Table 2 (Table 4) S4 -Articles assessed by the type of attribute (Table 4), with n = 203 potential alternatives (A n ) according to m = 3 criteria (C m ) to be applied to S3 S5 -Analysis of the values obtained for the criteria defined in S4 S6 -Assessment according to C m : C1-Citescore and C2 -Number of citations are functions of maximization, where the higher the value, the greater the utility; C3 -Time, assesses performance in qualitative terms. The weights used are those of the ROC curve (Centroid Order) (Roberts & Goodwin, 2002) (   Figure 5 clearly shows the prevalence of scientific articles on other typologies of documents and shows the areas of research they deal with, highlighting that most studies are related to searching on the term of "Urban Studies"(e.g. Caprotti et al., 2015;Flynn et al., 2016;Lee, 2015;Lin, 2018;Mehmood, 2016;Mneimneh et al., 2017;Tateishi, 2018;Wu, 2020;Zhuang, 2015). In other words, the fact there is growing concern about increased population density in cities, with environmental and social consequences, has aroused researchers' interest in urban areas, especially those that have already implemented models of eco-cities, as in the case of China. Successful eco cities include Reykjavik-Iceland, Zurich-Switzerland, Bristol-South West England, Portland-Oregon, Vancouver-Canada, Copenhagen-Denmark and others (Better World Soluctions.eu, 2021; . The countries with most publications are shown in Fig. 6. Once more, China stands out with 26 publications about eco-cities, since a great many studies included here are about this country and the first authors originate there.   (Saiu, 2017). This does not mean there are no successful cases-we already have many examples of successful implementation of ecocity principles around the world. (Mersal, 2017, p. 31).
Lexical analysis of the 70 documents also shows the terms occurring most frequently in the articles' contents (Fig. 8).
The word-cloud reveals the emergence of the words "eco-city, China, governance, transitions" in the documents selected on this topic.
The most cited of the total 70 articles are shown in Table 7.  The above table provides various interpretations. The first is that the most cited article (135) is about eco-cities in Africa, which intends to highlight the fantasy of implementing this type of city in that region, but without presenting any scientific bibliographic references (Watson, 2014). The second is that the JOURNAL OF URBAN TECHNOLOGY is the journal with most articles (5) in this list with a citescore of 2.25 (Quartile 1), while CITIES with a citescore of 3.853 only presents 2 articles (Anthopoulos, 2017;Caprotti, 2014), although the one by Anthopoulos, (2017) is in the second position in the Top 10 with 95 citations. Finally, the year 2014 includes four articles, with the most recent article being from 2017 (Anthopoulos, 2017).
No less important is the number of publications per year (Fig. 9), with 2017 having most publications, followed by 2014. It stands out that 2014 has five articles in the Top 10.
The prevalence of 2017 with most publications shows it is important to study how strategies are used in eco-city projects, to understand the transition of other cities to this typology, because, as explained by the European Union, it is urgent to achieve global, long-term sustainability (UE, 2021).
Bibliometric analyses generally show author clusters for the articles on their databases. However, this study presents the cross-correlations between authors and keywords, as shown in Fig. 10. There are two main clusters, where the size of the node reflects the volume of associated registers and the lines the measures of similarity.
Cluster 1 includes the words "sustainable city, eco-city" and Cluster 2 the words "eco-city, China, sustainable urban development and eco-cycle model". There is also a correlation between all the authors shown around the two largest nodes, for example, Caprotti (2014) in Cluster 1, whose keywords were used by the authors in the smaller nodes on the right; Caprotti (2014) analysed Tianjin Eco-City, China, as the starting point for discussion on eco-city projects in the light of their definition, social resilience  and urban population growth, aiming to raise new questions in future research on the topic. Li and Qiu (2015) also made a comparative study between Suzhou (China) and Kitakyushu (Japan), as examples of reference on eco-cities, to analyse the structural development of this type of city by the Chinese government and their quality. These authors concluded that the Chinese city showed a notable evolution in its green spaces and environmental infrastructure, despite persistent gaps in indicators to measure environmental quality. Miao and Lang (2015) considered two ambitious eco-city projects launched in China in the last 15 years-the "Dongtan eco-city" project in Shanghai (currently halted) and the Sino-Singaporean Tianjin Eco-City (in progress) in Tianjin, aiming to understand the status of each of them. The Dongtan project failed due to the  lack of more government support, financial funding and international knowledge inputs. Finally, Liu et al. (2014) argued that the complexity surrounding sustainability and future uncertainties suggests the construction of more robust, resilient, as well as sustainable, cities to address climate change and the shortage of resources. Standing out in Cluster 2 is Yigitcanlar (2015), who made a critical review of the smart city model and the application of intelligent urban technology, analysing particularly emerging practices of ubiquitous eco-cities as examples of smart city initiatives, concluding that city planning should consider the implications of those technologies in achieving sustainable environments in the future. Included in this cluster is the ROC cut-off article (Cerón-Palma et al., 2012) which examined the obstacles and opportunities related to implementing Rooftop Eco.Greenhouses in Mediterranean cities in Europe, these being ecological greenhouses that improve the sustainability of buildings in cities. The authors concluded that the sustainable urban model for the future should facilitate the exchange between cities, for example. With this being a topic of great interest to academics, town planners, politicians and other entities, it is concluded that the construction of an eco-city is undoubtedly an extremely complex process. In other words, the ideal eco-city would be built from scratch, which in most cases is not possible, given the case of China with its unfinished projects, or by implementing eco-spaces within existing cities. Therefore, presenting a theoretical framework for these cities means including questions of various typologies of existing cities and regrouping them according to the definition of the eco-city. In these circumstances, the model presented here (Fig. 11) is supported by the keywords of Clusters 1 and 2 and Fig. 11 Theoretical framework of the future of cities the respective articles at their origin. It is paramount that eco-cities should not neglect any urban actor, and specifically residents, who in the midst of the current COVID 19 pandemic look increasingly for a sustainable future, environmental, social and economic safety, and a high quality of life.

Conclusion and contributions
The topic studied here is complex and controversial, as it involves numerous economic, social and environmental questions. These questions are directed to the building of cities that can stimulate global sustainability. In this context, research related to eco-cities has not included any article mapping only the term eco-city, as studies have interlinked this with other topics, such as smart cities, technological infrastructure, maximizing energy efficiency and reducing greenhouse gas emissions among many other issues. Therefore, the bibliometric mapping carried out revealed that studies eligible according to various criteria defined were empirical, with comparative studies and focusing mainly on China. The predominance of this country is justified by the fact that in May 2003, the Chinese state established a plan to build ecological urban areas (Li & Qiu, 2015). However, the unfinished projects aroused even greater interest in understanding why.
The mapping and bibliometrics presented here suggest that cities have to be approached holistically in all their dimensions, as argued by Vakula et al. (2018). This means that the urban buzz of cities must involve all the ergonomics of their ecosystems to achieve the desired sustainability and reduce the ecological footprint, although this is a high ambition (Xie et al., 2020). It should be noted, the rapid urbanization that has been seen in the global urban area has triggered environmental and socio-economic challenges, which include climate change and pollution, among others, which can be understood as an opportunity for the transition to more ecological cities, supported by the models of success of eco cities (e.g. Madrid, Singapore, Zurich, Bristol and Vancouver) and, at the same time, economically sustainable, assuming cities as the driving vehicles for this potential lasting change.
The bibliometric methods used, particularly SMARTER, PRISMA and VANTAGE POINT, are original in management studies and studies on urban spaces, which is an original contribution of this research. This study showed the importance of continuing to study sustainable city models already developed by the scientific community and proceed with empirical studies on them, to conclude about their weaknesses and strengths and, thus, promote future improvements. Farther the importance of the transition to a model of a circular economy, as highlighted by the European Union, the theoretical framework proposed here aims to show that cities, of any typology, should promote their economic, social and environmental development without neglecting community life, history, local values and culture, proximity, mobility, good governance practices, information and communication technology (ICT), the network functioning of hard and soft infrastructure, and people, to be able to provide their residents with quality of life and well-being. Furthermore, the pillars and issues of cities should be directed towards a circular urban economy, in order to reduce the ecological footprint by diminishing the use of non-renewable resources, maximizing energy efficiency and reducing greenhouse gas emissions, among other measures. It is important to mention that these considerations to the presented framework do not aim to define the direction of the future development of cities, nor to recall the obligations of local authorities, but rather a useful guide for all economic and non-economic actors crucial for the sustainable development of ecosystems in cities in a near future.
Another contribution to theory lies in adopting a mixed methodological approach where the research techniques used-qualitative PRISMA method and the quantitative SMARTER and VANTAGE POINT methods-are complementary rather than mutually exclusive (Minayo & Sanches, 1993;Patton, 1991Patton, , 2002. Therefore, this approach overcame the weaknesses of the quantitative and qualitative methods, as this triangulation allowed the external and internal validity of the research presented here.
Concerning implications for practice, the articles reviewed show the importance of continuing with research to identify cases of success and failure, to provide lessons for future projects. In addition, sustainability and concern about climate change, among many other aspects, should always be present in those projects. This is even more urgent, given the current COVID 19 pandemic and all its tangible and intangible consequences, and the increased ecological footprint witnessed. In addition, the empirical contributions of the studies presented here provide practical implications for public policy and local authorities. These types of studies are essential for local governments and political decision-makers to be able to take corrective measures in the various axes of their urban economic growth strategies at the macro-, micro-and meso-levels to achieve the intelligent, inclusive and sustainable growth aimed for by the European Union.
Like any study, this one is not without limitations. The first concerns only using WoS, despite this being recognized internationally as the most widely used. The second has to do with the search terms used, as many others could have been employed. Finally, the bibliometric methods adopted do not preclude others.
These limitations suggest various indications for the future. The first would be to use the WoS and SCOPUS databases to replicate this study. The second would be to add search terms, such as infrastructure, technology, intelligence, greenhouse gases, waste, pollution and energy efficiency. As this is a fertile field of research, also suggested is study of all smart cities of reference, to determine if they can be framed in the ecocity concept, and research relating this concept to the transition to the circular economy model. It would also be interesting to study the eco-city model implemented in Korea. Finally, using other bibliometric software would provide numerous paths for the future, given the proliferation of this, as well as using SMARTER through SPSS software.
Finally, this study showed that the trend topics for future research are fertile, due to the wide-ranging nature of the eco-city concept. Continued study of Chinese eco-cities is certainly pertinent for better understanding of their strengths and weaknesses, and also as in that country it is the government's will to construct eco-cities, studying why in these cities with excellent systems to monitor epidemics (e.g. Singapore), COVID 19 did not occur to a lesser extent. Another topic would be to study how eco-cities worldwide reach a balance between the rural city and the urban city, as mentioned by Register (1987Register ( , 2006. The relation between eco-cities and sustainability is also seen to be a trend topic in need of more research, in order to understand how a sustainable city can be transformed into an eco-city, as it would be utopian to believe that eco-cities of the future could be built from scratch.