As data becomes the fuel driving technological and economic growth, a fundamental challenge is how to quantify the value of data in algorithmic predictions and decisions. For example, in healthcare and consumer markets, it has been suggested that individuals should be compensated for the data that they generate, but it is not clear what is an equitable valuation for individual data. In this work, we develop a principled framework to address data valuation in the context of supervised machine learning. Given a learning algorithm trained on @math data points to produce a predictor, we propose data Shapley as a metric to quantify the value of each training datum to the predictor performance. Data Shapley value uniquely satisfies several natural properties of equitable data valuation. We develop Monte Carlo and gradient-based methods to efficiently estimate data Shapley values in practical settings where complex learning algorithms, including neural networks, are trained on large datasets. In addition to being equitable, extensive experiments across biomedical, image and synthetic data demonstrate that data Shapley has several other benefits: 1) it is more powerful than the popular leave-one-out or leverage score in providing insight on what data is more valuable for a given learning task; 2) low Shapley value data effectively capture outliers and corruptions; 3) high Shapley value data inform what type of new data to acquire to improve the predictor.
Shapley value was proposed in a classic paper in game theory @cite_29 and has been widely influential in economics @cite_12 . It has been applied to analyze and model diverse problems including voting, resource allocation and bargaining @cite_34 @cite_13 . To the best of our knowledge, Shapley value has not been used to quantify data value in a machine learning context like ours. Shapley value has been recently proposed as a feature importance score for the purpose of interpreting black-box predictive models @cite_9 @cite_15 @cite_17 @cite_4 @cite_0 @cite_25 . Their goal is to quantify, for a given prediction, which features are the most influential for the model output. Our goal is very different in that we aim to quantify the value of individual data points (not features). There is also a literature in estimating Shapley value using Monte Carlo methods, network approximations, as well as analytically solving Shapley value in specialized settings @cite_8 @cite_22 @cite_20 @cite_31 @cite_3
@cite_31 @cite_4 @cite_22 @cite_8 @cite_9 @cite_29 @cite_20 @cite_3 @cite_0 @cite_15 @cite_34 @cite_13 @cite_25 @cite_12 @cite_17
2128449113 2436401953 2167073044 2163338525
Preface 1. Introduction to the Shapley value Alvin E. Roth Part I. Ancestral Papers: 2. A value for n-person games Lloyd S. Shapley 3. A method for evaluating the distribution of power in a committee system Lloyd S. Shapley and Martin Shubik Part II. Reformulation and Generalizations: 4. The expected utility of playing a game Alvin E. Roth 5. The Shapley-Shubik and Banzhaf power indices as probabilities Philip D. Straffin, Jr. 6. Weighted Shapley values Ehud Kalai and Dov Samet 7. Probabilistic values for games Robert James Weber 8. Combinatorial representations of the Shapley value based on average relative payoffs Uriel G. Rothblum 9. The potential of the Shapley value Sergiu Hart and Andreu Mas-Colell 10. Multilinear extensions of games Guillermo Owen Part III. Coalitions: 11. Coalitional value Mordecai Kurz 12. Endogenous formation of links between players and the coalitions: an application of the Shapley value Robert J. Aumann and Roger B. Myerson Part IV. Large Games: 13. Values of large finite games Myrna Holtz Wooders and William R. Zame 14. Payoffs in nonatomic economics: an axiomatic approach Pradeep Dubley and Abraham Neyman 15. Values of smooth nonatomic games: the method of multilinear approximation Dov Monderer and Abraham Neyman 16. Nondifferentiable TU markets: the value Jean-Francois Mertens Part V. Cost Allocation and Fair Division: 17. Individual contribution and just compensation H. P. Young 18. The Aumann-Shapley prices: a survey Yair Tauman Part VI. NTU Games: 19. Utility comparison and the theory of games Lloyd S. Shapley 20. Paths leading to the Nash set Michael Maschler, Guillermo Owen and Bezalel Peleg. The Shapley value is a key solution concept for coalitional games in general and voting games in particular. Its main advantage is that it provides a unique and fair solution, but its main drawback is the complexity of computing it (e.g., for voting games this complexity is #p-complete). However, given the importance of the Shapley value and voting games, a number of approximation methods have been developed to overcome this complexity. Among these, Owen's multi-linear extension method is the most time efficient, being linear in the number of players. Now, in addition to speed, the other key criterion for an approximation algorithm is its approximation error. On this dimension, the multi-linear extension method is less impressive. Against this background, this paper presents a new approximation algorithm, based on randomization, for computing the Shapley value of voting games. This method has time complexity linear in the number of players, but has an approximation error that is, on average, lower than Owen's. In addition to this comparative study, we empirically evaluate the error for our method and show how the different parameters of the voting game affect it. Specifically, we show the following effects. First, as the number of players in a voting game increases, the average percentage error decreases. Second, as the quota increases, the average percentage error decreases. Third, the error is different for players with different weights; players with weight closer to the mean weight have a lower error than those with weight further away. We then extend our approximation to the more general k-majority voting games and show that, for n players, the method has time complexity O(k^2n) and the upper bound on its approximation error is O(k^2 n). We present a new approach to representing coalitional games based on rules that describe the marginal contributions of the agents. This representation scheme captures characteristics of the interactions among the agents in a natural and concise manner. We also develop efficient algorithms for two of the most important solution concepts, the Shapley value and the core, under this representation. The Shapley value can be computed in time linear in the size of the input. The emptiness of the core can be determined in time exponential only in the treewidth of a graphical interpretation of our representation. For f a weighted voting scheme used by n voters to choose between two candidates, the nShapley-Shubik Indices (or Shapley values) of f provide a measure of how much control each voter can exert over the overall outcome of the vote. Shapley-Shubik indices were introduced by Lloyd Shapley and Martin Shubik in 1954 [SS54] and are widely studied in social choice theory as a measure of the "influence" of voters. The Inverse Shapley Value Problem is the problem of designing a weighted voting scheme which (approximately) achieves a desired input vector of values for the Shapley-Shubik indices. Despite much interest in this problem no provably correct and efficient algorithm was known prior to our work. We give the first efficient algorithm with provable performance guarantees for the Inverse Shapley Value Problem. For any constant e>0 our algorithm runs in fixed poly(n) time (the degree of the polynomial is independent of e) and has the following performance guarantee: given as input a vector of desired Shapley values, if any "reasonable" weighted voting scheme (roughly, one in which the threshold is not too skewed) approximately matches the desired vector of values to within some small error, then our algorithm explicitly outputs a weighted voting scheme that achieves this vector of Shapley values to within error e. If there is a "reasonable" voting scheme in which all voting weights are integers at most poly(n) that approximately achieves the desired Shapley values, then our algorithm runs in time poly(n) and outputs a weighted voting scheme that achieves the target vector of Shapley values to within error e=n−1 8.