We study the problem of fair classification within the versatile framework of [ITCS 2012], which assumes the existence of a metric that measures similarity between pairs of individuals. Unlike previous works on metric-based fairness, we do not assume that the entire metric is known to the learning algorithm. Instead, we study the setting where a learning algorithm can query this metric a bounded number of times to ascertain similarities between particular pairs of individuals. For example, the queries might be answered by a panel of specialists spanning social scientists, statisticians, demographers, and ethicists. We propose "metric multifairness," a new definition of fairness that is parameterized by a similarity metric @math on pairs of individuals and a collection @math of"comparison sets" over pairs of individuals. One way to view this collection is as the family of comparisons that can be expressed within some computational bound. With this interpretation, metric multifairnesss loosely guarantees that similar subpopulations are treated similarly, as long as these subpopulations can be identified within this bound. In particular, metric multifairness implies that a rich class of subpopulations are protected from a multitude of discriminatory behaviors. We provide a general-purpose framework for learning a metric multifair hypothesis that achieves near-optimal loss from a small number of random samples from the metric @math . We study the sample complexity and time complexity of learning a metric multifair hypothesis (providing rather tight upper and lower bounds) by connecting it to the task of learning the class @math . In particular, if the class @math admits an efficient agnostic learner, then we can learn such a metric multifair hypothesis efficiently.
 We consider the problem of online learning in the linear contextual bandits setting, but in which there are also strong individual fairness constraints governed by an unknown similarity metric. These constraints demand that we select similar actions or individuals with approximately equal probability (arXiv:1104.3913), which may be at odds with optimizing reward, thus modeling settings where profit and social policy are in tension. We assume we learn about an unknown Mahalanobis similarity metric from only weak feedback that identifies fairness violations, but does not quantify their extent. This is intended to represent the interventions of a regulator who "knows unfairness when he sees it" but nevertheless cannot enunciate a quantitative fairness metric over individuals. Our main result is an algorithm in the adversarial context setting that has a number of fairness violations that depends only logarithmically on @math , while obtaining an optimal @math regret bound to the best fair policy. We study fairness in classification, where individuals are classified, e.g., admitted to a university, and the goal is to prevent discrimination against individuals based on their membership in some group, while maintaining utility for the classifier (the university). The main conceptual contribution of this paper is a framework for fair classification comprising (1) a (hypothetical) task-specific metric for determining the degree to which individuals are similar with respect to the classification task at hand; (2) an algorithm for maximizing utility subject to the fairness constraint, that similar individuals are treated similarly. We also present an adaptation of our approach to achieve the complementary goal of "fair affirmative action," which guarantees statistical parity (i.e., the demographics of the set of individuals receiving any classification are the same as the demographics of the underlying population), while treating similar individuals as similarly as possible. Finally, we discuss the relationship of fairness to privacy: when fairness implies privacy, and how tools developed in the context of differential privacy may be applied to fairness.