Experience replay is an important technique for addressing sample-inefficiency in deep reinforcement learning (RL), but faces difficulty in learning from binary and sparse rewards due to disproportionately few successful experiences in the replay buffer. Hindsight experience replay (HER) was recently proposed to tackle this difficulty by manipulating unsuccessful transitions, but in doing so, HER introduces a significant bias in the replay buffer experiences and therefore achieves a suboptimal improvement in sample-efficiency. In this paper, we present an analysis on the source of bias in HER, and propose a simple and effective method to counter the bias, to most effectively harness the sample-efficiency provided by HER. Our method, motivated by counter-factual reasoning and called ARCHER, extends HER with a trade-off to make rewards calculated for hindsight experiences numerically greater than real rewards. We validate our algorithm on two continuous control environments from DeepMind Control Suite - Reacher and Finger, which simulate manipulation tasks with a robotic arm - in combination with various reward functions, task complexities and goal sampling strategies. Our experiments consistently demonstrate that countering bias using more aggressive hindsight rewards increases sample efficiency, thus establishing the greater benefit of ARCHER in RL applications with limited computing budget.
Hindsight bias was first documented by Fischhoff @cite_9 , referring to the inflation in people's predicted likelihood of the true outcome of an event, after the outcome is known. This phenomenon, also termed as creeping determinism", is one of the most pervasive cognitive biases and routinely affects judgments in multiple domains, including medical diagnoses @cite_28 , criminal justice @cite_21 and financial systems @cite_29 .
@cite_28 @cite_9 @cite_21 @cite_29
1991314332 2120179935 2752099845 2103158016 1524486083 1980054641 2095762034 2170651797 158727920 2130330641 1968293352 2886424491 2011348940 2093351294 2163142059 2536769020 2750585749 2145451908 2217850561 1982374456
Abstract Hindsight bias is said to exist when individuals overestimate the extent to which an outcome could have been anticipated prior to its occurrence. Ex post, litigants tend to blame auditors in hindsight for failing to foresee and anticipate subsequent financial problems of their audit clients. However, it is not known whether the blame attributed in hindsight is due to hindsight bias or other factors such as the generally hypothesized user-preparer expectation gap. A behavioral experiment was conducted with 65 federal and state general jurisdiction judges and 58 auditors from one of the Big-Six public accounting firms. Results indicate that judges provided significantly lower evaluations of auditors' performance than did auditors, reflective of an expectation gap. Of greater importance, our results reveal that evaluative judgments of auditors' performance are subject to hindsight bias. Additional analysis revealed that our results are consistent with a cognitive interpretation of hindsight bias. Those who consider the likelihood of an event after it has occurred exaggerate their likelihood of having been able to predict that event in advance. We attempted to eliminate this hindsight bias among 194 neuropsychologists. Foresight subjects read a case history and were asked to estimate the probability of three different diagnoses. Subjects in each of the three hindsight groups were told that one of the three diagnoses was correct and were asked to state what probability they would have assigned to each diagnosis if they were making the original diagnosis. Foresight-reasons and hindsight-reasons subjects performed the same task as their foresight and hindsight counterparts, except they had to list one reason why each of the possible diagnoses might be correct. The frequency of subjects succumbing to the hindsight bias was lower in the hindsight-reasons groups than in the hindsight groups not asked to list reasons, x!( 1, N = 140) = 4.12, p < .05. Daniel Kahneman, recipient of the Nobel Prize in Economic Sciences for his seminal work in psychology challenging the rational model of judgment and decision making, is one of the world's most important thinkers. His ideas have had a profound impact on many fields - including business, medicine, and politics - but until now, he has never brought together his many years of research in one book. In "Thinking, Fast and Slow", Kahneman takes us on a groundbreaking tour of the mind and explains the two systems that drive the way we think and make choices. One system is fast, intuitive, and emotional; the other is slower, more deliberative, and more logical. Kahneman exposes the extraordinary capabilities - and also the faults and biases - of fast thinking, and reveals the pervasive influence of intuitive impressions on our thoughts and behaviour. The importance of properly framing risks, the effects of cognitive biases on how we view others, the dangers of prediction, the right ways to develop skills, the pros and cons of fear and optimism, the difference between our experience and memory of events, the real components of happiness - each of these can be understood only by knowing how the two systems work together to shape our judgments and decisions. Drawing on a lifetime's experimental experience, Kahneman reveals where we can and cannot trust our intuitions and how we can tap into the benefits of slow thinking. He offers practical and enlightening insights into how choices are made in both our professional and our personal lives-and how we can use different techniques to guard against the mental glitches that often get us into trouble. "Thinking, Fast and Slow" will transform the way you take decisions and experience the world. Recently, 'determinization in hindsight' has enjoyed surprising success in on-line probabilistic planning. This technique evaluates the actions available in the current state by using non-probabilistic planning in deterministic approximations of the original domain. Although the approach has proven itself effective in many challenging domains, it is computationally very expensive. In this paper, we present three significant improvements to help mitigate this expense. First, we use a method for detecting potentially useful actions, allowing us to avoid estimating the values of unnecessary ones. Second, we exploit determinism in the domain by reusing relevant plans rather than computing new ones. Third, we improve action evaluation by increasing the chance that at least one deterministic plan reaches a goal. Taken together, these improvements allow determinization in hindsight to scale significantly better on large or mostly-deterministic problems. In this lecture I shall focus on situations involving repeated decisions with time inconsistent behavior. Although each choice may be close to maximizing and therefore result in only small losses, the cumulative effect of a series of repeated errors may be quite large. Thus, in my examples, decision makers are quite close to the intelligent, well-informed individuals usually assumed in economic analysis, but cumulatively they make seriously wrong decisions that do not occur in standard textbook economics. This lecture discusses and illustrates several "pathological" modes of individual and group behavior: procrastination in decision making, undue obedience to authority, membership of seemingly normal individuals in deviant cult groups, and escalation of commitment to courses of action that are clearly unwise. In each case, individuals choose a series of current actions without fully appreciating how those actions will affect future perceptions and behavior. The standard assumption of rational, forwardlooking, utility maximizing is violated. The nonindependence of errors in decision making in the series of decisions can be explained with the concept from cognitive psychology of undue salience or vividness. For example, present benefits and costs may have undue salience relative to future costs In this paper, we explore the rules that determine intuitive predictions and judgments of confidence and contrast these rules to the normative principles of statistical prediction. Two classes of prediction are discussed: category prediction and numerical prediction. In a categorical case, the prediction is given in nominal form, for example, the winner in an election, the diagnosis of a patient, or a person's future occupation. In a numerical case, the prediction is given in numerical form, for example, the future value of a particular stock or of a student's grade point average. In making predictions and judgments under uncertainty, people do not appear to follow the calculus of chance or the statistical theory of prediction. Instead, they rely on a limited number of heuristics which sometimes yield reasonable judgments and sometimes lead to severe and systematic errors (Kahneman & Tversky, 1972b, 3; Tversky & Kahneman, 1971, 2; 1973, 11). The present paper is concerned with the role of one of these heuristics – representativeness – in intuitive predictions. Given specific evidence (e.g., a personality sketch), the outcomes under consideration (e.g., occupations or levels of achievement) can be ordered by the degree to which they are representative of that evidence. The thesis of this paper is that people predict by representativeness, that is, they select or order outcomes by the degree to which the outcomes represent the essential features of the evidence. We study online learning where a decision maker interacts with Nature with the objective of maximizing her long-term average reward subject to some sample path average constraints. We define the reward-in-hindsight as the highest reward the decision maker could have achieved, while satisfying the constraints, had she known Nature's choices in advance. We show that in general the reward-in-hindsight is not attainable. The convex hull of the reward-in-hindsight function is, however, attainable. For the important case of a single constraint, the convex hull turns out to be the highest attainable function. Using a calibrated forecasting rule, we provide an explicit strategy that attains this convex hull. We also measure the performance of heuristic methods based on non-calibrated forecasters in experiments involving a CPU power management problem. In the television series The Wire, addicts Bubbles and Johnny regularly engage in bizarre and elaborate schemes to obtain drugs, ranging from robbing an ambulance to stealing drugs by lowering a fishing line from a rooftop. That these fictional crimes can be both so meticulously planned and yet focused on such narrow, shortsighted goals highlights a gap in our understanding of how the real brain deploys deliberative vs. automatic mechanisms to make decisions. On a standard account, people can deliberatively evaluate the consequences of candidate actions, which gives us our flexibility to dream up novel plans. Alternatively, the brain can crystallize repeatedly successful behaviors into habits: learned reflexes that free up resources by executing the behaviors automatically (although at the expense of inflexibility and, it is believed, underpinning pathological compulsions). As with most dichotomies, the problem with this view is that the world is not so black and white. Much as the drug-seeking behavior of addicts seems not to fit into either category, for healthy behaviors also, neither of these two sorts of decision making is very practical on its own. In PNAS, Cushman and Morris suggest a hybrid of these mechanisms, and show behavioral evidence that humans use it to plan actions (1). This paper describes three heuristics, or mental operations, that are employed in judgment under uncertainty. (i) An assessment of representativeness or similarity, which is usually performed when people are asked to judge the probability that an object or event A belongs to a class or process B. (ii) An assessment of the availability of instances or scenarios, which is often employed when people are asked to assess the frequency of a class or the plausibility of a particular development. (iii) An adjustment from a starting point, which is usually employed in numerical prediction when a relevant value is available. These heuristics are highly economical and usually effective, but they lead to systematic and predictable errors. A better understanding of these heuristics and of the biases to which they lead could improve judgments and decisions in situations of uncertainty. When a series of individuals with private information announce public predictions, initial conformity can create an 'information cascade' in which later predictions match the early announcements. This paper reports an experiment in which private signals are draws from an unobserved urn. Subjects make predictions in sequence and are paid if they correctly guess which of two urns was used for the draws. If initial decisions coincide, then it is rational for subsequent decisionmakers to follow the established pattern, regardless of their private information. Rational cascades formed in most periods in which such an imbalance occurred. Copyright 1997 by American Economic Association. Part 1 Reasoning and behaviour: interpretations of probability beliefs and behaviour inference and decision reasoning and rationality assessment strategies survey of related work. Part 2 Coherent previsions: possibility probability currency upper and lower previsions avoiding sure loss coherence basic properties of coherent previsions coherent probabilities linear previsions and additive probabilities examples of coherent previsions interpretations of prevision and probability objections to behavioural theories of probability. Part 3 Extensions, envelopes and decisions: natural extension extension from a field lower envelopes of linear previsions linear extension invariant linear previsions compactness and extreme points of M(P) desirability and preference equivalent models for beliefs decision making. Part 4 Assessment and elicitation: a general elicitation procedure finitely-generated models and simplex representations steps in assessment process classificatory probability comparative probability other types of assessment. Part 5 The importance of imprecision: uncertainty, indeterminacy and imprecision sources from imprecision information from Bernoulli trials prior-data conflict Bayesian noninformative prioirs indecision axioms of precision practical reasons for precision Bayesian sensitivity analysis second-order probabilities fuzzy sets maximum entropy the Dempster-Shafer theory of belief functions. Part 6 Conditional previsions: updated and contingent previsions separate coherence coherence with unconditional previsions the generalized Bayes rule coherence axioms examples of conditional previsions extension of conditional and marginal previsions conglomerability countable additivity conditioning on events of probability zero updating beliefs. Part 7 Coherent statistical models: general concepts of coherence sampling models coherence of sampling model and posterior previsions inferences from improper priors confidence intervals and relevant subsets proper prior previsions standard Bayesian inference inferences from imprecise priors joint prior previsions. Part 8 Statistical reasoning: a general theroy of natural extension extension to prior previsions extension to predictive previsions extension to posterior previsions posteriors for imprecise sampling models the likelihood principle. Part 9 Structural judgements: independent events independent experiments constructing joint previsions from independent marginals permutability exchangeability Robust Bernoulli models structural judgements. Appendices: verifying coherence N-coherence win and place betting on horses topological structure and L and P separating hyperplane theorems desirability upper and lower variances operational measuremnet procedures the World Cup football experiment regular extension W-coherence. Given a partial description like "she opened the hood of the car," humans can reason about the situation and anticipate what might come next ("then, she examined the engine"). In this paper, we introduce the task of grounded commonsense inference, unifying natural language inference and commonsense reasoning. We present SWAG, a new dataset with 113k multiple choice questions about a rich spectrum of grounded situations. To address the recurring challenges of the annotation artifacts and human biases found in many existing datasets, we propose Adversarial Filtering (AF), a novel procedure that constructs a de-biased dataset by iteratively training an ensemble of stylistic classifiers, and using them to filter the data. To account for the aggressive adversarial filtering, we use state-of-the-art language models to massively oversample a diverse set of potential counterfactuals. Empirical results demonstrate that while humans can solve the resulting inference problems with high accuracy (88 ), various competitive models struggle on our task. We provide comprehensive analysis that indicates significant opportunities for future research. Abstract In this article we propose an account of the early development of children's knowledge about the mind and report two studies designed to test a part of it. According to this “connections-representations” account, young children begin their discovery of the mental world by learning that they and other people have internal experiences or mental states that connect them cognitively to external objects and events-experiences such as seeing them, hearing them, and wanting them. Later, they realize that the same object can be seriously (other than in pretense) mentally represented in different, seemingly contradictory ways: for example, as A in appearance, B in reality, C according to their perceptual or conceptual perspective, and D according to another person's. The results of both studies confirmed the prediction that 3-year-olds would perform well on appearance-reality and perceptual perspective-taking tasks requiring only an understanding of cognitive connections, but perform poorly on tasks requiring an understanding of seemingly-contrary-to-fact mental representations. To illustrate, children of this age had little difficulty determining that they could hear but could not see a noise-making object located on the experimenter's side of a barrier, and that the experimenter could see it (connections-level tasks). In contrast, they were largely unable to say, for instance, that a toy bear held behind a large elephant mask and emitting a cat sound looked like an elephant, sounded like a cat, and really was a toy bear (representations-level tasks)—even though the experimenter had actually told them previously what it looked like, sounded like, and really was. The article concludes with speculations about the possible origins of connections and representations knowledge and observations about the significance of these acquisitions for the child's development. One of the main findings of cognitive sciences is that automatic processes of which we are unaware shape, to a significant extent, our perception of the environment. The phenomenon applies not only to the real world, but also to multimedia data we consume every day. Whenever we look at pictures, watch a video or listen to audio recordings, our conscious attention efforts focus on the observable content, but our cognition spontaneously perceives intentions, beliefs, values, attitudes and other constructs that, while being outside of our conscious awareness, still shape our reactions and behavior. So far, multimedia technologies have neglected such a phenomenon to a large extent. This paper argues that taking into account cognitive effects is possible and it can also improve multimedia approaches. As a supporting proof-of-concept, the paper shows not only that there are visual patterns correlated with the personality traits of 300 Flickr users to a statistically significant extent, but also that the personality traits (both self-assessed and attributed by others) of those users can be inferred from the images these latter post as "favourite". Abstract This article presents a case study in examining the bias of two particular formalisms: decision trees and linear threshold units. The immediate result is a new hybrid representation, called a ‘perceptron tree’, and an associated learning algorithm called the ‘percepton tree error correction procedure’. The longer term result is a model for exploring issues related to understanding representational bias and constructing other useful hybrid representations. Recent advances in Natural Language Processing and Machine Learning provide us with the tools to build predictive models that can be used to unveil patterns driving judicial decisions. This can be useful, for both lawyers and judges, as an assisting tool to rapidly identify cases and extract patterns which lead to certain decisions. This paper presents the first systematic study on predicting the outcome of cases tried by the European Court of Human Rights based solely on textual content. We formulate a binary classification task where the input of our classifiers is the textual content extracted from a case and the target output is the actual judgment as to whether there has been a violation of an article of the convention of human rights. Textual information is represented using contiguous word sequences, i.e., N-grams, and topics. Our models can predict the court’s decisions with a strong accuracy (79 on average). Our empirical analysis indicates that the formal facts of a case are the most important predictive factor. This is consistent with the theory of legal realism suggesting that judicial decision-making is significantly affected by the stimulus of the facts. We also observe that the topical content of a case is another important feature in this classification task and explore this relationship further by conducting a qualitative analysis. Machine learning is now being used to make crucial decisions about people's lives. For nearly all of these decisions there is a risk that individuals of a certain race, gender, sexual orientation, or any other subpopulation are unfairly discriminated against. Our recent method has demonstrated how to use techniques from counterfactual inference to make predictions fair across different subpopulations. This method requires that one provides the causal model that generated the data at hand. In general, validating all causal implications of the model is not possible without further assumptions. Hence, it is desirable to integrate competing causal models to provide counterfactually fair decisions, regardless of which causal "world" is the correct one. In this paper, we show how it is possible to make predictions that are approximately fair with respect to multiple possible causal models at once, thus mitigating the problem of exact causal specification. We frame the goal of learning a fair classifier as an optimization problem with fairness constraints entailed by competing causal explanations. We show how this optimization problem can be efficiently solved using gradient-based methods. We demonstrate the flexibility of our model on two real-world fair classification problems. We show that our model can seamlessly balance fairness in multiple worlds with prediction accuracy. Unbiased language is a requirement for reference sources like encyclopedias and scientific texts. Bias is, nonetheless, ubiquitous, making it crucial to understand its nature and linguistic realization and hence detect bias automatically. To this end we analyze real instances of human edits designed to remove bias from Wikipedia articles. The analysis uncovers two classes of bias: framing bias, such as praising or perspective-specific words, which we link to the literature on subjectivity; and epistemological bias, related to whether propositions that are presupposed or entailed in the text are uncontroversially accepted as true. We identify common linguistic cues for these classes, including factive verbs, implicatives, hedges, and subjective intensifiers. These insights help us develop features for a model to solve a new prediction task of practical importance: given a biased sentence, identify the bias-inducing word. Our linguistically-informed model performs almost as well as humans tested on the same task. We study the online decision problem in which the set of available actions varies over time, also called the sleeping experts problem. We consider the setting in which the performance comparison is made with respect to the best ordering of actions in hindsight. In this article, both the payoff function and the availability of actions are adversarial. [2010] gave a computationally efficient no-regret algorithm in the setting in which payoffs are stochastic. [2009] gave an efficient no-regret algorithm in the setting in which action availability is stochastic. However, the question of whether there exists a computationally efficient no-regret algorithm in the adversarial setting was posed as an open problem by [2010]. We show that such an algorithm would imply an algorithm for PAC learning DNF, a long-standing important open problem. We also consider the setting in which the number of available actions is restricted and study its relation to agnostic-learning monotone disjunctions over examples with bounded Hamming weight. Abstract We ask if certain dimensions of perceptual similarity are weighted more heavily than others in determining word extension. The specific dimensions examined were shape, size, and texture. In four experiments, subjects were asked either to extend a novel count noun to new instances or, in a nonword classification task, to put together objects that go together. The subjects were 2-year-olds, 3-year-olds, and adults. The results of all four experiments indicate that 2- and 3-year-olds and adults all weight shape more heavily than they do size or texture. This observed emphasis on shape, however, depends on the age of the subject and the task. First, there is a developmental trend. The shape bias increases in strength and generality from 2 to 3 years of age and more markedly from early childhood to adulthood. Second, in young children, the shape bias is much stronger in word extension than in nonword classification tasks. These results suggest that the development of the shape bias originates in language learning—it reflects a fact about language—and does not stem from general perceptual processes.