Experience replay is an important technique for addressing sample-inefficiency in deep reinforcement learning (RL), but faces difficulty in learning from binary and sparse rewards due to disproportionately few successful experiences in the replay buffer. Hindsight experience replay (HER) was recently proposed to tackle this difficulty by manipulating unsuccessful transitions, but in doing so, HER introduces a significant bias in the replay buffer experiences and therefore achieves a suboptimal improvement in sample-efficiency. In this paper, we present an analysis on the source of bias in HER, and propose a simple and effective method to counter the bias, to most effectively harness the sample-efficiency provided by HER. Our method, motivated by counter-factual reasoning and called ARCHER, extends HER with a trade-off to make rewards calculated for hindsight experiences numerically greater than real rewards. We validate our algorithm on two continuous control environments from DeepMind Control Suite - Reacher and Finger, which simulate manipulation tasks with a robotic arm - in combination with various reward functions, task complexities and goal sampling strategies. Our experiments consistently demonstrate that countering bias using more aggressive hindsight rewards increases sample efficiency, thus establishing the greater benefit of ARCHER in RL applications with limited computing budget.
Hindsight bias was first documented by Fischhoff @cite_9 , referring to the inflation in people's predicted likelihood of the true outcome of an event, after the outcome is known. This phenomenon, also termed as creeping determinism", is one of the most pervasive cognitive biases and routinely affects judgments in multiple domains, including medical diagnoses @cite_28 , criminal justice @cite_21 and financial systems @cite_29 .
@cite_28 @cite_9 @cite_21 @cite_29
1991314332 2120179935 2752099845 2103158016
Abstract Hindsight bias is said to exist when individuals overestimate the extent to which an outcome could have been anticipated prior to its occurrence. Ex post, litigants tend to blame auditors in hindsight for failing to foresee and anticipate subsequent financial problems of their audit clients. However, it is not known whether the blame attributed in hindsight is due to hindsight bias or other factors such as the generally hypothesized user-preparer expectation gap. A behavioral experiment was conducted with 65 federal and state general jurisdiction judges and 58 auditors from one of the Big-Six public accounting firms. Results indicate that judges provided significantly lower evaluations of auditors' performance than did auditors, reflective of an expectation gap. Of greater importance, our results reveal that evaluative judgments of auditors' performance are subject to hindsight bias. Additional analysis revealed that our results are consistent with a cognitive interpretation of hindsight bias. Those who consider the likelihood of an event after it has occurred exaggerate their likelihood of having been able to predict that event in advance. We attempted to eliminate this hindsight bias among 194 neuropsychologists. Foresight subjects read a case history and were asked to estimate the probability of three different diagnoses. Subjects in each of the three hindsight groups were told that one of the three diagnoses was correct and were asked to state what probability they would have assigned to each diagnosis if they were making the original diagnosis. Foresight-reasons and hindsight-reasons subjects performed the same task as their foresight and hindsight counterparts, except they had to list one reason why each of the possible diagnoses might be correct. The frequency of subjects succumbing to the hindsight bias was lower in the hindsight-reasons groups than in the hindsight groups not asked to list reasons, x!( 1, N = 140) = 4.12, p < .05. Daniel Kahneman, recipient of the Nobel Prize in Economic Sciences for his seminal work in psychology challenging the rational model of judgment and decision making, is one of the world's most important thinkers. His ideas have had a profound impact on many fields - including business, medicine, and politics - but until now, he has never brought together his many years of research in one book. In "Thinking, Fast and Slow", Kahneman takes us on a groundbreaking tour of the mind and explains the two systems that drive the way we think and make choices. One system is fast, intuitive, and emotional; the other is slower, more deliberative, and more logical. Kahneman exposes the extraordinary capabilities - and also the faults and biases - of fast thinking, and reveals the pervasive influence of intuitive impressions on our thoughts and behaviour. The importance of properly framing risks, the effects of cognitive biases on how we view others, the dangers of prediction, the right ways to develop skills, the pros and cons of fear and optimism, the difference between our experience and memory of events, the real components of happiness - each of these can be understood only by knowing how the two systems work together to shape our judgments and decisions. Drawing on a lifetime's experimental experience, Kahneman reveals where we can and cannot trust our intuitions and how we can tap into the benefits of slow thinking. He offers practical and enlightening insights into how choices are made in both our professional and our personal lives-and how we can use different techniques to guard against the mental glitches that often get us into trouble. "Thinking, Fast and Slow" will transform the way you take decisions and experience the world. Recently, 'determinization in hindsight' has enjoyed surprising success in on-line probabilistic planning. This technique evaluates the actions available in the current state by using non-probabilistic planning in deterministic approximations of the original domain. Although the approach has proven itself effective in many challenging domains, it is computationally very expensive. In this paper, we present three significant improvements to help mitigate this expense. First, we use a method for detecting potentially useful actions, allowing us to avoid estimating the values of unnecessary ones. Second, we exploit determinism in the domain by reusing relevant plans rather than computing new ones. Third, we improve action evaluation by increasing the chance that at least one deterministic plan reaches a goal. Taken together, these improvements allow determinization in hindsight to scale significantly better on large or mostly-deterministic problems.