Drawing an inspiration from behavioral studies of human decision making, we propose here a general parametric framework for a reinforcement learning problem, which extends the standard Q-learning approach to incorporate a two-stream framework of reward processing with biases biologically associated with several neurological and psychiatric conditions, including Parkinson's and Alzheimer's diseases, attention-deficit hyperactivity disorder (ADHD), addiction, and chronic pain. For AI community, the development of agents that react differently to different types of rewards can enable us to understand a wide spectrum of multi-agent interactions in complex real-world socioeconomic systems. Empirically, the proposed model outperforms Q-Learning and Double Q-Learning in artificial scenarios with certain reward distributions and real-world human decision making gambling tasks. Moreover, from the behavioral modeling perspective, our parametric framework can be viewed as a first step towards a unifying computational model capturing reward processing abnormalities across multiple mental conditions and user preferences in long-term recommendation systems.
In the study by @cite_19 , a simple heuristic model is developed to simulate individuals’ choice behavior by varying the level of decision randomness and the importance given to gains and losses. The findings revealed that risky decision-making seems to be markedly disrupted in patients with chronic pain, probably due to the high cost that pain and negative mood impose on executive control functions. Patients’ behavioral performance in decision-making tasks, such as the Iowa Gambling Task (IGT), is characterized by selecting cards more frequently from disadvantageous than from advantageous decks, and by switching more often between competing responses, as compared with healthy controls.
@cite_19
2141859379 1524486083 2016708334 2181918151
Risky decision-making seems to be markedly disrupted in patients with chronic pain, probably due to the high cost that impose pain and negative mood on executive control functions. Patients’ behavioral performance on decision-making tasks such as the Iowa Gambling Task (IGT) is characterized by selecting cards more frequently from disadvantageous than from advantageous decks, and by switching often between competing responses in comparison with healthy controls. In the present study, we developed a simple heuristic model to simulate individuals’ choice behavior by varying the level of decision randomness and the importance given to gains and losses. The findings revealed that the model was able to differentiate the behavioral performance of patients with chronic pain and healthy controls at the group, as well as at the individual level. The best fit of the model in patients with chronic pain was yielded when decisions were not based on previous choices and when gains were considered more relevant than losses. By contrast, the best account of the available data in healthy controls was obtained when decisions were based on previous experiences and losses loomed larger than gains. In conclusion, our model seems to provide useful information to measure each individual participant extensively, and to deal with the data on a participant-by-participant basis. In this lecture I shall focus on situations involving repeated decisions with time inconsistent behavior. Although each choice may be close to maximizing and therefore result in only small losses, the cumulative effect of a series of repeated errors may be quite large. Thus, in my examples, decision makers are quite close to the intelligent, well-informed individuals usually assumed in economic analysis, but cumulatively they make seriously wrong decisions that do not occur in standard textbook economics. This lecture discusses and illustrates several "pathological" modes of individual and group behavior: procrastination in decision making, undue obedience to authority, membership of seemingly normal individuals in deviant cult groups, and escalation of commitment to courses of action that are clearly unwise. In each case, individuals choose a series of current actions without fully appreciating how those actions will affect future perceptions and behavior. The standard assumption of rational, forwardlooking, utility maximizing is violated. The nonindependence of errors in decision making in the series of decisions can be explained with the concept from cognitive psychology of undue salience or vividness. For example, present benefits and costs may have undue salience relative to future costs Research and clinical expertise indicates that impulsivity is an underlying feature of pathological gambling. This study examined the extent to which impulsive behavior, defined by the rate of discounting delayed monetary rewards, varies with pathological gambling severity, assessed by the South Oaks Gambling Screen (SOGS). Sixty-two pathological gamblers completed a delay discounting task, the SOGS, the Eysenck impulsivity scale, the Addiction Severity Index (ASI), and questions about gambling and substance use at intake to outpatient treatment for pathological gambling. In the delay discounting task, participants chose between a large delayed reward (US @math 1-$999) across a range of delays (6h to 25 years). The rate at which the delayed reward was discounted (k value) was derived for each participant and linear regression was used to identify the variables that predicted k values. Age, gender, years of education, substance abuse treatment history, and cigarette smoking history failed to significantly predict k values. Scores on the Eysenck impulsivity scale and the SOGS both accounted for a significant proportion of the variance in k values. The predictive value of the SOGS was 1.4 times that of the Eysenck scale. These results indicate that of the measures tested, gambling severity was the best single predictor of impulsive behavior in a delay discounting task in this sample of pathological gamblers. This thesis investigates risk-sensitive sequential decision-making problems in an uncertain environment. We rst introduce the axiomatic concept of valuation functions that generalize known concepts of risk measures in mathematical nance to cover most of the existing risk related models in various elds, in particular, behavioral economics and cognitive neuroscience. By applying this concept to Markov processes, we construct valuation maps and develop thereby a uni ed framework for incorporating risk into Markov decision processes on general spaces. Within the framework, we study mainly two types of in nite-horizon risk-sensitive criteria, discounted and average valuations, and solve the associated optimization problems by value iteration. For the discounted case, we propose a new discount scheme, which is di erent from the conventional form but consistent with existing literature, while for the average criterion, we state Lyapunov-type stability conditions that generalize known conditions for Markov chains to ensure the existence of solutions to the optimality equation and a geometric convergence rate for the value iteration. Applying a set of valuation functions, called utility-based shortfall, we derive a family of model-free risk-sensitive reinforcement learning algorithms for solving the optimization problems corresponding to risk-sensitive valuations. In addition, we nd that when appropriate utility functions are chosen, agents’ behaviors express key features of human behavior as predicted by prospect theory, for example, di erent risk preferences for gains and losses, as well as the shape of subjective probability curves. As a proof of principle for the applicability of the new algorithms, we apply them to two tasks, 1) to quantify human behavior in a sequential investment task and 2) to perform risk control in simulated algorithmic trading of stocks. In the rst task, the risk-sensitive variant provides a signi cantly better t to the behavioral data and it leads to an interpretation of the subject’s responses which is indeed consistent with prospect theory. The analysis of simultaneously measured fMRI signals show a signi cant correlation of the risk-sensitive temporal di erence error with BOLD signal change in the ventral striatum. In the second task, our algorithm outperforms the risk-neutral reinforcement learning algorithm by keeping the trading cost at a substantially low level at the spot when the 2010 Flash Crash happened, and signi cantly reducing the risk over the whole test period.