In our daily lives timing of our actions plays an essential role when we navigate the complex everyday environment. It is an open question though how the representations of the temporal structure of the world influence our behavior. Here we propose a probabilistic model with an explicit representation of state durations which may provide novel insights in how the brain predicts upcoming changes. We illustrate several properties of the behavioral model using a standard reversal learning design and compare its task performance to standard reinforcement learning models. Furthermore, using experimental data, we demonstrate how the model can be applied to identify participants’ beliefs about the latent temporal task structure. We found that roughly one quarter of participants seem to have learned the latent temporal structure and used it to anticipate changes, whereas the remaining participants’ behavior did not show signs of anticipatory responses, suggesting a lack of precise temporal expectations. We expect that the introduced behavioral model will allow, in future studies, for a systematic investigation of how participants learn the underlying temporal structure of task environments and how these representations shape behavior.
Our ability to represent time and to generate complex actions and plans based on this representation are central to all aspects of our behavior. Knowing when to act, precisely, is clearly crucial for our survival [1,2]. Thus, it is not surprising that the question of how we perceive and represent time gains an increasing interest in neurosciences [3–5]. However, in comparison to our understanding of spatial cognition, the neural basis of time perception is still poorly understood [6]. Here, we address the question of how the brain, in principle, can use knowledge about a hidden temporal structure of a task when making decisions. Traditionally, the question how we learn the structure of the world and use these representations for decision making, has been investigated from the perspective of reinforcement learning [7,8]. The focus of these investigations has been on how learning is driven by prediction errors [9], defined as a mismatch between expected and observed outcomes of one’s actions. More recent studies on human behavior in dynamic environments [10–13] have demonstrated that the relative precision of one’s prior beliefs and current sensory information weights prediction error signals [14–21]. These findings indicate that humans update their beliefs about the structure of the world akin to a rational (Bayesian) agent [22,23]. This suggests that one can approximately describe human behavior using the methodology of probabilistic inference. The key advantage of the probabilistic inference framework over the standard reinforcement learning modelling approach is that one can embed the knowledge about the structure of the world and the uncertainty about that knowledge within a generative model that describes the known rules that shape the dynamics of the world. A potential limitation of previous approaches—both probabilistic and reinforcement learning based—is that they do not take into account the underlying temporal structure of the task. Recently a series of studies have demonstrated the relevance of learned temporal associations on attention, perception, and sensory integration [24–27]. Experimental paradigms employed in this studies utilized observable temporal structure within trial. Similarly, in [28,29] authors demonstrate that learning of the underlying temporal structure of the task is used by human participants (or animal subjects) to anticipate the moment in time at which rules are most likely to change. In contrast to the experimental design which use within trial temporal structure, here participants were exposed to a hidden temporal structure across trials. Motivated by these findings, we propose a way to extend current probabilistic models of behavior, aimed at describing decision making in dynamic environments, to incorporate an explicit representation of the underlying temporal structure in the form of prior beliefs about state durations. In particular we will focus on the case when the state durations are not directly observable, but inferred across multiple trials using observable changes in action outcomes. The two essential components of the proposed behavioral model are (i) the update of beliefs about states and duration derived using approximate inference under hidden semi-Markov models [30,31] and (ii) the update of beliefs about actions, that is the planning process, cast as an inference problem [32–34]. As a test bed for illustrating the key properties of the model and for demonstrating its applicability to experimental studies we adopted a probabilistic reversal learning task [35]. This task and its variants have been frequently used in human and animal studies to investigate key properties of flexible behavioral adaptation in dynamic environments [29,36–38]. In a typical setup participants first learn to associate a particular choice with a reward, followed by a sequence of reversals of reward contingencies. The interesting question is typically how fast participants adapt their choices to these new contingencies. In the work here, we address the question, whether participants actually use the underlying temporal task structure to predict the moment of the reversal so that behavior is adapted faster as compared to the case when the reversal is unexpected. Using the reversal learning task, we first demonstrate using simulations that we can link sub-optimal behavior in changing environments to a misrepresentation of the underlying temporal structure of the changes. Subsequently, when fitting the behavioral model to experimental data, we relate the measured behavior of each participant to model parameters that define the beliefs about the temporal structure of reversals. Strikingly, our results suggest a heterogeneous distribution of the task representation across participants where some participants correctly inferred the latent temporal structure of the task, whereas others did not. We discuss potential reasons for this finding and how the presented approach enables systematic investigations of how participants learn the underlying temporal structure of task environments and how these representations shape behavior.
All participants provided written informed consent and were paid on an hourly basis. The Medical Faculty of Leipzig University approved the study. A probabilistic reversal learning task is typically structured as follows. An agent is presented with two choices A and B where each choice is associated with a probability of receiving a reward or punishment. For example, initially choice A has high probability pH and choice B low probability pL of getting a reward. Importantly, after several trials the reward contingencies switch, such that choice B now corresponds to the high reward probability choice. Participants are not informed about the switch and they have to infer that a change occurred and adapt their behavior. Here we used a multiple reversals design in which reversals occur several times during the experiment. Furthermore, the reversals occur at predefined trial numbers and are independent of participant’s performance. This reversal schedule was successfully used in past studies in healthy as well as patient samples and is well suited to detect inter-individual differences in behavioral adaptation [39,40]. For the model-based analysis, we used a subset of behavioral data—consisting of 22 healthy participants—from a previously published fMRI study [39]. In the experimental task participants were deciding between two cards shown on a screen, each showing a different stimulus (a geometric shape, e. g. rectangle, triangle, etc.) as shown in Fig 1A. The reward probabilities associated with the two choice options were anti-correlated on all trials: whenever reward probability of choice A was high (pH = 0. 8) the reward probability of choice B was low (pL = 0. 2), and vice versa. Note that pH = 1 − pL on all trials. Reward contingencies change as follows: they were stable for the first 55 trials, afterwards they changed four times after 15 or 20 trials, and remained stable for the last 35 trials, see Fig 1B. In total, the experimental block consisted of 160 trials. The location of each stimulus on the screen (right or left side) was randomized over trials. After each choice the stimulus was highlighted and depicted for 1. 5 s minus the reaction time. The feedback in form of reward (won 10 Eurocents) or punishment (lost 10 Eurocents) was shown for 0. 5 s. If no response occurred during the decision window, the message “too slow” was presented, and no outcome was delivered. During the inter-trial interval, a fixation cross was presented for a variable duration (jittered and exponentially distributed; range 1–12. 5 s). All 22 participants underwent a training session during which they had the opportunity to learn the statistics of the rewards associated with high and low reward probability choices. The set of stimuli used in the training phase differed from the one used during the testing phase. The participants were instructed that they could either win or lose 10 cents on each trial, and that they will be paid the total amount of money they gained during the testing phase at the end of the experiment. Each participant performed 20 training trials without any reversal of reward contingencies. Before the start of the testing phase participants were told that reward probabilities might change over the course of the experiment. No other information about reversals or the correlation of choices and outcomes was provided. Thus, the participants had no explicitly instructed knowledge about the anti-correlated task structure before the experiment. To derive the probabilistic model of behavior we start by defining a generative model of the task that formalizes an agent’s beliefs about the structure and the dynamics of the task environment. The update rules are then obtained using (approximate) Bayesian inference. The assumption here is that participants learn to represent the latent task structure. This structure consists of hidden states which define two possible configurations of the environment: in one configuration stimulus A corresponds to a high reward choice, in the second, stimulus B corresponds to a high reward choice. Note that the notion of state used here differs from what is typically used in reinforcement learning models, in which states are cues that are associated with values over time. Here, the states are hidden and not directly observable. Furthermore, they capture a context, which defines how two stimuli (option cues) are related to actions and outcomes of those actions. Importantly, the task environment transits from one state to another in a probabilistic manner. Here we will consider two assumptions about the dynamics of state transition probabilities: (i) the state transition probability is constant and independent of the moment of the previous change, as is the case under hidden Markov models (HMM) [44]; (ii) the state transition probability is time dependent and linked to the moment of the last change, which we will represent using hidden semi-Markov models (HSMM) [31,45]. As the HMM corresponds to a special case of HSMM, in which state transition probabilities are constant, it is sufficient to define the behavioral model based on the HSMM assumption. In what follows we will define the components of the generative model (observation likelihood and state dynamics) and derive the corresponding update rules. To formalize the presence of sequential reversals, that is, transitions from one task configuration to another, we define the state transition probability as p (s t + 1 | s t) = { 1 - δ, ifs t = s t + 1 δ, ifs t ≠ s t + 1 (4) where δ denotes probability of transiting between distinct hidden states (e. g. from st−1 = ¬R to st = R. This representation of the state transition process corresponds to a standard HMM previously used in reversal learning tasks [29,38,46–48]. Note that under this formulation of the state transition matrix the agent implicitly assumes that the interval (the elapsed number of trials) between two reversals follows a geometric distribution. Hence, the probability that any between reversal interval is of length d is given as p (d) = (1 - δ) d - 1 δ, whered ∈ { 1,2, 3, … } (5) with an expected dwelling time in each task configuration (mean interval length) set to μ = 1 δ. To derive agents with arbitrary beliefs about between reversal intervals we will use a special type of hidden semi-Markov models, the so-called explicit duration hidden Markov models (ED-HMM) [49,50]. This will allow us to identify individual differences in the beliefs about the temporal task structure and to investigate what impact these beliefs might have on an agent’s performance. The ED-HMM embeds the generative model with the representation of state durations, that is, the state dwelling time dt (the time spent in each state, ¬R or R). In other words, an agent will be able to form expectations about the number of trials between consecutive reversals. Under the ED-HMM we define the state transition matrix as follows p (s t | s t - 1, d t - 1) = { I 2, if d t - 1 > 1, J 2 - I 2, if d t - 1 = 1, (6) where I2 denotes the 2 × 2 identity matrix and J2 denotes the 2 × 2 all-ones matrix. The above relations describe a simple deterministic process for which the current state st remains unchanged as long as dt−1 > 1 and switches to alternative state (e. g. if st−1 = A then st = B) with probability one when dt−1 = 1. The transitions between state durations follow a deterministic countdown (dt = dt−1 − 1) as long as dt−1 > 1; once dt−1 = 1 the subsequent value dt is sampled from the prior over state durations p0 (dt), that is, from the beliefs over between reversal interval. We can write this formally in the form of the transition probability as p (d t | d t - 1) = { δ d t, d t - 1 - 1, if d t - 1 > 1, p 0 (d t), if d t - 1 = 1. (7) In Fig 2 we illustrate conditional dependencies between states, durations, reward probabilities, and outcomes in the form of a a graphical model. Although, the semi-Markov formalisms allows for defining state dependent prior beliefs p0 (dt|st), here, to reduce model complexity, we have assumed that the priors are independent of the current state, thus we set p0 (dt|st) = p0 (dt) for any st ∈ {¬R, R}. In practice, prior beliefs about state durations p0 (dt) can have an arbitrary form, however for the purpose of inferring the participants’ representation of between reversal intervals we will use a parametric distribution, specifically the negative binomial NB distribution defined as p 0 (d t) = N B (d t; δ, r) = (d t + r - 2 d t - 1) (1 - δ) d t - 1 δ r, (8) where δ ∈ [0,1] and r > 0. The NB distribution has several convenient properties. First, in the case r = 1 we obtain the geometric distribution (see Eq (5) ), hence we recover the HMM model described above. Second, for r > 1 the NB distribution exhibits a nonzero mode which shifts towards the expected value μ = r + δ (1 - r) δ as a function of a decreasing variance σ = (1 - δ) r δ 2, which is illustrated in Fig 3. Hence, the parameters of the negative binomial distribution will allow us to quantify an agent’s specific belief in the regularity of reversals. The lower the variance σ, the more an agent believes that reversals occur at regular intervals. Importantly, the variance of prior beliefs p0 (d) has direct effects on the temporal expectation profile (that a reversal will occur at some future trial τ). In Fig 4 we show the dependence of the expected reversal probability δτ on the variance σ, and under fixed mean μ = 20 of prior beliefs about the between-reversals interval. The expected reversal probability δτ is defined as δ τ = p (s τ = R | s τ - 1 = ¬ R) = ∑ d, s p (s τ = R | s τ - 1 = ¬ R, s 1 = s, d 1 = d) p 0 (d) p (s), (9) where τ > 1, and p (s = ¬R) = 1. Hence, δτ corresponds to the transition probability (from non-reversal state ¬R into reversal state R) at some future trial τ starting from the non-reversal state s1 = ¬R. Note that for prior beliefs p0 (d) with large variance (σ = μ (μ − 1) ), we get a constant transition probability, which corresponds to the HMM formulation of the state transition probability. In contrast, for prior beliefs with low variance (σ = μ) one obtains a trial dependent transition probability with values alternating between the low and high probabilities in a periodic manner. This temporal dependence of the transition probability will affect the inference process. The agent will become insensitive to subsequent reversals occurring few trials after the initial reversal, and highly sensitive to reversals occurring twenty to thirty trials after the initial reversal. For model inversion we use a variational inference scheme [51–53] which allows us to derive the update rules for the inference process akin to the ones presented in Eq (2). After making a choice ct and observing outcome ot at trial t, the agent updates its beliefs over reward probabilities ρ →, states st, and durations dt following Bayes’ rule p ¯ (ρ →, s t, d t) ∝p (o t | ρ →, s t, c t) p ˜ (ρ →) p ˜ (d t | s t) p ˜ (s t) (10) where we used the following notation p ˜ (x) = p (x | o t - 1: 1, c t - 1: 1), and p ¯ (x) = p (x | o t: 1, c t: 1) —for x ∈ { ρ →, s t, d t }—to denote prior and posterior beliefs, respectively, at time step t. If we express prior beliefs as p ˜ (s t = ¬ R) = θ ˜ t, p ˜ (d t = d | s t) = p d s t, p ˜ (ρ →) = ∏ c ∈ {A, B} B (ρ c; a t - 1 c, b t - 1 c), (11) where B (x; a, b) denotes a beta distribution, we can define the approximate posterior in similar form p ¯ (ρ →, s t, d t) ≈ q (ρ →) q (d t | s t) q (s t), (12) where we use the following parametrization of the factors of the approximate posterior q (s t = ¬ R) = θ t, q (d t = d | s t) = p ¯ d s t, q (ρ →) = ∏ c ∈ {A, B} B (ρ c; a t c, b t c). (13) The prior expectation at the next trial t + 1 depends on the current posterior q (st, dt) via the sum product rule p ˜ (s t + 1, d t + 1) = ∑ d t, s t p (s t + 1, d t + 1 | s t, d t) q (s t, d t). (14) The update of the agent’s beliefs about reward contingencies for each choice ρ, current state st, and the number of trials until the next reversal dt is completely defined by the update rules of the sufficient statistics of the posterior, namely parameters θt, a t c t, and b t c t. We will obtain the update rules for these parameters using variational inference. The parameters of the approximate posterior can be found as minimizers of the variational free energy defined as F [ q ] = D K L (q | | p ¯) -lnp ¯ (o t) = ∑ s t, d t ∫ dρ → q (ρ →) q (s t, d t) lnq (ρ →) q (s t, d t) p (o t | ρ →, s t, c t) p ˜ (ρ →, s t, d t). (15) The posterior beliefs q that minimize the free energy F can be obtained using the following relations q (ρ→) ∝p˜ (ρ) e〈 lnp (ot|ρ→, st, ct) 〉q (st), q (st, dt) ∝p˜ (st, dt) e〈 lnp (ot|ρ→t, st, ct) 〉q (ρ→t). (16) To obtain update rules similar to the delta learning rules of the RW model we simplified the above iterative procedure needed to estimate the posterior parameters. To break the cyclic update we will assume that one can first update beliefs about q (st, dt) by setting 〈 lnp (ot|ρ→, st, ct) 〉q (ρ→) ≈lnp˜ (ot|st, ct) and then use the obtained posterior beliefs about states and durations to estimate the reward probabilities q (ρ →). Using this simplification we obtain the following update rules p¯dst=pdst, θt=θ˜tθ˜t+elnp (ot|st=R, ct) p (ot|st=¬R, ct) (1−θ˜t), (17) where θ ˜ t = (1 - θ t - 1) p d = 0 R + θ t - 1 (1 - p d = 0 ¬ R). Note that the conditional posterior q (dt|st) corresponds to the conditional prior p ˜ (d t | s t), as p ¯ d s t = p d s t, hence it remains constant during the update of beliefs. However, the prior expectations (at the next time step) about state duration will be linked to the inference process via the state-duration transition matrix (see Eqs (6) and (7) ). Subsequently, we update the beliefs about the choice reward contingencies as a t c t = a t - 1 c t + θ t o ¯ t, b t c t = b t - 1 c t + θ t (1 - o ¯ t), a t P ^ · c t = a t - 1 P · c t + (1 - θ t) o ¯ t, b t P ^ · c t = b t - 1 P · c t + (1 - θ t) (1 - o ¯ t), (18) where o ¯ t = o t + 1 2 (o ¯ t ∈ {0,1}). To demonstrate the relation of the above update rules to the ones of the DU-RW model, we transform the shape parameters (a t c t, and b t c t) of the posterior beta distribution into the mean μ t c t and the samples size ν t c t as ν t x = a t x + b t x, μ t x = a t x ν t x, (19) where x ∈ { c t, P ^ · c t }. Expressing the expected reward probability μt as a function of the expected value Vt, that is, as μ t x = V t x + 1 2, we get the following set of update equations V t c t = V t - 1 c t + α t c t (o t - V t - 1c t) V t P ^ · c t = V t - 1 P ^ · c t + κ t α t P ^ · c t (o t - V t P ^ · c t) (20) where α t x = θ t ν t x, and κ t = 1 - θ t θ t. Although in form similar to the DU-RW learning rules, a notable difference is that the fictive prediction error is of the same sign as the actual prediction error. The reason for this is that under the generative model the fictive learning signals are a product of the agent’s uncertainty about the current state of the world, that is, the current configuration of reward contingencies and not the uncertainty about the anti-correlation of reward contingencies on different choices. This alternative representation could be introduced into the generative model by introducing beliefs about a dependence between high and low reward probabilities. However, we will leave this extension for future work as it increases model complexity and does not contribute to our analysis of how temporal representations influence behavior. Here we will describe the response model, which defines the mapping from the agent’s beliefs to responses, based on the active inference framework [54]. We will demonstrate how a response likelihood that is often used in reinforcement learning models (in the form of a softmax transform) can be derived within this framework. We do this for didactic purposes to show how one can formally relate active inference to a well-known reinforcement learning account. The core concept of active inference is that agents generate behavior that is most likely to minimize the expected free energy, that is, that tends to maximise, at the same time, the extrinsic and the epistemic value of agents’ choices [55]. The expected free energy can be defined as G t c = E q (o t | c) [- U (o t) - D K L (q (x | o t, c) | | p ˜ (x) ) ], (21) where x = (ρ →, s t, d t), U (ot) denotes the utility of future outcomes (reward or punishment), and DKL denotes the Kullback-Leibler divergence between the posterior (conditioned on possible future outcome and action) and prior expectations at the time step t. Note that we have assumed that the behavior is characterized by planning only a single time step into the future. Importantly, the response model maintains the causal structure of the task, where at trial t an agent first makes a choice ct and only afterward observes an outcome ot. The first term on the right hand side of Eq (21) is typically denoted as the extrinsic value of an action (policy) whereas the second term is denoted as epistemic value or information gain. If we express the utility U as U (o t) = λ o t, where ot ∈ {−1,1}, and λ > 0, the extrinsic value becomes 〈 U 〉q=λV˜tc, V˜tc=θ˜tVt−1c+ (1−θ˜t) Vt−1P^·c. In practice, for sufficiently large λ the behavior will be fully driven by the extrinsic value, as each independent observation ot carries little information about the underlying hidden states (a sequence of observations is required to identify the precise past moment of a reversal). In other words, each individual choice leads to a small information gain. Thus, we will assume that the expected free energy can be approximated as G t c ≈ - λ V ˜ t c. The optimal action (choice) corresponds to the one that minimizes the expected free energy, thus c t = argmin c G t c = argmax c V ˜ t c. Still, for describing participants’ behavior we have to assume that the action selection process is corrupted by external sources of noise; e. g. mental processes irrelevant for the task at hand. Therefore, we will soften the requirement of minimizing the expected free energy using the softmax transform, which is typically used to define the response likelihood [56,57]. The choice probability then becomes p (c t = c | o t - 1: 1) = e - β V ˜ t c + ln p 0 (c) ∑ u e - β V ˜ t u +lnp 0 (u), (22) where β denotes the response precision and p0 (c) the response bias. These two parameters are the free parameters of the response model and allow us to capture participants’ deviation from optimal behavior when fitting models to the data. Finally, in the case of the DU-RW model, we will use the same form of the response likelihood as above, with the difference that the choice value V ˜ t c = V t c. When fitting models to behavior we have focused our analysis on either the DU-RW or the ED-HMM model, where we have used in both cases a hierarchical prior over free model parameters (see below for details). We will not consider the SU-RW and HMM models in model comparison as they represent special cases of the DU-RW and the ED-HMM model, respectively, which are obtained in the limit of κ → 0, in the case of the DU-RW model, and r → 1 in the case of the ED-HMM model. Hence, we can easily recover these special cases from the estimates of the posterior distributions of the free model parameters, which are summarized in Table 1. Note that the parameters ν 0 A and ν 0 B of the ED-HMM model (see Eq (19) ), which define the initial number of observations, have been fixed to values which reflect that participants underwent a 20 trials long training session. Hence, setting ν 0 A = ν 0 B = 10 reflects our assumption that participants selected both options in equal amounts during the training session. To estimate the posterior distribution over free model parameters we have used the above defined response model as observation likelihood of the behavioral model. Hence, the likelihood of behavior (the sequence of a participant’s responses) is defined as p (c T: 1 n | o T: 1 n, Λ, m) = ∏ t = 1 T p (c t n | o t - 1: 1 n, Λ, m), (23) where c T: 1 n denotes the sequence of responses of the nth participant (n ∈ {1, …, 22}), o T: 1 n denotes the sequence of observations (wins or losses) that the nth participant made, Λ denotes the set of free model parameters, and m ∈ {1,2} denotes the corresponding behavioral model. To define the prior distribution over model parameters we have used the so-called horseshoe prior as a weakly informative hierarchical prior. If we denote the ith model parameter (where i ∈ {1, …, 6}) of the nth participant as λ i n, the horseshoe prior is defined as p (λ i n, τ i) = C + (λ i n; 0, τ i) C + (τ i; 0,1) (24) where C+ (x; 0, γ) denotes a half-Cauchy distribution with scale parameter γ. Hence, the full hierarchical prior for the model m can be expressed as p (λ, τ | m) = ∏ n = 1 22 ∏ i = 1 6 C + (λ i n, 0, τ i) C + (τ i; 0,1). (25) Note that τi acts as a hyper-prior, which plays the role of regulating the prior scale, for the corresponding free parameter, over the whole group. As λ i n is a positive definite variable, we have used specific transforms to relate each λ i n parameter to the corresponding model parameter (see Table 1). The posterior probability over model parameters can be obtain from the Bayes rule as follows p (λ, τ | C, O, m) ∝ ∏ n = 1 22 p (c T: 1 n | o T: 1 n, λ, m) p (λ, τ | m), (26) where C = { c T: 1 1, …, c T: 1 22 }, and O = { o T: 1 1, …, o T: 1 22 }. However, the exact posterior distribution of the model parameters and their hyper-priors is analytically intractable, hence we have applied the variational mean-field approximation, in which one assumes that the posterior is fully factorized p (λ, τ | C, O, m) ≈ ∏ i ∏ n q (λ i n | m) q (τ i | m). (27) The full hierarchical model was implemented in the probabilistic programing library PyMC3 [58]. The PyMC3 library provides an interface to multiple state-of-the-art inference schemes. In our specific case, for estimating the approximate posterior distribution over model parameters, we have used the PyMC3 implementation of the automatic differentiation variational inference (ADVI) [59]. ADVI is a stochastic black-box variational inference scheme which returns an approximate estimate of the posterior distribution in a fully factorized form. To attribute participants’ behavior to a specific behavioral model we have assumed that models can be treated as random effects (variables), that is, the attribution of a model to participants’ behavior can differ across participants [60]. Furthremore, we have based Bayesian model comparison on the posterior predictive model evidence [61] instead of the full model evidence, that is marginal likelihood. The motivation for basing model comparison on the posterior predictive model evidence comes from the fact that the posterior predictive model evidence is less sensitive to a prior specification of free model parameters: the approximate estimate of the predictive evidence is based on the samples from the posterior distribution which is already constrained by the subset of the behavioral data. This procedure provides for a more robust model comparison and testing results as it resolves some issues arising when using weakly-informative or misspecified prior probabilities [61]. The posterior predictive model evidence is defined as a marginal expectation of the subset of behavioral responses CT: k condition on the behavioral responses up to the kth trial (Ck: 1) and the full set of outcomes presented to participants (O). Formally, we can define the posterior predictive model evidence as p (C T: k | C k: 1, O, m) = ∏ n p (c T: k n | C k: 1, O, m) p (c T: k n | C k: 1, O, m) = ∫ dλ ∏ t = k T p (c t n | o t - 1: 1 n, λ n, m) q (λ n | m) ≈ 1 N ∑ l = 1 N ∏ t = k T p (c t n | o t - 1: 1 n, λ l n, m) (28) where q (λ n | m) = ∏ i q (λ i n | m), and N = 104. Note that to estimate the posterior predictive model evidence we have first estimated the approximate posterior over free model parameters on a reduced data set consisting of the pre-reversal and reversal phases (k = 125). Then, we generated N samples from the posterior distribution to estimate the posterior predictive model evidence on the remaining part of behavioral data CT: k which covers only the post-reversal phase (see Fig 1). We then used the posterior predictive model evidence as the likelihood of the random effect model proposed in [60]. The goal here is to identify the posterior probability that each of the two behavioral models can generate the same sequence of behavioral response as participants. Here we have defined the random effect model as the following mixture model p (C T: k, π, γ | C k: 1, O ^) = p (C T: k | C k: 1, O ^, π) p (π | γ) p (γ), (29) where p (C T: k | C k: 1, O ^, π) = ∏ n = 1 22 ∑ m n = 1 2 p (c T: k n | C k: 1, O, m n) p (m n | π) p (m n | π) = ∏ l = 1 2 π l δ m n, l, p (π | γ) = 1 B (1 γ, 1 γ) ∏ l = 1 2 π l 1 γ - 1 p (γ) = C + (γ; 0,1) (30) Similar to the role of the hyperpriors on the parameters of the behavioral model, γ plays here the role of a regularization parameter that allows for data driven constraints of the random effects assumption. If the marginal posterior probability over γ shrinks towards zero, this implies that data supports a null hypothesis which states that all models are present in the population with the same frequency and any differences we observe are purely chance driven [60]. As above, the random effects model selection was implemented in PyMC3, where the estimation of the posterior was performed using the PyMC3 implementation of the ADVI procedure. A fully factorized approximate posterior p (π, γ | C, O) ≈ q (γ) q (π) (31) provides us with an estimate of the posterior model probability q (π). The posterior model probability can then be used to identify group level posterior estimate over possible model frequencies in the group of participants and the participant specific posterior model probability.
To illustrate how the agent’s performance depends on the agent’s prior beliefs over the between reversal interval p0 (d) we will start by comparing two special cases of the ED-HMM based behavioral model using synthetic data. In the first case, we will fix the variance σ of the prior beliefs p0 (d) to σ = μ (1 − μ), which we named irregular reversal interval (IRI) agent. In the second case, we will fix the variance to σ = μ, which we named regular reversal interval (RRI) agent. In addition, we will compare the performance of the two probabilistic behavioral models to the single (SU) and dual update (DU) Rescorla-Wagner (RW) models introduced in Eqs (1) and (2), respectively. To illustrate the behavioral difference of different computational models we have modified the experimental setup and introduced two conditions in which reversals occur either at regular or irregular intervals, as shown in Fig 5. These two conditions allow us to make clear distinction between different behavioral models with respect to their behavior in the presence or absence of regularities. In all simulated experiments we fixed the number of trials to T = 160 and the number of experimental blocks to n = 1000. Hence, each simulated agent repeated the experiment n times, where at the beginning of each experiment the free model parameters (besides mean and variance) of the ED-HMM based agents (IRI and RRI) were set to the following values a ˜ 0 A = 8; b ˜ 0 A = 2, a ˜ 0 B = 2; b ˜ 0 B = 8, V 0 A = 0; V 0 B = 0, p ˜ (d 1) = p 0 (d 1). (32) In other words, simulated agents have a loose initial knowledge of the underlying reward probabilities, but do not know the initial configuration of the task (whether the environment is initially in the reversal or the no-reversal state). Fig 6 shows the dependence of the agent’s performance on its prior beliefs about the between reversal interval in two different environments. We have defined the performance as the fraction of correct choices (the choice associated with the higher reward probability). The irregular environment corresponds to the interval duration drawn from the geometric distribution with mean μ = 20 and variance σ = μ (μ − 1), and the semi-regular environment corresponds to interval durations drawn from the negative binomial distribution with mean μ = 20 and variance σ = μ, as illustrated previously in Fig 5. The IRI agent exhibits stable performance levels independent on the environment or a belief misspecification (incorrect mean interval duration, or incorrect variability around the mean). In contrast, the performance of the RRI model strongly depends on the correctness of the prior beliefs. These results make the rather intuitive point that if one is unfamiliar with the temporal structure of the environment, assigning high uncertainty to the expected state duration (as is the case for the IRI agent) ensures reasonably high levels of performance. However, if the environment changes at regular intervals, it is worthwhile to build an accurate representation of the temporal structure, as the performance levels can drastically increase (in this example from 81% to 87% of median performance levels). Fig 7 shows the comparison of the performance distribution between four models; the probabilistic agents IRI and RRI, and the two reinforcement learning based agents with single update (SU-RW) and dual update (DU-RW) learning rules. The free model parameters (α and κ, see Table 1) of the SU-RW and DU-RW agents were fixed to those values that maximize the average performance levels in each environment. Importantly, when comparing the performance distribution of the DU-RW agent to the performance distribution of the IRI agent we find similar average performance per trial which is stable across environments. This finding suggests that in spite of subtle differences in learning rules (see Eqs (2) and (20) ) the two agents generate very similar behavior. To investigate the accuracy of the procedure for model comparison which we described in Model fitting and model comparison, we have simulated behavior of the ED-HMM based and the reinforcement learning based agents on the experimental task, and applied the same model inversion procedure which we used for the analysis of behavioral data below. In Fig 8A we show the average performance per trial for each agent type estimated over n = 1000 repetitions of the experimental task. Fig 8B shows the probability of assigning behavior of each type of agents to the correct model type. The confusion matrix was estimated over n = 100 simulated experimental blocks, where in half of the experimental blocks ED-HMM based agents (IRI and RRI) were used to generate behavioral responses and in the other half of the experimental blocks the reinforcement learning agents (SU-RW and DU-RW) were used to generate behavioral responses. The rather high mixing probability of the confusion matrix suggests that the model comparison will have difficulties distinguishing properly between the models. Nevertheless, adjusting the experimental design to contain larger number of trials, that is, more data points for estimating posterior and model evidence, is likely to make model comparison more accurate. Here we will present model comparisons and model fitting based on the behavioral data of 22 participants. As the SU-RW model was shown in previous studies to provide a worse account for behavior than the DU-RW model [40], we do not expect the SU-RW model to explain the behavioral data better than the other models. Moreover, as our simulations indicate that the HMM and DU-RW models provide for comparable response patterns (see Fig 7), we will use only the DU-RW model as a reference model, as this model was used in a previous study based on the same data [39]. To circumvent a potential sensitivity of the model evidence to the specification of the prior distribution for the free model parameters, we have based our analysis on the posterior predictive model evidence (see Model fitting and model comparison for details). As the posterior predictive model evidence is estimated from the posterior distribution, rather than the prior (see Eq (28) ), it is more robust to mis-specification of the prior, given a sufficient amount of data in the predictive sample. Hence, to estimate the posterior predictive evidence we have split the behavioral data for each participant into two sets. The first set, containing the initial 125 trials, that is the pre-reversal phase and the reversal phase of the experiment (see Fig 1), we used for estimating posterior distributions of free model parameters (see Table 1). Note that as this set contains all but the last reversal, it should provide sufficient information to constrain the posterior model parameters. The reversal phase of the experiment is the only period which participants can use to shape their beliefs about the time structure of reversals. The second set, containing the last 35 trials, that is, the post-reversal phase (see Fig 1), we used for Bayesian model comparison and model validation. Critically, as no reversals are present in the post-reversal phase, this phase is especially suited for model selection: If participants believe that a reversal will occur at specific moments during the post-reversal phase this belief should be reflected in their behavior; e. g. , they might change their choices in anticipation of a reversal. In Fig 9 we show the results at both the group level and the individual level, using Bayesian model comparison based on the posterior predictive model evidence. Although the group level results suggest substantial evidence in favor of the DU-RW model (see Fig 9A), we can still identify six participants for which we find higher model attribution (a participant specific posterior model probability) of the ED-HMM based model (see Fig 9B). The exact values of the predictive log model evidence (used for model comparison) and per subject model attribution are shown in S1 Table. We next asked the question whether there is some systematic behavioral difference between these six and the remaining eighteen participants. Thus, we compared the averaged responses of participants belonging to the ED-HMM group and the participants belonging to the DU-RW group. In Fig 10 we show the choice probability averaged for each of the two groups. Tellingly, the ED-HMM group (blue line) exhibits a sudden switch toward the alternative choice approximately 20 trials after the start of post-reversal phase. Note that no reversals were induced in this phase. This result suggests that participants belonging to the group for which the ED-HMM model has higher posterior evidence behaved as if they were expecting another reversal 20 trials after the last one. This indicates that they have used the reversal phase to infer the reversal frequency and regularity. If the ED-HMM model indeed captures this aspect of behavior better than the DU-RW model, we would expect to see a similar trend in the response probabilities estimated from the two behavioral models. For this we estimated the response probability for each participant under each of the two models and averaged responses according to group. In Fig 11 we show the between model comparison of the estimated response probabilities for the two groups of participants. Although the mean response of the DU-RW model also shows a trough towards the alternative choice (when conditioned on the participants in the ED-HMM group), we see a much wider excursion in the mean response probability obtained using the ED-HMM, explaining its higher predictive model evidence for this group of participants. Still, it is important to note that the presence of the trough in the mean response probability of both models suggest that the change of response probability towards the alternative choice was driven by the specific sequence of outcomes that participants observed during this time window of the post-reversal phase. In other words, it seems that the participants that were assigned to the ED-HMM group were sensitive to a short sequence of negative outcomes, as if they were expecting another reversal during the post-reversal phase. In Fig 12 we show the posterior estimates of the expected reversal probability δτ for all participants of the ED-HMM group. The expected reversal probability was estimated from State durations and corresponds to the measure illustrated in Fig 4. Note that for four out of the six participants we find the peak of the reversal probability to fall before τ = 20, that is, before the 20th trial within the post-reversal phase. For the other two we see a rather flat trajectory which suggests that these subjects behave close to the IRI agents. Although these results seem to be in contrast to our previous findings (that these six participants anticipated the change on the 20th trial of the post reversal phase), the expected reversal probability does not correspond to the expected response probability for a specific participant. The relation between these two quantities is non-linear and is also shaped by the sequence of outcomes.
The key component of the proposed behavioral model is its conceptualization as an explicit duration hidden Markov models ED-HMM [30], which involves an explicit representation of the between reversal intervals as the hidden structural variable. This representation results in an anticipation of specific moments of reversals. Such anticipation would be clearly advantageous for an agent, as it enables faster behavioral adaptation in cases when reversals actually do occur in a (semi) regular manner. The ED-HMM belong to more general group of hidden semi-Markov models which are often applied to the analysis of non-stationary time series [65–68]. In the context of decision making the semi-Markov formalism allows for temporal structuring of behavioral policies [69]. Importantly, semi-Markov dynamics was also applied to temporal difference learning to account for dopamine activity in cases when the timing between action and reward is varied between trials [70]. The proposed model builds upon recent approaches to model behavior in changing environments [11,71,72] and can be seen as a direct extension of hidden Markov models (HMM) which were applied to reversal learning tasks before [29,38,46–48]. In previous works, the HMM were used to identify the moment of reversal, changes in the beliefs about reversal probability, and the most likely moment in which agents reversed their behavior. This was crucial for understanding the effects of dopamine modulation on the underlying inference and consequently behavior. Although it is out of the scope of the present paper, it is possible to perform backward inference with hidden semi-Markov models (HSMM), hence identifying the most likely moments of reversal in the past. Furthermore, we will explore in the future possible learning rules for parameters of the prior beliefs p0 (d), similar to the work of [73]. Such extension would make the models also suitable for addressing questions related to changes in prior beliefs of state durations. In recent years, reinforcement learning models have found multiple applications in studies relying on a reversal learning task. For example, the classical Rescorla-Wagner model [74], the dual update extension of the Rescorla-Wagner model (as described in the present paper) [39,40], or models separating the prediction error signals on positive and negative prediction errors [75]. As we have shown here a reinforcement learning model generates behavior very similar to a probabilistic counterpart in relatively simple settings of the reversal learning task. Hence, we would expect that additional extensions of the considered dual update RW model could make the behavior of the reinforcement learning even more similar to the probabilistic model introduced here. Still, we can point out several advantages of probabilistic models of behavior over reinforcement learning models, in the context of decision making under uncertainty and in dynamic environments. The probabilistic modeling approach allows for a principle way of mapping complex knowledge about spatio-temporal task structure into a relatively simple set of learning rules (as demonstrated here). In turn this provides clear functional interpretation of various prediction error signals, and corresponding adaptive learning rates, which are typically difficult to derive or motivate within the context of classical reinforcement learning. Specifically, we would say that prescribing to a probabilistic modeling approach is crucial for understanding interaction between representation of temporal structure and decision making. The sense of time and time representation at a neuronal level has been the focus of numerous studies in the past [76–82]. Studies investigated how neuronal circuitry might implement a robust internal clock [83], and how such an internal clock can be utilized to estimate time elapsed between consequent events (e. g. , between two tones). Although not described here, behavior in such and similar tasks can be modelled using the formalism of the hidden semi-Markov models. Estimating elapsed time between events (here, reversals) can be obtained using a backward inference process which provides an estimate on the most likely moment of the previous reversal. Nevertheless, it is an open question if the neural circuitry that explains behavior related to interval timing, which involves sub-second time scales, can also be useful to understand behavior which requires a temporal reasoning over much longer time scale. This is of specific interest for understanding how the brain makes accurate predictions about the expected moment of reward, and how this information is used to construct timed prediction error signals [84,85]. The reversal learning task has been shown to be especially useful for behavioral phenotyping, e. g. in areas of executive control [86–90]; for quantifying individual proneness towards impulsive and compulsiveness [91]; and defining individual vulnerability towards addiction [92] and other psychiatric disorders [35,93,94]. The experimental paradigms associated with a reversal learning task can differ substantially from the design presented here (see Fig 1). For example, the probability of reward and punishment for different choices could be correlated to some degree [47,95], or the reversal schedule can be made dependent on the number of correct choices [96–98]. It is relatively straightforward to adapt the proposed model to any of these variants. For example, when the reversal schedule depends on the number of previously correct choices, one can make the duration transitions dependent on the belief that the previous choice was correct. This would relate state durations to the beliefs about the number of correct choices since the last reversal. Thus, we believe that in a combination with the proposed model, the reversal learning task opens a wide range of opportunities for linking anomalous beliefs in the time domain to different cognitive disorders. Distortions in the temporal organization of cognition and behavior have been implicated, for a long time, in a wide range of psychiatric conditions, perhaps most prominently in attention-deficit hyperactivity disorder, autism and schizophrenia [99]. Hence, the proposed model might help to integrate findings of patients’ aberrant behavior in simple time estimation tasks (e. g. , a deficit to reproduce durations [100–102] with their known deficits in the decision-making domain [103], such as, in variants of the reversal learning task. Over the recent years, the question of how people apply their knowledge about the underlying structure of the environment in their decision-making has been widely investigated and discussed. Studies have come to the rather unspecific notion that a reduction in model-based control based on using environmental structure is ubiquitous across different psychiatric symptoms and diagnoses [104,105]. We believe that the model at hand could be a starting point for developing a set of tasks particularly appropriate for refining this notion. For example, Bayesian theoretical accounts have conceptualized addictive behavior as decision-making based on an aberrant model of the world [106]. Both animal and human studies suggest that addicted patients might have a specific deficit to infer or simulate statistical regularities in the environment [40,107,108]. Such deficits suggest limitations in their ability to infer regularities, which consequently is observed as suboptimal decision-making. Reversal learning deficits and impairments in model-based control have also been shown in patients suffering from obsessive compulsive disorder (OCD) [109,110]. A subgroup of OCD patients suffers from an obsession for symmetry and organization. Similarly, obsessive compulsive personality disorder (OCPD) patients show an exaggerated focus on order and symmetry as well as rule-bound traits. Given this clinical picture, it is interesting to speculate that beliefs about regularities in the environment might differ between OCD/OCPD patients and controls. A reduced ability to infer and represent irregular changes in the environment might lead to suboptimal decision-making and distress in OCD/OCPD patients in the presence of irregular changes. Finally, recent studies have demonstrated that schizophrenia patients may be characterized by an “over-dominance” of an internal model [111] (e. g. , in the framework presented here, overrating regularities in an irregular environment). When such an internal model is detached from the external input from the world (i. e. , the true regularity structure), this mismatch might be involved in the development of psychotic states (in paranoia, delusions, hallucinations). In summary, we have presented a novel probabilistic model for understanding human decision making behavior in changing environments. The proposed model is based on the explicit duration hidden Markov model, which forms a special case of more general hidden semi-Markov models. The presented results, obtained from both simulated behavior and the model-based analysis of behavioral data, suggest concrete applications of the proposed model in understanding human behavior in changing environments. Most notably, it might be possible to link the quality of the underlying representation of temporal task structure to behavior, thereby opening new directions for cognitive phenotyping.
Although time perception and timed behavior are essential for our everyday experience, it is still unclear how the human brain represents the underlying temporal regularities of our dynamic environment. These regularities and their representations in the brain are important to generate well-timed behavior. When deciding on the sequence of actions to complete most of our everyday tasks like cooking, driving, or even brushing our teeth, it is essential to represent and keep track of the durations of different parts of the tasks. Here we introduce a behavioral model of decision making in environments in which a change is at least partially predictable by the time it took since the last change. We show that human participants are using such predictions in the so-called reversal learning task, which simulates abrupt but not immediately obvious changes of the environment. We find that some but not all participants harness previously experienced regularities in these changes to anticipate when the next change is going to happen. We expect that a wide range of similar questions of how humans and other animals use temporal expectations to make their decisions in a dynamic environment can be addressed using the new modelling approach.
learning
markov models
decision making
statistics
experimental design
social sciences
neuroscience
learning and memory
research design
cognitive psychology
mathematics
forecasting
cognition
thermodynamics
research and analysis methods
behavior
hidden markov models
mathematical and statistical techniques
human learning
probability theory
free energy
physics
psychology
biology and life sciences
physical sciences
cognitive science
statistical methods
Predicting change: Approximate inference under explicit representation of temporal structure in changing environments