It has been proposed that the sensorimotor system uses a loss (cost) function to evaluate potential movements in the presence of random noise. Here we test this idea in the context of both error-based and reinforcement-based learning. In a reaching task, we laterally shifted a cursor relative to true hand position using a skewed probability distribution. This skewed probability distribution had its mean and mode separated, allowing us to dissociate the optimal predictions of an error-based loss function (corresponding to the mean of the lateral shifts) and a reinforcement-based loss function (corresponding to the mode). We then examined how the sensorimotor system uses error feedback and reinforcement feedback, in isolation and combination, when deciding where to aim the hand during a reach. We found that participants compensated differently to the same skewed lateral shift distribution depending on the form of feedback they received. When provided with error feedback, participants compensated based on the mean of the skewed noise. When provided with reinforcement feedback, participants compensated based on the mode. Participants receiving both error and reinforcement feedback continued to compensate based on the mean while repeatedly missing the target, despite receiving auditory, visual and monetary reinforcement feedback that rewarded hitting the target. Our work shows that reinforcement-based and error-based learning are separable and can occur independently. Further, when error and reinforcement feedback are in conflict, the sensorimotor system heavily weights error feedback over reinforcement feedback.
Error feedback and reinforcement feedback can each guide motor adaptation in a visuomotor rotation task [1]. It has been proposed that error feedback and reinforcement feedback engage different neural mechanisms [2]. Indeed, depending on the form of feedback used during adaptation, newly acquired motor commands differ in terms of generalizability [1] and retention [3]. For error-based learning, it is suggested that adaptation occurs by minimizing error to update an internal model [4]. For reinforcement-based learning, it is proposed that adaptation is model-free and occurs by sampling motor outputs to find a set that maximizes the probability of task success [5]. Importantly, both these forms of learning can occur in the presence of internally [6,7] and externally [8,9] generated random variability (noise). Loss functions are central to several current computational theories of sensorimotor control [10]. An error-based loss function describes the relationship between potential movements and the associated costs of error [9]. In other words, an error-based loss function describes how errors of different magnitudes are penalized. A reinforcement-based loss function describes the relationship between potential movements and the associated probabilities of reward (or punishment) [7]. The idea that the sensorimotor system uses a loss function to select a statistically optimal movement in the presence of noise has been examined in tasks involving error feedback [9,11], reinforcement feedback [1], and both error and reinforcement feedback together [1,3, 7,12]. In tasks involving error feedback, it has been proposed that the sensorimotor system may use an error-based loss function to select movements [9]. As an example, here we will describe how an absolute error loss function and a squared error loss function can be used to select aim location during a game of darts. In this context, error refers to the distance of an individual dart to the bullseye. Let us assume that, after several throws, when attempting to aim the darts at a particular location that there is some spread, or distribution, of darts on a board. If the chosen strategy is to minimize absolute error around the bulls-eye, one should adjust their aim until the sum of the absolute distances of the darts is at its lowest possible value. This strategy corresponds to selecting a single aim location that minimizes the cost (output) of an absolute error loss function (i. e. , ∑|errori|1). This absolute error loss function linearly weights individual error magnitudes and would result in the median of the dart distribution being directly over the bullseye. Similarly, if the chosen strategy is to minimize squared error around the bullseye, one should adjust their aim such that the sum of the squared distances of the darts is at its lowest possible value. This strategy corresponds to selecting a single aim location that minimizes the cost of a squared error loss function (i. e. , ∑|errori|2). The squared error loss function places a heavier emphasis on minimizing large errors relative to smaller errors, in a quadratic fashion, and would result in the mean of the dart distribution being directly over the bullseye. Using tasks that involve noisy error feedback, some researchers have reported that sensorimotor behavior is best represented with an error-based loss function where the exponent on the error term is between 1 (absolute error) and 2 (squared error) [9,13], while others report that an exponent of 2 best fits behavior [11,14,15]. Based on these works, it is possible that the error-based loss function that most aligns with behavior may to some extent be task dependent. The concept of loss functions also extends to sensorimotor tasks involving reinforcement feedback, where the goal is maximize task success [5]. The optimal movement that maximizes the probability of success can be found by minimizing the 0-1 loss function [9,16]. Again, we can describe this reinforcement-based loss function using a distribution of darts on a board. With the 0-1 loss function, every dart that hits the bulls-eye is assigned a value of 0 and each dart that misses the bulls-eye is assigned a value of 1. To minimize this loss function, one should adjust their aim such that the greatest number of darts hit the bulls-eye. This strategy maximizes the probability of success (by minimizing failure) and corresponds to placing the mode of the distribution of darts directly over the bullseye. This loss function can easily be extended to account for graded reinforcement feedback, where either the magnitude [7,17] or probability [18] of reinforcement feedback varies according to some function that is externally imposed by the experimenter. It has recently been shown that the sensorimotor system can maximize task success when using only binary reinforcement feedback. Shadmehr and colleagues used a visuomotor rotation task, where the true target was displaced from the displayed target by some small amount [1,18]. In line with a reinforcement-based loss function and without any error feedback, they found that participants were able to adapt where they aimed their hand using only reinforcement feedback that signaled whether or not the true target had been hit. Researchers have also explored how reinforcement feedback and error feedback are used when provided simultaneously [1,3, 7,12,17,19]. This has been done for a range of tasks and has yielded mixed results. In studies reported by Trommershäuser and colleagues [7; 20–23], participants performed rapid reaches with continuous visual feedback of their hand (visual error feedback) to a large rewarding target (positive reinforcement) with an overlapping punishment region (negative reinforcement). Participants learned to aim to a location that maximized reward. Conversely, others have provided evidence that continuous error feedback dominates over, or perhaps suppresses, reinforcement feedback during a visuomotor rotation task [3,12,24]. Izawa and Shadmehr (2011) suggested that with a decrease in error feedback quality, the sensorimotor system might increase its reliance on reinforcement feedback. However, a feature of these experiments is that they did not separate the predictions for where participants should aim their hand when receiving reinforcement feedback or error feedback. Such separation would provide a powerful way to investigate how the sensorimotor system weights the relative influence of error feedback and reinforcement feedback when they are provided in combination. Here, we designed two experimental reaching tasks that separate the predictions of error-based and reinforcement-based loss functions on where to aim the hand. In doing so, we promoted dissociation in behavior simply by manipulating the form of feedback provided to participants. Participants reached to visual targets without vision of their arm. Unbeknownst to participants, visual feedback of their hand (represented by a cursor) was laterally shifted from trial to trial by an amount drawn from a skewed probability distribution. Skewed lateral shift probability distributions allowed us to separate the predictions of error-based and reinforcement-based loss functions on where to aim the hand. For example, a squared error loss function would predict that we should aim our hand to a location that corresponds to the mean of the skewed lateral shift probability distribution. Conversely, a reinforcement loss function would predict that we should aim our hand to a location that corresponds to the mode. Critically, skewed distributions separate several statistics, such as the mean and mode, that align with the optimal predictions of error-based and reinforcement based loss functions. Thus, by laterally shifting feedback using skewed noise and observing where participants reached, we were able to directly test how the sensorimotor system weights the relative influence of reinforcement feedback and error feedback when deciding where to aim the hand. In Experiment 1 we tested how reinforcement feedback and error feedback influence where participants aimed their hand. We predicted that participants receiving only error feedback would minimize some form of error. We also predicted that participants receiving both error and reinforcement feedback would increasing rely on reinforcement feedback with a decrease in error feedback quality. Such a strategy predicts a different pattern of compensation for participants who received both forms of feedback when compared to those who only received error feedback. Surprisingly, we found that both error-only feedback and error plus reinforcement feedback resulted in participants minimizing approximately squared error. In Experiment 2 we used a modified task to verify that reinforcement feedback alone was capable of influencing where to aim the hand. Indeed, we found that participants who received only reinforcement feedback maximized the probability of hitting the target. However, we again found that participants minimized approximately squared error when both error and reinforcement feedback were present. Taken together, our results describe how the sensorimotor system uses error feedback and reinforcement feedback, in isolation and combination, when deciding where to aim the hand.
Participants performed 2000 reaching movements in a horizontal plane (Fig 1). They were instructed to “hit the target”. A cursor that represented the true hand position disappeared once the hand left the home position. On each trial, the unseen cursor was then laterally shifted by an amount drawn from a skewed probability distribution. Participants were randomly assigned to one of three groups (n = 10 per group). The ErrorSR group and ErrorSL group both received error feedback that was laterally shifted by a right-skewed (RS; Fig 2A) or left-skewed (SL) probability distribution, respectively. The third group, Reinforcement + ErrorSR, received error feedback and reinforcement feedback that were both laterally shifted by a SR probability distribution. Importantly, the skewed lateral shift probability distributions were designed to separate the mean and the mode, corresponding with the optimal solutions of error-based and reinforcement-based loss functions, respectively. This separation allowed us to investigate how the sensorimotor system weights the relative influence of reinforcement feedback and error feedback when deciding where to aim the hand. We hypothesized that with a decrease in error feedback quality the sensorimotor system would increase its reliance on reinforcement feedback. To manipulate error feedback quality, halfway through each reach the laterally shifted cursor was either not presented (ς∞) or briefly presented (for 100ms) as a single dot (ς0mm), a medium cloud of dots (ς15mm) or a large cloud of dots (ς30mm), before disappearing once again (Figs 1 and 2). The medium and large clouds were composed of 25 dots, such that the dots were distributed according to a bivariate normal distribution with a standard deviation of 15mm and 30mm, respectively. Participants then attempted to hit the target by accounting for the laterally shifted error feedback (ς0mm, ς15mm, ς30mm, ς∞) they had experienced mid-reach. All participants received additional error feedback (also a single dot) at the end of the reach on trials in which single dot (ς0mm) error feedback was presented mid-reach [8]. Participants in the Reinforcement + ErrorSR group were presented with reinforcement feedback only on trials in which the error feedback was presented as a single dot (ς0mm). The reinforcement feedback was binary (the target doubled in size, a pleasant sound was played over a loudspeaker, and participants received 2¢ CAD) and was presented when the laterally shifted cursor hit the target. On each trial, we estimated how a participant compensated for the lateral shift by recording their hand location relative to the displayed target (see Fig 1). This was done for the four different levels of error feedback quality and lateral shift magnitudes. The average compensatory behavior of each group is shown in Fig 3. It can be seen that, with very little visual uncertainty about the magnitude of a lateral shift (ς0mm), all groups had a pattern of compensation that was well matched to the true magnitude of the shift. As error feedback quality decreased from little uncertainty (ς0mm) to some uncertainty (ς15mm and ς30mm) to complete uncertainty (ς∞), participants’ pattern of compensation became increasingly less sensitive to the true magnitude of the lateral shift. This is consistent with Bayesian inference in that participants were increasing their reliance on their prior with a decrease in error feedback quality. Interestingly, the average compensation of each group, even for participants who received both reinforcement and error feedback, corresponded quite closely to the predictions made by a Bayesian model whose loss function minimized squared error (compare Figs 3A–3C to 2E). To quantify the extent to which reinforcement feedback had an influence on behavior, we performed three separate analyses. First, we compared the average compensatory behavior between groups. Second, we used a Bayesian model to characterize how error feedback is used to guide behavior, and the extent to which reinforcement feedback influenced compensation. Finally, we used a simple linear model to characterize how the relationship between compensation and lateral shift is modulated by error signal quality, and the extent to which reinforcement feedback influenced compensation. All three analyses supported the idea that reinforcement feedback did not influence behavior when it was presented in combination with error feedback. Below, we describe each group’s compensatory behavior and the results of the Bayesian model. For brevity, detailed results of the linear model are presented in S2 Data. In Experiment 1 we found that when both error feedback and reinforcement feedback were presented in combination, participant behavior seemed only driven by error feedback. There is a possibility that the reinforcement feedback we used was not capable of influencing behavior. To test this, in Experiment 2 some participants only received reinforcement feedback, without error feedback. It is important to note that these participants only received reinforcement feedback at the end of each movement. This represents a difference in experimental design from Experiment 1, whose participants (ErrorSR, ErrorSL, Reinforcement + ErrorSR) often received feedback twice in a single movement—mid-reach and as they passed by the target. To properly control for this, in Experiment 2 we tested two additional groups that received only error feedback, or error plus reinforcement feedback. Importantly, however, all participants in Experiment 2 only received feedback once per trial, at the end of movement. This ensured that the frequency and location of feedback received by the three groups was the same. As a consequence of not providing error feedback mid-reach and only providing feedback at the target, compensation to the skewed lateral shift probability distribution in Experiment 2 reflects a trial-by-trial updating of where to aim the hand. This differs from Experiment 1 in which compensation reflects both online (via mid-reach feedback) and trial-by-trial (via target feedback) updating of where to aim the hand. In the context of a Bayesian framework, this would indicate in Experiment 2 that the prior representation of lateral shifts is updated after a trial is completed, instead of both during and after a trial is complete as in Experiment 1. One group of participants only received reinforcement feedback (Reinforcement), a second group received only error feedback (Error), and a third group received both error and reinforcement feedback (Reinforcement + Error). In total, there were ninety participants (30 per group). All participants performed 500 reaching movements in a horizontal plane. They were instructed to “hit the target”. On every trial, a cursor that represented the true hand position disappeared once the hand left the home position. The unseen cursor was laterally shifted by an amount drawn from a skewed-right (SR; n = 15 per group; Fig 4A) or skewed-left (SL, n = 15 per group) probability distribution. Binary reinforcement feedback occurred when the laterally shifted cursor hit the target (the target doubled in size, a pleasant sound was played over a loudspeaker, and participants received 2 ¢ CAD). Error feedback was presented as a single dot at the end of the reach, at the location where the laterally shifted cursor passed by or through the target. For each reach we recorded each participant’s pattern of compensation, that is, how laterally displaced his or her hand was relative to the displayed target (Fig 1). We calculated the compensation location that would maximize the probability of hitting the target (x a i m m a x (h i t s); Eqs 7,8 and 12 and Fig 4C). This calculation incorporated a measure of movement variability at the target, which was larger in this experiment, relative to Experiment 1, given that there was no mid-reach error feedback (see Methods for further details). We also calculated the compensation location that would minimize squared error of cursor positions about the target (x a i m m i n (e r r o r 2); Eqs 7–11 and Fig 4B). Fig 5 shows the pattern of compensation of a participant from each group. In response to the skewed lateral shift probability distribution, it can be seen that the Reinforcement participant learned to compensate by an amount that was on average close to maximizing the probability of hitting the target. Conversely, both the Error and the Reinforcement + Error participants had an average compensation that corresponded to minimizing approximately squared error. Fig 6A shows the average group compensatory behavior in response to the skewed lateral shift probability distribution. Compensation reached an asymptotic level after approximately 100 reaches (bin 10). Thus, for each participant, we averaged their last 400 trials to obtain a stable estimate of their behavior (Fig 6B). However, the results reported below were robust to whether we averaged the last 100,200,300 or 400 trials (Table 1). To test whether the form of feedback and skew direction influenced behavior, we compared the average pattern of compensation between the three groups. We found that there was a significant main effect of group [F (2,84) = 8. 928, p < 0. 001, ω ^ G 2 = 0. 150]. There was no significant main effect of skew direction [F (1,84) = 0. 164, p = 0. 687, ω ^ G 2 < 0. 001] nor an interaction between group and skew direction [F (2,84) = 0. 498, p = 0. 61, ω ^ G 2 < 0. 001]. Thus, under the influence of the same skewed lateral shift probability distribution, we found that different forms of feedback resulted in significantly different compensatory behavior. Specifically, we found a statistically reliable difference between the Reinforcement and Error groups (p < 0. 001, one-tailed; θ ^ = 77. 7 %). The Reinforcement group approached a compensatory position that would maximize their probability of hitting the target [x a i m m a x (h i t s) ] (p = 0. 081, two-tailed; θ ^ = 63. 3 %; CI[7. 1,10. 8mm]). Further, the Reinforcement group’s pattern of compensation was significantly different from one that corresponded to the minimization of squared error [x a i m m i n (e r r o r 2) ] (p < 0. 001, one-tailed; θ ^ = 83. 3 %). These two findings suggest that the reinforcement feedback used in Experiments 1 and 2 is capable of influencing behavior in a way that aligns with a reinforcement-based loss function. One prediction in the context of reinforcement-based learning is that an individual’s movement variability should influence their pattern of compensation to the lateral shifts. We tested this for the Reinforcement participants in Experiment 2. We characterized movement variability as the standard deviation of final hand position during the asymptotic phase of learning (the last 400 trials). While at the group level, participants appeared to compensate by an amount that approached an optimal strategy (see above), on an individual basis we did not find a statistically reliable relationship between movement variability and compensation (R2 = 0. 003). This finding appears to differ from that reported by Trommershäuser et al. (2003a). However, as expanded on in the Discussion, there are many differences between their task and ours, and such findings are not uncommon [16,25]. Nevertheless, the group level data suggests the existence of a reinforcement-based loss function that maximizes the probability of hitting the target. As expected, the Error group minimized squared error (Fig 6B). Their pattern of compensation was aligned with one that, on average, minimized squared error (p = 0. 795, two-tailed; θ ^ = 50. 0 %). It was also significantly different from one that maximized their probability of hitting the target (p < 0. 001, one-tailed; θ ^ = 100. 0 %). The results of Experiment 2 support the idea that the human sensorimotor system can update where to aim the hand during a reach by using only reinforcement-based feedback (to maximize the probability of hitting the target) or only error-based feedback (to minimize approximately squared error). As in Experiment 1, we again found that participants in the Reinforcement + Error group minimized squared error, and that reinforcement feedback did not influence behavior. Participants in the Reinforcement + Error group exhibited a pattern of compensation that was significantly different from the Reinforcement group (p = 0. 004, two-tailed; θ ^ = 76. 7 %), but was indistinguishable from participants in the Error group (p = 0. 922, two-tailed; θ ^ = 52. 2 %). Taken together, the results from both Experiment 1 and 2 support the idea that the sensorimotor system heavily weights error feedback over reinforcement feedback when updating where to aim the hand during a reaching task.
A key aspect of this study was the use of skewed noise to separate the optimal aim locations predicted by reinforcement-based and error-based loss functions. This allowed us to probe how the sensorimotor system uses reinforcement feedback and error feedback, in isolation and combination, to update where to aim the hand during a reaching task. We found that participants minimized approximately squared error when they received only error feedback. Participants maximized the probability of hitting the target when they received only reinforcement feedback. When both forms of feedback were presented in combination, participants minimized approximately squared error at the expense of maximizing the probability of hitting the target. This finding suggests that the sensorimotor system heavily weights error feedback over reinforcement feedback when deciding to aim the hand. In both Experiments, we found that the sensorimotor system minimized approximately squared error when using error feedback to guide reaching movements. This agrees well with previous work that examined how humans adapt to a small range of asymmetrical or multimodal noise during proprioceptive [11,14] and visual [15] tasks. In Experiment 1 and 2, we also used small ranges of asymmetrical noise that respectively spanned 3cm and 2. 8cm of the workspace. There is some evidence that as noise exceeds this range, the sensorimotor system is less sensitive to large errors [9,26]. While in the present study our data points to an error-based loss function based on squared error, other studies have focused on other mathematical forms. Another commonly examined loss function is the inverted-Gaussian [9,13,16], which places greater emphasis on penalizing smaller errors and less emphasis on penalizing larger errors. Sensinger and colleagues (2015) used a biofeedback task that involved controlling a myoelectric signal corrupted with skewed noise similar to that used by Körding and Wolpert (2004b). They then examined several different loss functions and their corresponding best-fit parameters given the data. The parameters of a loss function define how errors of different sizes are weighted relative to one another. For the inverted-Gaussian loss function they found its best-fit parameter was much larger (9 times) than that found by Körding and Wolpert (2004b). They suggest that the inverted Gaussian loss function may not be generalizable across different motor tasks. They did, however, estimate a best-fit power loss function exponent of 1. 69, a value that was quite close to the figure of 1. 72 estimated by Körding and Wolpert (2004b). In the present study we estimated an average best-fit power loss function exponent closer to 2. 0, which was not significantly different from 1. 72. These differences may be due to the range of noise and possibility the shape of the skewed noise. Nevertheless, and similar to others [11,14,15], we were able to explain 80% to 89% of the variability in our data (see S1 Data) using a power loss function that minimized approximately squared error (i. e. , αopt ≈ 2. 0). In the current paper, we use a Bayesian framework to interpret and model how the sensorimotor system uses error feedback and reinforcement feedback when deciding where to aim the hand. This framework combines prior experience and current sensory information, such as sensory cues [27] and sensory uncertainty [8], in a statistically optimal fashion. By accounting for both prior and current information, the Bayesian framework has successfully explained a broad range of phenomena, such as reduced movement variability [8,28], perceptual illusions [29] and online feedback control [30]. An alternative computational framework for error-based learning has been instrumental to our understanding of how the sensorimotor system learns to adapt on a trial-by-trial basis [31–35]. Of these models, the ones that account for sensorimotor noise [32,33,35] have been termed, ‘aim point correction’ models [36]. van Beers (2009) extended upon this framework with the ‘planned aim point correction’ model. This model separates central movement planning noise and peripheral movement execution noise. This model was able to explain reach adaptation patterns in a naturalistic task while demonstrating that the sensorimotor rate of learning was optimal given the properties of planning and execution noise. It has been successfully applied to explain differences in novice and expert dart throwers [37], and can account for both learning in task-relevant dimensions and exploratory (random walk) behaviour in task-irrelevant dimensions [38]. Aim point correction models, which can be derived from a Bayesian framework, are attractive because they are computationally tractable and learning is modeled using terms and constructs from sensorimotor control, such as planning noise, motor commands, efference copies, and execution noise [36]. In their current formulation, however, these models do not incorporate how the sensorimotor system responds to errors of differing magnitudes and different amounts of sensory uncertainty. We accounted for both of these factors with our Bayesian model, which was essential for testing our hypotheses. To further study how the sensorimotor system adapts on a trial-by-trial basis in experiments such as the ones used in this paper, a useful future direction would be incorporating the effects of sensory uncertainty and how errors of differing magnitudes are penalized. While it is most common to study adaptation while providing only error feedback, researchers have also examined adaptation in the context of both reinforcement and error feedback [1,3, 7,12,20–23,39,40]. Using a visuomotor rotation task, Izawa and Shadmehr (2011) examined how the sensorimotor system uses both a reinforcement and error feedback when deciding where to aim the hand during a reach. They manipulated the quality of error feedback presented on each trial in the following three ways: first, by displaying the cursor both at mid-reach and also at the target (mid-reach and target error-feedback condition); second, by displaying the cursor only at the target (target error-feedback condition); and thirdly, by withholding visual feedback of the cursor completely (no error-feedback condition). In each of these conditions, participants received reinforcement feedback for hitting the target (the target expanded and a pleasant sound was played over a loudspeaker). They modeled participants’ aiming behavior using a modified Kalman filter, which increasingly relied on reinforcement feedback as the quality of error feedback decreased. However, in the Izawa and Shadmehr (2011) experiment, the predicted aim location for minimizing error and maximizing the probability of hitting the target overlapped, making it difficult to determine the extent to which participants’ adaptation was driven by error feedback versus reinforcement feedback. Izawa and Shadmehr (2011) found greater movement variability in trials that provided feedback only at the target when compared to trials that provided feedback both mid-reach and at the target. Their model attributed a greater proportion of adaptation due to reinforcement feedback on trials in which error feedback was provided at the target, compared to trials in which error feedback was provided both mid-reach and at the target. While this is certainly a possibility, an alternative explanation for these behavioral differences is that the participants receiving feedback both mid-reach and at the target, unlike those receiving feedback only at the target, were able to compensate for accumulated sensorimotor noise error they sensed mid-reach. That is, it is difficult to determine whether behavioral differences between conditions were a result of reinforcement feedback, differences in the amount of (or location of) feedback, or both. In the present study we designed experiments aimed at resolving both of these potential issues. First, we were able to separate the optimal aim location of error-based and reinforcement-based loss functions. Second, for each group in Experiment 2 we equated the amount of and location of feedback by providing it only at the target. We found no differences in compensation between participants who received error feedback and participants who received both error and reinforcement feedback. This finding aligns with the work of Vaswani et al. (2015), who similarly found that reinforcement feedback did not influence behaviour when it was combined with error feedback. This suggests that the sensorimotor system heavily weights error feedback, when available, over competing reinforcement feedback. Trommershäuser and colleagues (2003a) provided evidence that humans can use reinforcement feedback to adjust where they aim their hand during a reaching task. In their study, participants reached to a screen displaying a rewarding target area (positive reinforcement: monetary gain) and an overlapping penalty area (negative reinforcement: monetary loss). Thus, participants received both reinforcement feedback for hitting the target, and error feedback that indicated where they touched the screen relative to the target. The reward and penalty for hitting these respective areas were held constant for a given block of trials. In their task, internal sensorimotor noise created a smooth and differentiable reinforcement landscape within and surrounding the reward and penalty areas. They found participants accounted for their internal sensorimotor noise and aimed towards the peak of this reinforcement landscape to maximize the probability of hitting the target. In our task, we found that participants receiving only reinforcement feedback maximized the probability of hitting the target, but participants who received both reinforcement and error feedback minimized squared error. Here, we suggest two potential reasons why participants receiving both reinforcement and error feedback continued to minimize error in our task, and appeared not to be driven by reinforcement feedback. First, it may be related to how the reinforcement region was spatially defined [10]. Second, the finding may only be present during implicit learning [41]. In the Trommershäuser et al. (2003a) experiments, the reinforcement regions were static and clearly defined in the workplace by the bounds of the penalty and reward areas. Thus, participants may have used error feedback to guide their aim towards the location they explicitly learned would maximize the probability of hitting the target. In our task, a stationary target was displayed on the screen. However, due to the lateral shifts, this target did not always represent the true reinforcement region. Moreover, similar to other experiments that used a small range of external noise and in which participants were unaware of their average change in reaching aim direction [8], our task was most likely implicitly learned [41]. Thus, unlike Trommershäuser et al. (2003a), our participants may have had to implicitly build a representation of the probabilistically varying reinforcement regions. In the absence of any error feedback, we found that participants receiving only reinforcement feedback compensated by an amount that approached the optimal compensation that maximized the probability of hitting the target. In accordance with a loss function that maximizes the probability of hitting the target, this suggests that the sensorimotor system was able to learn and compensate for the probabilistically provided reinforcement feedback. Despite an average compensation across participants that did not differ from the optimal solution, we did not find a correlation between final compensation and movement variability on an individual basis. This differs from Trommershäuser et al. (2003a), but as mentioned above there are many differences in experiment design between their task and ours. However, optimal performance across participants with suboptimal performance on an individual basis has been previously reported when using complex distributions (i. e. , skewed or bimodal distributions) to perturb feedback [16,25]. From a probabilistic viewpoint, suboptimal performance on an individual basis may partially be explained by individual differences in exploratory movement variability [42], as well as inaccurate [10], approximate [15] or stochastic [16] representations of the prior, likelihood and posterior. To produce accurate goal-directed movements, our nervous system must account for uncertainty and nonlinearities present in our environment [8,9, 11], in the biomechanics of our body [43,44], and in our nervous system [6,45–47]. By using skewed probability distributions, we were able to separate the optimal aim locations predicted by reinforcement-based and error-based loss functions. The results of our experiments demonstrate that by changing the form of feedback provided to participants, reinforcement-based and error-based learning are dissociable and can occur independently. Further, we also found that when both a reinforcement and error feedback are available, the sensorimotor system relies heavily, perhaps exclusively, on error feedback when deciding where to aim the hand during targeted reaching. Our work consolidates and builds upon previous research to provide insight into how the sensorimotor system performs model-based and model-free learning.
30 individuals (20. 2 ± 2. 7 years) participated in Experiment 1 and 90 individuals (20. 6 ± 2. 4 years) participated in Experiment 2. All participants were healthy, right-handed and provided informed consent to procedures approved by Western University’s Ethics Board. In both experiments, participants performed right-handed reaching movements in a horizontal plane while grasping the handle of a robotic arm (InMotion2, Interactive Motion Technologies, Massachusetts, United States; Fig 1). A semi-silvered mirror occluded vision of the arm and projected images from an LCD screen onto a horizontal plane aligned with the shoulder. These images included visual targets and, in certain conditions, real-time visual feedback of either the true or laterally shifted hand position. The workspace was calibrated such that the centroid of unperturbed visual feedback matched the center of the robotic handle. An air-sled supported each participant’s right arm and provided minimal friction with the desk surface. An algorithm controlled the robot’s torque motors and compensated for the dynamical properties of the robotic arm. Hand position was recorded at 600Hz and stored for offline analysis. We performed all offline analyses with custom Python (2. 7. 9) scripts. SPSS (version 21. 0; IBM, Armond, NY) was used for analysis of variance (ANOVA) tests. We used Greenhouse-Geisser adjustments to correct for violations of sphericity. Effect sizes for each ANOVA were calculated using generalized omega squared (ω ^ G 2), where values of ω ^ G 2 equal to 0. 02,0. 13, and 0. 26 were considered small, medium and large effects [56,57]. Follow-up comparisons (one-sampled, two-sampled or paired) were computed using nonparametric bootstrap hypothesis tests (n = 1,000,000). These tests provide more reliable p-value estimates than traditional comparison tests (e. g. , t-tests). Briefly, they make no parameter assumptions (e. g. , normality) and are less sensitive to unequal sample sizes or unequal variances. One-tailed tests were used when we had a priori predictions based on our Bayesian model, such as when we predicted significantly different compensation between participants only receiving error feedback and those receiving only reinforcement feedback. For all other comparisons we used two-tailed tests. Holm-Bonferroni corrections were used to allow for multiple comparisons [58]. 95th percentile confidence intervals (CI) were calculated by bootstrapping. Effect sizes for follow-up comparisons were made using the common language effect size statistic (θ ^), where values of θ ^ equal to 56. 0%, 64. 0% and 71. 0% were respectively considered small, medium and large effects [59,60]. Significance was set to p < 0. 05.
Whether serving a tennis ball on a gusty day or walking over an unpredictable surface, the human nervous system has a remarkable ability to account for uncertainty when performing goal-directed actions. Here we address how different types of feedback, error and reinforcement, are used to guide such behavior during sensorimotor learning. Using a task that dissociates the optimal predictions of error-based and reinforcement-based learning, we show that the human sensorimotor system uses two distinct loss functions when deciding where to aim the hand during a reach—one that minimizes error and another that maximizes success. Interestingly, when both of these forms of feedback are available our nervous system heavily weights error feedback over reinforcement feedback.
learning
control theory
medicine and health sciences
engineering and technology
nervous system
social sciences
limbs (anatomy)
neuroscience
learning and memory
control engineering
probability distribution
mathematics
cognitive psychology
systems science
statistical distributions
musculoskeletal system
computer and information sciences
hands
probability theory
arms
psychology
anatomy
biology and life sciences
sensory systems
physical sciences
cognitive science
Dissociating error-based and reinforcement-based loss functions during sensorimotor learning