article: Much progress has been made in brain-machine interfaces (BMI) using decoders such as Kalman filters and finding their parameters with closed-loop decoder adaptation (CLDA). However, current decoders do not model the spikes directly, and hence may limit the processing time-scale of BMI control and adaptation. Moreover, while specialized CLDA techniques for intention estimation and assisted training exist, a unified and systematic CLDA framework that generalizes across different setups is lacking. Here we develop a novel closed-loop BMI training architecture that allows for processing, control, and adaptation using spike events, enables robust control and extends to various tasks. Moreover, we develop a unified control-theoretic CLDA framework within which intention estimation, assisted training, and adaptation are performed. The architecture incorporates an infinite-horizon optimal feedback-control (OFC) model of the brain’s behavior in closed-loop BMI control, and a point process model of spikes. The OFC model infers the user’s motor intention during CLDA—a process termed intention estimation. OFC is also used to design an autonomous and dynamic assisted training technique. The point process model allows for neural processing, control and decoder adaptation with every spike event and at a faster time-scale than current decoders; it also enables dynamic spike-event-based parameter adaptation unlike current CLDA methods that use batch-based adaptation on much slower adaptation time-scales. We conducted closed-loop experiments in a non-human primate over tens of days to dissociate the effects of these novel CLDA components. The OFC intention estimation improved BMI performance compared with current intention estimation techniques. OFC assisted training allowed the subject to consistently achieve proficient control. Spike-event-based adaptation resulted in faster and more consistent performance convergence compared with batch-based methods, and was robust to parameter initialization. Finally, the architecture extended control to tasks beyond those used for CLDA training. These results have significant implications towards the development of clinically-viable neuroprosthetics.
  Brain-machine interfaces (BMI) have the potential to enable motor function in individuals with neurological injury or disease [1–5]. BMI research has demonstrated that human and non-human primates can use their neural activity to directly control computer cursors or robotic arms (e. g. , [6–28]). BMIs record the neural activity from motor cortical areas, use a mathematical transform termed the “decoder” to convert this activity into control commands for an external device, and provide visual feedback of the generated movement to the subject (Fig 1A). Various decoders such as linear regression, population vector, and Kalman filters (KF) have been used in real-time BMIs [29]. Once a decoding model is selected, its parameters need to be estimated for each subject. These parameters are often found in open loop in a training session by solving for the parameter values that best describe the neural activity in response to executed or imagined movement. However, the relationship between neural activity and movements can be different for movements of a natural arm versus a BMI cursor, and the decoder needs to be adjusted to better handle the latter relationship[6,8, 15,30]. Hence methods that fit the decoder parameters in closed-loop BMI operation [6,11,17,19–21,23,31], referred to as closed-loop decoder adaptation (CLDA), can improve BMI performance [20,21]. Recent works have used KF decoders and combined them with a specific form of CLDA to achieve proficient BMI control [20,21]. Typical decoders such as the KF take as input the spike counts in bins, typically of 50–100 ms width, and assume a Gaussian distribution on these counts. Hence using a KF decoder, users can control the BMI once per bin (i. e. , once every 50–100 ms). The CLDA process, in turn, consists of three steps: (1) a method to initialize the decoder, (2) a method to estimate the user’s velocity intention during CLDA, which is referred to as intention estimation, and (3) an algorithm to fit the decoder parameters based on the estimated intentions and the recorded neural activity. To perform CLDA, decoder parameters are first initialized based on arm reaching movements [20], visual feedback of cursor movements (referred to as visual feedback seed; see Methods) [21], or even arbitrarily [21]. The initialized decoder is then used by the subject to make brain-controlled movements towards visual targets on the computer screen (Fig 1A and 1B). To perform intention estimation, current methods referred to as CursorGoal find the intended velocity at each time by rotating the cursor’s decoded velocity vector towards the target while keeping its magnitude (i. e. , speed) unchanged, and by equating it to zero when at the target [20]. Decoder parameters are then refit by collecting batches of neural activity on the time-scale of minutes and inferring the subject’s intended velocity in these batches. In each batch, a new set of parameters are fit using maximum-likelihood techniques[20,21,31]. Parameter estimates from previous batches are then either replaced with these new estimates [20] or are smoothly changed either continuously [31], or intermittently (e. g. , once every 90 sec) using a technique termed SmoothBatch [21], which converges to a good solution even when not initialized based on arm movements. Here we develop a novel closed-loop BMI training architecture termed adaptive optimal feedback-controlled (OFC) point process filter (PPF) (Fig 1C and 1D) to achieve two main goals; (1) to build a closed-loop BMI architecture that enables the subject to control the BMI at the time-scale of the spike events—defined as the presence or absence of a spike at a given time—, and that allows the decoder parameters to be adapted dynamically at this time-scale and with every spike event (see S1 Text for a summary of time-scales involved in CLDA-based BMIs); (2) to provide a unified and general control-theoretic CLDA framework within which intention estimation, assisted training, and adaptation can be performed, in contrast to current approaches. First, while CLDA-based decoders such as the KF have yielded good performance, these decoders constrain the time-scale at which the spikes can be processed and hence the time-scale at which the subject can control the BMI based on the bin-width used for spike count calculation (typically once every 50–100ms [14,19–21,28,32]; see also S1 Text). Moreover, current batch-based CLDA techniques typically update the decoder parameters at slow adaptation time-scales. However, time-scales of control and adaptation may play an important role in ultimate BMI performance. Therefore designing decoders that are not limited to a particular time-scale because of modeling assumptions (S1 Text) would provide a powerful tool to investigate the effect of the time-scales of control and adaptation. Here we develop a novel CLDA-based PPF that exhibits this property and use it to test the possibility that faster spike-event-based adaptation time-scales will improve BMI adaptation. The PPF can run at any time-scale by modeling the spikes as a series of random 0 and 1 events occurring in time [33–35]. Hence it can incorporate any time-scale down to the time-scale of the spikes (5ms here). Second, to perform CLDA, previous approaches have successfully used various ad-hoc methods for assisted training and intention estimation. However, a unified and systematic framework that can be used to perform all these manipulations does not exist. For example, some studies do not incorporate assisted training during CLDA [20,21] since the initial decoder is good enough to keep the subject engaged in the task. Instead, these studies rotate the decoded velocity vector to estimate intention. In other studies, assisted training has been used to help subjects reach targets with a poor initial decoder and to estimate intention (e. g. , [6,11,36]). These studies perform assisted training by either adding to the decoded velocity vector an assistive vector that directs straight towards the target, or subtracting from the decoded velocity vector a vector perpendicular to the straight line to the target, thus attenuating the deviation. These methods often continuously or manually change the ratio of the assistive vector to the decoded vector in the sum to adjust how much assistance is provided to the subject. Some studies have used a combination of these methods (e. g. , [36]). These ad-hoc approaches can make convergence trainer-dependent and significantly reduce robustness. Highly variable levels of assistance can also complicate the interpretation of performance during adaptation. Finally, the existing CursorGoal method of velocity intention estimation is based on an assumption of straight reaches towards the target [20] and does not infer a speed intention; instead it sets the intended speed equal to the decoded speed, which may not be accurate during CLDA as decoder parameters are still being estimated. Here we propose a general unified control-theoretic framework within which full intention estimation (e. g. , both direction and speed of velocity intention), assisted training, and adaptation can be performed across diverse BMI setups (see also Discussion). Motivated by the above, we develop the novel closed-loop adaptive OFC-PPF (Fig 1C and 1D) using two main components: a point process model of the spikes and an infinite-horizon optimal feedback control model of the brain’s control behavior in the BMI paradigm. In our previous open-loop work [26,37–39], we solved the problem of joint decoding of target and trajectory for the special case of straight target-directed movements and when decoder parameters were assumed to be known and found in open loop. Here we construct the first closed-loop PPF BMI training architecture, which fits the decoder parameters robustly and in closed loop, results in a decoder that can control various tasks and is not specific to straight target-directed movements, and allows for spike-event-based BMI control and adaptation. To achieve our first goal, adaptive OFC-PPF models the spikes as a discrete time-series of 0’s and 1’s by binning them in small intervals containing at most one spike. This process creates a time-series of spike events; the spike events indicate the presence or absence of a spike in each small interval and hence take values 1 or 0. This enables subjects to control the BMI with every 0 or 1 spike event (every 5ms here; Fig 1C); moreover, it allows the BMI to dynamically update the decoder parameters with every spike event in contrast to batch-based maximum-likelihood methods that do so on a much larger time-scale (e. g. , minutes). We use the architecture to investigate whether faster spike-event-based time-scales of adaptation improve decoder convergence. To achieve our second goal, adaptive OFC-PPF models the brain in closed-loop BMI control as an infinite-horizon optimal feedback-controller to infer velocity intention (both direction and speed) during adaptation; this is in contrast to the existing method of intention estimation (i. e. , CursorGoal), which is based on the assumption of straight reaches towards a target and does not infer speed intention [20]. The OFC method uses the target and error information to estimate the monkey’s intended velocity command in reaching the target; this estimated intended velocity is consequently used to train the decoder. In sum, the OFC model not only provides a general framework for intention estimation, but also allows for speed intention estimation that is not possible in CursorGoal. The OFC model is also used to devise a new assisted training paradigm, which is automated and dynamic and enables the computation of a chance level assisted performance. Thus the infinite-horizon OFC model results in a unified framework for these manipulations. This CLDA framework may also generalize to various BMI setups by simply quantifying the corresponding prosthetic’s dynamics and task goals within the OFC model. Finally, the proposed architecture is designed to be fully modular. We use the architecture to investigate the effect of OFC intention estimation and assisted training on BMI adaptation and performance. We evaluated the architecture in closed-loop BMI experiments in a rhesus monkey over 89 days. We ran multiple different experiments to dissociate the effect of the novel CLDA architecture components (i. e. , the effect of spike-event-based adaptation, OFC intention estimation, and OFC assisted training). We find that spike-event-based adaptation results in faster performance convergence compared to batch-based methods. Moreover, this parameter convergence is independent of initialization. Our data also shows that using the OFC intention estimation in place of the existing methods of intention estimation [20] results in a PPF decoder with higher performance, suggesting that the former better approximates the user’s strategy. Moreover, the new OFC-based assisted training technique results in consistent parameter convergence across sessions and allows for a systematic way to change the level of assistance. We also find that the architecture extends control to tasks beyond those used for CLDA training, in particular a target-jump task and a target-to-target task. Taken together, these data show that adaptive OFC-PPF provides a unified framework for CLDA, and results in robust spike-event-based BMI adaptation and control that extends to various tasks.
  All procedures were conducted in compliance with the National Institutes of Health Guide for the Care and Use of Laboratory Animals and were approved by the University of California, Berkeley Institutional Animal Care and Use Committee. Here we derive the adaptive OFC-PPF BMI architecture (Fig 1D). The architecture consists of two main components, a point process model of the spikes and an optimal feedback control model of the brain behavior in BMI control. In what follows, we first develop these two components and then show how to combine them to build the architecture, which consists of a point process decoder for the kinematics and decoders for each neuron’s parameters. For readers who are mainly interested in final expressions, the kinematics decoder is given in Eqs (14) – (17) and the parameter decoder for each neuron is given in Eqs (9) – (12).
  The OFC intention estimation model that we have proposed here can be adjusted by changing the dynamic state-space model in Eq (1) and the cost function in Eq (2). For any prosthetic device with a given dynamic state-space model, once the task goal is chosen, OFC can be used for intention estimation. This may thus provide a general method for intention estimation in different BMI setups. Here for 2D cursor movements, we tried two candidate classes of OFC models for intention estimation. First, we tried a naturalistic OFC model (N-OFC) in which a in Eq (19) was fit to the monkey’s own naturalistic cursor movement and in which the cost function was defined as in Eqs (2) and (18). Second, we tried an OFC model, termed instant-OFC, that assumed the monkey’s control command directly sets the intended velocity, i. e. , that a = 0 in Eq (19). Consequently, in the latter model we set ωv = 0 in Eq (18) since in this case ut is equivalent to the intended velocity and hence a non-zero wr is sufficient to both enforce the stopping condition and impose the control cost. Fig 4A and 4B show two sample decoded trajectories, the decoded velocities, and the inferred intended velocities by the N-OFC and the instant-OFC models. The N-OFC model’s inferred intended velocity direction and speed depend on the feedback of both the decoded position and the decoded velocities. The assumption in the N-OFC model is that the monkey intends to gradually change the velocity vector towards the target as opposed to immediately. This is enforced by the non-zero a that models the correlation between two adjacent velocity vectors in time. In contrast to N-OFC, the instant-OFC model assumes that the intended velocity need not be correlated with the current velocity (a = 0). Hence both its intended velocity direction and speed are mainly dependent on the feedback of the decoded position. We ran experiments over a total of 18 sessions in which a PPF decoder was trained using either the N-OFC model or the instant-OFC model. We compared the steady-state performance of these two PPF decoders. We found that for 2D cursor movements, using the instant-OFC model compared with the N-OFC model resulted in a PPF decoder that improved success rate by 21% (one-sided t-test, P < 10−4). Movement error and reach times using the instant-OFC model also improved by 6% (one-sided t-test, P < 0. 09) and 15% (one-sided t-test, P < 10−3), respectively (S1 Table, S2 Fig). We also found that the improvement in performance for the instant-OFC model was not due to a simple increase in speed; on average the maximum speed when the decoder was trained with the N-OFC model did not change significantly compared to when it was trained using the instant-OFC model (two-sided t-test, P = 0. 55). This may suggest that the instant-OFC model better captures the user’s control strategy. We thus elected to use the instant-OFC model for intention estimation in this study. Hence OFC-PPF in the remainder of this paper refers to a PPF that was trained with the instant-OFC model. To examine the effect of using the OFC intention estimation on the steady-state performance of the PPF decoder, we performed experiments that dissociated its effect and compared it to the state-of-the-art method of intention estimation, i. e. , CursorGoal [20]. In particular, in addition to running experiments in which we converged the PPF decoder with the adaptive OFC-PPF, we also ran experiments in which we converged the PPF with exactly the same adaptive approach except for using the CursorGoal method of intention estimation in place of OFC (Fig 2). The CursorGoal method infers the intended velocity direction by rotating the decoded velocity vector towards the target, while setting the intended speed equal to the decoded speed before reaching the target and to zero at the target. We call the PPF decoder trained using the CursorGoal method of intention estimation, CursorGoal-PPF. Fig 4A and 4B show the inferred velocity intentions used in the CursorGoal-PPF for two sample decoded trajectories. These inferred velocities are directed straight towards the target; thus the inferred velocity direction in this case depends only on the current position. Moreover, this method does not infer the speed intention and sets it equal to the decoded speed. Hence a major difference between the instant-OFC model we elected to use and the existing CursorGoal method of intention estimation [20] is that the former infers the speed intention in contrast to the latter that does not infer the speed intention. We compared the steady-state performance between 12 days with OFC-PPF and 12 days with CursorGoal-PPF (Fig 4C). We found that using the OFC intention estimation significantly improved the performance of PPF decoder (S1 Table), increasing success rate by 26% (one-sided t-test, P < 10−6). Also, reach times were 24% shorter with OFC-PPF compared with CursorGoal-PPF (one-sided t-test, P < 10−5), and maximum speeds with OFC-PPF were higher by 6% compared to CursorGoal-PPF (one-sided t-test, P < 0. 002). Finally, movement error and accuracy did not change significantly between the two decoders (two-sided t-test, P > 0. 6). These results indicate that the monkey could control the OFC-PPF significantly faster than CursorGoal-PPF without a cost on movement error or accuracy. In addition to performance improvement, a principal advantage of using an explicit model of the brain for intention estimation is its ability to extend to prosthetics with various dynamics, simply by incorporating their dynamic model in the OFC formulation (see Discussion). We found that assisted training enabled the monkey to achieve proficient control consistently across days. This was in contrast to days in which assisted training was not applied. Initially during the experiments, we adapted the PPF decoder using the method of SmoothBatch and without the assisted training paradigm. We performed 8 days of experiments in which we initialized the decoder using visual feedback seeds. We found that in 3 of these 8 days, the monkey was unable to converge to proficient control and hence discontinued playing the task. In many cases, the cursor initially got stuck at the boundaries or corners of the workspace and hence was not able to explore the space. In contrast, in tens of days that adaptation was performed with assisted training, whether using the method of SmoothBatch or using continuous spike-event-based adaptation, the monkey always achieved proficient control eventually (see below for comparison of the speed of convergence between SmoothBatch and spike-event-based adaptation). These results suggest that assisted training was essential in keeping the subject engaged and in exploring the space so that decoder parameters could converge. We investigated whether fast spike-event-based parameter adaptation for the PPF decoder can increase the speed of convergence compared to batch-based methods that update the parameters on the slower time-scale of minutes. To do so, we performed experiments to dissociate the effect of the time-scale of adaptation. In particular, in addition to running experiments in which we trained the PPF using the spike-event-based adaptation method, we also ran experiments in which we trained it with slower adaptation time-scales using the method of SmoothBatch [21], while keeping all the other components of the algorithm the same (Fig 2). SmoothBatch adapts the parameters smoothly once every 90 sec (see Methods). We refer to this decoder as SmoothBatch OFC-PPF. We compared the speed of performance convergence using SmoothBatch OFC-PPF and adaptive OFC-PPF by finding the time it took these decoders to reach 90% of the maximum performance. Fig 5A shows the performance convergence using SmoothBatch OFC-PPF and adaptive OFC-PPF in consecutive days in which parameters were initialized to the same values (see also S3A Fig). Performance converged faster in adaptive OFC-PPF compared to SmoothBatch OFC-PPF in these days. This result held on average across all days as demonstrated in Fig 5B and 5C (see also S2 Table and S3B and S3C Fig). Across 12 days of experiments with each decoder, while the eventual steady-state success rate was the same (two-sided t-test; P = 0. 35), adaptive OFC-PPF converged much faster to this steady-state level compared with SmoothBatch OFC-PPF. In particular, across 12 days, success rate in SmoothBatch OFC-PPF converged in 18. 7±3. 2 min (mean ± s. e. m.). In comparison and across 12 days, success rate in adaptive OFC-PPF converged in 6. 5±0. 7 min, which was significantly faster than SmoothBatch OFC-PPF (one-sided t-test; P < 10−3). Hence adaptive OFC-PPF resulted both in faster convergence and in more consistent speed of convergence as indicated by the lower s. e. m. across days. Moreover, assisted training in 2 of the 12 days for SmoothBatch OFC-PPF had to reinitiate after the test period and went on to a second assist period (Fig 2). In comparison, using adaptive OFC-PPF, assisted training always stopped after the first test period since the performance had already exceeded the threshold success rate of 5 trials/min (Fig 5). We also investigated whether adaptive OFC-PPF could converge to a proficient decoder when starting from a poor initial decoder. First, we found that this monkey was not able to control the BMI using a visual feedback seed decoder (i. e. , a decoder whose parameters were found from the neural activity collected during a training session in which cursor movements were visually displayed to the monkey). Success rate using a PPF decoder that used the visual feedback parameters was 0 across all days tried (the monkey soon lost motivation using this decoder, which indeed motivated the design of the assisted training paradigm). Hence across tens of days, adaptive OFC-PPF could always result in proficient performance in this monkey in spite of starting from a poor initial decoder (i. e. , the visual feedback seed decoder). To further examine the robustness of adaptive OFC-PPF to initialization, we ran 2 days of experiments in which we started the adaptive OFC-PPF once from a visual feedback seed and once from a seed that was obtained by randomly permuting the visual feedback seed across neurons. We found that adaptive OFC-PPF resulted in proficient control starting from the permuted seed. Moreover, success rate, movement error, percent correct, and reach times did not change significantly between the two converged decoders (two-sided t-test; P > 0. 16; Fig 6A). For example, average success rate for the visual feedback and permuted visual feedback seeds were 14. 6 ± 1. 8 vs. 14. 45 ± 2. 3 trials/min, respectively. We also examined the convergence of decoder parameters starting from the visual feedback and permuted visual feedback seeds using adaptive OFC-PPF (Fig 6B–6D). On the days in which we started adaptive OFC-PPF once from each of these two seeds, we calculated the difference between the estimated parameters in the two cases as a function of time into the adaptive session. We found that this difference was significantly reduced at the end of the adaptive session compared to the start of this session (paired t-test; P < 0. 05). This suggests that the parameters converged to similar values regardless of the initial seed (Fig 6B–6D). In particular, for the baseline firing rate and αc parameters of each neuron (c. f. Eq (5) ), this difference at the start of the session was on average 5. 3 and 5. 6 times larger than at the end of the session, respectively (see S4 Text for details of calculation). We also investigated whether adaptive OFC-PPF could extend to tasks beyond the standard center-out target-directed task used for CLDA training. The computational element that allows for this extendability after adaptation is complete is the flexible prior model placed on the kinematics in the PPF (see Eq (13) ). After the decoder is trained and assisted training stops (i. e. , when La = 0), we use a random-walk prior model on kinematics that only constrains the kinematics thorough the dynamic matrix A in Eq (19). The A matrix just enforces two general rules: 1) velocity evolution in time is correlated, i. e. , there is correlation between velocity vectors at two adjacent time points, 2) position is the integral of velocity. These general constraints hold for various types of movement (e. g. , straight movements or curved movements). Here we ran closed-loop experiments and showed that the new CLDA technique could enable extension to other tasks. Using a decoder trained with adaptive OFC-PPF, we had the monkey perform a target-jump task across 7 sessions. In this task, in a random selection of center-out trials, the target jumped to one of the other 7 targets 500ms after the cursor left the center (see Methods; Fig 7A). The monkey achieved proficient control in the target-jump task (Fig 7A). In particular, while the target-jump trials were harder to perform, monkey’s accuracy (percent correct) on these trials was only slightly lower than the normal center-out trials in the same sessions, i. e. , 88% vs. 92% (S3 Table). The monkey achieved a success rate of 13. 9±0. 4 trials per minute on the target-jump task across sessions (note that jumps require moving a longer distance compared to center-out trials; for example, on the same days the monkey’s success rate on the center-out task was 15. 08±0. 3 in comparison). In addition to the target-jump sessions, the monkey performed a target-to-target movement task in one session. This task required the monkey to perform a point-to-point movement similar to the center-out task (i. e. , movement to and brief hold at an initial target, followed by movement to and brief hold at a terminal target). However, this time, there was no center target and the initial and end target locations could be different from the center-out task (Fig 7B). The monkey again achieved proficient control on this task reaching a success rate of 13. 2 trials per minute and an accuracy of 90%.
  Significant progress has been made in BMI development by combining Kalman filters with CLDA techniques. However, current decoders such as Kalman filters may limit the time-scale at which the brain can control the prosthetic and the time-scale at which decoder parameters can be adapted. Time-scales of control and adaptation may significantly affect BMI performance. Hence developing decoders that are not limited to a particular time-scale because of modeling assumptions provides a powerful tool to investigate these effects (S1 Text). Moreover, while various successful CLDA approaches have been used in different studies, a unified CLDA framework for intention estimation, assisted training, and adaptation is lacking. Here we propose a novel CLDA architecture for training a PPF BMI in closed-loop operation, which consists of two main computational components: 1. A general unified control-theoretic OFC framework for (i) intention estimation and (ii) assisted training, 2. Dynamic spike-event-based adaptation as opposed to batch-based adaptation. This architecture allows an arbitrarily fast time-scale for spike processing in control and adaptation, and incorporates a general unified control-theoretic CLDA framework. We use the architecture to investigate the effects of the time-scale of adaptation and of the OFC framework on closed-loop decoder training. Using closed-loop experiments in a rhesus monkey over 89 days, we dissociated the effect of the novel components of our CLDA framework, i. e. , the OFC method of intention estimation, the OFC method of assisted training, and the spike-event-based time-scale of adaptation in training a PPF. In summary, our paradigm produced improvements on two fronts: 1) spike-event-based adaptation and assisted training improved the speed and ease of decoder convergence during training, and 2) OFC intention estimation resulted in higher performance PPF decoders after training was complete. Overall, the BMI architecture—termed adaptive OFC-PPF—enabled robust performance and extended control to various tasks. The architecture is based on an infinite-horizon optimal feedback-control model of brain behavior in closed-loop BMI control and a point process model of the spiking activity (Fig 1). The OFC model infers the brain’s intention during the process of adaptation. This OFC model is also used to develop a new assisted training technique, which allows for dynamic and automated assisted training. The assisted training technique also enables the calculation of a chance level performance during assisted training; this may provide a potential approach to evaluate subject engagement during assistance by comparing to this chance level. Thus OFC modeling of brain behavior provides a unified framework for intention estimation and assisted training. The point process model allows subjects to control the BMI with every spike event, and enables the BMI to adapt its parameters with every spike event. The architecture is also fully modular (Fig 2). Our closed-loop experiments in a non-human primate dissociated the advantage of each of the architecture modeling components. We first dissociated the effect of the OFC intention estimation by comparing OFC-PPF with CursorGoal-PPF at steady state. We found that using the OFC model for intention estimation results in a PPF with higher steady-state success rate compared with current intention estimation techniques (i. e. , CursorGoal; Fig 4); this suggests that the former better estimates the brain’s control strategy. We dissociated the effect of the OFC assisted training by comparing OFC-PPF with assisted training turned off or on. We found that the automated assisted training technique results in consistent performance convergence across days by allowing the decoder to explore the space and by keeping the subject motivated. We dissociated the effect of spike-event-based adaptation by comparing SmoothBatch OFC-PPF with adaptive OFC-PPF during decoder training. We showed that fast spike-event-based adaptation results in faster performance convergence compared with existing batch-based methods, which have a slower adaptation time-scale (Fig 5). We also found that the architecture is robust to initialization of parameters (Fig 6). Finally, we showed that adaptive OFC-PPF results in a decoder that extends to tasks beyond those used for CLDA (Fig 3), in particular a target-jump task and a target-to-target task (Fig 7). While we implemented the OFC model for intention estimation in spike-based BMIs and to fit a point process observation model, the OFC intention estimation technique can also be used for other signal modalities or observation models. These include local field potentials or electrocorticography (ECoG) signals, which can be modeled as a Gaussian process. In these cases, the estimated intention would be used to fit a Gaussian model of neural activity in an appropriate decoder such as a Kalman filter. Moreover, the OFC models the brain’s strategy in closed-loop control of a prosthetic. Since the brain is the controller of movement in any BMI setup, the OFC framework could generalize to various BMI setups by simply quantifying the corresponding prosthetic’s dynamics and the task goals within the OFC’s state-space model and cost function, respectively. The OFC assisted training technique is also not specific to a point process. We can, for example, extend this framework to provide assisted training in a Kalman filter BMI by using the OFC state-space model as the prior kinematic model in the Kalman filter. The dynamic assisted training will otherwise be identical to the one used in combination with a point process observation model here (Fig 2). In this work we used a 5ms bin-width to generate the 0 and 1 time-series of the spikes. This was because in our case more than 1 spike rarely occured within a single bin. In the PPF decoder we use a discrete-time point process as an approximation to a continuous-time point process. This approximation is reasonable as long as the bins are small enough to contain at most 1 spike [35]. If in a scenario units or multiunits are recorded with higher firing rates such that there is a considerable probability of having more than 1 spike per 5ms bin, then smaller bin-widths should be used to get a discrete-time point process, which is a good approximation to the continuous-time point process. An important observation in the BMI field is that the brain can adapt to control a BMI decoder over time [6,8, 15,28]. Neural plasticity can result in changes in neural firing properties (e. g. , directional tuning) that yield improved BMI performance. Closed-loop decoder adaptation also introduces an additional adaptive element into the BMI, creating a “two-learner system”. Neural and decoder adaptation can interact [28], raising the possibility for interference between these two learning elements. Neural plasticity is most robustly observed with slowly-varying or fixed decoders [15,28]. A potential concern, then, is that the high frequency of adaptation and control in the OFC-PPF could disrupt neural plasticity. More broadly, how decoder control and adaptation time-scales influence neural plasticity is currently unknown. We therefore devised a set of experiments to examine neural learning effects. We ran an additional experiment with this monkey over 14 days with a semi-stationary neural ensemble using the adaptive OFC-PPF. On each day, adaptation was run for 5–15 minutes starting from the previous day’s decoder. The monkey showed gradual performance improvement over these days, with increased success rate and reduced movement error. These results suggest that learning can indeed occur while the monkey controls a PPF adapted with the spike-event-based adaptation paradigm; thus adaptive OFC-PPF can indeed be combined with brain learning to improve neuroprosthetic control. Detailed data related to the learning experiment are presented elsewhere [28]. The examination of how the time-scales of neural and decoder adaptation may interfere in other scenarios from a theoretical perspective is important in future studies [52]. Another important problem to investigate is the selection of the learning rate in CLDA algorithms. The learning rate is a critical parameter of any adaptive algorithm and is currently selected in an ad-hoc manner and by trial and error in CLDA-based BMIs. The learning rate in adaptive OFC-PPF is dictated by the noise covariance in the state model of the parameter decoder. Developing principled methods for optimal selection of this learning rate will be the topic of our future investigation [50]. The point process model allows subjects to control the BMI and receive feedback at a much faster time-scale than a Gaussian model used in Kalman filters. In this work, we focused on developing the novel closed-loop point process BMI training architecture that is fully modular, adaptive and robust, and can extend control to various tasks. While outside the scope of this work, it is important to investigate whether the PPF can be controlled better than the Kalman filter by using the point process neural encoding model and by providing the fast time-scale of control, which in turn increases the rate of control and feedback. For example, it may indeed be that the faster time-scale of control results in more accurate control. Moreover, the spiking activity that any BMI algorithm uses as its input is fundamentally a sequence of spike events, or zeros and ones, which indicate the time-points at which the spikes occur. The larger bins used in prior decoders just preprocess this spiking signal first, and then use the preprocessed signal (e. g. , the count of spikes within the bin) to decode movement intention. As the data processing inequality in information theory states [53], preprocessing data cannot add information. Hence processing the spikes directly with an optimal filter specifically designed for the zero and one spike events could extract the maximal amount of information from the spikes. This, along with the faster update rates, may result in performance improvements. To examine questions related to performance comparisons, novel experiments are needed to dissociate the effects of control rate, feedback rate, and the mathematical encoding model on BMI control. This will be the subject of our future investigation [54]. Finally, in our experiments the smaller time-scale of adaptation resulted in faster convergence to the same steady-state level. However, it would be interesting to examine whether the steady-state performance could also change as a result of the time-scale of decoder adaptation in more complex BMI tasks that rely more on neural adaptation and introduce complex interactions between the decoder and neural adaptation processes. Here we not only developed the novel adaptive OFC-PPF BMI training architecture, but also dissociated and explored the effect of each of its modeling components. Given the modularity of the architecture and its general control-theoretic CLDA framework, these results guide the incorporation of various proposed modeling components in neuroprosthetic designs. These findings have significant implications towards the development of future clinically-viable neurotechnologies.
summary: Brain-machine interfaces (BMI) decode subjects’ intended movements from neural activity. Despite significant advances, performance, robustness, and extendibility remain key challenges. Current BMIs use decoders such as Kalman filters whose parameters are estimated in closed-loop BMI operation. However, current decoders do not model the spikes directly and hence may limit the time-scale of neural processing, control, and parameter estimation. Here we develop a novel BMI training architecture for spike-event-based control and parameter estimation, and show that it enables robust control and extends to various tasks. Moreover, we propose a control-theoretic framework for closed-loop decoder training. The BMI incorporates an optimal feedback-control (OFC) model of brain’s control behavior to infer its motor intention. The BMI uses a point process to model the spikes; this enables the subject to control the movement and the decoder to estimate the parameters using spike events. Using closed-loop experiments in a non-human primate, we show that the OFC model improves performance compared with current intention estimation techniques; that spike-event-based adaptation enables faster performance convergence compared with current batch-based methods, and is robust to parameter initialization; and that the architecture extends to various tasks. This architecture has significant implications for future clinically-viable BMIs.
keywords: velocity
  action potentials
  medicine and health sciences
  classical mechanics
  education
  adaptive training
  membrane potential
  sociology
  vertebrates
  electrophysiology
  social sciences
  neuroscience
  animals
  mammals
  primates
  training (education)
  vision
  animal management
  kinematics
  animal performance
  monkeys
  animal cells
  agriculture
  physics
  cellular neuroscience
  psychology
  cell biology
  physiology
  neurons
  biology and life sciences
  cellular types
  physical sciences
  sensory perception
  amniotes
  neurophysiology
  organisms
  motion
title: Robust Brain-Machine Interface Design Using Optimal Feedback Control Modeling and Adaptive Point Process Filtering