The acts of learning and memory are thought to emerge from the modifications of synaptic connections between neurons, as guided by sensory feedback during behavior. However, much is unknown about how such synaptic processes can sculpt and are sculpted by neuronal population dynamics and an interaction with the environment. Here, we embodied a simulated network, inspired by dissociated cortical neuronal cultures, with an artificial animal (an animat) through a sensory-motor loop consisting of structured stimuli, detailed activity metrics incorporating spatial information, and an adaptive training algorithm that takes advantage of spike timing dependent plasticity. By using our design, we demonstrated that the network was capable of learning associations between multiple sensory inputs and motor outputs, and the animat was able to adapt to a new sensory mapping to restore its goal behavior: move toward and stay within a user-defined area. We further showed that successful learning required proper selections of stimuli to encode sensory inputs and a variety of training stimuli with adaptive selection contingent on the animat' s behavior. We also found that an individual network had the flexibility to achieve different multi-task goals, and the same goal behavior could be exhibited with different sets of network synaptic strengths. While lacking the characteristic layered structure of in vivo cortical tissue, the biologically inspired simulated networks could tune their activity in behaviorally relevant manners, demonstrating that leaky integrate-and-fire neural networks have an innate ability to process information. This closed-loop hybrid system is a useful tool to study the network properties intermediating synaptic plasticity and behavioral adaptation. The training algorithm provides a stepping stone towards designing future control systems, whether with artificial neural networks or biological animats themselves.
One of the most important features of the brain is the ability to adapt or learn to achieve a specific goal, which requires continuous sensory feedback about the success of its motor output in a specific context. We developed tools [1]–[3] for closing the sensory-motor loop between a cultured network and a robot or an artificial animal (animat) [4] in order to study learning directly through behavior of the artificial body and its interaction with its environment. Compared to animal models, the cultured network is a simpler and more controllable system to investigate basic network computations; confounding factors such as sensory inputs, attention, and behavioral drives are absent, while diverse and complex activity patterns remain [5]–[9]. Previously, an embodied cultured network' s ability to control an animat or a mobile robot was demonstrated without a specifically defined goal [2], [10]. In another case, animats were designed to avoid obstacles [11] or follow objects [12], but deterministically and without learning. By using a lamprey brainstem to control a mobile robot, Mussa-Ivaldi et al. demonstrated the embodied in vitro network' s tendency to compensate the sensory imbalance caused by artificially altering the sensitivity of the sensors at one side of the robot. Without a pre-defined goal and external training stimulation, long-term changes in behavior in response to the sensory imbalance were found in embodied lamprey brainstems [13], however, the changes were unpredictable [14]. In order to further understand the learning capability of an embodied cultured network for goal-directed behavior, we need to investigate how the network can be shaped and rewired, and how to direct this change. Previous studies have demonstrated the potential for disembodied cultured networks to achieve functional plasticity. This neural plasticity provides a potential learning capability to cultured networks. Jimbo et al. [15] used a localized tetanic stimulus to induce long-lasting changes in the network responses that could be either potentiated or depressed depending on the electrode used to evoke the responses. Moreover, we and others previously found that such tetanus-induced plasticity was spatially localized and asymmetrically distributed [16], [17]. By delivering two different tetanic stimulation patterns, Ruaro et al. trained a cultured network to discriminate the spatial profiles of the stimuli. These results suggest that different stimulation patterns can shape diverse functional connectivity in cultured networks. By incorporating closed-loop feedback, Shahaf and Marom [18] showed unidirectional learning: to induce an electrode-specific increase in response. This simple form of learning was achieved by a binary training: to stop a periodic stimulation at one electrode when the desired response level at the target electrode was obtained. In order to scale to more complex behavior, we need to create more structured training stimuli and detailed activity metrics to investigate whether an embodied cultured network can learn multiple tasks simultaneously. Unlike in vivo systems, the sensory-motor mapping and training algorithm in an embodied cultured network are defined by the experimenters. In order to efficiently find an effective closed-loop design among infinite potential mappings, we first embodied a biologically-inspired simulated network to study an adaptive goal-directed behavior in an animat: learning to move toward and stay within a user-defined area in a 2-D plane. The simulated network of 1000 leaky integrate-and-fire neurons expressed spontaneous and evoked activity patterns similar to that of the dissociated cortical cultures [19]. Furthermore, a similar but larger simulated network showed that localized coherent input resulted in shifts of receptive and projective fields similar to those observed in vivo [20]. Thus simulated networks show promise for analyzing biological adaptation with various closed-loop designs. The closed-loop design we discuss here consists of four unique elements: Here, we demonstrate adaptive goal-directed behavior in the simulated network, where multiple tasks were learned simultaneously. The desired behavior could only be achieved with proper selection of stimuli to encode sensory inputs and a variety of training stimuli with adaptive selection contingent on the animat' s behavior. While lacking the characteristic layered structure of in vivo cortical tissue, the biologically-inspired simulated network still could be functionally shaped, and showed meaningful behavior, demonstrating that these neural networks have an innate ability to process information. The proposed design is not restricted to a particular sensory-motor mapping, and could be applied with different and more complex goal-directed behaviors, which may provide a useful in vitro model for studying sensory-motor mappings, learning, and memory in the nervous system.
We used three networks with different connectivity, each with 5 different sets of CPSs (randomly selected CPSQ1–CPSQ4). These 15 setups with different network connectivity and sensory-motor mappings were used for the following simulation experiments:
In order to validate the use of RBS to maintain desired behavior, the animat was run with RBS between context-control probing sequences (CPSs) without training (no PTS), and the results were compared to the animat' s performance without RBS (CPSs only). An example of the time course of the animat' s distance from the origin is shown in Figure 2A. The motor mapping was transformed (by, see Figure 1B) to obtain desired movements before the simulation. Therefore, in the beginning of both simulations with RBS and without RBS, the animat moved in desired directions in each quadrant and stayed within the inner circle. The animat maintained this desired behavior for the entire hour over 90% of the time when RBS was applied, whereas it moved outward after 10 minutes when no RBS was applied. The mutual information between the movement angle and the sensory input is shown in Figure 2B. When the animat started moving outward in an undesired direction, the mutual information decreased significantly. This indicates decreasing stability of the animat' s movement under the same sensory input. The mutual information during the last 10 minutes (P2 period in Figure 2B) was compared to the mutual information during the first 10 minutes (P1) in the 15 simulations (3 networks, 5 different selections of CPSs each) (Figure 2C). With RBS, the mutual information in P2 was 1. 42±0. 15 bits (mean±SEM, n = 1800 measures, 15 networks, 120 measures in 10 min per network), which was comparable to 1. 53±0. 09 bits in P1 (p = 0. 77, Wilcoxon signed-rank test). Without RBS, the mutual information in P2 was 0. 14±0. 10 bits, which was significantly lower than 1. 40±0. 24 bits in P1 (p<1e-4). This indicates that RBS with an aggregate frequency of 3 Hz maintained stability of the network input-output function, validating the use of RBS to maintain desired behavior in the animat. Furthermore, the results also suggested that repetitive non-training stimuli (CPSs and RBS) were unable to induce enough plasticity to systematically alter the animat' s behavior. We investigated the networks' ability to learn a user-defined goal behavior by “switching” the sensory mapping. A motor mapping was created (through transformations) to obtain desired movements before the experiment began (Figure 1B). The animat' s performance was observed for 10 minutes, demonstrating robust goal-directed behavior (Figures 3 and 4). Then the sensory mapping was suddenly and drastically altered, so that the animat' s behavior was no longer correct. Specifically, a CPS appropriate for evoking movement toward the center from Q1 was now delivered when the animat was in Q3, and vice versa. Learning was then quantified by the animat' s ability to adapt to the new, fixed sensory mapping and exhibit goal-seeking behavior. Ten simulations, out of 15, showed successful adaptation to the switch. One successful simulation is shown in Figure 3A, and the corresponding movie is shown in Supplemental Material Movie S1. Immediately after the switch, as expected, the animat moved outward in the quadrants where the sensory mapping switch was performed (Q1 and Q3). Patterned training stimulation (PTS), paired stimulation designed to induce STDP throughout any shared activation pathways in the network, began to shape the network synaptic weights, and the desired behavior was restored under the switched mapping. An unsuccessful simulation is shown in Figure 3B. In 5 unsuccessful simulations, the animat kept moving outward and was repeatedly put back into the inner circle whenever it reached the outer circle. The training was unable to restore the desired behavior throughout a 4-hr simulation. In Figure 3B, only the first 90 minutes are shown for clarity. Distance plots for all 15 simulations are shown in Figure 4. For successful simulations, the average time for the adaptation was 88. 6±12. 2 minutes (mean±SEM, n = 10 successful-learning simulations). Two different types of unsuccessful learning are also indicated (Type I and Type II failures, see below). One-third of the simulations showed unsuccessful learning but were nevertheless informative (see Figure 4). Two types of failures were observed in these following 5 unsuccessful experiments. In order to verify that the successful adaptation in the overall system was contributed by learning in the network, and not solely by the adaptive process in the artificial training algorithm, we repeated the original successful-learning simulations with the STDP algorithm turned off. We found that the desired behavior could not be restored without the STDP algorithm, or long-term plasticity, in the network. This also rules out frequency-dependent synaptic depression as the adaptation mechanism, since that algorithm was left turned on. The comparison of the animat' s movement in one successful-learning simulation and its corresponding simulation without STDP is shown in Figure 7, and the comparison of learning curves is shown in Figure 7B. Among all original successful-learning simulations, the average probability of successful behavior before the switch was 63. 3±3. 5% (n = 10 successful-learning simulations), dropped significantly to 9. 8±1. 1% after the switch (p<5e-4, Wilcoxon signed-rank test), and increased significantly back to 53. 6±3. 5% after 88. 6±12. 2 minutes when the desired behavior was restored (p<5e-4) (Figure 7C). The probability of successful behavior after the switch was comparable to that before the switch (p = 0. 09). For all corresponding simulation without STDP algorithm, the probability of successful behavior before the switch was 68. 4±4. 6% (n = 10 simulations without STDP), dropped significantly to 6. 2±0. 8% after the switch (p<5e-4), but showed no significant increase at the end of the simulation (6. 4±0. 9%) (p = 0. 91) (Figure 7C). This indicated that network long-term plasticity was essential for successful learning in the closed-loop system. Different PTSs were delivered at different times before the desired behavior was restored. The training history from the same successful-learning example shown in Figure 7 is shown in Figure 8A. We hypothesized that the same PTS might have different effects at different points in time because the network would be in different states. Therefore, successful adaptations would require application of PTSs in a certain sequence. In order to test this hypothesis, we ran 10 additional simulations with only one PTS pattern available for training in each quadrant, instead of a pool of 660 PTSs as in the original stimulations (see Methods). These were the four most often used PTSs in the original simulations, one for each quadrant. For the example shown in Figure 8A, only PTS #575 was delivered in the new simulation when training was required due to unsuccessful movement in Q1. We compared the original simulation and the corresponding new simulation by their learning curves (one example is shown in Figure 8B). The probability of successful behavior generally kept increasing after the switch for the original successful-learning simulation where multiple PTS patterns were available for training (gray curve), but not for the new simulation where only a single PTS pattern was available (blue curve). A significant increase of the probability of successful behavior after the sensory mapping switch was found in the original successful-learning simulations (p<5e-4) (Figure 8D, and also Figure 7C). However, all 10 new simulations with only the four most frequent PTSs available showed no significant increase of the probability of successful behavior from immediately after the switch (9. 2±1. 0%) to the end of the simulation (10. 1±3. 7%) (p = 0. 61, Wilcoxon signed-rank test) (Figure 8D). This shows that not only one PTS, but a sequence of different PTSs was needed in order to restore the desired behavior. We have demonstrated that successful adaptations to altered sensory mappings required a sequence of different PTSs, which was determined by the real-time feedback contingent on the animat' s performance. In order to investigate the importance of behavior-contingent training for successful learning, we recorded the whole stimulation sequence (PTS and RBS) for each successfully adapted case and replayed it into the same network with the same initial state and same sensory-motor mapping. Different random seeds for fluctuations in neurons' membrane potentials and synaptic currents were used between the successful-learning simulations and the replayed training simulations. This difference would lead to different network responses, and thus different movement trajectories and different CPS sequences. However, the effect of non-training stimuli (CPSs and RBS) on shaping the network was insignificant, as shown in Figure 2. Therefore, whether the network could adapt to the new sensory mapping solely depended on the effect of training stimulation. The replayed training stimulation was no longer contingent on whether or not desired movement occurred. In 10 stimulation-replay experiments, the animat was unable to show successful adaptation to the sensory mapping switch (shown as “non-contingent” in the example of Figure 9A), which had been successful with behavior-contingent training (shown as “contingent”). A comparison of the learning curves for this example is shown in Figure 9B. With contingent training, a significant increase of the probability of successful behavior after the sensory mapping switch was found (p<5e-4) (Figure 9C, and also Figure 7C). However, with replayed training stimulation, the average probability of successful behavior in the last 10 minutes of the simulations was 11. 6±2. 2%, which is comparable to 9. 2±1. 8% measured within 10 minutes after the switch (p = 0. 47) (Figure 9C). In order to understand how successful (closed-loop) and replayed (open-loop) training stimulation shaped the network differently, we visualized the changes in weights of all synapses by using Principal Components Analysis (PCA). The first three components (PC1 to PC3) of the network synaptic weights for the contingent training simulation and the non-contingent training simulation example shown in Figure 9A are plotted over time (Figure 9D). Starting from the same initial synaptic weights, the network diverged to different synaptic weights distributions as the training became progressively less contingent on the network activity and the animat' s performance. We have demonstrated that two different sets of network synaptic weights that were responsible for the desired behavior under two different sensory mappings (Pre and Post-contingent in Figure 9D). We then further investigated whether under a specific sensory mapping, the desired behavior could only be exhibited by a specific set of network synaptic weights. After the network adapted to the switched sensory mapping, we switched the sensory mapping back to the original sensory mapping to see whether the network could re-adapt to the original mapping (Figure 10). After the switch-back, the behavior-contingent patterned training stimulation was able to restore the desired behavior under the original sensory mapping (Figure 10A), but with a different set of network synaptic weights (Figure 10B). This indicates that multiple synaptic configurations, or “solutions”, existed for the desired behavior.
RBS was hypothesized to negate “attractors” in network synaptic weight distributions caused by spontaneous activity (mainly network-wide synchronized bursts of activity called barrages), and to prevent network synaptic weights from drifting to such attractors after inducing plasticity with electrical stimulation [19]. RBS with an aggregate frequency of 1 Hz reduced the occurrence of spontaneous barrages by at least 10 times in the simulated network and dissociated cortical cultures [19]. By reducing the occurrence of spontaneous barrages, the network synaptic weights were mainly affected by activity evoked by RBS. Since RBS was random spatially and temporally, the evoked activity had an unbiased randomizing effect on changing network synaptic weights. In a different approach, a barrage-control stimulation protocol consisting of a group of electrodes cyclically stimulated with an aggregated frequency of 50 Hz was found to completely eliminate spontaneous barrages [32]. Similar to RBS, the barrage-control stimulation stabilized tetanus-induced plasticity in dissociated cortical cultures (Madhavan R, Chao ZC, Potter SM, unpublished data). However, different mechanisms might be involved. RBS evoked network-wide responses with unbiased spatiotemporal structure, while the barrage-control stimulation desynchronized spontaneous activity into spatially localized and temporally dispersed responses. In this study, the aggregate stimulation frequency of RBS was increased from 1 to 3 Hz so that the amount of stimulation in RBS and PTS were comparable. RBS did stabilize network synaptic weights (the network synaptic weights were clustered in Pre period in Figure 9D) and also stabilized the network input-output function (see Figure 2). Even though sharing the same network connectivity and the same PTS pools, some simulations showed successful learning and others were unsuccessful. Therefore, we concluded that the selection of CPSs for sensory encoding, which was the only remaining difference, was crucial for determining the success of adaptation. We found that the stimulations used to encode sensory inputs should evoke neither overly localized nor largely overlapped responses. Too much localization reduced the possibility to improve movement directions in switched quadrants, and too much overlap caused unwanted changes in un-switched quadrants. These results suggest a certain level of independence is required between responses to stimulations used to encode different sensory inputs, which could be achieved by using smaller and distinct recording areas to determine movement, or by offsetting the CA through the motor mapping transformation so that the probability of a CA to point in different directions is more uniform. Furthermore, correlated changes in responses to different sensory inputs could also be avoided by using training stimulation that only causes localized plastic changes. These findings could instruct the designs of implant electrode geometries and feedback stimulation patterns in prosthetics to achieve a more efficient and effective adaptation. We showed that long-term plasticity in the network (STDP) was essential for the adaptation in the overall system (see Figure 7). Short-term plasticity (frequency-dependent synaptic depression, see Methods and Supplemental Material Text S1) alone was not able to achieve successful adaptation (Figure 7). Furthermore, learning curves indicate that fewer training stimuli were required to maintain the desired behavior after the system had adapted (see Figure 7B and Figure 8B). These suggest that the improved performance was not due to short-term elastic responses to the stimulation. Elastic change was observed in dissociated cultures where the neurons' responsiveness adapted to very low frequency stimulation but relaxed back within minutes after stimulation was removed [33], [34]. Using paired pulses with different stimulation electrodes and different inter-pulse intervals was one possible design for training. More optimal training algorithms likely exist. Using stimulation sequences with more than two stimuli could help shape the network synaptic weights to a desired state, since they might evoke a greater variety of response patterns and produce different behaviors. However, the tradeoff is that a larger pool of possible training stimuli could lead to a longer training duration before successful adaptation. Furthermore, a different algorithm to adaptively update the probability of selecting PTSs might better find appropriate PTSs and remove unhelpful ones in the pool. The simulated network was used to explore many different possible sensory-motor mappings and training algorithms (not described here) because of savings in preparation time and an ability to monitor all synaptic weights. The described algorithm successfully demonstrated adaptive goal-directed behavior with multiple sensory-motor mappings. This closed-loop algorithm is not restricted to a particular type or a particular number of sensory-motor mappings. Integrate-and-fire networks have been used previously for demonstrating goal-directed learning [35], [36]. In this work, we constructed a simulated network, specifically to mimic living MEA cultures, in order to find a closed-loop design that might be applicable to show goal-directed learning living cultures. In another study, we tested our closed-loop algorithm in a cortical network cultured over an MEA, where we successfully avoid Type I and Type II failure to train a living network to control the movement of an animat in a desired direction (Chao ZC, Bakkum DJ, Potter SM, unpublished data). Studying neural networks' basic computational properties, such as parallel signal processing and learning, by working with simulated/living in vitro networks could lead to direct development of more advanced artificial neural networks, more robust computing methods, and even the use of neurally controlled animats themselves as biologically-based control systems.
The ability of a brain to learn has been studied at various levels. However, a large gap exists between behavioral studies of learning and memory and studies of cellular plasticity. In particular, much remains unknown about how cellular plasticity scales to affect network population dynamics. In previous studies, we have addressed this by growing mammalian brain cells in culture and creating a long-term, two-way interface between a cultured network and a robot or an artificial animal. Behavior and learning could now be observed in concert with the detailed and long-term electrophysiology. In this work, we used modeling/simulation of living cortical cultures to investigate the network' s capability to learn goal-directed behavior. A biologically inspired simulated network was used to determine an effective closed-loop training algorithm, and the system successfully exhibited multi-task goal-directed adaptive behavior. The results suggest that even though lacking the characteristic layered structure of a brain, the network still could be functionally shaped and showed meaningful behavior. Knowledge gained from working with such closed-loop systems could influence the design of future artificial neural networks, more effective neuroprosthetics, and even the use of living networks themselves as a biologically based control system.
neuroscience/behavioral neuroscience
neuroscience
computational biology/computational neuroscience
Shaping Embodied Neural Networks for Adaptive Goal-directed Behavior