article: Neural networks with a single plastic layer employing reward modulated spike time dependent plasticity (STDP) are capable of learning simple foraging tasks. Here we demonstrate advanced pattern discrimination and continuous learning in a network of spiking neurons with multiple plastic layers. The network utilized both reward modulated and non-reward modulated STDP and implemented multiple mechanisms for homeostatic regulation of synaptic efficacy, including heterosynaptic plasticity, gain control, output balancing, activity normalization of rewarded STDP and hard limits on synaptic strength. We found that addition of a hidden layer of neurons employing non-rewarded STDP created neurons that responded to the specific combinations of inputs and thus performed basic classification of the input patterns. When combined with a following layer of neurons implementing rewarded STDP, the network was able to learn, despite the absence of labeled training data, discrimination between rewarding patterns and the patterns designated as punishing. Synaptic noise allowed for trial-and-error learning that helped to identify the goal-oriented strategies which were effective in task solving. The study predicts a critical set of properties of the spiking neuronal network with STDP that was sufficient to solve a complex foraging task involving pattern classification and decision making.
  Biologically inspired neural networks should be capable of performing sophisticated information processing that the human and animal brains can perform. Information processing by the brain is deeply multilayered and involves many sequential steps before sensory information can be interpreted and translated into behavior. What makes this cascade powerful and unique is its capability to learn and respond to an ever changing environment. In the most studied sensory pathway—the visual one—the sensory input gets progressively more general, though the stages at which visual learning occurs are still a matter of controversy and different plasticity mechanisms might be operating at different processing steps [1]. Eventually sensory information reaches decision centers (such as lateral intraparietal cortex) which govern behavior and those centers are under the influence of reward signals [2,3]. It has been shown that reinforcement based on the reward can be vital to visual learning [4]. To what extent reward influences learning in the sensory cortices seems to be task dependent and different studies showed both learning with [5,6] and without [7,8] the presence of reward. While a great deal of research has gone into understanding the mechanisms of learning, it is still not fully known how learning at the cellular level gives rise to the learning at the level of animal behavior. Most learning models successfully employ hebbian type learning principles formulated as an abstract rule [9] without being concerned with the cellular level mechanisms. The most promising model of learning at the synaptic level is spike time dependent plasticity (STDP) [10]. In STDP, when a presynaptic cell fires shortly before a post synaptic cell, the synapse between them increases in strength. If the postsynaptic cell fires before the presynaptic cell, the synapse decreases in strength. Such basic form of STDP is only capable of unsupervised learning. By storing STDP events in the form of synaptic tags [11], a delayed reward signal can enable a network to perform reinforcement learning [12]. This mechanism relies on the dopamine modulation of synaptic tags created by STDP [13,14,15]. A small number of studies have attempted to perform task learning using rewarded STDP including studies that attempted to maintain the stability of a virtual robotic arm [16,17]. In theoretical studies, STDP commonly leads to the net increase of synaptic strength across the network [18], causing large numbers of synapses to climb toward their maximum values. This would result in a highly dysfunctional brain and can eventually promote an epileptic state [19]. To prevent runaway synaptic dynamics, various regulatory synaptic mechanisms maintain the distribution of synaptic weights in biological networks. This includes homeostatic mechanisms when a group of cells that experiences high firing frequencies over a prolonged period becomes less responsive to excitatory input by effectively reducing incoming excitatory synaptic weights [20]. Another regulatory mechanism, heterosynaptic plasticity, responds to the increase of a synaptic weight by reducing all other incoming synaptic strengths of the same post synaptic neuron [21,22,23,24] preventing neurons from becoming overactive [25,26,27,28]. In this new work, we built a model containing two plastic layers inspired by the pathways known in mammalian brain from sensory to higher order sensory-motor areas responsible for decision making [29,4]. The first layer of the model uses unsupervised (unrewarded) learning to classify the input while the second layer (based on rewarded STDP) is responsible for decision making. As a whole the network simulated the brain of an agent moving through an unknown environment, continuously learning distinct input patterns of food and adjusting synaptic weights controlling its movement according to the reward and punishment signals based on the shape of the different configurations of food particles that it acquires. We demonstrate that such multilayered network combining rewarded and nonrewarded STDP and synaptic regulatory mechanisms is capable of solving a more complex foraging task, involving discrimination between elementary patterns of food, than it was previously reported in the networks with only one plastic layer based on rewarded STDP [30].
  In this study we have expanded upon a network with a single plastic layer which was designed to learn and perform a foraging task in a virtual environment [30]. The new model incorporated two layers of plastic synaptic connections and implemented both rewarded and non-rewarded spike time dependent plasticity (STDP), see Fig 1. It included one-to-many connections from the input layer to the large middle layer and all to all connections from the middle layer to the output layer. Each middle layer cell received inputs from a few randomly selected input neurons. Reducing the number of inputs to each middle layer cell greatly reduced the computational power required for the simulation and was also a realistic approximation of the brain anatomy [31]. Connections between all three layers were set up initially at random and as such any correlation between the input pattern and the network response was random. As the simulation progressed, the learning in the form of changes in synaptic weights caused the network to become more proficient in obtaining food. Connections from the input to the middle layer employed non-rewarded STDP. This form of unsupervised learning allowed the network to respond more readily to the common features (such as spatial patterns), and common combinations of features, within the environment. Connections from the middle to output layers were controlled by the rewarded STDP. Thus, STDP traces were recorded as synaptic tags [13] and only applied if a reward signal was received later. The strength of traces declined, with time causing older traces to have less impact if a reward was eventually applied. A global reward signal was received whenever the network directed the virtual agent to a goal (”food” location). Synapses that observed pre-before-post combination of the action potentials within a few movement cycles before the reward became stronger; therefore, it increased the probability for the same postsynaptic cell to spike when the same input occurred in the future. The supervised learning allowed the network to develop approach or avoidance behavior based on the reward predicting value of the stimuli. To evaluate the training performance, we compared our STDP based learning algorithm to a set of simple heuristic algorithms. In the best performing heuristic algorithm, when no food was present in the visual field, the agent either continued in the same direction it moved on the last step with a 98% probability or turned at 45 degrees left or right with a 2% probability. When the food was present, the heuristic algorithm searched through all possible combinations of 5 moves within the visual field. It then chose the set of moves that would result in the most food being obtained and made the first move from that set. If multiple sets of moves obtained the same number of food particles, the set which obtained food sooner was selected. If multiple sets had the same sequence of food being obtained, one of those sets was chosen randomly. Under standard conditions (such as given density of food particles in the environment) this strategy had an average food acquisition rate of 56%. Our multilayered network combining rewarded and nonrewarded STDP was able to achieve very close acquisition rate of 52%. For comparison, in the previous study we were able to achieve average food acquisition rates of about 48% [30]. This paper is divided into two main sections. In the first part we considered a simple task to obtain food at any location regardless of the spatial pattern of the food particles. In the second part, we increased the complexity of the task, so the agent was trained to pick up food organized within a specific spatial pattern (horizontal bar) and avoid any food particles organized within another pattern (vertical bar). The last task could not be achieved by the simpler network with only one plastic layer [30]. Thus, the main achievement of the current model is that it sets the stage for further development into the learning of more complex tasks, however, it also shows a modest improvement compared to the simpler models performing with food acquisition rates above 51%. In Fig 2 we show the dynamics of a complete model for a simple foraging task of food acquisition regardless of the spatial arrangement of the food particles. Various homeostatic mechanisms were applied in order to obtain the best performance for a simple foraging task of acquiring food; the contribution of individual mechanisms will be discussed in detail in the following sections. Over time performance increased (Fig 2A) as the agent learned to move into the direction of food (Fig 2B). The Fig 2C shows the dynamics of the strengths of excitatory connections from the input to the middle layer over time. As the network learned a task, a small fraction of synapses increased and moved apart from the majority of other synapses which were slightly reduced. This small subset of synapses played the decisive role in learning, and maintaining their size and stable distance from the bulk of synapses kept the network behavior stable. Our previous model with only one plastic layer was able to solve a simple foraging task of food acquisition at any location regardless of the food pattern shape [30]. It failed, however, to distinguish between patterns of different shape and to acquire only food particles arranged in a particular way. In contrast, the new model with two plastic layers, was able to accomplish this more complex task. In the following experiments the food in the environment was arranged into vertical and horizontal pairs of locations. When the virtual agent arrived at the location of either particle from a pair, both particles were removed and another pair of food was generated at a random location on the map. Obtaining horizontally arranged food was rewarded, while obtaining vertically arranged food was punished. Thus, vertically arranged food was considered to be “bad” or “toxic”. When food particles were placed in the environment, it was arranged in such a way that one pattern was never adjacent to another to avoid ambiguity of the input. The two layer model was able to quickly learn avoiding vertical food arrangements and only acquire food particles arranged horizontally (see Fig 6).
  While STDP has been previously applied in the network models to explain pattern recognition [36], and rewarded STDP has been shown to be capable of overcoming the distal reward problem [12,37,38,39,16,17], our study combined both types of synaptic plasticity in a single model to achieve high learning performance in discrimination and decision making tasks. By punishing the network when incorrect decision (wrong type of food is acquired) was made, the network was required (a) to identify the input pattern (food type) by recognizing the arrangement of the food participles and (b) to learn to move toward the rewarded pattern (horizontal arrangement of food particles) and away from the punished pattern (vertical arrangement of food particles). It has been shown that STDP is most relevant to cells firing in the mid range frequencies, while other rules, depending primarily on the presynaptic firing rate, may become important for the networks operating in the high frequency range [40,41,42]. Combining both rate-based and time-based plasticities gives rise to more complex network connectivity patterns [43,44] which could potentially be used for more complex tasks. Our model does not employ the rate-based plasticity but future extensions of our model may incorporate a combination of these complementary forms of synaptic learning. The idea of synaptic strength normalization was formulated shortly after Hebb’s seminal work [45]. It was shown that for a successful Hebbian cell assembly to develop, the total sum of input synaptic weights has to be held near constant in order to prevent runaway dynamics [46,47]. This leads to a strong need for extensive homeostatic regulation to prevent pathological synchronized activity and indeed there is evidence that homeostatic mechanisms are deeply involved in both preventing and causing epileptic conditions [48,49,50,51]. Such normalization rules became an integral part of modeling studies of supervised learning and cortical map formation [52]. In our model, maintaining the balance of synaptic strength within the network was vital to learning and to achieving high model performance. However, not all synaptic rules contributed equally to the final performance of the model. By far the most important was heterosynaptic plasticity which reduces (increases) the strength of all other synapses when one synapse increases (reduces) in strength by homosynaptic plasticity. Such a tendency towards maintaining the net input to a neuron has already been found in experimental studies [53,54,26,27] and has been tested in detailed biophysical models [25,28]. Its potential mechanism may depend on the intracellular Ca2+ dynamics which is influenced by the backpropagating action potentials [55,56] at the synapses which do not actively participate in homosynaptic plasticity induction [24,23,22,57,21]. We also found that the model could only operate efficiently when the net network activity remained within a certain range. If the network became overactive, high levels of activity in the output layer made direction selectivity highly inconsistent. If the network activity was too low, not enough input would be received by the output layer to cause spiking, leading to poor performance as well. Control of the long term average network activity can be achieved by homeostatic scaling [58,59], which adjusted the strength of synaptic inputs up for the neurons with low average firing rates and down for the neurons with high average activity. Whenever there was variation in food density near the agent, it was important to maintain comparable firing rates in the output neurons. Keeping a functional range of this activity from one trial to the next required that the network activation did not increase in direct proportion to the input activity. As the highly active and more consistently rewarded synapses regularly increased in strength, it was possible for certain neurons to gain a major control over the network dynamics. Although it was not necessary that all neurons be equally represented, having a small number of neurons exert much greater control over the network had a pronounced negative effect on performance. Output-side synaptic balancing allowed us to reduce inequalities in the representation of the neurons and created competition between the outputs of a neuron as was proposed in [30]. This was particularly useful when a middle-layer neuron had two (or more) strong connections to the output neurons, thus “supporting” two different directions of movement. If such middle-layer neuron received a reward, it would increase strength of both connections—despite the fact that only one of them was actually responsible for correct food acquisition. Output-side balancing allowed only synapses correlated with reward to be strengthened via rewarded STDP traces. While we do not know direct experimental evidence for this mechanism, a similar effect can be potentially achieved by more complex circuits involving reciprocal inhibition between output neurons. We observed that by normalizing the amount of potentiaton in STDP (based on the history of STDP traces occurring in the synapse past), we increased the speed of initial learning and increased the stability of efficiency gains for the network in the long term. Though not directly observed, this mechanism is consistent with experimental data [32,33,34] and might be of interest for additional experimental scrutiny. Interaction between synaptic excitation and inhibition is a ubiquitous phenomenon found across many different brain circuits [60,61,62,63,64,65,66,67]; it can be observed during spontaneous activity of the brain [68,69,70] and is an integral part of the oscillatory mechanisms in the neuronal networks [71]. To simplify implementation of the inhibitory effects in our model, the neurons of the preceding layer have sent inhibitory connections to the same set of post synaptic cells they excited in the following layer. This essentially provided feedforward inhibition between layers of neurons, using a population of virtual inhibitory neurons that was not explicitly modeled. The total strength of inhibitory connections from any given cell was the same as the total of its excitatory connections. While not biologically realistic, this simplification effectively simulated a feed forward inhibition found in various brain areas [72] where neurons in one layer project to both excitatory and inhibitory neurons in the down stream layer and inhibitory neurons in turn project to excitatory neurons in the same layer [73]. As inhibitory neurons are often more electrically compact [74] and have a faster membrane time constant, this can result in the inhibition arriving to the post synaptic excitatory neurons with only a small delay compared to the excitation. No significant difference was seen in performance of this network as a result of this simplified implementation of inhibition when compared to the previous model where inhibition was implemented explicitly [30]. Noise is another ubiquitous phenomenon in the neural networks and spans multiple time scales and different physical domains [75]. In contrast with artificially crafted devices where noise is usually detrimental to functioning, biological neural networks are capable of using noise in a productive way to improve efficiency and performance [76,77]. Examples of this include enhanced input detection by means of stochastic resonance [78,79,80,81] and probabilistic inference [82,83]; noise can help with performance of the associative memories [84], smoothing the threshold for action potentials [85,86], allowing rapid response time for neuronal populations [87,88] or faithful propagation of firing rates across a layered network [89]. The largest source of the noise in any biological neuronal network is synaptic noise [90]. In our model we employed noise at the level of synaptic currents that represented variability in synaptic vesicle release. Noise was not only tolerated by this model but it was required for its function. Before training there was no meaningful mapping between the input received by the network and its output; the output activity (and therefore direction of movement) was essentially driven by random initial configuration of synaptic weights. Variability (noise) in synaptic release occasionally caused an unexpected output for a given input. When the network output resulted in food acquisition, it was reinforced by a reward. If a synapse that was likely to be associated with a reward was able to surpass the strength of another synapse of the same presynaptic cell, noise in the synaptic output created instances where the “correct” output cell fired and the “incorrect” output cell did not, despite the “incorrect” output cell having a stronger incoming synaptic strength. This effectively allowed for a form of trial and error learning. Without such variety of the actions attempted, it would be difficult to find which goal-oriented strategies are effective and it can be seen as representing a strategy of associating a given stimulus with a given response. Unlike common machine learning approaches or the currently booming deep network architectures [91] which, in some cases, have already outperformed humans [92,93,94,95,96], in this study we only applied biologically plausible and experimentally identified plasticity rules. This new study, following our previous work [30], generated specific predictions regarding what biological learning mechanisms are necessary and/or sufficient to accomplish the learning task. The key components for a successfully working model were STDP, synaptic balancing processes keeping the learning stable, dopamine type reward feedback derived from successful outcome of action and noise at the synaptic level allowing trial-and-error learning. Although these components have already been studied, sometimes in great detail, our study links them together (unsupervised STDP for sensory learning and rewarded STDP for decision making in particular) in a new way and shows advantages of interaction between different plasticity mechanisms for complex task solving.
  Foraging behavior took place in a virtual environment of randomly distributed “food” particles. The environment consists of 50 by 50 grid of locations. Initially, each location was either assigned or not a food particle. When a food particle was acquired as a result of the virtual agent move, it was then assigned randomly to a new spot on the map. This resulted in a continuously changing environment with the same food density. The density of food particles in the environment was set to 10%. The virtual agent was seeing a 7 by 7 grid of squares the (“visual field”) centered on its current location and it could move to any adjacent square including diagonally for a total of 8 directions. The network was composed of 842 spiking map based neurons [97,98], arranged into 3 feed forward layers to mimic a basic biological circuit: a 7 by 7 input layer (I), a 28 by 28 middle (hidden, H) layer, and a 3 by 3 output layer (O) (Fig 1). This structure provides a basic feedforward inhibitory circuit [73] found in many biological structures [99,73,100,101,102,103]. Each cell in the middle layer received synaptic inputs from 9 random cells in the input layer. These connections initially had random strengths drawn from a normal distribution. Each cell in the excitatory middle layer (cell Hi) connected to every cell in the output layer (Oj) with synaptic strength Wij or WIij, respectively. Initially all these connections had uniform strengths and the responses in the output layer were due to the random synaptic variability. Random variability was a property of all synaptic interactions between neurons and it was implemented as variability in the magnitude of the individual synaptic events. Simulation time was divided up into epochs of 600 time steps, each roughly equivalent to 300 ms. At the start of each epoch the virtual agent received input corresponding to locations of nearby food within the input (7x7) layer. Thus 48 of the 49 cells receive input from a unique position relative to the virtual agent location. At the end of the epoch the virtual agent made one move based on the activity of the output layer. If the virtual agent moved to a grid square with “food”, the “food” was removed from that square and assigned to a randomly selected new square. Each epoch was of sufficient duration for the network to receive inputs, produce outputs, and return to a resting state. Input layer neurons representing positions in the environment containing food received brief pulse of excitatory stimulation that triggered a spike; this stimulation was evoked at the start of each movement cycle (epoch). Output was chosen and the virtual agent moved at the end of the epoch. The activity of the output layer of the network controlled direction of virtual agent’s movement. Each of the output layer cells was mapped to a specific direction. The output layer cell (Oj) that spiked the greatest number of times during the first half of an epoch defined the direction of movement on that epoch. If there was a tie, direction was chosen randomly from the tied outputs. If no cells in the output layer fired, the virtual agent continued in the direction it traveled during the previous epoch. There was 1% chance on every move that the virtual agent would ignore any output and instead move in a random direction. This random variability prevented infinite loops of virtual agent’s motion during the learning process. Synaptic noise was not sufficient to break out of all movement loops as some loops were the result of forming strong connections that would mediate the same spiking pattern regardless of the noise. Other times the probability of escape from a loop due to the noise was simply low enough that it would take a significant amount of time to break the loop. While biological systems could utilize different mechanisms to achieve the same goal, the method we implemented was efficient and accomplished the goal. In the model where no pattern recognition was required, the chance of random movement started at 0. 5% but increased by 0. 5% for each move in which no food was obtained. This value was reset to its starting value whenever food was obtained. For more complex task where pattern recognition was also required, high performing solutions to the task included both approach and avoidance behavior. As such, the network was far more susceptible to becoming stuck in a movement cycle, usually when the virtual agent became surrounded by “toxic” food. In order to break these cycles the rate of random motion was gradually increased by 1% per move in which the agent did not obtain food. Synaptic plasticity closely followed the rules introduced in [30]. A rewarded STDP paradigm [12,37,38,104] was implemented between layers H and O and a non-reward modulated STDP paradigm between I and H. A spike in a post-synaptic cell (Oj of the output layer) that directly followed a spike in a pre-synaptic cell (Hi of the hidden layer) created a “pre before post” event. Each new post synaptic cell spike was compared to all pre synaptic spikes within the time window and each new pre synaptic spike was compared to all postsynaptic spikes within the window. The value of an STDP event (trace) was calculated using the following equation [10,105]: p = - | t r - t p | T c, t r k = K e p (1) where tr and tp are the times at which the pre and post synaptic spiking events occurred respectively, Tc is the time constant and is equal to 40 ms. K is equal to —0. 04 in the case of a post before pre event and 0. 04 in the case of a pre before post event. STDP event was immediately applied to the respective synapse Wij between neurons Ii and Hj. In contrast, for synapses between neurons Hi and Oj the events were stored as traces for later use. Each trace remained stored for 6 epochs after its creation and then was erased. While still stored, STDP trace had an effect whenever there was a rewarding or punishing event. If the network was rewarded or punished the new synaptic strength of the synapse Wij was described as: W i j (n + 1) = W i j (n) ∏ k t r a c e s (1 + W i 0 W i * Δ k), (2) Δ k = S r p · t r k t - t k + c · S u m t r (n + 1) A v g t r (n + 1), S u m t r (n + 1) = ∑ k t r a c e s t r k t - t k + c, A v g t r (n + 1) = A v g t r (n) (1 - δ) + δ S u m t r (n + 1), where t is current time step, Srp is a scaling factor for reward/punishment, trk is magnitude of trace (defined in Eq (2) ), tk is time of the trace event, c is a constant (=1 epoch) used for decreasing sensitivity to very recent spikes, Wi = ∑j Wij is a total synaptic strength of all connections from specific cell Hi to all cells Oj of the output layer, Wi0 is a constant that is set to the value of Wi at the at the beginning of the simulation (“target value”). The term Wi0/Wi helped to keep the output weight sum close to the initial target value. The values for Avgtr and Sumtr were almost always positive in our simulations due to the feed forward architecture that we used. We should note, that for a more general model with feedback loops it would be more appropriate to handle negative and positive traces separately. The network was rewarded when the virtual agent moved to a “food” location and Srp = 1. In case of pattern recognition model it was punished when it moved to a location with a toxic food and Srp = −0. 5. There was also smaller punishment applied when no food is obtained, Srp = −0. 1 in the base model and Srp = −0. 01 in the pattern recognition model. The effect of these rules was that the cells with lower total output strength increased their output strength more easily. To ensure that all the output neurons maintained a relatively constant long term firing rate, the model incorporated homeostatic synaptic scaling, which takes place every epoch (= 600 time steps). The total strength of synaptic inputs Wj = ∑i Wij to a given output cell Oj was set to be equal at each time step to the target synaptic input Wj = Wj0—a slow variable that varied over many epochs and depended on the activity of the cell Oj and activity of its all pre-synaptic cells. If a cell Oj consistently fired below the target rate, the Wj0 was increased by Dtar = 0. 0001. If the cell responded above its target firing rate the Wj0 was gradually reduced: Wj (n+1) ={ Wj (n) * (1+Dtar) spikerate<targetrateWj (n) * (1−Dtar) spikerate>targetrate (3) To ensure that the net synaptic input Wj to any neuron was unaffected by plasticity events of the individual connections at the individual time steps and equal to Wj0, we implemented scaling process that occurs after each STDP event. When any excitatory connection increased in strength, all the other excitatory connections incoming to that cell decreased in strength by a “scale factor” Sf to keep Wj = Wj0: W i j (n + 1) = W i j n S f S f = W j 0 ∑ i W i j n (4) Wijn are synaptic weights right after STDP event but before scaling, Wij (n+1) are synaptic weights after scaling, Wj0 is Wi0 from Eq (2). The underlying reduced model of fast spiking neuron was identical to the model used in [30] and can be described by the following set of difference equations [97,106,107]: V n + 1 = f α (V n, I n + β n), (5) I n + 1 = I n - μ (V n + 1) + μ σ + μ σ n, where Vn is the membrane voltage, In is a slow dynamical variable describing the effects of slow conductances, and n is a discrete time step (~0. 5 msec). Slow temporal evolution of In was achieved by using small values of the parameter μ ≪ 1. Input variables βn and σn were used to incorporate external current I n e x t (e. g. , synaptic input): β n = β e I n e x t, σ n = σ e I n e x t. Parameter values were set to σ = 0. 06, βe = 0. 133, σe = 1, μ = 0. 0005. The nonlinearity fα (V, I) was designed in the form of a piece-wise continuous function: f α (V n, I n) = { α (1 - V n) - 1 + I n, V n ≤ 0 α + I n, 0 < V n < α + I n & V n - 1 ≤ 0 - 1, α + I n ≤ V n or V n - 1 > 0, (6) where α = 3. 65. To convert the dimensionless “membrane potential” V to the physiological membrane potential Vph, the following equation was applied: Vph = 50V − 15 [mV] [98]. This model is very computationally efficient and, despite its intrinsic low dimensionality, produces a rich repertoire of dynamics capable to mimic the dynamics of the Hodgkin-Huxley type neurons both at the single cell level and in the context of network dynamics [97,107]. A fast spiking neuron model was chosen to simulate the neurons. To model synaptic interconnections, we used conventional first order kinetic models of synaptic conductances rewritten in the form of difference equations: g (n + 1) s y n = γ g n s y n + { (1 + X R) g s y n / W j, s p i k e p r e, 0, o t h e r w i s e, (7) and the synaptic current computed as: I n s y n = − g n s y n (V n p o s t − V r p). Here gsyn is the strength of synaptic coupling, modulated by the target rate Wj of receiving cell j, indices pre and post stand for the presynaptic and postsynaptic variables, respectively. The first condition, “spikepre”, is satisfied when presynaptic spikes are generated. Parameter γ controls the relaxation rate of synaptic conductance after a presynaptic spike is received (0 ≤ γ < 1). The parameter R is the coefficient of variability in synaptic release. The standard value of R is 0. 12. X is a randomly generated number between -1 and 1. Parameter Vrp defines the reversal potential and, therefore, the type of synapse: excitatory or inhibitory. The term (1+XR) introduces a variability in synaptic release such that the effect of any synaptic interaction has an amplitude that is pulled from a flat distribution ranging from 1+R to 1-R times the average value of the synapse.
summary: This study explores how intelligent behavior emerges from the basic principles known at the cellular level of biological neuronal network dynamics. Compared to the approaches used in the artificial intelligence community, we applied biologically realistic modeling of neuronal dynamics and plasticity. The building blocks of the model are spiking neurons, spike-time dependent plasticity (STDP) and homeostatic rules, known experimentally, which are shown to play a fundamental role in both keeping the network stable and capable of continous learning. Our study predicts that a combination of these principles makes possible a foraging behavior in a previously unknown environment, including pattern classification to distinct between environment shapes which are rewarded and those which are punished and decision making to select the optimal strategy to acquire the maximal number of the rewarded elements. To solve this complex task we used multi-layer neuronal processing that implemented pattern generalization by unsupervised STDP at the earlier processing step, as commonly observed in the animal and human sensory processing, followed by reinforcement learning at the later steps. In the model, the intelligent behavior emerged spontaneously due to the network organization implementing both local unsupervised plasticity and reward feedback resulting from a successful behavior in the environment.
keywords: learning
  medicine and health sciences
  action potentials
  neural networks
  nervous system
  membrane potential
  social sciences
  electrophysiology
  neuroscience
  learning and memory
  synaptic plasticity
  cognitive psychology
  animal behavior
  zoology
  neuronal plasticity
  developmental neuroscience
  computer and information sciences
  foraging
  animal cells
  behavior
  cellular neuroscience
  psychology
  cell biology
  anatomy
  synapses
  physiology
  neurons
  biology and life sciences
  cellular types
  cognitive science
  neurophysiology
title: Multi-layer network utilizing rewarded spike time dependent plasticity to learn a foraging task