article: The capacity to learn abstract concepts such as ‘sameness’ and ‘difference’ is considered a higher-order cognitive function, typically thought to be dependent on top-down neocortical processing. It is therefore surprising that honey bees apparantly have this capacity. Here we report a model of the structures of the honey bee brain that can learn sameness and difference, as well as a range of complex and simple associative learning tasks. Our model is constrained by the known connections and properties of the mushroom body, including the protocerebral tract, and provides a good fit to the learning rates and performances of real bees in all tasks, including learning sameness and difference. The model proposes a novel mechanism for learning the abstract concepts of ‘sameness’ and ‘difference’ that is compatible with the insect brain, and is not dependent on top-down or executive control processing.
  Abstract concepts involve the relationships between things. Two simple and classic examples of abstract concepts are ‘sameness’ and ‘difference’. These categorise the relative similarity of things: they are properties of a relationship between objects, but they are independent of, and unrelated to, the features of the objects themselves. The capacity to identify and act on abstract relationships is a higher-order cognitive capacity, and one that is considered critical for any operation involving equivalence or general quantitative comparison [1–4]. The capacity to recognise abstract concepts such as sameness has even been considered to form the “very keel and backbone of our thinking” [5]. Several non-verbal animals have been shown to be able to recognise ‘sameness’ and ‘difference’ including, notably, the honey bee [6–9]. The ability of the honey bee to recognise ‘sameness’ and ‘difference’ is interesting, as the learning of abstract concepts is interpreted as a property of the mammalian neocortex, or of regions of the avian pallium [10–12] and to be a form of top-down executive modulation of lower-order learning mechanisms (complex neural processing affecting less complex processing, see [13] for more details) [4,12]. This interpretation has been reinforced by the finding that activity of neurons in the prefrontal cortex of rhesus monkeys (Macaca mulatta) correlates with success in recognising sameness in tasks [11,12]. The honey bee, however, has nothing like a prefrontal cortex in its much smaller brain. In this paper we use a modelling approach to explore how an animal like a honey bee might be able to solve an abstract concept learning task. To consider this issue we must outline in more detail how learning of sameness and difference has been demonstrated in honey bees, and originally in other animals. A family of ‘match-to-sample’ tasks has been developed to evaluate sameness and difference learning in non-verbal animals. In these tasks animals are shown a sample stimulus followed, after a delay, by two stimuli: one that matches the sample and one that does not. Sometimes delays of varying duration have been imposed between the presentation of the sample and matching stimuli to test duration of the ‘working memory’ required to perform the task [1,14]. This working memory concept is likened to a neural scratchpad that can store a short term memory of a fixed number of items, previously seen but no longer present [15]. Tests in which animals are trained to choose matching stimuli are described as Match-to-Sample (MTS) or Delayed-Match-To-Sample (DMTS) tasks, and tests in which animals are trained to choose the non-matching stimulus are Not-Match-To-Sample (NMTS) or Delayed-Not-Match-To-Sample (DNMTS) tasks. On their own, match-to-sample tasks are not sufficient to show concept learning of sameness or difference. For this it is necessary to show, having been trained to select matching or non-matching stimuli, that the animal can apply the concept of sameness or difference in a new context [4]. Typically this is done by training animals with one set of stimuli and testing whether they can perform the task with a new set of stimuli [6–9]; this is referred to as a transfer test. In a landmark study [8] showed that honey bees can learn both sameness and difference. They could learn both DMTS and DNMTS tasks and generalise performance in both tasks to tests with new, previously unseen, stimuli [8]. In this study free-flying bees were trained and tested using a Y-maze in which the sample and matching stimuli were experienced sequentially during flight, with the sample at the entrance to the maze and the match stimuli at each of the y-maze arms. Bees could solve and generalise both DMTS and DNMTS tasks when trained with visual stimuli, and could even transfer the concept of sameness learned in an olfactory DMTS task to a visual DMTS task, showing cross-modal transfer of the learned concept of sameness [8]. Bees took 60 trials to learn these tasks [8]; this is much longer than learning a simple olfactory or visual associative learning task, which can be learned by bees in 3 trials [16]. Their performance in DMTS and DNMTS was not perfect either; the population average for performance in test and transfer tests was around 75%, but they could clearly perform at better than chance levels [8] in both. The concept of working memory is crucial for solving a DMTS/DNMTS task, as information about the sample stimulus is no longer available externally to the animal when choosing between the match stimuli. If there is no neural information that can identify the match then the task cannot be solved. We therefore must identify in the honeybee a candidate for providing this information in order to produce a model that can solve the task. A previous model by [17] demonstrates DMTS and DNMTS with transfer, however the model contains many biologically unfounded mechanisms that are solely added for the purpose of solving these tasks, and the outcome of these additions disagrees with neurophysiological, and behavioural evidence. We instead take an approach of constraining our model strongly to established neurophysiology and neuronanatomy, and demonstrating behaviour that matches that of real bees. We will compare this model to the model presented here further in the Discussion. As the problem (without transfer) is a binary XOR, it can be solved by a network with at least one hidden layer, however this would not solve the generalisation aspect of the problem. The honey bee brain is structured as discrete regions of neuropil (zones of synaptic contact). These are well described, as are the major tracts connecting them [18]. The learning pathways have been particularly intensely studied (e. g. [19–22]). The mushroom bodies (corpora pedunculata) receive processed olfactory, visual and mechanosensory input [23] and are a locus of multimodal associative learning in honey bees [19]. They are essential for reversal and configural learning [4,24,25]. Avarguès-Weber and Giurfa [4] have argued the mushroom bodies to be the most likely brain region supporting concept learning, because of their roles in stimulus identification, classification and elemental learning [19,22,26]. Yet it is not clear how mushroom bodies and associated structures might be able to learn abstract concepts that are independent of any of the specific features of learned stimuli and, crucially, how the identity of the sample stimulus could be represented. Solving such a problem requires two computational components. First, a means of storing the identity of the sample stimulus, a form of working memory; second, a mechanism that can learn to use this stored identity to influence the behaviour at the decision point. Below we propose a model of the circuitry of the honey bee mushroom bodies that can perform these computations and is able to solve DMTS and DNMTS tasks.
  We explored whether a neural circuit model, inspired and constrained by the known connections of the honey bee mushroom bodies, is capable of learning sameness and difference in a DMTS and DNMTS task (Fig 1). Full details of the models can be found in Methods. A sufficiency table for the key mechanisms and the following results is shown in Table 1. The mushroom body has previously been modelled as an associative network consisting of three neural network layers [26,27], comprised of input neurons (IN) providing processed olfactory, visual and mechanosensory inputs [23,28], an expansive middle layer of Kenyon cells (KC) which enables sparse-coding of sensory information for effective stimulus classification [22], and finally mushroom body extrinsic neurons (EN) which output to premotor regions of the brain and can be considered (at this level of abstraction) to activate different possible behavioural responses [22,26]. Here, for simplicity, we consider the EN as simply two subpopulations controlling either ‘go’ or ‘no-go’ behavioural responses only, which allow choice between different options via sequential presentations where ‘go’ chooses the currently presented option. Connections between the KC output and ENs are modifiable by synaptic plasticity [26,29–31] supporting learned changes in behavioural responses to stimuli. As outlined in the introduction, we require two computational mechanisms for solving the DMTS/DNMTS task. First is a means of storing the identity of the sample stimulus. Second is learning to use this identity to drive behaviour and solve the task. Moreover this learning must generalise to other stimuli. The computational complexity of this problem should not be underestimated; either the means of storing the identity of the sample, or the behavioural learning, must generalise to other stimulus sets. The bees were not given any reward with the transfer stimuli in [8]’s study, so no post-training learning mechanism can explain the transfer performance. In addition, during the course of the experiment of [8] each of the two stimuli were used as the match, i. e. for stimuli A and B the stimulus at the maze entrance were alternated between A and B throughout the training phase of the experiment. This requires, therefore, that the bees have a sense of stimulus ‘novelty’, and can associate novelty with a behaviour: either approach for DNMTS, or avoid for DMTS. With one training set the problem is solvable as delayed paired non-elemental learning tasks (e. g. responding to stimuli A and B together, but not individually), however with the transfer of learning to new stimulus sets such an approach does not solve the whole task. There is one feature of the Kenyon Cells which can fulfill this computational requirement for novelty detection, that of sensory accommodation. In honey bees, even in the absence of reward or punishment, the KC show a stark decrease in activity between initial and repeated stimulus presentations of up to 50%, an effect that persists over several minutes [32]. This effect is also found in Drosophila melanogaster [33], where there is additionally a set of mushroom body output neurons that show even starker decreases in response to repeated stimuli, and which respond to many stimuli with stimulus specific decreases (thus making them clear novelty detectors), however such a neuron has not been found in bees to date. This response decrease in Kenyon Cells found in flies and bees is sufficient to influence behaviour during a trial but, given the decay time of this effect, not likely to influence subsequent trials. The mechanism behind this accommodation property is not known, and therefore we are only able to model phenomenologically, which we do by reducing the strength of the KC synapses for the sample stimulus by a fixed factor, tuned to reproduce the reduction in total KC output found by Szyszka et al. [32] (see Fig 2 panel E). However it should be noted that stimulus-specific adaptation is shown in many species and brain areas, and can be explained by short-term plasticity mechanisms [34–36] and architectural constraints only; see for instance [37]. Exploiting the faculty of the units to adapt to repeated stimuli allows the network to distinguish between repeated stimuli (“same”) and non-repeated, i. e. novel “different”) stimuli. Having identified our first computational mechanism, a memory trace in the form of reduced KC output for the repeated stimulus, we need only to identify the second, a learning mechanism that can use this reduced KC output to drive behaviour to choose the correct (matching or non-matching) arm of the y-maze. If this learning mechanism exists at the output synapses of the KC it is either specific to the stimulus—if using a pre-postsynaptic learning rule—and therefore cannot transfer, or it utilises a postsynaptic-only learning rule. Initially the postsynaptic learning rule appears a plausible solution, however we must consider that bees can learn both DMTS and DNMTS, and that learning can only occur when the bee chooses to ‘go’. This creates a contradiction, as postsynaptic learning will proportionally raise both the weaker (repeated) stimulus activity, as well as the stronger (non-repeated) stimulus activity in the GO EN subpopulation. To select ‘go’ the GO activity for the currently presented stimulus must be larger than the activity in the NOGO subpopulation, which is fixed. Therefore we face the contradiction that in the DMTS case the weaker stimulus response must be higher than the stronger one in the GO subpopulation with respect to their NOGO subpopulation counterpart responses, yet in the DNMTS case the converse must apply. No single postsynaptic learning rule can fulfil this requirement. A separate set of neurons that can act as a relay between the KC and behaviour is therefore required to solve both DMTS and DNMTS tasks. A plausible candidate is the inhibitory neurons that form the protocerebellar tract (PCT). These neurons have been implicated in both non-elemental olfactory learning [25] and regulatory processes at the KC input regions. They also project to the KC output regions [38–41], where there are reward-linked neuromodulators and learning-related changes [20,42]. These neurons are few in number in comparison to the KC population, and some take input from large numbers of KC [43]. We therefore propose that, in addition to their posited role in modulating and regulating the input to the KC based on overall KC activity [43], these neurons could also regulate and modulate the activity of the EN populations at the KC output regions. Such a role would allow, via synaptic plasticity, a single summation of activity from the KCs to differentially affect both their inputs and outputs. If we assume a high threshold for activity for the PCT neurons (again consistent with their proposed role) such that repeated stimuli would not activate the PCT neurons but non-repeated stimuli would, it is then possible for synaptic plasticity from the PCT neurons to the EN to solve the DMTS and DNMTS tasks and, vitally, transfer that learning to novel stimuli. We do not propose that this is the purpose of these neurons, but instead that it is a consequence of their regulatory role. We present two models inspired by the anatomy and properties of the honey bee brain that are computationally capable of learning in DMTS and DNMTS tasks, and the generalisation of this learning to novel stimuli (Fig 1). Our first, reduced, model is a simple demonstration that the key principles outlined above can solve DMTS and DNMTS tasks, and generalise the learning to novel stimulus sets. By simplifying the model in this way the computational principles are readily apparent. Such a simple model, however, cannot demonstrate that associative learning in the KC to EN synapses does not interfere with learning in the PCT to EN synapses or vice versa. For this we present a full model that includes the associative learning pathway from the KC to the EN, and demonstrate that this model can not only solve DMTS and DNMTS with transfer to novel stimuli, but can also solve a suite of associative learning tasks in which the MB have been implicated. The results of computational experiments performed with these models are presented below. The full model addresses the interaction of the PCT to EN learning and the KC to EN learning, as well as suggesting a possible computational role of the PCT to EN synaptic pathway in regulating the behavioural choices driven by the MB output, which we present in the Discussion. The reduced model is shown in Fig 1 Panel B and model equations are presented in Methods. The input nodes S1 and S2 represent the two alternative stimuli, where we have reduced the sparse KC representation into two non-overlapping single nodes for simplicity, and as such we do not need to model the IN input neurons separately. Node I (which corresponds to the PCT neurons, again reduced to a single node for simplicity) represents the inhibitory input to the output neurons GO and NOGO. Nodes S1 and S2 project to nodes I and to GO and NOGO with fixed excitatory weighted connection. Finally, node I projects to GO and NOGO with plastic inhibitory weighted connections. Node I is thresholded so that it only responds to novel stimuli. Fig 2 panels A and B show the performance of the reduced model bees (as well as the full model) for task learning and transfer to novel input stimuli, alongside experimental data from [8]. While the reduced model solves the transfer of sameness and difference learning the pretraining process strongly biases the model towards non-repeated stimuli, proportional to the number of pretraining trials. Notably, this bias in the reduced model is different to that found in the full model, which we discuss below. The model operates by adjusting the weights between the I and the GO to change the likelihood of choosing the non-matching stimulus. Since only connections from the I (representing the PCT neuron) to GO neurons are changed, the I to GO/NOGO weights are initialised to half the maximum weight value. Note that the I node is only active for the non-repeated stimulus, and this pathway has no effect for repeated stimuli. This means that if the weights are increased then non-repeated stimuli will have greater inhibition to the GO neuron, and therefore be less likely to be chosen. If the weights are decreased then non-repeated stimuli will have less inhibition to the GO neuron and therefore will be more likely to be chosen. As the conditions for changing the weights are only met when the non-repeated stimulus is chosen for ‘go‘, this means that the model only learns on unsuccessful trial for DMTS (increasing the weight), or successful trials for DNMTS (decreasing the weight). The full model is shown in Fig 1 Panel C and model equations are presented in Methods. Fig 2 panel D shows the performance of the full model for the first block of learning following from different numbers of pretraining repetitions. When only the PCT pathway is plastic there is a large bias towards the non-repeated stimulus due to the pretraining, as found in the reduced model. This bias is reduced by the presence of the associative learning pathway, and the bias is independent of the number of pretraining trials for more than 5 trials. It should be noted that the experimental data [8] show indications of such a bias, in line with the results from the full model. The reduced model therefore requires fewer pretraining trials than the full model to produce a similar bias, which leads to the reduced model having large maladaptive behavioural biases for non-repeated stimuli if all stimuli are rewarded. This is important, as it suggests a role for the PCT pathway in modulating the behavioural choice of the bee. This possible role is explored further in the Discussion. Fig 2 panels A and B show the performance of the model bees compared with the performance of real bees from [8], and the reduced model. In both cases the trends found in the performance of the model bees match the trends found in the real bees for both task learning and transfer to novel stimuli. It is important to note the different forms of the learning in the DMTS and DNMTS paradigms, with DNMTS slower to learn. This is a direct consequence of the inhibitory nature of the PCT neurons; excitatory neurons performing the role of the PCT neurons in the model would lead to a reversal of this feature, with DMTS learning more slowly. We next sought to confirm that learning in the PCT neuron to EN pathway enabled generalisable learning of sameness and difference. Computational modelling provides powerful tools with which to do this, by comparing model performance when different elements are suppressed with the full model. We selectively suppressed the KC associative learning pathway, the PCT pathway learning, and all learning in the model. When a learning mechanism is suppressed this means that the synaptic weights stay the same throughout the training, but the pathway is otherwise active. The results are summarised in Fig 2 panel C. It can clearly be seen that within our model learning in the PCT pathway is necessary for transfer of the sameness and difference learning to novel stimuli. Associative learning via the KC pathway alone has no effect on the transfer task performance compared to the fully learning-suppressed model. Unsuppressed associative learning leads to a preference for the matched stimulus, which has weaker KC activity, but this learning is specific to the trained stimuli, and does not transfer to novel stimuli. Many models have reproduced the input neuron to Kenyon Cell to Extrinsic neuron pathway [26,27,44,45], and these models demonstrate many forms of elementary and complex associative learning that have been attributed to the mushroom bodies. It is therefore important to demonstrate that in our model the PCT neuron pathway does not affect the reproduction of such learning behaviours. We therefore tested elemental and non-elemental associative learning undertaken by conditioning the Proboscis Extension Reflex (PER) in restrained bees, and reversal learning in free flying bees, as described in Methods. Our model is capable of reproducing the results found in experiments involving real bees, with the model’s acquisition curves showing similar to the performance to the real bees. The results are shown in Fig 3, and details of the experimental paradigms used can be found in Methods.
  We have presented a simple neural model that is capable of learning the concepts of sameness and difference in Delayed Match to Sample (DMTS) and Delayed Not Match to Sample (DNMTS) tasks. Our model is inspired by the known neurobiology of the honey bee, and is capable of reproducing the performance of honey bees in a simulation of DMTS and DNMTS tasks. Our model therefore proposes a hypothesis for how animals like the honey bee might be able apparently to learn abstract concepts. Abstract concept learning is typically described as a higher-order cognitive capacity [1,4], and one that is dependent on a top-down modulation of simpler learning mechanisms by a higher cognitive process [49]. By contrast our model proposes a solution to sameness and difference learning in DMTS-style tasks with no top-down structure (that is, no complex mechanisms imposing themselves on simpler processes). The actions of the PCT neurons are integrated with the KC learning pathway and provide a parallel processing pathway sensitive to stimulus magnitude, rather than a top-down imposition of a learned concept of sameness or difference (Fig 1). This is a radical new proposal for how an abstract concept might be learned by an animal brain. The first question we must ask when constructing a model regards plausibility. Our model (Fig 1) shows a close match to the neuroanatomical data for the mushroom bodies. Several computational requirements of our model match with experimental data, notably the sensory accommodation in the response of the KC neurons. Previous neural models based on this structure have proposed mechanisms for various forms of associative learning, including extinction of learning, and positive and negative patterning [17,26,45]. Our model is also capable of solving a range of stimulus-specific learning tasks, including patterning (Fig 3). No plausible previous model of the MB or the insect brain has been capable of learning abstract concepts, however. As mentioned in the Introduction, a previous model by [17] demonstrates DMTS and DNMTS with transfer. Their motivation is the creation of a model for robotic implementation, rather than reproduction of behavioural observations from honey bees. While we suggest a role for the PCT neurons given experimental evidence of changes in the response of Kenyon Cells to repeasted stimuli, Arena et al. ’s model assumes resonance between brain regions that is dependent upon the time after stimulus onset and the addition of specific neurons for ‘Match’ and ‘Non-match’; there is no biological evidence for either of these assumptions. Furthermore, the outcome of these additions is an increase in Kenyon cell firing in response to repeated stimuli; this is in opposition to neurophysiological evidence from multiple insect species, including honey bees [32,33]. In addition, Arena et al‘s proposed mechanism does not replicate the difficulty honey bees have in learning DMTS/DNMTS tasks, exhibiting learning in three trials, as opposed to 60 in real bees. In contrast, our model captures the rate and form of the learning found in real honey bees. To enable a capacity for learning the stimulus-independent abstract concept of sameness or difference our model uniquely includes two interacting pathways. The KC pathway of the mushroom bodies retains stimulus-specific information and supports stimulus-dependent learning. The PCT pathway responds to summed activity across the KC population and is therefore largely independent of any stimulus-specific information. This allows information on stimulus magnitude, independent of stimulus specifics, to influence learning. Including a sensory accommodation property to the KCs [32] makes summed activity in the KCs in response to a stimulus sensitive to repetition, and therefore stimuli encountered successively (same) cause a different magnitude of KC response to novel stimuli (different) irrespective of stimulus specifics. This model is capable of learning sameness and difference rules in a simulation of the Y-maze DMTS and DNMTS tasks applied to honey bees (Fig 2), but in theory it could also solve other abstract concepts related to stimulus magnitude such as quantitative comparisons [4,50]. Our model demonstrates a bias towards non-repeated stimuli, induced by the combination of sensory accommodation in the KC neurons and PCT learning during the pretraining phase, and largely mitigated by associative learning in the KC to EN synapses. This bias (see Fig 2) is indicated in the data from [8], and could be confirmed by further experimentation. We note, however, that our model only supports a rather limited form of concept learning of sameness and difference. Learning in the model is dependent on sensory accommodation of the KCs to repeated stimuli [32]. This effect is transient, and hence the capacity to learn sameness or difference will be limited to situations with a relatively short delay between sample and matching stimuli. This limitation holds for honey bee learning of DMTS tasks [51], but many higher vertebrates do not have this limitation [52]. For example, in capuchins learning of sameness and difference is independent of time between sample and match [53]. We would expect that for animals with larger brains and a developed neocortex (or equivalent) many other neural mechanisms are likely to be at play to reinforce and enhance concept learning, enabling performance that exceeds that demonstrated for honey bees. Monkey pre-frontal cortex (PFC) neurons demonstrate considerable stimulus-specificity in matching tasks, and different regions appear to have different roles in coding the salience of these stimuli [54,55]. Recurrent neural activity between these selective PFC neurons and lower-order neural mechanisms could support such time independence. Language-trained primates did particularly well on complex identity matching tasks and the ability to form a language-related mental representation of a concept might be the reason [56–58]. Wright and Katz [1] have utilised a more elaborate form of a MTS task in which vertebrates simultaneously learn to respond to sameness and difference, and are trained with large sets of stimuli rather than just two. They argue this gives less-ambiguous evidence of true concept learning since both sameness and difference are learned during training, and the large size of the training stimulus set encourages true generalisation of the concept. In theory our model could also solve this form of task, but it is unlikely a honey bee could. Capuchins, rhesus and pigeons required hundreds of learning trials to learn and generalise the sameness and difference concepts [1]. Bees would not live long enough to complete this training, Finally as a consequence of our model we question whether it is necessary to consider abstract concept learning to be a higher cognitive process. Mechanisms necessary to support it may not be much more complex than those needed for simple associative learning. This is important because many behavioural scientists still adhere to something like Lloyd Morgan’s Canon [59], which proposes that “in no case is an animal activity to be interpreted in terms of higher psychological processes if it can be fairly interpreted in terms of processes which stand lower in the scale of psychological evolution and development” ([59] p59). Yet the Canon is therefore reliant on an unambiguous stratification of cognitive processes according to evolutionary history and development [60]. If abstract concept learning is in fact developmentally quite simple, evolutionarily old and phylogenetically widespread, then Morgan’s Canon would simply beg the question of why even more animals do not have this capacity [61]. We argue that far more information on the precise neural mechanisms of different cognitive processes, and the distribution of cognitive abilities across animal groups, is needed in order to properly rank capacities as higher or lower. In other words, so-called higher cognitive processes may be solved by relatively simple structures.
  Many of the parameters of the model were fixed by the neuroanatomy of the honey bee, as well as the previous values and procedures described in [26], with the following modifications. First, we increased the sparseness of the connectivity from the PN to the KC. Second, the reduction in the magnitude of the KC output to repeated stimuli was tuned to replicate the magnitude of reduction described in [32]. Third, the learning rates were set so that acquisition of a single stimulus is rapid. In addition there are two ratios from this initial value that must be set. These are the ratio of the speed of excitatory associative learning in the Kenyon Cell to Extrinsic Neuron pathway to the inhibitory learning in the Protocerebellar Tract to Extrinsic Neuron pathway, and the ratio of the speed of acquisition when rewarded to the speed to extinction when no reward is given. We conservatively set both of these ratios to 2: 1, with excitatory learning faster than inhibitory learning, and extinction faster than acquisition. The learning parameters used in the reduced model are used in the fitting of the full model. Finally, we tuned the threshold value for the PCT neurons so that they only responded to a new stimulus, and not a repeated one. A full list of the parameters can be found in Table 2. The reduced model is shown in Fig 1, and described in the text in Results. Here are the equations governing the model. The input node S1 projects to node I via a fixed excitatory weight of 1. 0 and to GO and NOGO with excitatory weights w GO, S 1 e and w NOGO, S 1 e correspondingly (superscript denotes excitatory and subscript the connection from neuron S1 to GO/NOGO). Similarly, node S2 projects to I via an excitatory weight w I, S 2 e and to GO and NOGO with excitatory weights w GO, S 2 e and w NOGO, S 2 e. Finally, node I projects to GO and NOGO with inhibitory weights w GO, I i and w NOGO, I i correspondingly. Node I is a threshold linear neuron with a cut-off at high values of activity xmax. Nodes GO and NOGO are linear neurons, with activities restricted between zero and one. The model is described by the following equations, where only one input node S1 or S2 are active (but not both, as the bee observes one option at a time), where the activities of neurons I, GO and NOGO are calculated by: I = S i Θ (S i > θ) (1) GO = w GO, S i e S i - w GO, I i I (2) NOGO = w NoGo, S i e S i - w NoGo, I i I, (3) where i = 1,2 is an index taking values depending on which stimulus is present (S1 or S2), and neuronal activities of I, GO and NOGO are constrained between xmin and xmax. If a stimulus has been shown twice, during its second presentation there is a suppression of the neuronal activity that represents the specific stimulus, consistent with experimental findings [32]. This is modelled as a reduction by a factor of 0. 7 of the value Si for the repeated stimuli. To calculate the proabability of the behavioral outcome of GO or NOGO being the winner we use the following equation: P (GO) = 1 1 + e - (c - d) (GO - NOGO), (4) P (NOGO) = 1 - P (GO). (5) where c is a fixed coefficient and d a bias that increases linearly with the time it takes to make a decision, in the following way: d = k d o, (6) with k being the number of consequent iterations for which GO has not been selected, set at zero at the beginning of each stimulus presentation. The parameter do is a constant, and selected so that the factor c − d will always be positive. This parameterisation makes sure that the longer it takes for a decision GO to be made, the higher the probability that GO will be chosen at the next step. The full model explicitly implements this strategy iteratively, and validation of this approach can be seen in the close agreement between the behavioural outcomes of the two models. Inhibitory synaptic weights wi are learned using the following equation: Δ w i = - λ i (R - R b) presynaptic activity × postsynaptic activity, (7) where λi is the learning rate of the inhibitory weights, reward R = 1 if reward is given, and zero in all other cases, Rb is a reward baseline, and the presynaptic (postsynaptic) activity is 1 if the presynaptic (postsynaptic) neuron is active and 0 elsewhere. This is a reward-modulated Hebbian rule also known as a three factor rule [62,63]. Additional neuronal inputs with similar connectivity as S1 and S2, not shown explicitly in the diagram, are also present in the model simulations, and constructing the equations for these simply requires substitution of Si for Ti in Eqs 1,2 and 3. These represent the transfer stimuli and can be used following training to demonstrate transfer of learning. Details of training the model can be found in the Experiment subsection of the Methods. The full model is shown in Fig 1. Our model builds on a well established abstraction of the mushroom body circuit (see [26,27,44]) to model simple learning tasks. The main structure of the model consists of an associative network with three neural network layers. Adapting terminology and features from the insect brain we label these: input neurons (IN) (correponding to S in the Reduced Model), a large middle layer of Kenyon cells (KC) (correponding to the S to GO / NOGO connections in the Reduced Model), and a small output population of mushroom body extrinsic neurons (EN) separated into GO and NOGO subsets (as in the Reduced Model). The connections, cij, between the IN and KC are fixed, and are randomly selected from the complete connection matrix with a fixed probability pIN − >KC = 0. 02. Connections from the KC to the EN are plastic, consisting of a fully connected matrix. The connection strength between the jth KC and the kth EN (wjk) can take a value between zero and one. The neural description used in the entire model is linear with a bottom threshold, and contains no dynamics, consisting of a summation over the inputs followed by thresholding at a value b via a Heaviside function Θ, with a linear response above the threshold value. This gives the associative model as the following, where the outputs of ith, jth and kth neurons of the IN, KC and EN populations are xi, yj and zk respectively, and M describes the modulation of KC activity for the stimulus seen at the maze entrance: M = { 1. 0: at maze entrance 1. 0: in arms if y j = 0 (KC inactive) at entrance 0. 7: in arms if y j > 0 (KC active) at entrance (8) y j = M Θ (∑ i = 0 N I N c i j x i - b) (∑ i = 0 N I N c i j x i - b) (9) z k = Θ (∑ j = 0 N K C w j k y j) (s u m j = 0 N K C w j k y j) (10) where NIN is the number of IN and NKC is the number of KC. DMTS generalisation is performed by the inhibitory protocerebral tract (PCT) neurons sl (correponding to I in the Reduced Model) described by the following equations: s l = Θ (∑ j = 0 N K C x j - b s) (∑ j = 0 N K C x j - b s) (11) y j = Θ (∑ i = 0 N I N c i j x i - b - ∑ l = 0 N P C T s l *) (∑ i = 0 N I N c i j x i - b - ∑ l = 0 N P C T s l *) (12) z k = Θ (∑ j = 0 N K C w j k x j - 0. 5 ∑ l = 0 N P C T w l k i s l) (∑ j = 0 N K C w j k x j - 0. 5 ∑ l = 0 N P C T w l k i s l) (13) Where wlk are the inhibitory weights between the PCT neurons. The * denotes 10 iteration delayed activity from the PCT neurons due to delays in the KC->PCT->KC loop. Learning takes place according to Eq (7), and the following equation for excitatory synapses: Δ w e = λ e (R - R b) presynaptic activity × postsynaptic activity, (14) where λe is the learning rate of the excitatory weights, reward R = 1 if reward is given, and zero in all other cases, Rb is a reward baseline, and the presynaptic (postsynaptic) activity is 1 if the presynaptic (postsynaptic) neuron is active and 0 elsewhere. Similarly to the reduced model, a decision is made when the GO EN subpopulation activity is greater than the NOGO EN population by a bias Rd, where d increases every time a NOGO decision is made by 10. 0, and R is a uniform random number in the range [−0. 5,0. 5]. To prevent early decisions the sum of the whole EN population activity must be greater than 0. 1. Our challenge is to reproduce Giurfa et al’s data demonstrating bees solving DMTS and DNMTS tasks [8]. To aid exploration of our model we simplify the task it must face, while retaining the key elements of the problem as faced by the honeybee. We therefore embody our model in a world described by a state machine. This simple world sidesteps several navigation problems associated with the real world, however we believe that for the sufficiency proof we present here such simplifications are acceptable—the ability of the honeybee to form distinct and consistent neural representations of the training set stimuli as it flies through the maze is a prerequisite of solving the task, and is therefore assumed. The experimental paradigm for our Y-maze task is shown in Fig 2. The model bee is moved between a set of states which describe different locations in the Y-maze apparatus: at the entrance, in the central chamber facing the left arm, in the central chamber facing the right arm, in the left arm, in the right arm. When at the entrance or in the main chamber the bee is presented with a sensory input corresponding to one of the test stimuli. We can then set the test stimuli presented to match the requirements of a given trial (e. g. entrance (A), main chamber left (A), main chamber right (B) for DMTS when rewarding the left arm, or DNMTS when rewarding the right arm). The experimental environment consists of a simplified Y-maze (see Fig 2: main paper), in which the model bee can assume one of three positions: at the entrance to the Y-maze; at the choice point in front of the left arm; at the choice point in front of the right arm. At each position there are two choices available to the model: go and no-go. Go is always chosen at the entrance to the Y-maze as bees that refuse to enter the maze would not continue the experiment. Following this there is a random choice of one of the two maze arms, left or right. If the model chooses no-go this procedure is repeated until the model chooses to go. As no learning occurs at this stage it is possible for the model to constantly move between the two arms, never choosing to go. To prevent this eventuality we introduce a uniformly distributed random bias B to the go channel that increases with the number of times the model chooses no-go (N): B = 10 N (U [ 0,1 ] - 0. 5). The IN neurons are divided into non-overlapping groups of 8 neurons, each representing a stimulus. These are: Each group contains neurons which are zero when the stimulus is not present, and a value of 1 - U [ - 0. 05,0. 05 ]—consistent across the experiment for each bee, but not between bees—when active. In addition to solving the DMTS and DNMTS tasks, we must validate that our proposed model can also perform a set of conditioning tasks that are associated with the mushroom bodies in bees, without our additional PCT circuits affecting performance. Importantly, these tasks are all performed with exactly the same model parameters that are used in the DMTS and DNMTS tasks, yet match the timescales and relative performances found in experiments performed on real bees. We choose four tasks, which comprise olfactory learning experiments using the proboscis extention reflex (PER) that are performed on restrained bees as well as visual learning experiments performed with free flying bees (Fig 4). The reduced model was simulated in GNU Octave [64]. The full model was created and simulated using the SpineML toolchain [65] and the SpineCreator graphical user interface [66]. These tools are open source and installation and usage information can be found on the SpineML website at http: //spineml. github. io/. Input vectors for the IN neurons and the state engine for navigatation of the Y-maze apparatus are simulated using a custom script written in the Python programming language (Python Software Foundation, https: //www. python. org/) interfaced to the model over a TCP/IP connection. Statistical tests were performed as in [8] using 2x2 X2 tests performed in R [67] using the chisq. test () function. The code is available online at http: //github. com/BrainsOnBoard/bee-concept-learning.
summary: Is it necessary to have advanced neural mechanisms to learn abstract concepts such as sameness or difference? Such tasks are usually considered a higher order cognitive capacity, dependent on complex cognitive processes located in the mammalian neocortex. It has always been astonishing therefore that honey bees have been shown capable of learning sameness and difference, and other relational concepts. To explore how an animal like a bee might do this here we present a simple neural network model that is capable of learning sameness and difference and is constrained by the known neural systems of the insect brain, and that lacks any advanced neural mechanisms. The circuit model we propose was able to replicate bees’ performance in concept learning and a range of other associative learning tasks when tested in simulations. Our model proposes a revision of what is assumed necessary for learning abstract concepts. We caution against ranking cognitive abilities by anthropomorphic assumptions of their complexity and argue that by application of neural modelling, it can be shown that comparatively simple neural structures are sufficient to explain different cognitive capacities, and the range of animals that might be capable of them.
keywords: learning
  invertebrates
  decision making
  neural networks
  honey bees
  social sciences
  neuroscience
  learning and memory
  animals
  cognitive psychology
  cognition
  bees
  computer and information sciences
  hymenoptera
  animal cells
  olfactory receptor neurons
  insects
  arthropoda
  psychology
  cellular neuroscience
  eukaryota
  cell biology
  neurons
  biology and life sciences
  cellular types
  afferent neurons
  cognitive science
  organisms
title: Abstract concept learning in a simple neural network inspired by the insect brain