Animals excel at adapting their intentions, attention, and actions to the environment, making them remarkably efficient at interacting with a rich, unpredictable and ever-changing external world, a property that intelligent machines currently lack. Such adaptation property strongly relies on cellular neuromodulation, the biological mechanism that dynamically controls neuron intrinsic properties and response to external stimuli in a context dependent manner. In this paper, we take inspiration from cellular neuromodulation to construct a new deep neural network architecture that is specifically designed to learn adaptive behaviours. The network adaptation capabilities are tested on navigation benchmarks in a meta-learning context and compared with state-of-the-art approaches. Results show that neuromodulation is capable of adapting an agent to different tasks and that neuromodulation-based approaches provide a promising way of improving adaptation of artificial systems. We are now seeing the emergence of highly efficient algorithms that are capable of learning and solving complex problems. However, it remains difficult to learn models that generalise or adapt themselves efficiently to new, unforeseen problems based on past experiences. This calls for the development of novel architectures specifically designed to enhance adaptation capabilities of current deep neural networks (DNN). In biological nervous systems, cellular neuromodulation provides the ability to continuously tune neurons input/output behavior to shape their response to external inputs in different contexts, generally in response to an external signal carried by biochemicals called neuromodulators [2, 9] . Neuromodulation regulates many critical nervous system properties that cannot be achieved solely through synaptic plasticity [7, 8] , which represents the ability for neurons to tune their connectivity during learning. Neuromodulation has been shown to be critical to the adaptive control of continuous behaviours, such as in motor control among others [7, 8] . We propose a new neural architecture specifically designed for DNNs and inspired from cellular neuromodulation which we call NMN, standing for "Neuro-Modulated Network". At its core, the NMN architecture is made of two neural networks: a main network and a neuromodulatory network. The main network is a feed-forward DNN composed of neurons equipped with a parametric activation function specifically designed for neuromodulation. It allows the main network to be adapted to new unforeseen problems. The neuromodulatory network, on the other hand, controls the neuronal dynamics of the main network via the parameters of its activation functions. Both networks have different inputs: whereas the main network is in charge of processing samples, the neuromodulatory network processes feedback and contextual data. In [11] , the authors take inspiration from Hebbian plasticity to build networks with plastic weights, allowing them to tune their weights dynamically. In [10] the same authors extand their work by learning a neuromodulatory signal that dictates which and when connections should be plastic. Our architecture is also related to hypernetworks [5] , in which a network's weights are computed through another network. Other recent works focused on learning fixed activation functions [1, 6] . The NMN architecture revolves around the neuromodulatory interaction between the neuromodulatory and main networks. We mimick biological cellular neuromodulation [3] in a DNN by assigning the neuromodulatory network the task to tune the slope and bias of the main network activation functions. Let σ(x) : R → R denote any activation function and its neuromodulatory capable version σ NMN (x, z; w s , w b ) = σ z T (xw s + w b ) where z ∈ R k is a neuromodulatory signal and w s , w b ∈ R k are two parameter vectors of the activation function, respectively governing a scale factor and an offset. In this work, we propose to replace all the main network's neurons activation function with their neuromodulatory capable counterparts. The neuromodulatory signal z, which size k is a free parameter, is shared for all these neurons and computed by the neuromodulatory network as z = f (c). The function f can be any DNN taking as input the vector c representing some contextual inputs (e.g. c may have a dynamic size in which case f would be parameterized as a recurrent neural network (RNN) or a conditional neural process [4] ). The complete NMN architecture and the change made to the activation functions are depicted on Figure 1 . Notably, the number of newly introduced parameters scales linearly with the number of neurons in the main network whereas it would scale linearly with the number of connections between neurons if the neuromodulatory network was affecting connection weights, as seen for instance in the context of hypernetworks [5] . Therefore this approach can be extanded to very large networks. In this work, we use a high level view of a nervous system mechanism called cellular neuromodulation to improve artificial neural networks adaptive capabilities. The results obtained on three meta-RL benchmark problems showed that this new architecture was able to perform better than classical RNN. The work reported in this paper could be extended along several lines. First, it would be interesting to explore other types of machine-learning problems where adaptation is required. Second, research work could also be carried out to further improve the NMN introduced here. For instance, one could introduce new types of parametric activation functions which are not linear, or spiking neurons. It would also be of interest to look at sharing activation function parameters per layer. Furthermore, analysing more in-depth the neuromodulatory signal (and its impact on activation functions) with respect to different more complex tasks could also be worth-while. Finally, let us emphasize that even if the results obtained by our NMN are good and also rather robust with respect to a large choice of parameters, further research is certainly still needed to better characterise their performances.
This paper introduces neuromodulation in artificial neural networks.
{"src": "scitldr-AIC_train", "paper_id": "H1xI7XYULr"}