Prefrontal cortex (PFC) is a part of the brain which is responsible for behavior repertoire. Inspired by PFC functionality and connectivity,  as well as human behavior formation process, we propose a novel modular architecture of neural networks with a Behavioral Module (BM) and corresponding end-to-end training strategy.   This approach allows the efficient learning of behaviors and preferences representation. This property is particularly useful for user modeling (as for dialog agents) and recommendation tasks, as allows learning personalized representations of different user states.   In the experiment with video games playing, the resultsshow that the proposed method allows separation of main task’s objectives andbehaviors between different BMs. The experiments also show network extendability through independent learning of new behavior patterns. Moreover, we demonstrate a strategy for an efficient transfer of newly learned BMs to unseen tasks. Humans are highly intelligent species and are capable of solving a large variety of compound and open-ended tasks. The performance on those tasks often varies depending on a number of factors. In this work, we group them into two main categories: Strategy and Behaviour. The first group contains all the factors leading to the achievement of a defined set of goals. On the other hand, Behaviour is responsible for all the factors not directly linked to the goals and having no significant effect on them. Examples of such factors can be current sentiment status or the unique personality and preferences that affect the way an individual makes decisions. Existing Deep Networks have been focused on learning of a Strategy component. This was achieved by optimization of a model for defined sets of goals, also the goal might be decomposed into sub-goals first, as in FeUdal Networks BID29 or Policy Sketches approach BID1 . Behavior component, in turn, obtained much less attention from the DL community. Although some works have been conducted on the identification of Behavior component in the input, such as works in emotion recognition BID15 BID11 BID17 . To the best of our knowledge, there was no previous research on incorporation on Behavior Component or Behavior Representation in Deep Networks before. Modeling Behaviour along with Strategy component is an important step to mimicking a real human behavior and creation of robust Human-Computer Interaction systems, such as a dialog agent, social robot or recommendation system. The early work of artificial neural networks was inspired by brain structure BID9 BID16 , and the convolution operation and hierarchical layer design found in the network designed for visual analytic are inspired by visual cortex BID9 BID16 . In this work, we again seek inspiration from the human brain architecture. In the neuroscience studies, the prefrontal cortex (PFC) is the region of the brain responsible for the behavioral repertoire of animals BID18 ). Similar to the connectivity of the brain cortex (as shown in Figure 1 ), we hypothesize that a behavior can be modeled as a standalone module within the deep network architecture. Thus, in this work, we introduce a general purpose modular architecture of deep networks with a Behavioural Module (BM) focusing on impersonating the functionality of PFC.Apart from mimicking the PFC connectivity in our model, we also borrow the model training strategy from human behavior formation process. As we are trying to mimic the functionality of a human brain we approached the problem from the perspective of Reinforcement Learning. This approach also aligns with the process of unique personality development. According to BID6 and BID5 unique personality can be explained by different dopamine functions caused by genetic influence. These differences are also a reason for different Positive Emotionality (PE) Sensory Cortex (Conv Layers) PFC (Behavior Module) Motor Cortex (FC layers) Figure 1: Abstract illustration of the prefrontal cortex (PFC) connections of the brain BID18 and corresponding parts of the proposed model. patterns (sensitivity to reward stimuli), which are in turn a significant factor in behavior formation process BID5 . Inspired by named biological processes we introduce extra positive rewards (referring to positive-stimuli or dopamine release, higher the reward referring to higher sensitivity) to encourage specific actions and provoke the development of specific behavioral patterns in the trained agent.To validate our method, we selected the challenging domain of classic Atari 2600 games BID2 , where the simulated environment allows an AI algorithm to learn game playing by repeatedly seek to understand the input space, objectives and solution. Based on this environment and an established agent (i.e. Deep Q-Network (DQN) BID20 ), the behavior of the agent can be represented by preferences over different sets of actions. In other words, in the given setting, each behaviour is represented by a probability distribution over given action space. In real-world tasks, the extra-reward can be represented by the human satisfaction by taken action along with the correctness of the output (main reward).Importantly , the effect of human behavior is not restricted to a single task and can be observed in various similar situations. Although it is difficult to correlate the effect of human behavior on completely different tasks, it is often easier to observe akin patterns in similar domains and problems. To verify this , we study two BM transfer strategies to transfer a set of newly learned BMs across different tasks. As a human PFC is responsible for behavior patterns in a variety of tasks, we also aim to achieve a zero-shot transfer of learned modules across different tasks.The contributions of our work are as follow:• We propose a novel modular architecture with behavior module and a learning method for the separation of behavior from a strategy component.• We provide a 0-shot transfer strategy for newly learned behaviors to previously unseen tasks. The proposed approach ensures easy extendability of the model to new behaviors and transferability of learned BMs.• We demonstrate the effectiveness of our approach on video games domain. The experimental results show good separation of behavior with different BMs, as well as promising results when transfer learned BMs to new tasks. Along with that, we study the effects of different hyper-parameters on the behavior separation process. In this work, we have proposed a novel Modular Network architecture with Behavior Module, inspired by human brain Pre-Frontal Cortex connectivity. This approach demonstrated the successful separation of the Strategy and Behavior functionalities among different network components. This is particularly useful for network expandability through independent learning of new Behavior Modules. Adversarial 0-shot transfer approach showed high potential of the learned BMs to be transferred to unseen tasks. Experiments showed that learned behaviors are removable and do not degrade the performance of the network on the main task. This property allows the model to work in a general setting, when user preferences are unknown. The results also align with human behavior formation process. We also conducted an exhaustive study on the effect of hyper-parameters on behavior learning process. As a future work, we are planning to extend the work to other domains, such as style transfer, chat bots, and recommendation systems. Also, we will work on improving module transfer quality. In this appendix, we show the details of our preliminary study on various key parameters. The experiments were conducted on the Behavior Separation task.
Extendable Modular Architecture is proposed for developing of variety of Agent Behaviors in DQN.
{"src": "scitldr-AIC_train", "paper_id": "Syl6tjAqKX"}