text:
Multi-view Regularized Gaussian Processes

Gaussian processes (GPs) have been proven to be powerful tools in various areas of machine learning. However, there are very few applications of GPs in the scenario of multi-view learning. In this paper, we present a new GP model for multi-view learning. Unlike existing methods, it combines multiple views by regularizing marginal likelihood with the consistency among the posterior distributions of latent functions from different views. Moreover, we give a general point selection scheme for multi-view learning and improve the proposed model by this criterion. Experimental results on multiple real world data sets have verified the effectiveness of the proposed model and witnessed the performance improvement through employing this novel point selection scheme.

Introduction
Gaussian processes (GPs) [10] are flexible and popular Bayesian nonparametric tools for probabilistic modeling. Without giving concrete functional forms, they can be employed to define distributions over functions. As a kind of effective probabilistic models, they provide estimations of the uncertainty of predictions. With many convenient properties, GPs are widely used for various applications in machine learning and statistics. For instance, GPs have progressed dramatically in semi-supervised learning [9,23,14], active learning [8,22], multi-task learning [19,3], reinforcement learning [5,11], and time series modeling [4,21].
Standard GPs only deal with single view data. However, in real world, many data involve multi-view information, which may come from different feature extractors or different domains. For example, in web-page classification, a web-page can be described by its content and its hyperlink structure. In image classification, an image can be represented by its color, texture, shape, and so on. Therefore, recently, multi-view learning has aroused wide concern in machine learning. There are increasing number of algorithms proposed for multi-view learning, which can mainly be divided into two major categories [15]: co-training style algorithm [2,16] and co-regularization style algorithms [20,6]. However, GPs, as efficient and elegant methods in machine learning, have very few applications in multi-view learning [20,13]. Our work extends the GPs to the scenario of multi-view learning.
Existing multi-view learning methods which involve the GPs can be classified into two groups: Bayesian co-training [20] and subspace learning [13,17]. The Bayesian co-training approach [20] is a Bayesian undirected graphical model, which pays attention to semi-supervised multi-view learning. The conditional independence between the output y of each data and latent functions f j for each view is ensured by involving a latent function f c [18]. On the other hand, the subspace learning methods [13,17] use the GPs as tools to construct a latent variable model which could tackle the task of non-linear dimensional reduction. In contrast to those existing methods, inspired by the thought of co-regularization, our work focuses on directly extending the GPs to the context of the multi-view learning via the posterior consistency regularization, leading to elegant inference and optimization.
Our method models the classifier of each separated view as a Gaussian process. We optimize hyperparameters of the GPs by maximizing weighted average of marginal likelihood on each view and minimizing the discrepancy among the posterior distributions of the latent function on each view. As the Kullback-Leibler (KL) divergence [7] is a frequently used measure for describing the difference between two probability distributions, we employ it to characterize the discrepancy among the posterior distributions. Moreover, as data sets in real word are complex and may be seriously contaminated by noises, the sufficiency assumption, i.e., each view is sufficient for classification on its own, and the compatibility assumption, i.e., the target functions of all the views predict the same labels with a high probability [18], may fail in some cases. In consideration of these situations, we improve our model by a selective regularization idea, which is different from previous multi-view methods. In the experiments, we have compared the improved method with the original model to verify the effectiveness of the idea of the selective regularization on real word data sets.
The highlights of our work can be summarized as follows. First, we present a new GP model for multi-view learning, which extends the GPs to the scenario of multi-view learning by simple and elegant posterior consistency regularization. Secondly, our models automatically learn which views of the data should be trusted more when predicting class labels. Finally, we give a general point selection scheme for multi-view learning to deal with the situations where the sufficiency and compatibility assumptions fail, and propose the multi-view GPs with selective posterior consistency inspired by this criterion.
The remainder of this paper is organized as follows. Section 2 reviews the Gaussian processes. In Section 3, we present the multi-view GPs with posterior consistency (MvGP1), our first algorithm, covering the principles and detailed inference and learning in the proposed model. Moreover, we improve the MvGP1 to the multi-view GPs with selective posterior consistency (MvGP2) based on a general idea of the consistent set in Section 4. Experimental results are provided in Section 5. Finally, we conclude this paper and discuss the future work in Section 6.

Gaussian Processes
This section briefly reviews the Gaussian process (GP) model.
GPs are powerful tools for classification and regression. A Gaussian process is a collection of random variables, any finite number of which have a joint Gaussian distribution [10]. The GP is widely used to describe a distribution over functions, and can be completely specified by its mean function and covariance function. Formally, suppose that the training set , where x i ∈ R M is the ith input, and y i ∈ R is the corresponding label. Denote X = [x 1 , x 2 , ..., x N ] T , and y = [y 1 , y 2 , ..., y N ] T . Following standard notations for GPs, the prior distribution for the latent functions f is assumed to be Gaussian, f |X ∼ N (0, K), with a zero mean and a covariance matrix K, whose element K ij is determined by the covariance function k(x i , x j ). Diverse covariance functions can be employed in GPs. In this paper, we select a commonly used covariance function, the squared exponential kernel, where s 2 f is the signal variance, and l is the length-scale of the covariance. The Gaussian likelihood for regression can be written as y|f ∼ N (f , σ 2 I) , and after integrating out the hidden variables, the marginal likelihood is y|X ∼ N (0, K + σ 2 I).
Under these settings, the posterior of the latent functions should be where µ = K(K + σ 2 I) −1 y and Σ = K − K(K + σ 2 I) −1 K are the mean and the covariance of the posterior distribution, respectively. We use Θ to denote the hyperparameters in the Gaussian process regression model, that is, Θ = {s 2 f , l, σ 2 }. These hyperparameters can be obtained by generalized maximum likelihood. In generalized maximum likelihood, we calculate the negative logarithmic marginal likelihood of the samples, L(Θ) = − log p(y|X, Θ), and then minimize L(Θ) with respect to Θ.
For a new point x * , the prediction is also Gaussian, Here, k is the covariance function, and k * is the vector of covariance function values between x * and the training data X.
Standard GPs only handle single view data. However, collected data sets in the real world can often be represented by multiple views which may come from different feature extractors or various measurement modalities. As GPs are popular tools in machine learning, we propose to develop the GPs to multi-view learning.

Multi-view GPs with Posterior Consistency
In this section, we present the formulation of the multi-view GPs with posterior consistency (MvGP1) and show the corresponding inference and optimization in the proposed model. We first pay attention to two views learning tasks, and then give illustrations about the extensions to the scenario having more than two views.

Model Representation
Assume that the two views training set First, on account of leveraging the information in the separated single view, we simply assume that each view of data is modeled by a GP. That is, the prior distribution for the latent functions f 1 on the first view and f 2 on the second view are supposed to be Gaussian, i.e. p(f 1 |X) = N (0, K 1 ), and p(f 2 |Z) = N (0, K 2 ), where K 1 and K 2 are covariance matrixes determined by the corresponding covariance functions of two views, respectively. In our model, the covariance function is the squared exponential kernel as mentioned in (1). Although the Gaussian noise model is originally developed for regression, it has also been proved effective for classification, and its performance is typically comparable to the more complex probit and logit likelihood models used in classification problems [1]. Therefore, we use the Gaussian regression likelihood for our classification task to enjoy the elegant exact inference. The Gaussian likelihood for regression on the first view is p(y|f 1 ) = N (f 1 , σ 2 1 I), and the likelihood on the second view is p(y|f 2 ) = N (f 2 , σ 2 2 I). Secondly, we also need leverage the consistence between two views. The KL divergence [7] can measure the discrepancy between two distributions. Thus, we use the KL divergence between the posterior distributions on two views to regularize the objective function in MvGP1, enforcing the consistence between two views.
Suppose the posterior distribution of the latent function f 1 on the first view is and the posterior distribution of the latent function f 2 on the second view is Based on the above setting, our objective function of MvGP1 is is the marginal likelihood on the first view, is the marginal likelihood on the second view, and the KL divergences between the posterior distributions f 1 and f 2 are and Since the KL divergence is not a symmetrical quantity, that is, KL(p 1 ||p 2 ) = KL(p 2 ||p 1 ), and we have no general method to determine which one is better for measuring the discrepancy between the two posterior distribution p 1 and p 2 , we construct a symmetrical quantity based on the above KL divergences, i.e., Parameters µ 1 , µ 2 , Σ 1 , and Σ 2 in (9) and (10) are provided as follows. The mean and covariance of the posterior distribution for the latent function The mean and covariance of the posterior distribution for the latent function f 2 are

Inference and Optimization
In our model, we consider the hybrid prediction function sign(af 1 + (1 − a)f 2 ), a ∈ [0, 1]. For a new point {x * , z * }, the prediction distribution f 1 * of the first view and the prediction distribution f 2 * of the second view are also Gaussian.
The mean of f 1 * is k 1 * T (K 1 + σ 2 1 I) −1 y, and the covariance is k 1 (x * , x * ) − k 1 * T (K 1 +σ 2 1 I) −1 k 1 * . Here, k 1 is the covariance function, and k 1 * is the vector of covariance function values between x * and the training data X. The mean of f 2 * is k 2 * T (K 2 + σ 2 2 I) −1 y, and the covariance is k 2 (z * , z * ) − k 2 * T (K 2 + σ 2 2 I) −1 k 2 * . Here, k 2 is the covariance function, and k 2 * is the vector of covariance function values between z * and the training data Z. According to the hybrid prediction function, we give our prediction on the new data point {x * , z * }.
As GPs are nonparametric models, the related hyperparameters need to be determined. In the context of a single view, we often obtain the hyperparameters via generalized maximum likelihood. In the case of two views, we optimize the hyperparameters collaboratively by two views. Co-regularization approaches often expect the predictions for the same observation of different views to be consistent to optimize the parameters. Similarly, in the context of Bayesian learning, we want the posterior distributions of the latent functions of the same observation across different views to be consistent to obtain the hyperparameters, which can be realized via minimizing the above objective function shown in (6).
The hyperparameters in our model can be divided into two classes: the tradeoff hyperparameters a, b and the GP related hyperparameters, which include the hyperparameters s 2 f , l in the covariance functions and the noise hyperparameters σ 2 in the likelihood. We use Θ = {s 2 f 1 , l 1 , σ 2 1 , s 2 f 2 , l 2 , σ 2 2 } to denote hyperparameters in the second group, where s 2 f 1 , l 1 , σ 2 1 are hyperparameters related to the first view, and others are hyperparameters related to the second view. The GP related hyperparameters Θ are optimized by the gradient descent method. As our model has an elegant formulation, the gradients with respect to Θ also have graceful forms, and the code can be easily implemented by the existing toolbox [12]. Following the above parameter notations, the gradient w.r.t the s f 1 is The gradients with respect to other hyperparameters in Θ are similar with (11), and hence we omit them here. The trade-off hyperparameters a and b are obtained through grid search. We summarize MvGP1 in Algorithm 1.

Extension to Multiple Views
In the above sections, we take two views as an example to illustrate our model MvGP1. This model can be easily extended to multiple views because of the elegant formulation. The posterior distribution of the latent function on each view is the Gaussian distribution. Moreover, the KL divergence [7] between two Gaussian distributions can be calculated analytically, i.e., for two Gaussian distributions N 0 = N (µ 0 , Σ 0 ) and N 1 = N (µ 1 , Σ 1 ), we have KL(N 0 ||N 1 ) =

Algorithm 1 Multi-view GPs with Posterior Consistency
. Output: accuracy acc, trade-off parameters a, b, and GP related hyperparameters Θ.
1: initialize Θ randomly. 2: for k = 1 to 10 do 3: Divide the training data {xi, zi, for a, b in the search grids do 5: while termination conditions are not satisfied do 6: Update Θ by gradient descent to minimize L1 in (6). 7: end while 8: Calculate the predictions by sign(af1 + (1 − a)f2) on the validation set. 9: Calculate the accuracy accv on the validation set if accv is larger than the accuracy on the last iteration then 11: record the trade-off parameters a, b.
Therefore, we can extend MvGP1 to multiple views by regularizing the weighted logarithm with the KL divergences between every pair of the distinct posterior distributions. Following the notations in MvGP1, given a data set which involves K views, p(y|X k ) denotes the marginal distribution of the kth view, and p i represents the corresponding posterior distribution of the latent function f i . The objective function in multiple views is

Multi-view GPs with Selective Posterior Consistency
When the predictions from different views are not consistent on some data points, the multi-view sufficiency assumption and compatibility assumption can not be satisfied on the whole data set. In this context, it is not appropriate to enforce the posteriors on the whole data set across the different views as similar as possible. For instance, in some cases, the input data of some views may be largely affected by the noises, which may lead the predictions of these points in these views totally different from those in other views. In these cases, the multi-view assumptions fail, and enforcing the predictions of all views on these data points consistent seems to be improper.
Considering the above problem, we improve MvGP1 and present multi-view GPs with selective posterior consistency (MvGP2). In this model, we modify the regularization term in the objective function to make the posterior distributions across different views on a subset of the data set other than the whole one as similar as possible. In order to find the proper subset, namely the consistent set, we first optimize the hyperparameters through MvGP1 on the training set and give the predictions for the training data on each view. Next, we select data points whose predictions on each view are all consistent and are also consistent with the true label to construct the consistent set. Finally, we optimize the hyperparameters with the analogous procedure to MvGP1 except that we regularize the posteriors only on the chosen consistent set.
In fact, the key idea of MvGP2 is to construct the consistent set and constrain the multi-view assumptions on the consistent set. We construct the consistent set by selecting the data points whose label predictions on each view are all equal and the same as true labels. Formally, given two views of the input data X = [x 1 , ..., x N ] T , Z = [z 1 , ..., z N ] T and the corresponding label data y = [y 1 , ..., y N ] T , we find a index set T = [t 1 , t 2 ..., t k ], (k <= N ) such that for each t ∈ T , the predictions for X t , Z t and the corresponding y t are all agreed. Then the consistent set is {x i , z i , y i } i∈T . Let X T ,Z T , and y T denote the corresponding data matrix of the consistent set. After constructing the consistent set, we modify the objective function to only restrict the KL divergence of the posterior distributions of the latent functions on the consistent set to be minimized, where p 1 ′ = p(f ′ 1 |X T , y T ) is the posterior distribution of the latent function f 1 ′ on the set {X T , y T }, and p 2 ′ = p(f 2 ′ |Z T , y T ) is the posterior distribution of the latent function f ′ 2 on the set {Z T , y T }. We summarize MvGP2 in Algorithm 2. The idea of constraining the multiview assumption on the consistent set other than the whole data set is a novel view in multi-view learning. In the real word, data are complex and noisy. It is likely that not all the data satisfy that the predictions on different views are equal. Moreover, we have verified this selective idea through the experiments on real word data sets in the following section.

Experiments
In this section, we performed experiments with our proposed MvGP1 and MvGP2 on multiple real word data sets. For comparison, we use three single-view methods corresponding to GPs named GP1, GP2, GP3, which use the first view, the second view, and the combination of two views, i.e., concatenating two views

Algorithm 2 Multi-view GPs with Selective Posterior Consistency
. Output: accuracy acc, trade-off parameters a, b, GP related hyperparameters Θ and the consistent index set T .
Calculate the predictions by fi1, fi2 on the training data xi and zi, respectively.

5:
if fi1 = fi2 && fi2 = yi then 6: add i into the set T 7: end if 8: end for 9: for k = 1 to 10 do 10: Divide the training data {xi, zi, for a, b in the search grids do 12: while termination conditions are not satisfied do 13: Update Θ by gradient descent to minimize L ′ 1 in (13). 14: end while 15: Calculate the predictions by sign(af1 + (1 − a)f2) on the validation set. 16: Calculate the accuracy accv on the validation set if accv is larger than the accuracy on the last iteration then 18: record to construct new high-dimensional data, respectively. In addition, we also compare our algorithms with a multi-view method SVM-2K [6], which is a two-view version of SVMs and is also inspired by the thought of co-regularization.

Data Sets
Web-Page The web-page data sets have been extensively used in multi-view learning, which consist of two-view web pages collected from computer science department web sits at four universities: Cornell university, university of Washington, university of Wisconsin, and university of Texas. The two views are words occurring in a web page and words appearing in the links pointing to that page. The documents are described by 1703 words in the content view, and by 569 links between them in the cites views. We list the statistical information about the four data sets in Table 1. The web pages are distributed over five classes: student, project, course, staff and faculty. We set the category with the greatest size to be the positive class (denoted as class 1), and all the other categories as the negative class (denoted as class 2) in each data set. Ionosphere Downloaded from UCI, 1 the ionosphere data set is collected by a system in Goose Bay, Labrador. This system involves a phased array of 16 highfrequency antennas with a total transmitted power on the order of 6.4 kilowatts.
The targets are free electrons in the ionosphere. Those showing evidence of some type of structure in the ionosphere are good radar returns, while those not showing the above phenomenon and whose signals pass through the ionosphere are bad returns. The data set consists of 351 examples in which 225 are "good" instances and 126 are "bad" instances. There is only one view in this data set, and we generate the other view via principal component analysis, resulting in two views, which have 35 and 24 dimensions, respectively.

Experimental Setting
In the experiments, we select 60% data in each data set as the training set, and the rest as the test set Other hyperparameters in MvGP1, MvGP2 and hyperparameters in GP1, GP2, GP3 are initialized randomly. As for the kernel function in GP1, GP2, GP3, MvGP1, and MvGP2, we all use the squared exponential kernel as mentioned in (1). We repeat the experiments for all the data sets five times and record the average accuracies and the corresponding standard deviations. We compared our proposed MvGP1, MvGP2 with multi-view method SVM-2K, and three single view methods GP1, GP2, and GP3. For SVM-2K, besides the prediction functions sign(f 1 ) and sign(f 2 ) from the separated views, we also consider the hybrid prediction function sign((f 1 + f 2 )/2).

Results
We present the average accuracies and standard deviations of all the methods on the webpage data sets and ionosphere data set in Table 2.
It is clearly shown in Table 2 that our proposed methods MvGP1 and MvGP2 are superior to GP1, GP2, GP3 and SVM-2K. We can also observe that MvGP2 further improves the performance over MvGP1, which benefits from the idea of using the selective posterior regularization other than the posterior regularization on the whole data sets to ensure the consistency.

Conclusion
In this paper, we have proposed MvGP1 which extends GPs to the scenario of learning with multiple views via the methods of posterior consistency regularization. This approach is very intuitive, resulting in an elegant objective function formulation. Experimental results on real-word web-page classification validate the effectiveness of the proposed MvGP1. Moreover, considering that the multiview assumptions may not be met on all data points, we have proposed MvGP2, which constructs a consistent set and constrains the posterior consistency regularization on the consistent set other than the whole data set, leading to further improvements of the performance. In fact, the idea of constraining the multiview assumptions on a selective consistent set other than the whole data set is general. It not only can be applied to GPs, but also can inspire other multi-view learning methods.
In the future, we will attempt to extend the proposed models to the scenario of big data, which may use Nyström methods or other approximate approaches.


A Deep Compositional Framework for Human-like Language Acquisition in Virtual Environment

We tackle a task where an agent learns to navigate in a 2D maze-like environment called XWORLD. In each session, the agent perceives a sequence of raw-pixel frames, a natural language command issued by a teacher, and a set of rewards. The agent learns the teacher's language from scratch in a grounded and compositional manner, such that after training it is able to correctly execute zero-shot commands: 1) the combination of words in the command never appeared before, and/or 2) the command contains new object concepts that are learned from another task but never learned from navigation. Our deep framework for the agent is trained end to end: it learns simultaneously the visual representations of the environment, the syntax and semantics of the language, and the action module that outputs actions. The zero-shot learning capability of our framework results from its compositionality and modularity with parameter tying. We visualize the intermediate outputs of the framework, demonstrating that the agent truly understands how to solve the problem. We believe that our results provide some preliminary insights on how to train an agent with similar abilities in a 3D environment.

(a) (b) Figure 1: Illustration of our XWORLD environment and the zero-shot navigation tasks. (a) Test command contains an unseen word combination. (b) Test command contains completely new object concepts that are learned from the recognition task in some previous sessions (a).
The development of a sophisticated language system is a very crucial part of achieving humanlevel intelligence for a machine. Language semantics, when grounded in perception experience, can encode knowledge about perceiving the world. This knowledge is transferred from task to task, which empowers the machine with generalization ability. It is argued that a machine has to go through physical experience in order to learn human-level semantics [Kiela et al., 2016], i.e., a process of human-like language acquisition. However, current machine learning techniques do not have a reasonably fast learning rate to make this happen. Thus we model this problem in a virtual environment, as the first step towards training a physical intelligent machine.
Human generalize surprisingly well when learning new concepts and skills through natural language instructions. We are able to apply an existing skill to newly acquired concepts with little difficulty. For example, a person who has learned how to execute the command "cut X with knife" when "X" equals to apple, will do correctly when "X" is something else he knows (e.g., pear), even though he may have never been asked to cut anything other than apple before. In other words, human have the zero-shot learning ability.
This paper describes a framework that demonstrates the zero-shot learning ability of an agent in a specific task, namely, learning to navigate in a 2D maze-like environment called XWORLD (Figure 1). We are interested in solving a similar task that is faced by a baby who is learning to walk and navigate, at the stage of learning his parents' language. The parents might give some simple navigation command consisting of only two or three words in the beginning, and gradually increase the complexity of the command as time goes by. Meanwhile, the parents might teach the baby the language in some other task such as object recognition. After the baby understands the language and masters the navigation skill, he could immediately navigate to a new concept that is learned from object recognition but never appeared in the navigation command before.
We train our baby agent across many learning sessions in XWORLD. In each session, the agent perceives the environment through a sequence of raw-pixel images, a natural language command issued by a teacher, and a set of rewards. The agent also occasionally receives the teacher's questions on object recognition whenever certain conditions are triggered. By exploring the environment, the agent learns simultaneously the visual representations of the environment, the syntax and semantics of the language, and how to navigate itself in the environment. The whole framework employed by the agent is trained end to end from scratch by gradient descent. We test the agent under four different command conditions, three of which require that the agent generalizes to interpret unseen commands and words, and that the framework architecture is modular so that other modules such as perception and action will still work properly under such circumstance. Our experiments show that the agent performs equally well (∼ 90% average success rate) in all conditions. Moreover, several baselines that simply learn a joint embedding for image and language yield poor results.
In summary, our main contributions are two-fold: • A new deep reinforcement learning (DRL) task that integrates both vision and language. The language is not pre-parsed [Sukhbaatar et al., 2016] or -linked [Mikolov et al., 2015, Sukhbaatar et al., 2016 to the environment. Instead, the agent has to learn everything from scratch and ground the language in vision. This task models a similar scenario faced by a learning child. • The zero-shot learning ability by leveraging the compositionality of both the language and the network architecture. We believe that this ability is a crucial component of human-level intelligence.

Related Work
Our work is inspired by the research of multiple disciplines. Our XWORLD is similar to the MazeBase environment [Sukhbaatar et al., 2016] in that both are 2D rectangular grid world. One big difference is that their quasi-natural language is already parsed and linked to the environment. They put more focus on reasoning and planning but not language acquisition. On the contrary, we emphasize on how to ground the language in vision and generalize the ability of interpreting the language. There are several challenging 3D environments for RL such as ViZDoom [Kempka et al., 2016] and DeepMind Lab [Beattie et al., 2016]. The visual perception problems posed by them are much more difficult than ours. However, these environments do not require language understanding. Our agent needs to learn to interpret different goals from different natural language commands.
Our setting of language learning shares some similar ideas of the AI roadmap proposed by Mikolov et al. [2015]. Like theirs, we also have a teacher in the environment that assigns tasks and rewards to the agent. The teacher also provides additional questions and answers to the agent in an object recognition task. Unlike their proposal of entirely using linguistic channels, our tasks involve multiple modalities and are more similar to human experience.
The importance of compositionality and modularity of a learning framework has been discussed at length in cognitive science by Lake et al. [2016]. The compositionality of our framework is inspired by the ideas in Neural Programmer [Neelakantan et al., 2016] and Neural Module Networks (NMNs) [Andreas et al., 2016a,b]. Neural Programmer is trained with backpropagation by employing soft operations on databases. NMNs assemble several primitive modules according to questions in Visual Question Answering (VQA). It depends on an external parser to convert each sentence to one or several candidate parse trees and thus cannot be trained end to end. We adapt their primitive modules to our framework with differentiable computation units to enable gradient calculation. Their subsequent work (N2NMNs) [Hu et al., 2017] learns end to end to optimize over the full space of parse trees, but relies on candidate parse trees as an expert policy used by a behavior cloning procedure to initialize the policy parameters.
Our recognition task is essentially image VQA [Antol et al., 2015, Gao et al., 2015, Ren et al., 2015, Lu et al., 2016, Andreas et al., 2016a,b, Teney and Hengel, 2016. The navigation task can also be viewed as a VQA problem if the actions are treated as answer labels. Moreover, it is a zero-shot VQA problem (i.e., test questions containing unseen concepts) which has not been well addressed yet.
Our language acquisition problem is closely related to some recent work on grounding language in images and videos [Yu and Siskind, 2013, Rohrbach et al., 2016, Gao et al., 2016. The navigation task is also relevant to robotics navigation under natural language command [Chen and Mooney, 2011, Tellex et al., 2011, Barrett et al., 2015. However, they either assume annotated navigation paths in the training data or do not ground language in vision. As XWORLD is a virtual environment, we currently do not address mechanics problems encountered by a physical robot, but focus on its mental model building.

XWORLD Environment
We first briefly describe the XWORLD environment. More details are in Appendix 8.5. XWORLD is a 2D grid world ( Figure 1). An agent interacts with the environment over a number of time steps T , with four actions: up, down, left, and right. It does so for many sessions. At the beginning of each session, a teacher starts a timer and issues a natural language command asking the agent to reach a location referred to by objects in the environment. There might be other objects as distractors. Thus the agent needs to differentiate and navigate to the right location. It perceives the entire environment through RGB pixels with an egocentric view ( Figure 2c). If the agent correctly executes the command before running out of time, it gets a positive reward. Whenever it hits a wall or steps on an object that is not the target, it gets a negative reward. The agent also receives a small negative reward at every step as a punishment for taking too much time. After each session, the environment is reset randomly.
Some example commands are (the parentheses contain environment configurations that are withheld from the agent, same below): • Please navigate to the apple. (There is an apple, a banana, an orange, and a grape.) • Can you move to the grid between the apple and the banana? (There is an apple and a banana. The apple and the banana are separated by one empty grid.) • Could you please go to the red apple? (There is a green apple, a red apple, and a red cherry.) The difficulty of this navigation task is that, at the very beginning the agent knows nothing about the language: every word appears equally meaningless. After trials and errors, it has to figure out the language syntax and semantics in order to correctly execute the command.
While the agent is exploring the environment, the teacher also asks object-related questions whenever certain conditions are triggered (all conditions are listed in Appendix 8.5). The answers are always single words and provided by the teacher for supervision. Some example QA pairs are: We expect the agent to transfer the knowledge exclusively learned from this recognition task to the navigation task to execute zero-shot commands. Both the commands and questions are generated from templates (Appendix 8.5) and are made like human-elicited sentences.

Compositional Framework for Zero-shot Navigation
The zero-shot navigation problem can be formulated as training a policy with a grounding model: π(y|E, a), with a = Grounding(E, C) that works correctly for a new test command C which never appears in the training set. In the above formulation, y is the agent's navigation action, E is the perceived environment, a is the navigation goal attention computed by grounding the teacher's command C in the environment E. Note that the policy π(y|E, a) depends on a but not directly on C, thus our key problem is essentially how to ground a new command to get the correct attention map.
The proposed framework for solving this problem contains four major modules: perception (Section 4.1), recognition (Section 4.2), language (Section 4.3), and action (Section 4.4). The design of  PI Language grounding and recognition have to be reduced to (approximately) the same problem. This ensures that a new word exclusively learned from recognition has already been grounded for attention. PII The language module must be compositional. It must process a sentence while preserving the (major) sentence structure. One example would be a parser that outputs a parse tree. PIII Inductive bias [Lake et al., 2016] must be learned from training sentences. The language module must know how to parse a sentence that has a previously seen structure even if a word position of the sentence is filled with a completely new word.
There is no existing framework for image captioning or VQA exhibits all such properties. While in theory, binding parameters or embeddings between the language module and the recognition module might satisfy the first property, in practice a naive binding without further architecture support hardly works. It is also challenging for a simple (gated) Recurrent Neural Network (RNN) to satisfy the last two properties when a sentence has a rich structure. Our framework detailed below is designed exactly for these three properties.

Generating Feature Map
We compute a feature map F as an abstraction of the environment and the agent's ego-centric geometry, on which we perform the recognition and grounding of the entire dictionary. Because of this general computation purpose, the map should not only encode visual information, but also provide spatial features. The feature map is formed by stacking two feature maps F v and F s along the channel dimension. The perception module ( Figure 2c) transforms an environment image into the visual feature map F v through a Convolutional Neural Network (CNN), whose output has a spatial dimension equal to the number of grids N in the environment, where each pixel corresponds to a grid. The spatial feature map F s is a set of randomly initialized parameters. Intuitively, F v is responsible for the agent's visual sense while F s is responsible for its direction sense. The agent needs to learn to associate visually-related words with F v while associate spatially-related words with F s .

Recognizing and Grounding a Single Word
To achieve zero-shot navigation, we start with reducing the single-word recognition and grounding to the same problem, after which we will extend the reduction to the grounding of a sentence of multiple words (Section 4.3). The single-word grounding problem is defined as where F ∈ R D×N is a feature map, m is the word index, a ∈ R N is a spatial attention map, and N is the number of pixels. Imagine that the feature map is a collection of slots of features, given a word m, we need to find the corresponding slots (highlighted by attention) that match the word semantics.
In a reverse way, the recognition problem is defined as F, a → m. The recognition module (Figure 2b) outputs the most possible word for a feature map weighted by attention. This reflects the question answering process after the agent attending to a region of the environment image according to a question. In the extreme case, when a is a one-hot vector, the module extracts and classifies one slot of feature. Suppose that the extracted feature f = Fa is directly fed to a Softmax layer P where Sof tmax M denotes the Softmax function over M classes and s m is the parameter vector of the mth class. Notice that the multiplication s m F essentially treats s m as a 1 × 1 filter and convolves it with every feature on F. This computation operates as if it is solving the grounding problem in Eq. 1, and it produces an attention map s m F which is optimized towards the following property where m * is the groundtruth class for f . In other words, this property implies that s m is grounded in F where the feature of class m locates. This is exactly where we need to ground word m with embedding e m . Thus we set the Softmax matrix as the transpose of the word embedding table (i.e., s m = e m ). So far, we have satisfied property PI for the single-word grounding case, namely, if a new word M + 1 is trained to be recognized accurately, then it can also be grounded correctly without being trained to be grounded before.
There is still one flaw in the above definition of recognition. An attention map only tells the agent where, but not what, to focus on. Imagine a scenario in which the agent stands to the east of a red apple. Given an attention map that highlights the agent's west, should the agent answer west, apple, or red? It turns out that the answer should also depend on the question itself, i.e., the question intention has to be understood. Therefore, we redefine the recognition problem as: F, a, Φ(Q) → m, where Q is the question and Φ computes its intention. In our framework, Φ is modeled as a gated RNN [Cho et al., 2014] which outputs an embedding mask ∈ [0, 1] D through Sigmoid. Then the Softmax distribution becomes P Figure 2b), where • denotes element-wise multiplication. To make property PI still possible, we modify the grounding problem so that a word embedding is masked before it is grounded: where Ψ is modeled as a projection 1 that outputs an embedding mask through Sigmoid. Intuitively, it enhances the grounding with the ability of selecting features that are relevant to the word being grounded, and this selection is only decided by the word itself. Both functions Φ and Ψ serve as the agent's decision makers on which features to focus on given particular language tokens. They operate on the same intermediate word representation referred to as functionality embedding that is a projection of the word embedding (Figure 4 Appendix 8.7). Functionality embedding is exclusively used for computing embedding masks. Details of mask computation are in Figure 5 Appendix 8.7.

Grounding a Sentence
Now we extend single-word grounding to sentence grounding while trying to preserve property PI. The high-level idea is that we can ground a sentence by grounding individual words of that sentence through several steps, one or more words per step at a time. The questions we will answer in this section are how to: 1) decide which words to ground at each step, 2) combine the grounding results of these steps to reflect the grounding of the sentence, and 3) make the whole process differentiable.
The core of our language module ( Figure 2a) is a differentiable programmer (Figure 2 Right). We treat each sentence (either navigation command or recognition question) as a program command. The programmer converts the command implicitly to a set of operations (see Appendix 8.2 for an analogy between the operations of NMNs and ours). The programming result is an attention map. For navigation, the attended regions indicate the target locations. For recognition, they indicate which part of the environment to be recognized. The programmer programs a command in several steps. At each step, the programming consists of three substeps: attending, grounding, and combining. The programmer first softly attends to a portion of a sentence. Once one or more words are attended, they are extracted, modulated by the embedding mask, and grounded in image. The resulting attention map is selectively combined with the maps in the previous steps (detailed below). In this way, the programmer uses the dynamic word attention, the grounding process modulated by masks, and the selective map combination to implicitly output a network layout and model the structural process of grounding a sentence in image. This approximately satisfies property PII.
The word attention is determined by the global sentential context, the local context at each word, and the history of a programmer RNN. Both global and local contexts are computed based on an intermediate word representation referred to as syntax embedding that is a projection of the word embedding (Figure 4 Appendix 8.7). Intuitively, the syntax embedding can be trained to encode a group of words similarly if they have the same syntactic meaning. Thus when a sentence contains a new word that was only trained in recognition, the programmer RNN has the inductive bias of interpreting the sentence as if the new word is an existing word with the same syntactic meaning (property PIII). The details of word attention are in Appendix 8.1 and omitted here due to page limit.
With the word attention at each step s, we compute the averaged word embedding, mask it given the corresponding averaged functionality embedding, and convolve it with the feature map as in the single-word case (Eq. 2). The convolution result is input to a Softmax layer to get a sum-toone attention map a s . Assume that we cached an attention map a s−1 in the previous step. The programmer approximates the 2D translation of spatial attention by a s = o(a s ) * a s−1 , where o denotes the 180-degree rotation. Then the programmer caches a new attention map through a forget gate: a s = (1 − σ)a s−1 + σa s , where the gate σ ∈ [0, 1] is computed from the current state of the programmer RNN (Figure 2 Right). We set a 0 = i where i is a map whose center pixel is one and the rest are all zeros. Finally, the attention map a at the last step is used as the output of the programmer.

Navigation
A navigation attention map a only defines the target locations. To navigate, the agent needs to know the surrounding environment including obstacles and distractors. Our perception module (Figure 2c) convolves F v with a 1 × 1 filter to get the environment map. We stack the two maps and input them to an action CNN whose output is projected to a state vector q that summarizes the environment. The state vector is further projected to a distribution π(q, y) over the actions. At each time step, the agent takes action y with a probability of α · 0.25 + (1 − α) · π(q, y), where α is the rate of random exploration. The state is also projected to a scalar V (q) to approximate value function.

Training
Our training objective contains two sub-objectives L(θ) = L RL (θ) + L SL (θ), one for navigation and the other for recognition, where θ are the joint parameters of the framework. Most parameters are shared between the two tasks 2 . We model the recognition loss L SL as the multi-class cross entropy which has the gradients , where E Q is the expectation over all the questions asked by the teacher in all training sessions, m is the correct answer to each question, and f θ is the corresponding feature. We compute the navigation loss L RL (θ) as the negative expected reward −E π θ [r] the agent receives by following its policy π θ . With the Actor-Critic (AC) algorithm [Sutton and Barto, 1998], we have the approximate gradients where θ − are the target parameters that are periodically (every J minibatches) copied from θ, r is the immediate reward, γ is the discount factor, q θ − is the next state after taking action y at state q θ , and π θ and V θ are the policy and value output by the action module. Since the expectations E Q and E π θ are different, we optimize the two sub-objectives separately over the same number of minibatches. For effective training, we employ Curriculum Learning [Bengio et al., 2009] and Experience Replay [Mnih et al., 2015] with Prioritized Sampling [Schaul et al., 2016] (Appendix 8.6).

Experiments
We use Adagrad [Duchi et al., 2011] with a learning rate of 10 −5 for Stochastic Gradient Descent (SGD). In all experiments, we set the batch size to 16 and train 200k batches. The target parameters θ − are updated every J = 2k batches. The reward discount factor γ is set to 0.99. All the parameters have a default weight decay equal to 10 −4 × batch size. For each layer, by default its parameters have zero mean and a standard deviation of 1/ √ K, where K is the number of parameters of that layer. The agent has 500k exploration steps in total, and the exploration rate α decreases linearly from 1 to 0. We fix the number of programming steps as 3, and train each model with 4 random initializations. The whole framework is implemented with PaddlePaddle (https://github.com/PaddlePaddle/Paddle/) and trained end to end. More implementation details are in Appendix 8.3.
Baselines We first compare with five baselines in a normal setting.
• SimpleAttention We modify our framework by replacing the programmer with a simple attention model. Given a sentence, we use an RNN to output an embedding which is convolved as a 3 × 3 filter with the visual feature map to get an attention map. The rest of our framework is unchanged (Figure 7 Appendix 8.7). This baseline is an ablation to show the necessity of the programmer. • NoTransShare We do not tie the word embedding table to the transposed Softmax matrix. In this case, language grounding and recognition are not the same problem. This baseline is an ablation to show the impact of the transposed sharing on the training convergence. • VIS-LSTM Following Ren et al. [2015], we use CNN to get an image embedding which is then projected to the word embedding space and used as the first word of the sentence. The sentence goes through an RNN whose last state is used for navigation and recognition (Figure 8 Appendix 8.7). • Multimodal We implement a multimodal framework . This framework uses CNN to get an image embedding and RNN to get a sentence embedding. Then the two embeddings are projected to a common feature space for navigation and recognition (Figure 9 Appendix 8.7). • SAN We replace our language module with the attention process of Stacked Attention Network (SAN) , with the difference of training a CNN from scratch, instead of using a pretrained one, to accommodate to XWORLD. The rest of our framework is unchanged. For each method, we test 10k sessions in total over four navigation subtasks nav_obj, nav_col_obj, nav_nr_obj, and nav_bw_obj (Appendix 8.5). We compute the success rates where success means that the agent reaches the target location in time in a test session. The training reward curves and the success rates are shown in Figure 3 and Table 1a, respectively. VIS-LSTM and Multimodal have poor results because they do not ground language in vision. SAN has the same overall architecture with our framework but with a different language module, which requires it to generate accurate navigation attention map because the navigation action solely depends on attention maps rather than visual image features as in VIS-LSTM and Multimodal. This assumption is a quite strong one and our experiment result shows that even though SAN gets ∼100% recognition accuracy, its action module hardly works. Our explanation is that SAN reasons spatial relationships based on the encoding of multiple neighboring objects in a single feature at each convolution location (i.e., a broad receptive field), which can easily lead to attention diffusion. Its recognition module might be able to still correctly classify given diffused attention, but the action module requires far more accurate attention as the navigation signal for the agent. Surprisingly, NoTransShare converges much slower than our framework does. One possible reason is that the correct behavior of the language module is hard to be found by SGD if no constraint on word embedding is imposed. Although SimpleAttention is able to perform well, without word-level attention, its ability of sentence understanding is limited.

Zero-shot Navigation
Our primary question is whether the agent has the zero-shot navigation ability of executing previously unseen commands. We setup four command conditions for training the agent: • Standard The training command set has the same distribution with the test command set.
• NC Some word combinations are excluded from the training command set, even though all the words are in it. We specifically consider three types of word combinations: (object, location), (object, color), and (object, object). We enumerate all combinations for each type and randomly hold out 10% from the teacher in navigation. • NWNav&NWNavRec Some object words are excluded from navigation training, and are trained only in recognition and tested in navigation as new concepts. NWNavRec guarantees that the new words will not appear in questions but only in answers while NWNav does not. We randomly hold out 10% of the object words.  We use SimpleAttention for comparison as the other baselines have far worse performance than it in the previous experiment. Both SimpleAttention and our framework are trained under each condition without changing the their hyperparameters. For testing, we put the held-out combinations/words back to the commands (i.e., Standard condition) and again test 10k sessions. Table 1b contains the success rates. Our framework gets almost the same success rates for all the conditions, and obtains high zero-shot success rates. The results of NWNavRec show that although some new object concepts are learned from a completely different problem, they can be tested on navigation without any model retraining or finetuning. In contrast, SimpleAttention almost fails on zero-shot commands (especially on NWNavRec). Interestingly, in nav_bw_obj it has a much higher zero-shot success rate than in the other subtasks. Our analysis reveals that with the 3 × 3 filter, it always highlights the location between two objects without detecting their classes, because usually there is only one qualified pair of objects in the environment for nav_bw_obj. As a result, it does not really generalize to unseen commands.
Visualization and Analysis Our framework produces intermediate results that can be easily visualized and analyzed. One example has been shown in Figure 2, where the environment map is produced by a trained perception module. It detects exactly all the obstacles and goals. The map constitutes the learned prior knowledge for navigation: all walls and goals should be avoided by default because they incur negative rewards. This prior, combined with the attention map produced for a command, contains all information the agent needs to navigate. The attention maps are usually very precise (Figure 10 Appendix 8.7), with some rare cases in which there are flaws, e.g., when the agent needs to navigate between two objects. This is due to our simplified assumption on 2D geometric translation: the attention map of a location word is treated as a filter and the translation is modeled as convolution. This results in attention diffusion in the above case. To address this issue, more complicated transformation can be used (e.g., FC layers).
We also visualize the programming process. We observe that the programmer RNN is able to shift its focus across steps (Figure 11 Appendix 8.7). With additional analysis of the grounding and map operations, we find that in the first example, the programmer essentially does Transform[southeast](Find[cabbage](F)), and in the second example, it essentially performs Transform[between](CombineOr(Find[apple](F), Find[coconut](F))).
We find the attention and environment maps very reliable by visualization. This is verified by the ∼100% QA accuracy in recognition. However, in Table 1 the best success rate is still ∼ 5% away from the perfect. Further analysis reveals that the agent tends to stuck in loop if the target location is behind a long wall, although it has a certain chance to bypass it (Figure 12 Appendix 8.7). Thus we believe that the discrepancy between the good map quality and the imperfect performance results from our action module. Currently the action module learns a direct mapping from an environment state to an action. There is no support for either history remembering or route planning [Tamar et al., 2016]. Since our focus here is zero-shot navigation, we leave such improvements to future work.

Conclusion
We have demonstrated an end-to-end compositional framework for a virtual agent to generalize an existing skill to new concepts without being retrained. Such generalization is made possible by reusing knowledge, encoded by language and learned in other tasks. By assembling words in different ways, the agent is able to tackle new tasks while exploiting existing knowledge. We reflect these ideas in the design of our framework and apply it to a concrete example: zero-shot navigation in XWORLD. Our framework is just one possible implementation. Our claim is not that an intelligent agent must have a mental model as the presented one, but it has to possess several crucial properties discussed in Section 4. Currently the agent explores in a 2D environment. In the future, we plan to migrate the agent to a 3D world like Malmo [Johnson et al., 2016]. There will be several new challenges, e.g., perception and geometric transformation will be more difficult to model. We hope that the current framework provides some preliminary insights on training a similar agent in a 3D environment. 8 Appendix

Word Attention
A sentence of length L is first converted to a sequence of word embeddings e l by looking up the embedding table E. Then the embeddings are projected to syntax embeddings (Figure 4). The syntax embeddings are fed into a Bidirectional RNN [Schuster and Paliwal, 1997] to obtain word context vectors e c l . The last forward state and the first backward state are concatenated and projected to a booting vector (Figure 6), which is used to initialize the programmer RNN. Given a fixed number of programming steps, at each step s the programmer RNN computes the attention for each word l from its context vector e c l : c s,l = Sof tmax L (CosSim (h s , g(We c l + b))) , CosSim(z, z ) = z z z z , where W and b are projection parameters, h s is the RNN state, g is the activation function, and Sof tmax L is the Softmax function over L words. Then the context vectors e c l are weighted averaged by the attention and fed back to the RNN to tell it how to update its hidden state.

Programmer Operations
The embedding masks produced by Φ and Ψ support switching among the DescribeColor, DescribeLocation, and DescribeName operations by masking word embeddings according to given language tokens (Section 4.2). The selective map combination in Figure 2 Right supports nine different TranslateAttention operations by convolution. Our soft word attention supports the CombineAnd and CombineOr operations by attending to multiple words simultaneously. For example, when multiple objects are attended, the resulting attention map would be a union of the maps of the individual objects. When an object and the color modifying that object are both attended, the resulting attention map would be an intersection of the two maps, thus selecting the object with the correct color from a set of objects with the same name. Thus compared to NMNs [Andreas et al., 2016a], we end up with an implementation of an implicit and differentiable modular network.

Implementation Details
The agent at each time step receives a 156 × 156 RGB image. This image is egocentric and includes both the environment and the black padding region. The agent processes the input image with a CNN that has four convolutional layers: (3, 3, 64), (2, 2, 64), (2, 2, 512), (1, 1, 512), where (x, y, z) represents z x × x filters with stride y. All the four layers have the ReLU activation function. The output is the visual feature map with 512 channels. We stack it along the channel dimension with another parametric spatial feature map of the same sizes. This spatial feature map is initialized with zero mean and standard deviation (Figure 2c).
The agent also receives a navigation command at the beginning of a session. The same command is repeated until the end of the session. The agent may or may not receive a question at every time step. The dimensions of the word embedding, syntax embedding, and functionality embedding are 1024, 128, and 128, respectively. The word embedding table is initialized with zero mean and a standard deviation of 1. The hidden FC layers for computing the syntax and functionality embeddings have 512 units (Figure 4). The bidirectional RNN for computing word contexts has a state size of 128 in both directions. The output RNN booting vector and word context also have a length of 128. The state size of the programmer RNN is equal to the length of the booting vector (Figure 2 Right). The hidden FC layer for converting a functionality embedding to an embedding mask has a size of 128. The RNN used for summarizing the question intention has 128 states ( Figure 5). All FC layers and RNN states in the language and recognition module use Tanh as the activation function. The only exception is the FC layer that outputs the embedding mask (Sigmoid).
In the action module, the action CNN for processing the attention map and the environment map has two convolutional layers (3, 1, 64) and (3, 1, 4), both with paddings of 1. They are followed by three FC layers that all have 512 units. All five layers use the ReLU activation function.

Baseline Models
The language module of SimpleAttention has the word embedding size of 1024. The RNN has the same size with the word embedding. The FC layer that produces the 3 × 3 filter has an output size of 4608 which is 9 times the channel number of the visual feature map. The rest of the layer configuration is the same with our framework. VIS-LSTM has a CNN with four convolutional layers (3, 2, 64), (3, 2, 64), (3, 2, 128), and (3, 1, 128). This is followed by three FC layers with size 1024. The word embedding and the RNN both have sizes of 1024. The RNN output goes through three FC hidden layers of size 512 either for recognition or navigation. Multimodal has the same layer size configuration with that of VIS-LSTM. SAN has a CNN with four convolutional layers (3, 3, 64), (2, 2, 64), (2, 2, 512), and (3, 1, 1024), where the last layer makes each grid have a 3 × 3 receptive field. Its word embedding size and RNN state size are both 512. Following Yang et al. [2016], we use two attention layers.
The outputs of all layers of the above baselines are ReLU activated except for those that are designed as linearly activated.

XWORLD Setup
We configure square environments with sizes ranging from 3 to 7. We fix the size of the environment image by padding walls for smaller environments. Different sessions may have different map sizes. In each session, • The number of time steps T is four times the map size.
• The number of objects on the map is from 1 to 3. • The number of wall grids on the map is from 0 to 10.
• The positive reward when the agent reaches the correct location is set to 1. The negative rewards for hitting walls and for stepping on non-target objects are set to −0.2 and −1, respectively. The time step penalty is set to −0.1.
The teacher has a vocabulary size of 104, including 2 punctuation marks. There are 9 locations, 4 colors, and 40 distinct object classes. Each object class has 2.85 object instances on average. Every time the environment is reset, a number of object classes are randomly sampled and an object instance is randomly sampled for each class. There are in total 16 types of sentences the teacher can speak, including 4 types of navigation commands and 12 types of recognition questions. Each sentence type has multiple non-recursive natural-language templates, and corresponds to a subtask the agent must learn to perform. In total there are 256,832 distinct sentences with 92,442 for the navigation task and 164,390 for the recognition task. The sentence length ranges from 2 to 12.
The object, location, and color words of the teacher's language are listed below. These are the content words with actual meanings that can be grounded in the environment. All the other words are treated as grammatical words whose embeddings are only for interpreting sentence structures. The differentiation between content and grammatical words is automatically learned by the agent based on the teacher's language and the environment. All words have the same form of representation. The sentence types that the teacher can speak are listed below. Each sentence type corresponds to a subtask. The triggering condition describes when the teacher says that type of sentences. Besides the conditions shown, an extra condition for navigation commands is that the target location must be reachable from the current agent location. An extra condition for color-related questions is that the object color must be one of the four defined colors, and objects with other colors will be ignored in these questions. If at a time step there are multiple conditions triggered, we randomly sample one sentence type for navigation and another for recognition. After the sentence type is sampled, we generate the command or question according to the corresponding sentence templates. unique names and are separated by one grid. rec_col2obj What is the red object?
[C4] There is only one object that has the referred color. rec_obj2col What is the color of the apple?
Please tell the name of the object in the south.
[C5] The agent is near the referred object. rec_obj2loc What is the location of the apple?
What color does the object in the east have?
Where is the red object located? What is the object between apple and banana?
[C7] Both referred objects have unique names and are separated by one object. rec_bw_obj2loc Where is the object between apple and banana?
The agent is near the object in the middle. rec_bw_obj2col What is the color of the object between apple [C7] and banana?

Experience Replay and Curriculum Learning
We employ Experience Replay [Mnih et al., 2015] for training both the navigation and recognition tasks. The environment inputs, rewards, and the actions taken by the agent at the most recent 10k time steps are stored in a replay buffer. During training, every time two minibatches of the same number of experiences are sampled from the buffer, one for computing ∇ θ L SL (θ) and the other for computing ∇ θ L RL (θ). For the former, only individual experiences are sampled. We uniformly sample experiences from a subset of the buffer which contains the teacher's questions. For the latter, we need to sample transitions (i.e., pairs of experiences) so that TD error can be computed. We sample from the entire buffer using the Rank-based Sampler [Schaul et al., 2016] which has proven to increase the learning efficiency by prioritizing rare experiences in the buffer.
Because in the beginning the language is quite ambiguous, it is difficult for the agent to start learning with a complex environment setup. Thus we exploit Curriculum Learning [Bengio et al., 2009] to gradually increase the environment complexity. We gradually change the following things linearly in proportional to min(1, G / G), where G is the number of sessions so far and G is the number of curriculum sessions: • The number of grids of the environment.
• The number of objects in the environment.
• The number of wall grids.
• The number of possible object classes that can appear in the environment.
• The length of a navigation command or a recognition question.
We find that this curriculum is crucial for efficient learning, because in the early phase the agent is able to quickly master the meanings of the location and color words given only small ambiguity. After this, these words are used to guide the optimization when more and more new sentence structures and objects are added. In the experiments, we set G = 10k during training while do not use any curriculum during test (maximal difficulty).  [Ren et al., 2015]. The framework treats the projected image embedding as the first word of the sentence. Navigation and recognition only differ in the last several FC layers.  Figure 9: A multimodal framework adapted from the one in Mao et al. [2015]. The image and the sentence are summarized by a CNN and an RNN respectively. Their embeddings are concatenated and projected to the same space. Navigation and recognition only differ in the last several FC layers.

Watermelon is the
Reach the grid between Red ladybug is the Can you go to the destination. grape and deer. target. south of the elephant?


Analysis of Naïve Bayes Algorithm for Email Spam Filtering across Multiple Datasets

E-mail spam continues to become a problem on the Internet. Spammed e-mail may contain many copies of the same message, commercial advertisement or other irrelevant posts like pornographic content. In previous research, different filtering techniques are used to detect these e-mails such as using Random Forest, Naïve Bayesian, Support Vector Machine (SVM) and Neutral Network. In this research, we test Naïve Bayes algorithm for e-mail spam filtering on two datasets and test its performance, i.e., Spam Data and SPAMBASE datasets [8]. The performance of the datasets is evaluated based on their accuracy, recall, precision and F-measure. Our research use WEKA tool for the evaluation of Naïve Bayes algorithm for e-mail spam filtering on both datasets. The result shows that the type of email and the number of instances of the dataset has an influence towards the performance of Naïve Bayes.

Introduction
Nowadays, e-mail provides many ways to send millions of advertisement at no cost to sender. As a result, many unsolicited bulk e-mail, also known as spam e-mail spread widely and become serious threat to not only the Internet but also to society. For example, when user received large amount of e-mail spam, the chance of the user forgot to read a non-spam message increase. As a result, many e-mail readers have to spend their time removing unwanted messages. E-mail spam also may cost money to users with dial-up connections, waste bandwidth, and may expose minors to unsuitable content. Over the past many years, many approaches have been provided to block e-mail spam [1].
For filtering, some email spam are not being labelled as spam because the e-mail filtering does not detect that email as spam. Some existing problems are regarding accuracy for email spam filtering that might introduce some error. Several machine learning algorithms have been used in spam e-mail filtering, but Naïve Bayes algorithm is particularly popular in commercial and open-source spam filters [2]. This is because of its simplicity, which make them easy to implement and just need short training time or fast evaluation to filter email spam. The filter requires training that can be provided by a previous set of spam and non-spam messages. It keeps track of each word that occurs only in spam, in non-spam messages, and in both. Naïve Bayes can be used in different datasets where each of them has different features and attribute. The research objectives are: (i) to implement the Naïve Bayes algorithm for e-mail spam filtering on two datasets, (ii) to evaluate the performance of Naïve Bayes algorithm for e-mail spam filtering on the chosen datasets.
The rest of the paper is organized as follows: Section II describes the related work on Naïve Bayes algorithm for e-mail spam filtering. Section III presents the methodology process of e-mail spam using WEKA. Section IV presents the experimental setup. Section V shows the result and analysis on two datasets. Finally, Section VI concludes the work and highlights the direction for future research.

Related Work
Spammers are now able to launch large scale spam campaigns, malware and botnets helped spammers to spread spam widely. Upon receiving and opening a spam email, Internet users is exposed to security issues as spams are normally broadcast for bad intention. One of the common email spam example received by users are an email requesting for IDs and passwords(Refer to Figure 1). Several machine learning algorithms have been employed in anti-spam e-mail spam filtering, including algorithms that are considered top-performers in Text Classification [3], like Boosting algorithm, Support Vector Machines (SVM) algorithm [5] and Naïve Bayes algorithm [7].
Konstantin Tretyakov et al., [6] have evaluated several most popular machine learning methods i.e., Bayesian classification, k-NN, ANNs, SVMs and of their applicability to the problem of spam-filtering. In this work, the author proposed most trivial sample implementation of the named techniques and the comparison of their performance on the PU1 spam corpus dataset is presented. The author used extracting feature to convert all messages to vectors of numbers (feature vectors) and then classify these vectors. This is because most of the machine learning algorithms can only classify numerical objects like vector.
Then the author created the straightforward C++ implementations of the algorithms, and tested them on the PU1 spam corpus. The PU1 corpus consists of 1099 messages, of which 481 are spam. The test setup use by efficiency measure which are precision, legitimate mail fallout and spam fallout. From the result, the performance of the k-nearest neighbours classifier appeared to be poor and the number of false positives was always rather large. According to the author, only the Naïve Bayesian classifier has passed the test.

Methodology
This section describes the methodology that is used for the research. The methodology that is used for the filtering method is machine learning techniques that divide by three phases.The methodology is used for the process of e-mail spam filtering based on Naïve Bayes algorithm.

Naïve Bayes classifier
The Naïve Bayes algorithm is a simple probabilistic classifier that calculates a set of probabilities by counting the frequency and combination of values in a given dataset [4]. In this research, Naïve Bayes classifier use bag of words features to identify spam e-mail and a text is representing as the bag of its word. The bag of words is always used in methods of document classification, where the frequency of occurrence of each word is used as a feature for training classifier. This bag of words features are included in the chosen datasets.
Naïve Bayes technique used Bayes theorem to determine that probabilities spam e-mail. Some words have particular probabilities of occurring in spam e-mail or non-spam e-mail. Example, suppose that we know exactly, that the word Free could never occur in a non-spam e-mail. Then, when we saw a message containing this word, we could tell for sure that were spam email. Bayesian spam filters have learned a very high spam probability for the words such as Free and Viagra, but a very low spam probability for words seen in non-spam e-mail, such as the names of friend and family member. So, to calculate the probability that e-mail is spam or non-spam Naïve Bayes technique used Bayes theorem as shown in formula below. Where: (i) P (spamword) is probability that an e-mail has particular word given the e-mail is spam. (ii) P (spam) is probability that any given message is spam. (iii) P (wordspam) is probability that the particular word appears in spam message. (iv) P (non − spam) is the probability that any particular word is not spam.
(v) P (wordnon − spam) is the probability that the particular word appears in non-spam message.
To achieve the objective, the research and procedure is conducted in three phases. The phases involved are as follows: (i) Phase 1: Pre-processing (ii) Phase 2: Feature Selection (iii) Phase 3: Naïve Bayes Classifier The following sections will explain the activities that involve in each phases in order to develop this project. Figure 2 shows the process for e-mail spam filtering based on Naïve Bayes algorithm.

Pre-processing
Today, most of the data in the real world are incomplete containing aggregate, noisy and missing values [9]. Pre-processing of e-mails in next step of training filter, some words like conjunction words, articles are removed from email body because those words are not useful in classification.
As mentioned earlier, we are using WEKA tool to facilitate the experiments. For both experiments, the datasets are presented in Attribute-Relation File Format(ARFF) file (Refer to Figure 3 for sample of data).  A full list of the attributes in this data set appears in the "Attributes" frame as shown in Figure 4. Random selection of attribute are performed for the further process.
Attributes capital run length average,capital run length longest and capital run length total are removed from the list by checking the box to their left and hitting the Remove button.

Feature Selection
After the pre-processing step, we apply the feature selection algorithm, the algorithm which deploy here is Best First Feature Selection algorithm [12].

Experimental Setup
The experimental setting of the research is like follows:

The Evaluation Metric
Evaluation metrics are used to evaluate the performance of WEKA tool based on two datasets that had been chosen. The most simple measure is filtering accuracy namely percentage of  [5]. Table 1 shows the evaluation measures for spam filters.

Dataset
Dataset is a collection of data or related information that is composed for separate elements. A collection of dataset for e-mail spam contains spam and non-spam messages. In this research, two datasets are be used to evaluate the performance of Naïve Bayes algorithm to filter e-mail spam. Spam Data [7] is used in order to test the performance of spam filter based on Naïve Bayes algorithm. This dataset contains 9324 e-mails and 500 attributes. Most of the attributes represent the frequency of a given word or character in the email that corresponds to the instance. This dataset is collected from Usenet posts that exist in the 20 Newsgroup collections and collect from many account e-mails located on different e-mail servers.
(ii) Dataset 2: SPAMBASE SPAMBASE was taken from UCI machine learning repository [8] and was created by Mark Hopkins, Erik Reeber, George Forman, and Jaap Suermondt. This dataset contains 4601 email messages and 58 attributes. This dataset collection of non-spam email came from filled work, personal e-mail and single e-mail account. This dataset is composed of a selection of mail messages, suitable for use in testing spam filtering systems. Each instance in SPAMBASE consists of 58 attributes. Most of the attributes represent the frequency of a given word or character in the email that corresponds to the instance.

Result and discussions
This section discussed the experimental result by utilising WEKA tool using Naïve Bayes algorithm. The two datasets are compared based on the percentage of correctly identified spam and non-spam message, percentage of spam message manage to block, percentage of correct message for spam e-mail and weighted average of precision and recall. For each dataset, 10 run of experiments were conducted using WEKA. The experiments have been done in two parts; (i) using random number of attribute, (ii) using same number of attribute. Figure 5  For the accuracy average, the difference total of two datasets is 8.59% which Spam Data get 91.13% while SPAMBASE get 82.54%(Refer to Figure 5). Figure 6 on the other hand shows that SPAMBASE get the highest percentage with 88% while Spam Data 83% for the average of precision. It means SPAMBASE get almost correctly prediction for spam e-mail. From Figure 7, SPAMBASE get the highest percentage of spam e-mail manage to block than Spam Data with 86%. Lastly from Figure 8, SPAMBASE again get the highest percentage with 87% while Spam Data 83%.
As mention earlier, the second part of the experiment conducted by selecting same number of attributes for each dataset. From Figure 9, we can see Spam Data again achieved the highest percentage for accuracy but still low for precision, recall and F-measure than SPAMBASE. For accuracy, Spam Data get 82.88% than SPAMBASE only get 72.57%. For precision Spam Data get 74% while SPAMBASE get 76%. For recall, the difference total of two datasets is 28% which   Spam Data get only 51% while SPAMBASE get 79%. Lastly, for F-Measure, Spam Data get 60% while SPAMBASE get 77%. As can see, all the results have not much difference from result above. But, from this result, can see the big different total percentage of two datasets than earlier result.
From result that we have been record, we can analyse that Naïve Bayes classifier performs better on SPAMBASE as compared to the Spam Data, even though Spam Data manage to achieve good result on accuracy. This is because filter that gets good result based on precision, recall and F-measure are also important. Spam Data has many attribute and instance of e-mail with overall 9324 e-mails and 500 attributes and Spam Data not manage to performance well with the implement Naïve Bayes classifier. For all the evaluation of metric which precision, recall and F-measure, SPAMBASE performance very well than Spam Data. Even SPAMBASE  dataset only has 4601 instance e-mails and 58 attributes, but Naïve Bayes classifier manage to get good result from this dataset. This is because Naïve Bayes classifier no needs many instances of e-mails and attributes to train the classifier for e-mail spam filtering. SPAMBASE dataset is multivariate dataset contain data from a single e-mail account while Spam Data dataset collect from many e-mail account. From this we can know, Naïve Bayes classifier can perform well with dataset that come from a single e-mail account than many email account. This is because Naïve Bayes classifier can focus train with various type of e-mail spam that come from single e-mail account located on same e-mail servers.
Besides that, we also test if these datasets have same total of attribute and minimizing the attribute with six attribute, wether it gives different result. From what we get, the result not given different result because Naïve Bayes classifier that used SPAMBASE dataset still has the best performance than Spam Data dataset. But the percentage for accuracy, precision, recall and F-measure drop a little bit. This is proving that total of attribute give important role to Naïve Bayes classifier for filtering e-mail spam.
Although minimizing of attribute decreased the performance of Naïve Bayes classifier, but in the other hand it improved time complexity. The time taken to build model is faster with less attribute and the best performance Naïve Bayes classifier that used SPAMBASE dataset need 0.14 second to build the model. However, it is not important because e-mail spam filtering must have highest accuracy, precision, recall and F-measure to filter that e-mail spam or non-spam. 6. Conclusion E-mail spam filtering is an important issue in the network security and machine learning techniques; Naïve Bayes classifier that used has a very important role in this process of filtering e-mail spam. The quality of performance Naïve Bayes classifier is also based on datasets that used. As can see, dataset that have fewer instances of e-mails and attributes can give good performance for Naïve Bayes classifier. Naïve Bayes classifier also can get highest precision that give highest percentage spam message manage to block if the dataset collect from single e-mail accounts. So we can see, why performance of Naïve Bayes classifier is good when used SPAMBASE dataset.