Features: In addition to different sources of data to learn from, our approach allows the agents to also use different sensors (or feature spaces). Our main dataset (RGB-D) does naturally provide two modalities: RGB and DEPTH. We extract Hierarchical Matching Pursuit (HMP) features [1]}, [2]}, [3]}, [4]} for each domain provided by the authors. The HMP extractor uses RGB images to compute features for the RGB modality and depth images with surface normals for the Depth modality. Due to lack of other such datasets in the community, we use different sensors on the AWA and SUN datasets by extracting different features from the images as proxies. For AWA, the first agent uses SIFT (bag-of-words) features, while the second agent uses COLOR histograms (both made available by the authors of the dataset [5]}). For SUN, the first agent uses GIST and the second uses COLOR histograms (available with the dataset [6]}).
