To visualize the impact of the dissimilarity regularization,  fig:tsne shows t-SNE [1]} visualization of the activations of the last layer of the trained networks. fig:tsnecatcher show the network trained for the Catcher environment, while  fig:tsnecopter, the network trained for the PixelCopter environment. The upper row of the figure shows the original, unregularized models, while the lower row a regularized version. For all combinations, we find that the activations from the original MaxminDQN and EnsembleDQN versions do not show any obvious pattern, while the regularized ones show distinct clusters. An additional benefit of t-SNE visualizations over CKA similarity heatmaps is that the CKA similarity heatmaps are useful to show representation similarity between two neural networks, but they become counter intuitive as the number of neural networks increases. More t-SNE visualizations for four neural network experiments can be seen in sec:heatmaps.
<FIGURE>