where \(W_I \in \mathbb {R}^{P \times D}\)  (\(P=256\) ) and \(\mbox{FFN}\)  is a two-layer feed-forward network. The use of a non-linear prediction head has shown to improve the quality of the representations learned in a contrastive learning setup [1]}; following the recent work in Transformers [2]}, [3]}, [4]}, we use a GELU non-linear activation function [5]} in \(\mbox{FFN}\) . Optimizing Eqn. REF  enforces \(\mbox{I}(\mathbf {x}_t, \tilde{\mathbf {o}}_{t})\)  to approximate the density ratio \(\frac{p(\mathbf {x}_t|\tilde{\mathbf {o}}_{t})}{p(\mathbf {x}_t)}\) ; this can be seen as maximizing the mutual information between \(\mathbf {x}_t\)  and \(\tilde{\mathbf {o}}_{t}\)  [6]}. Intuitively, this encourages the Transformer to capture the underlying dynamics of \(\mathbf {x}\)  from each modality without explicitly learning a generative model \(p(\mathbf {x}_t | \tilde{\mathbf {o}}_{t})\) .
