
Nonlinear feature transformation: The first three layers are convolutional layers (i.e. CNNs): two 8-dims, two 16-dims and two 32-dims followed by a ReLU activation function. The size of all the receptive fields is 3×3 and the stride in all the layers is 1. A max pooling layer is inserted between each pair of sequences. The last layer is a global pooling layer, which enforces correspondences between feature maps and categories. The outputs of the last layer are defined as \(L_1\)  and \(L_2\)  in Fig. REF , and are the inputs of the CCA layer.

CCA layer: This layer has the goal to jointly learn parameters for both views \(X_1\)  and \(X_2\)  to maximise corr(\(f_1(X_1)\) ,\(f_2(X_2)\) ), where \(f_1()\)  and \(f_2()\)  are the nonlinear functions learnt by the networks from the previous step. We denote by \(\theta _1\)  the vector of all parameters from the nonlinear feature transformation of the first view, and similarly for \(\theta _2\) . These are determined according to the following objective:
\((\theta _1^*, \theta _2^*) = \operatornamewithlimits{argmax}_{(\theta _1, \theta _2)} \text{corr}(f_1(X_1;\theta _1),f_2(X_2;\theta _2))\) 
The parameters \(\theta _1\)  and \(\theta _2\)  of DCCA are trained to optimise this quantity using gradient-based optimisation. For the full derivation of the gradients we refer to the original DCCA paper by [1]}. The outputs of the CCA layer are \(V_1\)  and \(V_2\) .

Feature fusion The fusion layer combines the outputs of the CCA layer (\(V_1\)  and \(V_2\) ) as follows: \(F_{fusion} = \alpha \cdot V_1 + \beta \cdot V_2\) , where \(\alpha \)  and \(\beta \)  are the fusion weights. In our experiments, in order to balance the composition of features, we set \(\alpha =\beta =0.5\) .

