The network architecture used for the permuted-label
experiment is a small CNN with two convolutions layers, a linear hidden layer and a linear output layer [1]}.
With a
learning rate of \(10^{-3}\) , we use the
hyper-prior, prior, hyper-posterior and posterior distributions given by (REF ), (REF ), (REF ) and (REF ), respectively.
We set \(\kappa _p^2=100\) , \(\kappa _s^2=0.001\) , and \(\delta =0.1\) . For each task \(\tau _i\) , and \(k=1,\dots ,d\) , the posterior parameter \(\log (\sigma _i^2(k))\)  initialized by \(\mathcal {N}(-10,0.01)\) ,
\(\mu _i(k)\) 
is initialized randomly with the
Glorot method [2]}. Then, for different
bounds, by using backpropagation, we evaluate the gradient of the bound with respect to \(\mathbf {\mu }_i=(\mu _i(1),\dots ,\mu _i(d))\) . Then, we set \(\mu _i(k)\)  and \(\sigma _i(k)\)  as the means and variance
of \(k\) -th weight.
The parameters
\(\mu _u(k)\)  and \(\sigma _u(k)\)  are similar in
structure, and the parameter \(\theta \)  is the vector containing
the weights of \(\mathrm {N}\)  tasks [1]}.
