We apply TAGI to the MNIST classification problem [1]} consisting of \(\mathtt {D}=70\,000\)  \((28\times 28)\)  greyscale images for \(\mathtt {K}=10\)  classes (\(60\,000\)  training and \(10\,000\)  test). Here, we compare the performance of two AG-FNN configurations, each having \(\mathtt {L}=2\)  hidden layers with a number of hidden units equal to \(\mathtt {A}\in \lbrace 100, 800\rbrace \) . Each AG-FNN has the same structure for the input (\(\mathtt {X}=784\)  nodes) and the output layer (\(\mathtt {Y}=11\)  nodes). The ReLU activation function is used for the two hidden layers. For each digit, the vector of covariates \(\mathbf {x}_{i}\in (0,1)^{784}\)  is assumed to be deterministic so that \({\mu }_{{X}_{i}}=\mathbf {x}_{i}\)  and \(\mathbf {\Sigma }_{{X}_{i}}=\mathbf {0}\) . The prior covariance for bias is initialized to \(\Sigma _{{B}}^{0}=0.01\cdot \mathbf {I}\) , and by using the Xaviers's initialization approach [2]} for weights \(\Sigma _{{W}}^{0}\) . The prior mean vector is randomly sampled from \({\mu }_{{\theta }}^{0}\sim \mathcal {N}(\mathbf {0},\Sigma _{{\theta }}^{0})\) .
The hyper-parameter associated with the output layer is set to \(\alpha =1/3\) . The posterior mean vector as well as the main diagonal of the posterior covariance are learnt using two setups: (1) a single observation per batch, that is, \(\mathtt {B}=1\) , and (2) 10 observations per batch, \(\mathtt {B}=10\) . Each network is evaluated for \(\sigma _{V} = \lbrace 0.1, 0.2, 0.3, 0.4\rbrace \)  and the optimal value for \(\sigma _{V}\)  is selected using a randomly selected validation set corresponding to 5% of the training set. The optimal number of epochs \(\mathtt {E}\)  is identified using an early-stop procedure evaluated on the validation set.
