The NN with full precision is trained with an initial learning rate of \(0.08\) . The projected full precision NN uses a projected gradient algorithm. The same hyperparameters as the "plain" algorithm are used, except that a deterministic projection onto the hypersphere is performed for each iteration \(w_{k+1} = \Pi (w_k - \gamma _k \widehat{\nabla \ell }(w_k))\) . For BinaryConnect, we use the method described in [1]}. For Mirror Descent (MD), we use the codehttps://github.com/kartikgupta-at-anu/md-bnn from [2]} and implement the version \(\operatorname{tanh}(\cdot )\)  (without annealing and with \(\alpha =0.01\)  and \(\mu =100\)  when training). ProxQuant was run with the parameters specified in [3]}. Note ProxQuant does not initially quantize the fully-connected layer, and add full precision biases. For fair comparison we have tested
ProxQuant with all layers binarized. The ASkewSGD method is described in alg:detailed. Multiple values for \(\alpha \)  in \([0.1, 5]\)  are considered. The precision threshold \(\epsilon \)  is decreased from epoch to epoch: it is set to 1 at the beginning and then exponentially annealed to \(.88^t\)  in the last 50 epochs, where \(t\)  is the epoch. After the last step, all weights are within an interval of length \(\epsilon _{\operatorname{final}}=0.01\)  of \(\lbrace -1, +1\rbrace \) .
