Table tab:WD compares the test accuracies for cyclical weight decay (CWD) to training with tuned hyper-parameters (with a constant weight decay) and learning rate warmstart and cosine annealing [1]}.
For each dataset in this Table there are two rows: the first row presents the mean test accuracy and the standard deviation over four runs (for ImageNet, this is the mean and standard deviation over two runs), and the second row provides the range of weight decay used in the training. The second column in the Table provides the results of training with a constant weight decay, and the subsequent columns, show the results of training with an increasing range for weight decay.
In our experiments, we found that the performance was relatively insensitive to the value of \(f_c\) .
<FIGURE>