In this inequality, \(\overline{V}\)  denotes the empirical estimate of the validation error, \(\mathbb {E}[V]\)  is its hypothetical true
value, \(p\)  is the desired maximum deviation between the estimation and the true value, and \(n\)  represents the
number of samples in the validation set. Based on this inequality, the number of samples in the validation set should be very large. For example, we will require about 20,000 samples in the validation set if we want to be 95% confident in estimation of validation error that differes less than 1% from the absolute true value. Note that in this analysis, validation error is computed as the average of independent binary indicator variables representing if a given sample in the validation set is classified correctly or not. This analysis may be unrealistic because of the assumption that the validation accuracy is
the average of independent variables. To address this problem, Oliver et al. [1]} measure this phenomenon empirically, and train the SSL methods using 1,000 labels in the training set from SVHN dataset and then evaluate them on the validation sets with different sizes. Note that these small synthetic validation sets are generated by different randomly sampled sets without
overlapping from the full SVHN validation set. Following the same setting for evaluation of our SSL algorithm (ROT) in a real world scenario, in Fig. REF  and Fig. REF , we reported the mean and standard deviation of validation errors over five times randomly non-overlapping splitting the SVHN and CIFAR validation sets with varying sizes. The results in Fig. REF  and Fig. REF  indicate that as we increase the size of validation set, the ROT algorithm will be more confident and stable to select its hyperparameters than the case where we use small-size validation set. For a fair comparison between our method and the other SSL methods in Table. 1 of the paper, we have been consistent with other methods in the size of the training and validation sets as it is designed in standard SVHN and CIFAR-10 datasets. Specifically, for SVHN, we used 65,932 images for the training set and 7,325 for the validation set, and for CIFAR-10 dataset, we used 45,000 images for the training set and 5,000 images for the validation set. Fig. REF  and Fig. REF  indicate the error rate of the ROT algorithm on the SVHN and CIFAR validation sets for different values of \(\lambda \)  in our transportation plan. Note that during the tuning of \(\lambda \) , we fixed \(\alpha \)  in Eq. (5) to one and changed \(\lambda \)  to different values including 0.1, 0.25, 0.5 , 0.75 and 1. Moreover, during the tuning of \(\alpha \) , we fixed \(\lambda \)  in Eq. (3) to 0.25 and changed \(\alpha \)  to different values including 0.1, 0.25, 0.5 , 0.75 and 1. Fig. REF  and Fig. REF  show the error rate of the ROT algorithm on the SVHN and CIFAR validation sets for different values of \(\alpha \)  for training the parameters of the CNN.
