Potentially, sequence length can be searched over as well, but this is not done in this example. The Block Library is built using the BKD process, requiring \(4\times 3\times 3\times 3=135\)  epochs of total training to model a fairly small search space of .5M architectures. The Architecture Library exists out of 23 uniformly sampled architectures in this search space, finetuned for 50 epochs on ImageNet [1]}, using a large CNN model as a teacher until convergence. The latter process is calibrated such that the original teacher model (vit-base-patch16-224), initialized with weights from the Block Library achieves the accuracy of the teacher model after these 50 epochs. Note that our reliance on such finetuning and knowledge distillation allows extracting knowledge without access to full datasets, in this case ImageNet21k. Finally, we use the Block- and Architecture libraries to train an accuracy predictor and execute an evolutionary search targeting minimization of the number of operations. Figure REF (left) illustrates the results of this search, showing that our search in this space allows finding a pareto set of models. In terms of number of operations, this ViT-based search space does not outperform ResNet-50. Figure REF (right) illustrates the quality of the accuracy predictor, on a limited set of ViT architectures.
