Essentially, the blocks \(B_{n,m}\)  are trained to closely replicate the teacher's non-linear function \(Y_n = B_n(Y_{n-1})\) . Intuitively, larger, more accurate blocks with a larger “modeling capacity” or “expressivity” replicate this function more closely than smaller, less accurate blocks. On ImageNet [1]} such knowledge distillation requires only a single epoch of training for effective results. After training each block, the resulting NSR metric is added to the Block library as a quality metric of the block \(B_{n,m}\) . Note that the total number of trainable blocks \(B_{n,m}\)  grows linearly as \(N\times M\) , whereas the overall search space grows exponentially as \(M^N\) , making the method scale well even for large search-spaces.
<FIGURE>