We use a NAS infrastructure similar to [1]}, [2]}, where we employ the same RNN-based controller. We build an infrastructure to retrieve TPU and GPU hardware latency directly during search and run NAS on TPUv3[3]}, [4]} and GPUv100 [5]}. We used data parallelism for distributed training/searching on both TPUs and GPUs. Since the largest model candidate can fit in a single TPU/GPU device, the data parallelism is sufficient for distributed training on a pool of TPUs/GPUs.
