In our experiments we perform iterative pruning. Specifically, we prune every 320 minibatches after loading the pretrained model with \(k=30\)  pruning steps in total to satisfy the constraint. We finetune the network for 90 epochs in total with an individual batch size at 128 for each GPU. For finetuning, we follow NVIDIA's recipe [1]} with mixed precision and Distributed Data Parallel training. The learning rate is warmed up linearly in the first 8 epochs and reaches the highest learning rate, then follows a cosine decay over the remaining epochs [2]}. For latency lookup table construction, we target a NVIDIA TITAN V GPU with batch size 256 for latency measurement to allow for highest throughput for inference, and target a Jetson TX2 with inference batch size 32. We pre-generate a layer latency look-up table on the platform by iteratively reducing of the number of neurons in a layer to characterize the latency with NVIDIA cuDNN [3]} V7.6.5. We profile each latency measurement 100 times and take the average to avoid randomness.
