The problem of reducing the amount of computation while keeping the model performance high have been tackled from many different angles. Knowledge distillation approach aims to train a smaller model to mimic the output of a larger model [1]}. Quantization assigns less bits to the model weights to reduce the computational cost [2]}, [3]}. Adaptive computation methods propose skipping some parts of the model based on the input [4]}, [5]}, [6]}. Pruning methods remove redundant weights in the network [7]}, [8]}, [9]}. Early exit networks have additional exit points so that not every sample will have to go through the entire network [10]}, [11]}, [12]}.
