Extreme compression, particularly ultra-low bit precision (binary/ternary)  quantization, has been proposed to fit large NLP models on resource-constraint devices.
However, to preserve the accuracy for such aggressive compression schemes, cutting-edge methods usually introduce complicated compression pipelines, e.g., multi-stage expensive knowledge distillation with extensive hyperparameter tuning.
Also, they oftentimes focus less on smaller transformer models that have already been heavily compressed via knowledge distillation and lack a systematic study to show the effectiveness of their methods.
In this paper, we perform a very comprehensive systematic study to measure the impact of many key hyperparameters and training strategies from previous works.
As a result, we find out that previous baselines for ultra-low bit precision quantization are significantly under-trained.
Based on our study, we propose a simple yet effective compression pipeline for extreme compression, named XTC.
XTC demonstrates that
(1)  we can skip the pre-training knowledge distillation to obtain a 5-layer BERT while achieving better performance than previous state-of-the-art methods, e.g., the 6-layer TinyBERT;
(2)  extreme quantization plus layer reduction is able to reduce the model size by 50x, resulting in new state-of-the-art results on GLUE tasks.