The increasing size of large language models has posed challenges for deployment and raised concerns about environmental impact due to high energy consumption.
In this work, we introduce BitNet, a scalable and stable 1-bit Transformer architecture designed for large language models.
Specifically, we introduceBitLinearas a drop-in replacement of thenn.Linearlayer in order to train 1-bit weights from scratch.
Experimental results on language modeling show that BitNet achieves competitive performance while substantially reducing memory footprint and energy consumption, compared to state-of-the-art 8-bit quantization methods and FP16 Transformer baselines.
Furthermore, BitNet exhibits a scaling law akin to full-precision Transformers, suggesting its potential for effective scaling to even larger language models while maintaining efficiency and performance benefits.