PE. As different part of FQ-BERT has different bitwidth, the input bitwidth of multiplication has different combinations, such as 8-bit \(\times \)  4-bit for \(XW^Q\)  and 8-bit \(\times \)  8-bit for \(QK^T\) . In order to reuse the same module to support different operations, we design a Bit-split Inner-product Module (BIM), which is similar to Bit-fusion[1]}, as depicted in Figure REF .
Each BIM contains \(M=2m\)  8-bit \(\times \)  4-bit multipliers, two \(m\) -adder tree and several shift-add logic. Each multiplier has a sign signal to indicate the input number is signed or unsigned. The shift-add logic is used for shifting the partial sum when the bitwidth of input is larger than 4 bits. The position of shift-add logic has two choices, as shown in Fig.REF , and using shift logic at adder tree's output (Type A) can save more resources, though this need to rearrange the input data.
