Easy and Efficient Transformer Scalable Inference Solution For large NLP mode
The ultra large scale pre training model can effectively improve the effect of a variety of tasks, and it also brings a heavy computational burden to inference. This paper introduces a series of ultra large scale pre training model optimization methods that combine algorithm characteristics and GPU processor hardware characteristics, and on this basis, propose an inference engine Easy and Efficient Transformer (EET) Which has a significant performance improvement over the existing schemes. We firstly introduce a pre padding decoding mechanism that improves token parallelism for generation tasks. Then we design high optimized kernels to remove sequence masks and achieve cost free calculation for padding tokens, as well as support long sequence and long embedding sizes. Thirdly a userfriendly inference system with an easy service pipeline was introduced which greatly reduces the difficulty of engineering deployment with high throughput. Compared to Faster Transformer's implementation for GPT 2 on A100, EET achieves a 1.5 15x state of art speedup varying with context length.EET is available https:/github.com/NetEase FuXi/EET.
ArXiv