The Evolved Transformer
David R. So1Chen Liang1Quoc V . Le1
Abstract
Recent works have highlighted the strength of
theTransforme
The Evolved Transformer
Recent works have highlighted the strength of the Transformer architecture on
sequence tasks while, at the same time, neural architecture search (NAS) has
begun to outperform human-designed models. Our goal is to apply NAS to search
for a better alternative to the Transformer. We first construct a large search
space inspired by the recent advances in feed-forward sequence models and then
run evolutionary architecture search with warm starting by seeding our initial
population with the Transformer. To directly search on the computationally
expensive WMT 2014 English-German translation task, we develop the Progressive
Dynamic Hurdles method, which allows us to dynamically allocate more resources
to more promising candidate models. The architecture found in our experiments
-- the Evolved Transformer -- demonstrates consistent improvement over the
Transformer on four well-established language tasks: WMT 2014 English-German,
WMT 2014 English-French, WMT 2014 English-Czech and LM1B. At a big model size,
the Evolved Transformer establishes a new state-of-the-art BLEU score of 29.8
on WMT'14 English-German; at smaller sizes, it achieves the same quality as the
original "big" Transformer with 37.6% less parameters and outperforms the
Transformer by 0.7 BLEU at a mobile-friendly model size of 7M parameters.
David R. So Chen Liang Quoc V. Le