Looped Transformers as Programmable Computers
Angeliki Giannouw*, Shashank Rajputw, Jy-yong Sohnw,
Kangwook Leew, Jason D. Leep
Looped Transformers as Programmable Computers
We present a framework for using transformer networks as universal computers
by programming them with specific weights and placing them in a loop. Our input
sequence acts as a punchcard, consisting of instructions and memory for data
read/writes. We demonstrate that a constant number of encoder layers can
emulate basic computing blocks, including embedding edit operations, non-linear
functions, function calls, program counters, and conditional branches. Using
these building blocks, we emulate a small instruction-set computer. This allows
us to map iterative algorithms to programs that can be executed by a looped,
13-layer transformer. We show how this transformer, instructed by its input,
can emulate a basic calculator, a basic linear algebra library, and in-context
learning algorithms that employ backpropagation. Our work highlights the
versatility of the attention mechanism, and demonstrates that even shallow
transformers can execute full-fledged, general-purpose programs.
Angeliki Giannou Shashank Rajput Jy-yong Sohn Kangwook Lee Jason D. Lee Dimitris Papailiopoulos