Various studies have shown that transformers [1]}
can manipulate symbolic data in a manner traditionally associated with symbolic computation.
For example, in [2]}, a transformer model learned to do symbolic integration and solve
ordinary differential equations symbolically, tasks traditionally reserved for symbolic computer algebra systems. Similarly, in [3]}, a transformer model learned to solve formulas in propositional logic
and demonstrated some degree of generalization to out of distribution formulas.
Finally, [4]} showed that a transformer trained for language modeling can also do simple analogical reasoning tasks without explicit training.
Although these models do not necessarily beat carefully tuned symbolic algorithms in all cases (especially on out of distribution data),
they are an important motivation for our proposed recipe for attaining strong reasoning capabilities from self-attention-based models on visually grounded tasks.
