@inproceedings{
mohtashami2023cotformer,
title={Co{TF}ormer: More Tokens With Attention Make Up For Less Depth},
author={Amirkeivan Mohtashami and Matteo Pagliardini and Martin Jaggi},
booktitle={Workshop on Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource Optimization (WANT@NeurIPS 2023)},
year={2023},
url={https://openreview.net/forum?id=rBgo4Mi8vZ}
}
The race to continually develop ever larger and deeper foundational models is underway. However, techniques like the Chain-of-Thought (CoT) method continue to play a pivotal role in achieving optimal downstream performance. In this study, we establish an approximate parallel between the utilization of the chain-of-thought and employing a deeper transformer. Building on this insight, we introduce CoTFormer, a transformer variant that employs an implicit CoT-like mechanism to achieve comparable performance to that of a deeper model. Our empirical findings demonstrate the effectiveness of CoTFormers, as they significantly outperform larger standard transformers.
CoTFormer: More Tokens With Attention Make Up For Less Depth
Amirkeivan Mohtashami Matteo Pagliardini Martin Jaggi