Many natural language processing tasks benefit from long inputs, but processing long documents with Transformers is expensive – not only due to quadratic attention complexity but also from applying feedforward and projection layers to every token. However, not all tokens are equally important, especially for longer documents. We proposeCoLT5, a long-input Transformer model that builds on this intuition by employing conditional computation, devoting more resources to important tokens in both feedforward and attention layers. We show thatCoLT5achieves stronger performance thanLongT5with much faster training and inference, achieving SOTA on the long-input SCROLLS benchmark. Moreover,CoLT5can effectively and tractably make use of extremely long inputs, showing strong gains up to 64k input length.