Large language models have an exceptional capability to incorporate new information in a contextual manner. However, the full potential of such an approach is often restrained due to a limitation in the effective context length. One solution to this issue is to endow an attention layer with access to an additional context, which comprises of (key, value)  pairs. Yet, as the number of documents increases, the proportion of relevant keys to irrelevant ones decreases, leading the model to focus more on the irrelevant keys. We identify a significant challenge, dubbed thedistraction issue, where keys linked to different semantic values might overlap, making them hard to distinguish. To tackle this problem, we introduce theFocused Transformer(FoT) , a technique that employs a training process inspired by contrastive learning. This novel approach enhances the structure of the (key, value)  space, enabling an extension of the context length. Our method allows for fine-tuning pre-existing, large-scale models to lengthen their effective context. This is demonstrated by our fine-tuning ofandOpenLLaMA checkpoints. The resulting models, which we nameLongLLaMA111We release thecheckpoints and source codeofLongLLaMA, see also our colabs., exhibit advancements in tasks requiring a long context. We further illustrate that ourLongLLaMAmodels adeptly manage acontext length for passkey retrieval.