Genomic (DNA)  sequences encode an enormous amount of information for gene regulation, protein synthesis, and numerous other cellular properties.
Similar to natural language models, researchers have proposed foundation models in genomics to learn generalizable features from unlabeled genome data that can then be fine-tuned for downstream tasks such as identifying regulatory elements.
Due to the quadratic scaling of attention, previous Transformer-based genomic models have used 512 to 4k tokens as context (<0.001% of the human genome) , significantly limiting the modeling of long-range interactions in DNA.
In addition, these methods rely on tokenizers or fixed k-mers to aggregate meaningful DNA units, losing single nucleotide resolution (i.e. DNA "characters")  where subtle genetic variations can completely alter protein function via single nucleotide polymorphisms (SNPs) .
Recently,, a large language model based on implicit convolutions was shown to match attention in quality while allowing longer context lengths and lower time complexity.
Leveraging’s new long-range capabilities, we present,a genomic foundation modelpretrained on the human reference genome withcontext lengths of up to 1 million tokens at the single nucleotide-level– anup to 500x increaseover previous dense attention-based models.scales sub-quadratically in sequence length (training up to 160x faster than Transformer) ,uses single nucleotide tokens, and hasfull global context at each layer.
We explore what longer context enables - including the first use of in-context learning in genomics for simple adaptation to novel tasks without updating pretrained model weights.
On a long-range species classification task,is able to effectively solve the challenge by increasing the context length to 1M without downsampling.
On fine-tuned benchmarks from the Nucleotide Transformer,reaches state-of-the-art (SotA)  on 12 of 18 datasets using a model with orders of magnitude less parameters and pretraining data.222On benchmarks from Nucleotide Transformer,uses a model with 1500x fewer parameters (2.5B vs 1.6M)  and 3200x less pretraining data (3202 vs 1 human reference genome) .On the GenomicBenchmarks,surpasses SotA on 7 of 8 datasets on average by +10 accuracy points, and by as much as +20 accuracy points on enhancer identification.
Code available athttps://github.com/HazyResearch/hyena-dna.