Hepatitis B virus (HBV) remains a global public health concern, with over 250 million individuals living with chronic HBV infection (CHB) and no curative therapy currently available. Viral diversity is associated with CHB pathogenesis and immunological control of infection. Improved methods to characterize the viral genome at both the population and intra-host level could aid drug development efforts. Conventionally, HBV sequencing data are aligned to a linear reference genome and only sequences capable of aligning to the reference are captured for analysis. Reference selection has additional consequences, including sample-specific ‘consensus’ sequence construction. It remains unclear how to select a reference from available sequences and whether a single reference is sufficient for genetic analyses. Using simulated short-read sequencing data generated from full-length publicly available HBV genome sequences and HBV sequencing data from a longitudinally sampled individual with CHB, we investigate alternative graph-based alignment approaches. We demonstrate that using a phylogenetically representative ‘genome graph’ for alignment, rather than linear reference sequences, avoids issues of reference ambiguity, improves alignment, and facilitates the construction of sample-specific consensus sequences genetically similar to an individual’s infection. Graph-based methods can therefore improve efforts to characterize the genetics of viral pathogens, including HBV, and may have broad implications in host pathogen research. In addition to age, clinical and environmental factors, and host genetic variation,4 viral genetic diversity contributes to the pathogenesis and the severity of CHB.10Approximately one-third of the world’s population has been exposed to the hepatitis B virus (HBV), a major cause of hepatocellular carcinoma and end-stage liver disease.11 With 10 known genetically and geographically distinct HBV genotypes and >30 subgenotypes, CHB is also caused by recombinants or mixtures of genotypes.19 Additionally, like other chronic viral infections, intra-host CHB diversity involves multiple viral strains which evolve, mutate, and change in frequency over time, termed a viral quasispecies.24 This intra-host diversity has also been shown to influence disease progression,27 treatment outcome,29 and confound molecular epidemiology or surveillance efforts.30HBV has a small (3.2kb) partially double-stranded circular genome with four overlapping gene-encoding regions and a higher mutation rate than most DNA viruses.32 However, it can be difficult to select the most appropriate reference sequence when analyzing clinical CHB samples of unknown HBV genotype or subgenotype. One option is to assess all potential HBV reference sequences to identify the most appropriate reference sequence but this is both computationally expensive and can still fail within the context of recombinant or mixed infections.Characterizing this extensive genetic variation is therefore important for advancing our understanding of the natural history of disease and potential treatment targets. Sequencing-based analyses of HBV and other microbial pathogens usually involve an initial alignment of sequencing data to a representative reference genome. Choosing the right reference is critical, as only data sufficiently similar to the reference can be aligned and retained within subsequent analyses.35 Reference selection can also affect the ability to accurately derive sample-specific ‘consensus’ sequences, which provide an approximation of the genome sequence causing an infection. This issue of reference ambiguity is especially problematic for CHB, as a set of phylogenetically representative HBV reference sequences has only recently been proposed.36 Furthermore, given the extreme diversity of HBV, the use of a single reference sequence, even of the correct HBV genotype, may be insufficient.33Nevertheless, the use of unrepresentative reference sequences can interfere with characterizing pathogen diversity, resulting in false or missing mutations and biased phylogenetic relationships.38 Graph-based structures containing genetic variation from multiple genomes have been shown to improve sequence alignment and variant calling for highly variable regions of the human genome and microbial organisms like Escherichia coli.39 A graph-based reference containing a representative sampling of the genetic variation observed across all known HBV genotypes/subgenotypes might also improve sequence alignment and variant calling, as well as enable the generation of accurate sample-specific consensus sequences for HBV-related genetic analyses. Consensus sequences reflect the most commonly observed nucleotide at each site across the genome, inferred from aligning sequencing data against a specific reference sequence. The construction of consensus sequences is a typical objective of viral-focused genetic analyses,41 including HBV.43 However, whether a graph-based approach can improve viral sequence alignment and sample-specific consensus sequence construction has, to our knowledge, yet to be demonstrated.Aligning to a phylogenetically representative ‘genome graph’ constructed from many different HBV genome sequences rather than a single HBV genome sequence could potentially avoid these reference ambiguity issues. Genome graphs are comprised of ‘nodes’ which reflect stretches of genetic sequence connected by ‘edges’ which determine the path a genome sequence traverses across a subset of nodes within the graph.In this study, we leverage 2,837 publicly available full-length HBV genomes, simulated high-throughput sequencing data from these HBV sequences, and real-world longitudinally collected sequencing data from an individual with CHB to identify the optimal alignment method as determined by the proportion of successfully aligned HBV sequencing data. Additionally, by comparing alignment derived sample-specific consensus sequences, the accuracy of graph vs. linear reference-based alignment methods will be evaluated.36 Briefly, 7,108 full-length HBV genomes were obtained from the HBVdb database (https://hbvdb.lyon.inserm.fr/HBVdb/) and recombinant or highly similar full-length HBV genome sequences were removed. A set of 44 sequences representative of all phylogenetically identified genotypes, subgenotypes, and genetically distinct clades was then identified for use as reference sequences in downstream analyses.A set of non-redundant full-length HBV genomes  was obtained from the publicly available resource provided by McNaughton et al.44 Sample-level clinical and demographic data were obtained through communication with study authors. The high-throughput CHB sequencing data included in this study reflects five longitudinally sampled visits from a single individual (identifier ‘C4’). C4 is a male who was 36 years old at the time the first sample was obtained in 1991. The final sample was obtained in 1996. Individual C4 remained chronically infected and HBeAg positive for the entirety of study follow-up. Sequencing was performed using an Illumina HiSeq 2500, as previously described.44 Cutadapt was used to trim adapters, poor-quality bases, and reads <36bp long.45 FastQC was used to ensure the post-QC data passed Illumina sequencing-related QC checks.46HBV-targeted sequencing data from an individual included in a longitudinal cohort study of treatment naïve individuals with CHB was obtained via the NCBI Sequence Read Archive (BioProject ID: 479693).44Patients with HBV infection, including individual C4, were recruited with fully informed written consent from the Division of Gastroenterology and Hepatology at the National University Health System, Singapore.47 Two datasets of paired-end sequences/reads were generated using an Illumina HiSeq error model, the first set  was simulated from each of the recommended HBV reference sequences (N=44). The second set  was simulated from a randomly selected HBV genome sequence from each of the 9 HBV genotypes  and 50 additional randomly selected HBV genomes not included within the HBV reference graph (N=59).Realistic high throughput HBV sequencing data were simulated using InSilicoSeq, which enables the generation of error-prone Illumina-like sequencing data with pre-specified abundance/coverages.36 The HBV reference graph was created using the pangenome graph builder (PGGB) pipeline (https://github.com/pangenome/pggb), which performs pairwise whole-genome alignment using wfmash and graph induction using the seqwish software.49 PGGB can then sort and order the graph via partial order alignment using smoothxg (https://github.com/pangenome/smoothxg).A sequence variation graph, termed the HBV reference graph, was created using the full set of phylogenetically representative reference sequences (N=44) .36 The H50 Fast short-read alignment via the VG giraffe mapper was accomplished by creating a haplotype-aware graph index where each reference genome was indexed as a unique haplotype.51 Highly accurate but more computationally intensive graph-based mapping was performed using the VG map mapper.The variation graph toolkit  was used to perform all graph-related format conversions, indexing, sequence-to-graph alignment, and collating of mapping/alignment statistics, as described in the VG documentation.52 These subsampled HBV sequencing datasets were aligned to the HBV reference graph using the haplotype-aware VG giraffe mapper. For the graph to be internally valid, we required >99% of the reads simulated from HBV genomes embedded within the graph to successfully align.All reads simulated from the graph-embedded HBV genomes were concatenated and then randomly subsampled to 20,000X coverage seven times. Coverage-based subsampling was performed using rasusa.To assess whether each path within the graph was utilized during sequence alignment, and to test whether aligned sequences had the highest alignment scores to graph-embedded HBV genomes which were more genetically similar to the aligned sequences, each full-length HBV sequence  was aligned to the graph using VG map. A ‘correct’ alignment was observed if the reference path with the highest alignment score was of the same HBV genotype as the query sequence. Path-specific alignment scores were also derived for alignments made using the set of simulated high throughput sequencing data from HBV genomes not used in graph construction (N=59). Briefly, by identifying the graph nodes for each path with successful alignments, the genome path with the most alignments was able to be identified . Path-sp53 respectively. While comparisons of the computational time and resources required for variation graph and linear-reference-based aligners have been performed previously,50 ‘/usr/bin/time’ estimates for the alignments using BWA-MEM, VG giraffe, VG giraffe in fast-mode, and VG map can be found in 55To determine whether a graph-based reference improves sequence alignment compared to linear reference-based approaches for HBV sequencing datasets, we aligned the combined simulated high-throughput sequencing data (generated from 59 HBV genomes not included within the graph) to the graph using VG giraffe and to each linear reference sequence (N=44) using BWA-MEM. The proportion of successfully aligned reads was obtained using ‘VG stats’ and ‘SAMtools flagstat’,We estimated the depth of coverage across the HBV genome for the alignment of all simulated high-throughput sequencing data to each linear reference using ‘SAMtools depth’. Genotype-specific depth estimates were obtained by estimating the mean alignment depth across alignments made using references of the same genotype via a sliding-window approach (50bp wide) in R. Local minima in depth were estimated using the ggmisc package in R. To approximate site-specific depth of coverage across the HBV genome from the graph-based alignment, the start site of each successfully aligned read was used to infer coverage by estimating a rolling sum of the median number of reads within a sliding window the length of the simulated reads (125bp).56To facilitate the comparison of whether regions of poor coverage corresponded to loci of increased pairwise diversity, the local nucleotide sequence diversity across the set of reference sequences (N=44) was estimated using a sliding window approach (150bp wide) in R using the pegas package.57 Alignment quality was assessed using Qualimap (v2.2.1).58 The proportion of successfully aligned reads were estimated using ‘SAMtools flagstat’. For each sample, the linear reference with the highest proportion of successfully aligned reads was an HBV subgenotype B2 sequence (GenBank ID: GU815637). For alignments to this reference, mean depth of coverage ranged from 82,930X-334,157X. To reduce computational time and resources required for our analyses, QC-passed reads were down-sampled to obtain an average coverage of 20,000X. To test whether subsampling altered the proportion of successfully aligned reads, subsampled reads were also aligned to each linear reference sequence and the proportion of successful alignments was compared to the alignments involving all QC-passed sequencing data using a binomial generalized linear mixed model (GLMM) with random intercepts in R. The GLMM treated each alignment as a binomial outcome , with the total number of reference-specific alignments used as weights. Whether alignments of these subsampled reads to an extended linear reference, obtained by concatenating the first 120bp of each reference to the end of each sequence, altered alignment statistics were also assessed using the same GLMM performed in R. To identify whether reads which failed to align to sub-optimal linear references (non-HBV subgenotype B2) were uniformly distributed across the genome, unaligned reads from each linear reference-based alignment were re-aligned to the best performing linear reference. The genome-wide distribution of these ‘rescued’ reads was visually assessed in R.To determine the approximate sequencing depth for each CHB sample (N=5), raw sequencing data were aligned to each linear reference sequence (N=44) using BWA-MEM.de novo (reference-free) assembled HBV haplotypes from each sample created using Bifrost and visualized with Bandage.60 We also performed BLAST in Bandage using these successfully re-mapped reads against the HBV reference graph only to confirm that rescued reads mapped to regions of increased graph complexity.Graph-based alignment of subsampled CHB sequencing data was performed using the VG map mapper. For samples with higher alignment proportions to the graph than any linear reference, unmapped reads from the best performing linear reference for each sample were re-mapped to the graph and the genome-wide distribution of the reads ‘rescued’ via graph alignment was visualized using R. To identify and visualize the loci where HBV sequence was rescued via graph alignment, the rescued reads were queried via BLAST against a compacted de Bruijn Graph comprised of the reference sequences and 61 iVar was developed to analyze amplicon-based viral sequencing data and leverages SAMtools to call variants and derive a consensus from the most common nucleotide across each position in an alignment file. We used a minimum base-level quality score of 20 and depth threshold of 10 while accounting for ambiguous nucleotides. For graph-based alignments, we used a wholly graph-based variant calling approach leveraging the alignments across all paths using VG (https://github.com/vgteam/vg), followed by consensus generation via bcftools.62Consensus sequences were obtained from linear reference-based alignments of simulated non-graph derived sequencing data using iVar.64 Reads were aligned to each linear reference or the HBV reference graph, followed by iVar-based consensus sequence identification. We also performed variant calling using the LoFreq software, a variant calling tool able to identify even low-frequency variants from high-coverage data across diverse genetic sequencing datasets,65 for each linear reference-based alignment followed by consensus generation using a majority allele rule for each site  via bcftools. For LoFreq-derived consensuses, we used the same depth threshold (10) used in iVar and estimated insertion/deletion qualities which were used in addition to LoFreq’s method of combining base-level, mapping, and alignment quality information to determine variant quality and identify the majority nucleotide at each position, accounting for insertions/deletions. Graph-based alignment was performed using VG giraffe. VG-based variant calling using the graph alignments and consensus generation were obtained via bcftools. For these CHB samples, we also derived consensus sequences after re-aligning the successful graph-aligned reads to a single path within the graph via VG, termed surjection, followed by iVar consensus construction. Graph-aligned reads were surjected into the path corresponding to the best-performing linear reference.Prior to performing alignment and variant calling for the real CHB samples, QC-passed paired-end reads were merged using PEAR and filtered to retain the highest quality reads >150bp long for analysis via bbmap.66 which estimates a genetic distance metric, the Mash distance, based on the estimated mutation rate between two sets of sequences and the jaccard index (the fraction of k-mers shared between the comparison sequences). The Mash distance also approximates average nucleotide identity (ANI) estimates, with ANI equivalent to one minus the distance estimate, while also having the benefit of facilitating comparisons between sequences/sequencing datasets of variable lengths/sizes.66 Given the short length of the HBV genome (3.2kb), a k-mer sequence length of 7 was used for Mash distance estimations.68 The consensus sequence with the lowest estimated genetic distance with the set of full-length HBV genome sequences can be inferred to be the most accurate or genetically representative consensus sequence.Comparisons between each simulation-based consensus sequence and the full set of HBV genomes from which reads were simulated were performed using Mash,de novo HBV strain-level assembly using SAVAGE and VG-Flow to identify the viral haplotypes comprising each CHB infection.70 For each sample, the best-performing linear reference was added to the SAVAGE output for VG-Flow to improve strain-level contiguity and assembly. The set of sample-specific viral haplotypes with frequencies >1% were included in all pairwise genetic distance comparisons. The consensus sequence with the lowest estimated genetic distance with the HBV-specific high throughput sequencing data can be inferred to be the most accurate and genetically representative consensus sequence for each sample.To facilitate comparisons between CHB-derived consensus sequences and to identify the most genetically similar consensus to the HBV quasispecies of each sample, we estimated the Mash distance between each consensus and the subsampled HBV sequencing data which aligned to the best performing reference (linear or graph-based) for each sample. We also performed Short Illumina-like reads were simulated from 59 genetically diverse HBV genomes encompassing 9 HBV genotypes and aligned to a non-overlapping set of 44 phylogenetically representative HBV genome sequences reflecting the known breadth of HBV diversity. Despite all reads being of HBV origin , only 84.3% to 96.6% of sequences successfully aligned to these 44 linear references . In contTo reflect a more realistic scenario utilizing simulated HBV sequencing data, we limited the simulated data used in our alignment comparisons to those generated from HBV sequences of genotypes B or C. While >99.9% of simulated reads from genotypes B and C successfully aligned to the reference graph, a high proportion of these reads also successfully aligned to linear reference sequences of genotype B (97.9%-98.4%) and C (97.6%-98.8%) .The simulated HBV sequences that failed to align to the linear references (3.4%-15.7%) were not uniformly distributed across the genome, with loci observed to have precipitous drops in coverage corresponding to loci of increased genetic diversity . Within To determine whether graph-aligned HBV sequences aligned best to the path/reference sequence embedded within the graph of the same HBV genotype as the query sequence, all full-length HBV genome sequences  and each set of simulated short-read HBV sequencing data generated from the HBV genomes used in the simulations (N=59) were aligned to the HBV reference graph. For each alignment, query sequences always resulted in paths of the same HBV genotype having the highest alignment score , demonstGU815637) resulted in 98.5%, 98.6%, 92.4%, 99.3%, and 98.6% of aligned sequences for each sample , which were all lower than the proportion of successful graph-based alignments. Notably, differences were also observed between references of the ‘correct’ HBV genotype (B), with a per-sample difference between the best and worst-performing reference ranging between 7.2% (85.3% vs. 92.4% for SRR7471501) and 7.8% (90.8% vs. 98.6% for SRR7471513). Thus, compared to graph-based alignment across these samples, up to 8.8% of HBV sequencing data can be missed due to the use of a linear reference sequence of the correct HBV genotype compared to the HBV reference graph.Unlike our simulated HBV sequencing datasets, real CHB-derived HBV sequencing data can reflect extensive genetic variation due to both host and pathogen-derived evolutionary pressures in addition to any sample processing or sequencing-related errors. Additionally, real CHB sequencing data can have highly variable quality and coverage distributions across the genome. For the analyses of real longitudinally collected CHB samples, graph-based sequence alignment consistently achieved higher proportions of successfully aligned HBV sequence data compared to any single linear reference (N=44), with 98.6%, 98.7%, 93.5%, 99.4%, and 98.7% of successfully aligned sequence for samples SRR747499, SRR747500, SRR747501, SRR747502, SRR747513 , respect55 For reads which still failed to align to the best performing linear B2 reference sequence across each sample, 30.5%-63.2% were rescued via graph-based alignment . The distribution in the start sites of all rescued reads was similarly non-uniformly distributed. Interestingly, the loci in which graph-based alignment rescued the most reads also corresponded to loci with increased pairwise nucleotide diversity estimated across the 44 phylogenetically representative proposed HBV reference sequences . Additionally, there was no significant difference in the proportion of aligned reads when the full-length linear reference sequences or extended linear reference sequences were used across the linear reference-based alignments (P=0.76) .−2 and 7.82x10−2. Using the Mash distance as an approximation of average nucleotide identity (ANI), consensus sequences had ANIs ranging between 92.2%-92.5%, with the consensus inferred from VG-based variant calling having the highest ANI (92.5%).While no single full genome-length HBV sequence could realistically capture the sequence variation observed across our simulated high throughput HBV sequencing data, the graph-based variant calling performed using the variation graph toolkit (VG) provided a consensus sequence with the lowest genetic distance to the full set of HBV genomes used in the simulations  mash dis−2 and 6.17x10−2, and ANIs ranged between 93.8% and 94.0% . While both iVar and LoFreq can be used to identify variants across a diverse set of viral pathogens,72 LoFreq has repeatedly been used to identify variants from real CHB-derived HBV sequencing data.74 Additionally, while both consensus identification methods used the same site-specific depth threshold, our LoFreq-based approach accounted for insertions and deletions, potentially explaining its consistently lower Mash distance compared to the consensus sequences obtained via iVar  over attempts involving linear reference sequences when variants were identified via LoFreq and every sample other than SRR7471499 when consensus sequences were generated using iVar . For thivia iVar .−1 and 1.27x10−1, for SRR7471499, SRR7471500, and SRR7471502. For SRR7471501, distances ranged between 1.07x10−1 and 1.30x10−1 and for SRR7471513 ranged between 1.07x10−1 and 1.28x10−1. Mash distances from the LoFreq-derived consensus sequences ranged between 1.07x10−1 and 1.10x10−1 for SRR7471499, SRR7471501, SRR7471502, and SRR7471513 and 1.07x10−1 to 1.09x10−1 for SRR7471500. VG-based variant calling derived consensus sequences each had Mash distances of 1.07x10−1. Additionally, we observed no differences in the Mash distances between consensus sequences derived using reads surjected into the HBV subgenotype B2 path  and the linear reference-based alignment derived consensus using the same B2 reference.For linear reference iVar-based variant calling across all samples, Mash distances ranged between 1.07x10In this study, we confirm that the choice of reference plays a critical role in the alignment of high throughput HBV sequencing data and can influence the construction of sample-specific consensus sequences in genetic studies of CHB. We also demonstrate that sequence variation graphs can improve upon widely accepted methodologies used for sequence alignment of highly diverse pathogens such as HBV. Using both real CHB and simulated high diversity HBV sequencing datasets, we show that alignment to a phylogenetically representative reference graph results in a higher proportion of successful sequence alignment and facilitates the generation of accurate sample-specific consensus sequences.As the benefits of sequence-to-graph alignment are greatest for highly diverse sequencing datasets, the utility of graph-based sequence alignment is dependent upon the research question of interest. For example, sequence-to-graph alignment recovers only marginally more simulated sequencing data generated from a subset of HBV genotype B and C sequences  compared40 continue to be developed. In addition to providing an accurate characterization of the genome comprising a clinical infection, publicly available consensus sequences enable molecular epidemiology-focused research, allow for large-scale phylogenetic analysis, and can aid disease surveillance efforts.78 Consensus sequences can also serve as the ‘reference’ sequence in subsequent bioinformatic analyses, reducing the number of spuriously identified variants for HBV.32 Thus, care should be taken to ensure the most accurate and genetically representative sequences are obtained from clinical CHB or other HBV-infected samples. We show that graph-based alignment and variant calling can often improve upon linear reference-based approaches to derive sample-specific consensus sequences, even when such efforts utilize reference sequences of the correct HBV subgenotype, which produces consensus sequences less genetically similar to an individual’s CHB quasispecies than sequences derived via graph alignment. However, given the minute differences in average nucleotide identity between consensus sequences obtained from any of the best performing linear references and our graph-based approach, the deleterious consequences of linear reference-based alignment are likely minimal when effort is made to first identify a genetically representative reference sequence for use in alignment.The generation of consensus sequences is an important product of microbial-focused genomic analyses, and novel software and workflows devoted to generating pathogen consensus sequences, including for SARS-CoV-2,79 It is therefore possible that some publicly available full-length HBV genome sequences are not the most accurate HBV-related sample-specific consensus sequences. While alternative approaches to infer sample-specific consensus sequences exist,32 our approach using the Mash distance to compare and identify the consensus sequence that best approximates the set of HBV sequencing data or de novo assembled haplotypes could provide a mechanism by which sample-specific consensus sequences are compared and selected for use as ideal reference sequences.However, when reference selection is not carefully considered, unrepresentative reference sequences can impact the fidelity of consensus sequences and other downstream phylogenetic-focused analyses.82 with further improvements expected.83While sequence-to-graph alignment requires more computational resources than linear alignment-based approaches, especially for the VG map mapper , if the 85 their performance has not been assessed using HBV sequencing data. Furthermore, an added benefit of graph-based approaches is that differences observed between the embedded paths/reference sequences can be utilized during variant calling to identify loci of genetic variation, in addition to mutations inferred from the alignment of sequencing data directly. The ability to leverage graph topology was demonstrated in our use of path depth to infer the genotype of HBV sequences aligned to the reference graph. Alternative graph-based prediction methods, including models for HBV subgenotype prediction or recombination detection, are worth further exploration. Whether metrics linked to graph topology or complexity, including path depth, can be used to better characterize the viral genetics of CHB quasispecies, either within specific regions or across the genome,87 or the genetic diversity of HBV generally, remains unexplored. For example, we observe the distribution of path depth within the phylogenetically representative HBV reference graph approximates a universal gene frequency distribution typical of many bacterial species (34 despite there being no distinction between a core and accessory genome for HBV.While alternatives to graph-based alignment which leverage multiple reference sequences have also been developed, such as alignment using multiple linear references in tandem, species ,34 despiFuture efforts should investigate the utility and potential clinical importance of these graph-derived measures of genetic complexity for HBV and other microbial pathogens of public health importance. For example, graph-to-graph comparisons could enable the analysis of genetic sequence data in ways that Euclidean data structures cannot.Graph-based approaches are being increasingly used to investigate highly genetically diverse microbial pathogens and regions of the human genome. In this study, we demonstrate the limitations of using linear HBV reference sequences to derive consensus sequences for CHB samples. Furthermore, we hope to mitigate issues of HBV reference ambiguity by making this HBV reference graph publicly available, which will also promote the use of graph-based advances in genetic analyses to improve our understanding of CHB genetics.Supplement 1


SpTransformer (SpTrf) gene family encodes a set of proteins that function in the sea urchin immune system. The gene sequences have a series of internal repeats in a mosaic pattern that is characteristic of this family. This mosaic pattern necessitates the insertion of large gaps, which has made alignments of the deduced protein sequences computationally difficult such that only manual alignments have been reported previously. Because manual alignments are time consuming for evaluating newly available SpTrf sequences, computational approaches were evaluated for the sequences reported previously. Furthermore, because two different manual alignments of the SpTrf sequences are feasible because of the multiple internal repeats, it is not known whether additional alternative alignments can be identified using different approaches. The bioinformatic program, PRANK, was used because it was designed to align sequences with large gaps and indels. The results from PRANK show that the alignments of the internal repeats are similar to those done manually, suggesting multiple feasible alignments for some regions. GUIDANCE based analysis of the alignments identified regions that were excellent and other regions that failed to align. This suggests that computational approaches have limits for aligning the SpTrf sequences that include multiple repeats and that require inserted gaps. Furthermore, it is unlikely that alternative alignments for the full-length SpTrf sequences will be identified.The  R genes in higher plants , to alin genome . Althougn genome . We furtSpTrf gene sequences (n = 134) from the genomes of four sea urchins were compiled from the SpTrf genes identified in the sea urchin BAC library . The first exon and the intron were removed to focus the analysis on the second exon that encodes the mature protein , and franimal G . This daSpTrf gene sequences were translated to the deduced protein sequences in BioEdit [ver 7.2.5  as a by-product of the PRANK alignment process. Additional neighbor joining trees were produced from the PRANK alignments in GUIDANCE2 (http://guidance.tau.ac.il) using MEGAX.MEGAX [are.net/ ] was usehttp://wasabiapp.org/software/prank/  produced in webPRANK (NJGT-PRANK) (http://wasabiapp.org/software/prank/prank_differences/), and are due to low confidence in branch placement and short branch lengths. Errors in tree structure may lead to errors in the translated codon alignments. Therefore, a maximum likelihood tree, which is considered a more robust method for phylogeny construction of highly variable sequences  into an MSA rather than only point mutations . PRANK iequences –13, 22. equences , was proThe imperfect, tandem Type 1 repeats positioned at the N terminal end of the mature SpTrf proteins  are desiElements Er6 through Er10 as defined by the repeat-based alignment are positioned in the histidine rich region of the deduced proteins 3A. TheyA to G element patterns) because it has a highly variable number of histidines among the genes, and specific sequence variants of Er10 are associated with specific mosaics of elements  are a series of Type 2 to Type 6 repeats that make up a duplicated, interspersed element pattern of 6-2-4-2-3-5 . This reBecause S-Er10 was aligned with L-Er18 in the MLGT-PRANK alignment, the region including L-Er11 to L-Er16 located to the N terminal side of L-Er18 were aligned with a large gap in the S proteins S3. Thishttp://guidance.tau.ac.il/ver2) is used to evaluate the robustness of the PRANK alignments based on the reliability of columns , sequences, and individual amino acids for each sequence in the alignment. GUIDANCE scores are defined as the confidence level for specific positions in an MSA. The GUIDANCE2 server evaluates MSAs as a tertiary PRANK alignment that uses GUIDANCE2 (GUIDANCE2-PRANK) to report and visualize the GUIDANCE scores as bar graphs for each column. It was noted that the GUIDANCE2-PRANK alignment was different from the alignment generated by NJGT-PRANK. This was not unexpected given that PRANK has been documented as resulting in slightly different alignments from different runs of the same set of sequences (http://wasabiapp.org/software/prank/#Methods). Furthermore, while the GUIDANCE2 server only provided the GUIDANCE scores for the base or standard MSA, defined as the first alignment generated upon which additional phylogenetic trees and alignments were generated to calculate the GUIDANCE scores, additional MSAs were available as a SuperMSA file that contained alternative alignments generated by the program as a concatenation of the top 20 alternative PRANK aligned MSAs (data not shown). The base alignment from GUIDANCE2-PRANK showed the same type of alignment variations as those from the NJGT-PRANK alignment, which were also evident with most of the SuperMSA alignments to various degrees. For example, for the Type 1 repeats, the B2-10, E6-2, and C2-4 element pattern proteins with variable numbers of repeats resulted in three different gap insertions in all of the other protein sequences  that result in elements that are defined by gaps . TherefoTo determine whether there were other alternative alignments, GUIDANCE2 was also used to evaluate a ClustalW alignment  and the confidence scores were compared to those for the GUIDANCE2-PRANK alignment. The GUIDANCE2-ClustalW alignment had no regions of no confidence but was composed of long regions of low or variable confidence  to LCS and awards from the Wilber V. Harlan Trust through the Department of Biological Sciences and a Dissertation Fellowship from the Columbian College of Arts and Sciences at GWU to MABH.The authors are grateful to Helen Dooley for recommending the PRANK program for use with the SpTrf sequences. Leon Grayfer and Damien O’Halloran provided helpful critiques to improve the manuscript.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.


Multiple sequence alignments are widely used to infer evolutionary relationships, enabling inferences of structure, function, and phylogeny. Standard practice is to construct one alignment by some preferred method and use it in further analysis; however, undetected alignment bias can be problematic. I describe Muscle5, a novel algorithm which constructs an ensemble of high-accuracy alignment with diverse biases by perturbing a hidden Markov model and permuting its guide tree. Confidence in an inference is assessed as the fraction of the ensemble which supports it. Applied to phylogenetic tree estimation, I show that ensembles can confidently resolve topologies with low bootstrap according to standard methods, and conversely that some topologies with high bootstraps are incorrect. Applied to the phylogeny of RNA viruses, ensemble analysis shows that recently adopted taxonomic phyla are probably polyphyletic. Ensemble analysis can improve confidence assessment in any inference from an alignment. Multiple sequence alignments are widely used to predict protein structure, function, and phylogeny, but are uncertain with more diverged sequences. Muscle5 generates ensembles of alternative high-accurate alignments, enabling novel confidence estimates in alignments, trees, and other inferences. Despite decades of research into automated alignment, current algorithms predict > 30% columns incorrectly on structure-based benchmarks5. Most alignment algorithms are based on highly simplified models of evolution parameterised by substitution scores and gap penalties. Default values for model parameters are somewhat arbitrary as they are trained on data of varying relevance to a particular set of sequences in practice. Systematic changes, and hence the opportunity for systematic errors, may be induced by changing parameters. For example, reducing gap penalties tends to increase the number of gaps. Most algorithms, including Clustal-Omega, MAFFT and MUSCLE, use progressive alignment according to a guide tree6 which may cause bias towards this tree e.g. in an estimated phylogeny7. However, standard practice is to construct a single MSA using some preferred method and proceed on the assumption that bias  can be neglected.Multiple sequence alignment (MSA) algorithms are ubiquitous in molecular biology, with popular software such as Clustal-OmegaH-ensemble) of high-accuracy MSAs (replicates) such that no particular MSA from the collection (or by any other method) is preferred a priori , a consistency transformation9 is applied, and the final MSA is constructed by maximum expected accuracy pair-wise alignments10 by progressive alignment according to a guide tree. HMM parameters and its guide tree are fixed in one replicate and varied between replicates to maximise differences in bias between replicates without degrading accuracy. If each replicate has different bias, then averaging results over replicates can correct for bias, and comparing results from different replicates can assess whether bias is important in a particular downstream analysis. Variations are introduced by multiplying HMM probabilities by a random number in the range − 0.25 …  + 0.25, the largest range found to maintain accuracy on structural benchmarks. The guide tree, an important potential source of bias, is also varied by permuting the joining order of large subgroups close to the root. A divide-and-conquer strategy enables scaling to tens of thousands of sequences  of inference from an MSA is the fraction of replicates which supports it. For example, if all replicates support the same inference, then HEC = 1. HEC is calculated using a diversified H-ensemble, which is designed to generate the greatest possible variety in the alignments, especially in systematic errors, so that averaging over the ensemble mitigates MSA bias. This approach can be applied to the alignments themselves: the Column Confidence (CC) is the HEC of an alignment column, i.e. the fraction of replicates where the column is reproduced. Unlike typical conservation-based metrics, a column with many gaps or with biochemically dissimilar amino acids will be assigned high CC if it is consistently reproduced. Alignment Confidence (AC) is the mean column confidence of a replicate, and MAC is the mean AC over the ensemble. If MAC = 1, all replicates are identical and the alignment is robust; if MAC is smaller, then the alignment is more sensitive to small parameter adjustments. Differences between alignments necessarily reflect errors, and lower MAC values, therefore, necessarily indicate higher error rates in a typical MSA from the ensemble. The robustness of a phylogenetic tree against MSA bias can be assessed by comparing replicate trees, i.e. trees estimated from different alignment replicates  where some parameters are held fixed while others are varied. Comparing results from different strata enables the detection of particular types of bias in any inference from an MSA.Guide tree bias can be assessed by comparing results on replicates where the guide tree is held fixed. If inferences differ and correlate with the choice of guide tree, then guide tree bias is present by definition. In general, a In this work, I show that high-accuracy alignment ensembles enable novel, unbiased metrics of confidence in alignments and inferences therefrom.4 for proteins and Bralibase5 for RNA) is higher than state-of-the-art represented by Clustal-Omega and MAFFT. There is a negligible difference in average accuracy between parameter variants, showing that all replicates are equally plausible a priori. On a benchmark with 10,000 protein sequences per set, Muscle5 aligns 59% of columns correctly, which is a 13% improvement over Clustal-Omega (52% columns correct) and 26% over MAFFT (47% correct)  gene, which is widely used for phylogenetic and taxonomic analysis of viruses11  and minimum-evolution (ME) using MEGA16. Trees were rooted using outgroups Torovirus for genus and reverse transcriptases for phylum. As shown in Fig. Alphacoronavirus, B = Betacoronavirus, G = Gammacoronavirus and D = Deltacoronavirus), while the faster but more approximate methods MEGA-NJ and FastTree reported more variants. The combined ensemble topology confidence of the consensus is 98.4% (82.5%) including (excluding) the latter two methods. A conventional analysis using the default Muscle5 MSA gave low bootstrap confidence to most edges  and subsequently used as the basis for introducing new taxonomic ranks including phylum17. I measured the monophylicity of the new phyla on a diversified ensemble. For each MSA, a tree was estimated using RAxML, and the best-fit subtree identified for each phylum. To investigate whether the Wolf2018 MSA might be more accurate than Muscle5, I checked the alignment of essential catalytic residues, finding that Muscle5 is better by this measure . Combined, these results strongly suggest that the high bootstrap values in the Wolf2018 tree are artefacts of biases in their MSA, and the newly-adopted phyla are far from monophyletic.A deep RNA virus phylogeny was recently reported inure Fig. , MethodsIt is de facto standard practice in biological sequence analysis to make a “best effort” analysis based on one preferred alignment, proceeding on the implicit assumption that this alignment is correct, or at least good enough to make the desired downstream inferences. Sometimes, columns considered to be less reliable (e.g. “gappier”) may be discarded, but the possible impact of any remaining biases are almost universally neglected, presumably due to a lack of awareness that they may be present combined with a lack of convincing methods for identifying and mitigating such errors. The results presented here show that alignment bias can have a significant impact, and as an important special case that tree bootstrapping from a single MSA may give high confidence to incorrect edges caused by bias in the MSA. In contrast to previous ensemble methods, Muscle5 generates alignments with a greater diversity of systematic errors (if present) while maintaining state-of-the-art benchmark accuracy, thereby enabling detection and mitigation of incorrect inferences due to MSA bias.18: the alignment is correct, sites evolve independently according to Markov models, the best tree is successfully identified for each resampled alignment, and the best tree will converge on the true tree as more columns are sampled from the underlying distribution, i.e., that the evolutionary model is a good enough approximation to identify the correct tree. All these assumptions are surely violated in practice. Alignments are often wrong. Sites are not independent, and their evolution is not strictly Markovian. The space of trees is too large to search exhaustively for more than a few leaves. The best tree may not be the true tree, which seems almost certain in non-trivial cases because evolutionary models are highly simplified. Thus, bootstrapping may be unreliable, as illustrated by examples presented in this work. Ensemble analysis assumes that alignment errors are sampled sufficiently across the ensemble to induce variations in downstream inferences comparable in size with a typical inference error. This assumption is violated if the alignment is robust against parameter perturbations but nevertheless wrong. Thus, for assessment of phylogenies, bootstrapping and ensemble analysis are complementary. If there is no variation in the ensemble, then standard bootstrapping alone is appropriate. If the ensemble is variable, ensemble confidence may be more credible than bootstrapping, as shown by the examples of coronavirus genera and RNA virus phyla where bootstrapping and ensemble confidence imply different conclusions but the ensemble is more credible.The Felsenstein bootstrap relies on several assumptionsensemble bootstrap), thereby accounting for alignment uncertainty in addition to under-sampling of columns19. However, I would not advocate relying on the ensemble bootstrap as a complete solution to accounting for MSA bias in phylogenetic tree estimation. While ensemble bootstrap values from a Muscle5 ensemble will surely be more reliable than conventional bootstrapping from a single MSA, there is no substitute for intelligently assessing robustness by replicating an analysis in distinctly different ways, paying particular attention to likely sources of systematic bias.Ensemble confidence can straightforwardly be applied to other inferences from an alignment, such as predicted secondary structure, paralog/ortholog discrimination and so on. The topology of putative clades can be assessed by comparing results when different subsets of sequences are selected; in particular resampling of sequences with replacement can be considered bootstrapping of rows rather than columns, noting that the rows should be realigned after resampling. If phylogeny is estimated from multiple genes, results can be compared on different subsets of genes. The Felsenstein bootstrap can be improved by sampling columns from an MSA ensemble rather than a single MSA  and cherry-picking22. In21 the author assessed the robustness of arthropod phylogenies under variation in transversion-transition probabilities and gap penalties, noting that “[the] disturbing circularity of the interaction between the specification of [insertion-deletion and substitution probabilities] a priori and their inference a posteriori is a general and central problem in molecular phylogenetics”. The Elision method22 concatenates variant MSAs before estimating a phylogenetic tree. A 1997 study23 of 18S ribosomal RNA in Apicomplexa assessed robustness of phylogenetic inferences using multiple alignments from different software packages, finding that “different alignments produced trees that were on average more dissimilar from each other than did the different tree-building methods used”. More recent proposals along similar lines include25. Muscle5 improves on previous ensemble methods in two crucial respects. First, all Muscle5 replicates have high accuracy such that no other MSA is preferred a priori, while in previous methods there is a clearly preferred MSA, i.e. the alignment generated by the algorithm and parameter combination giving the best benchmark score. In my terminology, previous methods generated S-ensembles while Muscle5 generates H-ensembles  transitions from the start state S (symmetric with transitions into the end state E), (2) transitions out of M, (3) transitions out of I, (4) transitions out of J, and (5) letter-pair emissions.Posterior probabilities for aligning every pair of letters in the input sequences are calculated using a hidden Markov model (HMM) with topology shown in Supp. Fig. SThe core component of Muscle5 is a parallelised re-implementation of ProbCons. Iterative consistency transformations are applied to the posterior probabilities (two rounds by default); a greedy maximum expected accuracy guide tree is constructed; and refinement by randomised partitioning is applied (100 rounds by default). Scaling to large datasets is achieved by a divide-and-conquer strategy ; details in P → (1 + αδ)P, where α (amplitude) is a constant and δ is a random number uniformly distributed between − 1 and + 1. All probabilities including transitions and emissions are perturbed. Perturbations are introduced before calculating posteriors, then the HMM is held fixed. This procedure is independent of the data; there is no attempt to optimise parameters. By default, α = 0.25, causing perturbations up to ± 25%. This value was chosen by trial and error on a subset of Balibase. To maintain normalisation , a second adjustment Pk → Pk/∑iPi is applied after all probabilities have been perturbed by the first rule. The sum in the denominator is over probabilities in the same set of mutually exclusive events as k, e.g. transitions from M. A single parameter (α) sets a scale for all perturbations, reducing possible concerns about over-fitting (too many parameters) and over-tuning  to a minimum.Perturbations are introduced by adjusting all probabilities according to the rule 6, which in turn requires that the guide tree joining order should be preserved close to its leaves. Substantive variations require that larger groups are joined in different orders. These constraints imply that the tree should be mostly unchanged close to the leaves and large changes should be made to the joining order of larger groups close to the root, but these goals can be conflicting in practice as guide trees are often highly unbalanced, i.e. many nodes join small groups to large groups, in which case naive re-arrangements of the tree may fail to induce substantive variations. With these considerations in mind, Muscle5 manipulates the guide tree T as follows. An edge is identified which divides the leaves of T into subsets a and bc such that the ratio ∣a∣/∣bc∣ ≈ 1/2, i.e. a has approximately one third of the leaves in T. The tree bc is then divided into subsets b and c of equal size so that ∣b∣/∣c∣ ≈ 1. Regardless of the original guide tree topology, when there are many leaves this procedure successfully divides T into three subtrees a, b and c of approximately equal size where the joining order close to the leaves is mostly preserved. Progressive alignment is performed using the original guide tree and permutations , c), , b) and , a), abbreviated to none, abc, acb and bca respectively. A replicate is identified as perm.s, e.g. abc.3, where perm is the guide tree permutation and s is the random number seed, where the special case s = 0 indicates that no perturbations are applied, i.e. default HMM parameters are used. See The goal of permuting the guide tree is to induce substantive variation into any systematic errors due to progressive alignment, without compromising accuracy. Optimising accuracy requires that closely-related sequences are aligned before more diverged sequences are addeds = 0, 1. . . N where N is the desired number of replicates, while guide tree permutations cycle through the four variants none, abc, acb and bca. For the results reported in this work, N = 100 was used. Alternatively, convergence criteria could be set which terminate generation of further replicates when sufficient diversity has been sampled, though this feature was not implemented in the code described here. For convergence, I suggest an upper limit of m replicates where m is the median number of columns in the ensemble so far, and also testing the number of singleton distinct columns  compared to the number of reproduced distinct columns . If n2 > n1, most of the potential diversity in the ensemble has been sampled. In practice, when sequences are closely related most of all replicates may be identical; in such cases setting convergence criteria would save substantial computing resources in high-throughput applications.A diversified ensemble is designed to maximise variation among replicates by setting the random number seed C is identified as the node t which minimises the number of errors in its subtree. The number of true positives (TP) is the number of leaves under t which belong to C. Errors include false positives , and false negatives .Given a tree and categories assigned to a subset of its leaves , the best fit for category C (e.g. phylum name), the best-fit subtree t is identified. Monophyly of C is then m = TP/(TP + FP + FN). If C is monophyletic, m = 1, otherwise m < 1, and m is smaller with increasing errors , the best-fit node is identified for each category  to identify three well-conserved motifs in the RdRp palm domain, which are conventionally designated A, B and C, respectively. These motifs include six essential catalytic residues: two aspartic acids (D) in motif A, a glycine (G) in motif B, and GDD in motif C30. The quality of an RdRp alignment was assessed by using Palmscan to identify the position of each of these catalytic residues in all sequences, which successfully matched the PSSMs. If a catalytic residue appears in a different column from the majority of other sequences, it is considered to be misaligned  performance is obtained for their method. These considerations beg the question of whether simulations could convincingly support the main claims of this paper, which are (1) Muscle5 MSA replicates have high and practically indistinguishable accuracy, and (2) the effects of alignment errors can be assessed by comparing inferences from different replicates. Claim (1) is supported by results on structure-based benchmarks. While structural similarity does not necessarily imply sequence homology, structural alignments are largely independent of sequence, and greater agreement with structural alignments therefore surely correlates strongly with more accurate alignment of homologous residues. If a simulation fails to recapitulate relative algorithm performance according to structure, the failure is more plausibly explained by a defect in the simulation than a defect in the structural benchmark. Claim (2) is self-evidently true because two different alignments of the same sequences cannot both be correct.Further information on research design is available in the Supplementary InformationReporting Summary


To model incongruent evolution, where sequence and structural features diverge positionally, we recently introduced bi-alignments. This generalization of sequence and structure-based alignments is best understood as alignments of two distinct pairwise alignments of the same entities: one modeling sequence similarity, the other structural similarity.Commonly, sequence and structure elements are assumed to evolve congruently, such that homologous sequence positions correspond to homologous structural features. Assuming Optimal bi-alignments with affine gap costs (or affine shift cost) for two constituent alignments can be computed exactly in quartic space and time. Even bi-alignments with affine shift and gap cost, as well as bi-alignment with sub-additive gap cost are optimized efficiently. Affine gap-cost bi-alignment of large proteins (Affine cost bi-alignments are of practical interest to study shifts of protein sequences and protein structures relative to each other.https://github.com/s-will/BiAlign/releases/tag/v0.3 and as bioconda package bialign.The affine cost bi-alignment algorithm has been implemented in Python 3 and Cython. It is available as free software from The online version contains supplementary material available at 10.1186/s13015-022-00219-7. While biological function is eventually encoded in a genomic sequence, it relies on the “decoding” of the sequence into a spatially structured RNA or protein, or into specific interactions, such as the binding of a DNA element by a transcription factor. Natural selection acts to conserve function over evolutionary times and therefore preserves functional RNA or protein structures, binding motifs, intron–exon boundaries, etc. Stabilizing selection on such a functional entity typically also causes the conservation of its encoding DNA sequence. Homologous functional units, i.e. those that share a common ancestry , are theintron-sliding, where the start and end of an intron “moves” in the same direction for the same number of nucleotides [Homology of a feature or trait, however, does not require that all its constituent parts are homologous. Most obviously, insertions and deletions in a DNA sequence imply that not all nucleotides trace back to a common ancestor even if the sequence as a whole does. Similarly, homology of a structural feature does not imply that all its constituent contacts are preserved. There are indeed well-documented exceptions to the by far most common case of homologous features being produced from homologous sequence positions. A well-studied, albeit apparently rare, example is leotides –9. Whileleotides . In the leotides , of whicleotides . This obTo our knowledge, incongruences between conserved protein sequence and conserved protein structures so far have not been studied systematically. However, the example of Fig. Incongruences between sequence homology and homology of structure or functional elements are rooted in the inherent redundancy of genotype-phenotype maps. For both RNA and proteins, very different sequences can encode the same fold or function –19, whilNot only is incongruent evolution of interest as an under-studied aspect of evolutionary dynamics, but it has practical implications for data analysis. Incongruences impact our ability to detect and reconstruct consensus structures, since corresponding structural features are formed by evolutionarily unrelated nucleotides or amino acids, while homologous sequence positions form disparate structural elements. This means that (in the presence of incongruent evolution) a single multiple sequence alignment cannot simultaneously represent the similarities of sequence and structure. In particular, conserved structure can no longer be represented as ‘consensus structure’, i.e. as an annotation of the columns of a sequence alignment.a posteriori from a pair of sequence and structure alignments that have been computed separately. Instead, bi-alignments allow simultaneously predicting sequence and structure homologies and their relation. For this purpose, we define a bi-alignment to consist of two alignments  that are related by a third alignment, which captures the shift events. All three constituent alignments contribute to a common score.We recently introduced bi-alignments , 23 as acombined sequence and structure alignments , bi-alignments extend such models by supporting shift events explicitly. Combined sequence/structure alignments therefore can be interpreted as the limit case of bi-alignments where arbitrarily high shift penalties completely prohibit shift events. As important consequence, bi-alignments overcome the requirement of a consensus structure, which is the key assumption underlying combined sequence/structure alignments.While bi-alignments have similarities to coherent framework to detect shift-like incongruences, i.e. a local “movement” of conserved structures relative to the underlying sequence. It is worth noting that the formal concept of bi-alignments is not tied to applications in structural biology. Instead, it can be seen as a way to quantify the effect of differences in scoring schemes that focus on different aspects of the same sequence. The only requirement for bi-alignments is a position-wise one-to-one correspondence between the two different representations of each input object.As their main purpose, bi-alignments provide a affine gap costs. We will illustrate our algorithmic developments using protein sequences and their secondary structures as an example, because the position-wise annotation of a secondary structure elements fits well with the framework of sequence alignments. The  example in Fig. In this contribution, we extend bi-alignments with linear costs to a more realistic model with bi-alignment is formally defined as an alignment relating two, generally different, alignments of the same objects.A bi-alignmentA annotation of sequence positions with a secondary structure features by an alignment of both the sequence and the string of structural features. Importantly, In Fig. mentclass2pt{minimlinear scoring model, i.e. scores for Assuming a mentclasspt{minima as well . Linear  as well , 25 in qA denotes a (pairwise) alignment, the terminal c denotes an alignment column, which may be a (mis)match c to the previously accumulated score of the alignment A. Denote by M(x) the optimal score of an alignment of the prefixes s is completely determined by a single column.To address this problem, it is helpful to describe the structure of alignments by multi-tape grammars, see e.g.  for a mo e.g. in , 28, thecursions  thus cancursions ) as1\docAffine gap cost. While linear gap costs are not very realistic in sequence alignment [O(n) in running time [affine gap cost model serves as a useful and convenient compromise that is most often used in practice. Here, the opening and the extension of a gap are scored differently. It is therefore necessary to distinguish three different non-terminal M the optimal score of an alignment of the prefixes c. We can then write Gotoh’s well-known recursions [lignment , arbitraing time , 32 and cursions  for pairk-ary alignments [Both the Needleman–Wunsch algorithm and the Gotoh algorithm run in mentclass2pt{minimignments , 34, 35, bounded . On the ment}k=3 –39.M.In this contribution we show that the bi-alignment model with affine gap costs for the constituent alignments can be solved in polynomial time by dynamic programming. As we shall see, the recursions are of the form of Eq.  but requRecall that we define a bi-alignment as an alignment of alignments Def. . It is wi-Alignment Problem for two input sequences u, v, and w. The special case where u, v, and w are linear scoring functions has been discussed in [The Bmentclasspt{minimaussed in .The alignment her Fig. . InsertiIfis a bi-alignment ofandthenFor column with Eq.  completeA natural scoring function for Lemma k of Eq.  as a resesult of  it is noi.x, y) identifying the prefixes a four-dimensional index  of a bi-alignment p of the last column of q of the last column of The p nor q consist only of gaps.The definition is illustrated in Fig. Now, we define a column-wise scoring function that captures the alignment score with affine gap cost. It scores a single column of a bi-alignment p, q). The terminals are the 15 possible column types of a 4-way alignment, which we write as In order to derive a dynamic programming algorithm that solves the bi-alignment problem with this type of scoring function, we consider a decomposition of the search space in grammar form. The non-terminals The language of bi-alignments with fixed end column type is generated by the productionsc, d) and end column type , and denote by c of its end column type from the previous alignment Consider an alignment Note that this grammar would allow terminating with any end column type. This is undesirable since we would like the first column to be scored as it was preceded by a match column in both p, q).Let In order to enforce that empty alignment is treated as having end column type The matricessatisfy the recursionp, q) and last column  as the optimal score of an alignment p, q) is gap pattern on the last column of We first note that every column of s in Eq. , and covr in Eq.  specifieAs an immediate consequence we haveThe bi-alignment problem with affine gap cost models for the two constituent alignments can be solved intime and space.While bi-alignment with affine gap cost and linear shift costs may be of the most obvious practical relevance, we also discuss two variations with affine shift costs. First of all, we clarify how to attribute affine shift cost in our bi-alignment scoring model.Let’s take a step back to our original definition of the bi-alignment score Eq.  and our For affine shift costs, we take the view that every consecutive run of gap symbols in the pairwise alignments of the two copies of p and q of the gap patterns in the respective alignments of rows We first consider affine shift cost and non-affine (i.e. linear) gap cost. Since affine shifts are scored exactly in the same way as affine gaps, this situation is symmetric to the case of affine gap cost combined with linear shift cost. The corresponding bi-alignment problem can thus be solved efficiently by applying exactly the same idea as in our previous algorithm Theorem , only noThe bi-alignment problem with affine shift cost models (and linear gap cost) can be solved intime and space.p that encode the respective gap patterns c.More remarkably, we can even solve the general case of affine gap cost and affine shift cost in polynomial time by dynamic programming. Essentially, we combine the ideas of the above two algorithms. Our algorithm follows a grammar with general decompositionp and c in Eq.  alignment with affine gap costs can be computed by dynamic programming following the above idea of keeping track of the right-most non-gap-only gap-patterns in all pairwise alignments. This requires considering k-way alignment thus needs exponentially many matrices in k.In bi-alignments of two sequences, we need to consider only four gap patterns, two for the two alignments and two for the shifts between the sequence copies. That is, there are (at most) p at the end of the alignment. With this notation we can writes(x) for The affine gap cost model, despite its algorithmic convenience, has been criticized because empirical gap length distributions usually are power laws thus suggesting a logarithmic gap costs . Howeverpractice . PairwisProtein sequences may be scored with an arbitrary, user-defined similarity matrix. The BLOSUM62 matrix , 44 is sProtein secondary structures are scored using a simple bonus  for matched secondary structure.The dynamic programming matrices are stored as sparse matrices to to limit space consumption to In case of ambiguity, simpler shifts are preferred  and Xanthomonas hortorum (length 933), while allowing shifts of sequence against structure by up to two positions to the left or to the right in either protein (As a proof of concept we generated optimal bi-alignments of DNA Polymerase I from Figure mentclass2pt{minimWe have shown here that bi-alignments with affine gap cost models for both constituent alignments and linear shift costs can be computed in quartic time by dynamic programming. Moreover, limiting the number of shifts to a constant reduces the cost to quadratic space and time. This makes the detection of locally-confined shifts computationally feasible for sequences of with length of realistic proteins or mRNAs. While we have illustrated our algorithmic innovations here using amino acid sequences and protein secondary structures as an example, the algorithm and its implementation is applicable to any linear representation of monomer-wise features along a biopolymer. In can be used, for instance, directly as an extension of the linear-gap-cost bi-alignments of RNAs described in .We have focused here on the analysis of optimization problem and development of the algorithm. In addition to cost models for the constituent alignments The formal framework of bi-alignments, Eq. , is muchructures  or as anructures , 47. Thiructures , 49 provructures , which cructures  to the cructures . In anotructures .k-way alignment k-way alignment k pairwise alignments with affine gap costs and additive cost contributions for the shifts between each pair of constituent alignments thus can be computed exactly in k. It is unlikely, therefore, that exact dynamic programming algorithms for these generalized problems will be practical. This begs the question whether poly-alignment problems can be approximated e.g. by progressive alignment schemes in a manner that is satisfactory from an applications point of view.In  we furthmmar Eq.  generaliAdditional file 1. Bi-alignments with different choices of λ. Bi-alignments of the same data as in Fig.


Biological data have gained wider recognition during the last few years, although managing and processing these data in an efficient way remains a challenge in many areas. Increasingly, more DNA sequence databases can be accessed; however, most algorithms on these sequences are performed outside of the database with different bioinformatics software. In this article, we propose a novel approach for the comparative analysis of sequences, thereby defining heuristic pairwise alignment inside the database environment. This method takes advantage of the benefits provided by the database management system and presents a way to exploit similarities in data sets to quicken the alignment algorithm. We work with the column-oriented MonetDB, and we further discuss the key benefits of this database system in relation to our proposed heuristic approach. Bioinformatics is an interdisciplinary field of molecular biology, genetics, computer science, mathematics and statistics . As biolA is adenine, G is guanine, C is cytosine and T is the equivalent of thymine. Sometimes other characters can occur in a sequence; for example, N indicates when sequencing has failed to determine which base is in a given position. Next-generation sequencing was established in the early 2000s. This technique does not read the whole genome at once, but only specific shorter regions, allowing for parallelization. The short DNA segments thus defined are called reads; the length of reads varies from technology to technology and is usually between 20 and 400 bases but can be up to 1000 bases long. The whole genome can be reconstructed from the reads. This is a splicing algorithm, where a reference genome is taken and the position of each read is searched, taking into account that the sequenced genome may contain mutations [One important area in bioinformatics is the analysis of DNA sequences. From an algorithmic point of view, these bases are of paramount importance, and therefore, DNA sequences are defined as a sequence of bases in bioinformatics.utations .The comparison of two sequences is called pairwise sequence alignment and can be local or global. The latter means that whole sequences need to be aligned, which helps identify the evolutional relationship between them. Based on the alignment scoring system, an optimal alignment exist: the one with the highest score, in our case. The Needleman–Wunsch dynamic programming algorithm  finds thAdvances in technology have created data sets of such size that storing and processing them has created new challenges for professionals. This is no different in biology, where a number of data warehouses have been created over the years . Often,There are many advantages to storing biological data in databases. For example, these systems usually have built-in features for managing different access rights, concurrent access or even data encryption. Fast, complex searches can be implemented by indexing up-to-date and expanding data sets. The question that we have been exploring is whether it is possible to efficiently perform pairwise alignment in a database environment where the sequences are already stored. Within the database, there are optimised representations of the data as well as optimised operators and functions for the different structures. With data-intensive problems, a common approach is to move the algorithm to the data, not vice versa, to reduce I/O operations.Given a database system as the environment, subresults can be stored during algorithm execution. Our idea was to store relatively good subalignments from previous alignments to build a knowledge base and to use them for upcoming alignments. For the subalignments, we take the optimal alignments of each pair from a randomly chosen sample set and split the representative strings into substrings. These are stored in a table and queried for suitable candidates if a new alignment needs to be performed. If the best candidate is found, it can be inserted into the alignment, which reduces the size of the problem significantly. After inserting subalignments, the remaining unaligned parts are aligned with a dynamic programming algorithm. Our measurements showed that with choosing the appropriate parameters, the algorithm outperforms the original solutions in time and gives a close-to-optimal result.Pairwise alignment may be the foundation of other important algorithms such as multiple sequence alignment. The development of methods for multiple alignment is an area of active research. For example,  presentsA and B of length n and m, respectively, this method creates a matrix of the size c is the scoring function for any two characters of the sequences or gap symbols (–). In this paper, we used score instead of distance; so in this case, matching characters have a positive value, while mismatches or gaps have a negative value.Over the years, a number of different approaches have been published to improve pairwise alignment. First of all, we introduce the original solution: the Needleman–Wunsch dynamic programming algorithm . Given tEquation , and evaEquation , where iTo define the optimal alignment, a backtrack phase should be performed. When a cell gets a value, the case that gave the maximum needs to be stored, and this would be marked as the previous cell. From the Equation , where kF, P, Q) and backtracking. The initialization of the matrices is shown in Equation . If one of the sequences contains the subsequence ‘ACA’ and the other contains ‘ACCA’, then the stored subalignment can be utilized. To avoid the need to always remove the gap symbols, we store the base strings to be searched for in a separate table called aligned_subseq. This also allows us to decrease redundancy: ‘–ACA’, ‘A–CA’, ‘AC–A’, ‘ACA–’; ‘ACA’ subalignment strings can be all bind to ‘ACA’ in the aligned_subseq table. In this table, each subsequence is given an id, and in the lookup table, the records refer to those ids which help in the searching phase.N, in the same way as for the A, C, G or T bases (with the same match and mismatch scores), most of them do it differently, e.g., BWA-MEM [N in their work [N. If this approach is followed, the question arises as to how to handle subalignment strings where there are Ns. There can be a number of ways to do this, which are beyond the scope of this paper, but one possible approach is outlined. For example, if one string contains N base(s) , then in addition to the string containing N, we can enter the bases A, C, G or T in all possible ways in the positions containing N and store the resulting strings in the aligned_subseq table  so that they can be used for further pairwise alignments.Although there are sequence analysis programs that handle degenerate bases, namely the unspecified or unknown base  BWA-MEM . Kim andeir work  and intrAfter filling the lookup and the aligned_subseq tables with records, the remaining pairwise alignments can be performed, which is Step 4. For each pair, the subalignment candidates must first be selected, i.e., for both sequences it must be determined which of the subsequences in aligned_subseq table they contain. The result of this query is the ids of the records and the indices where the subsequences begin in the original sequence, and for the next product, the result is sorted by the ids of the records in ascending order. Once we have this information for both sequences, we take the product of the results of the two queries and select the pairs of ids present from the lookup table. Here it should also be indicated which id belongs to which sequence, and this can be achieved, for example, by specifying a constant expression as an element of the select clause that makes up the query. The results are sorted by score because we want to start with the candidates with the highest scores.By completing the previous steps, we get the subalignment candidates for the given two sequences, supplemented by the indices on which the subsequences begin and the order of the sequences. At this point, we apply a filter: if the two start indices are too far away, then we can assume that many multiple gap symbols need to be inserted before the subalignment, which could result in an overall unsatisfactory alignment; therefore, we discard these candidates. The threshold we used was the following in the case of a fixed gap penalty: p and q, then we change the array values in these positions. Note that r is the length of the subalignment string without gap symbols. We can check if an upcoming alignment is overlapping by checking the values of the arrays, and if it is overlapping, then we discard it.From the remaining subalignment candidates, we always choose one with the highest score and check if it can be inserted in the alignment. It can be inserted if it is not overlapping with any previous subalignment. For this, we store an array of the size of each sequence, and if a subalignment is inserted between positions p, then we change the value of the array at p according to the following schema: For inserting the subalignment, we convert the sequence strings into arrays. If the starting position is By executing the steps described above, a heuristic pairwise alignment can be constructed. The proposed method is a heuristic approach: it is not guaranteed to find the optimal alignment in each case, but with the right configurations, the result is close to optimal. The process can be divided into two phases: pre-processing and alignment.The pre-processing phase is when the lookup table is filled with records. This can be performed periodically. For example, if sequences are stored for a specific virus and the data set expands with new data every week, then performance of the heuristic method should be monitored, and if the quality decreases with the new data, then the lookup table needs to be updated accordingly. We handle the complexity of this phase separately because it can be done between alignment phases, and if we store the previously run alignment results, then that information can be used to speed up this operation. In the Results section, we present how much time this operation takes with different parameter settings.k the length of the k-mers and z the size of the sample set. If we take a sequence pair and perform the alignment, then it can be split into k is small. , its performance is For every subsequence, a pattern-matching algorithm needs to be performed. The best case for single pattern matching is 2·havg)) . After wd tables .p, then the complexity changes as follows: the alignment of the subsequence before position p is p, it is If there is no subalignment for a pair of sequences, then the complexity of the alignment step is N bases, it becomes more complicated to handle subalignments. If we follow the approach given earlier as an example, then the estimation requires that the proportion of N within a sequence is denoted by Ns within a k-mer is, on average, N bases.As mentioned above, if we want to prepare our system for the presence of se cases , but in To measure the performance of the algorithm, two types of data sets were used: generated and real sequences. The generated sequences were created by the Rose  softwareThe real sequences were Porcine circoviruses  from thek: length of the subalignments;z: size of the sample set;d choice ) and −3,k value: The input parameters of the method are:We used MonetDB v11.44.0 installed from the source. The implementation contains SQL and Python codes and is available at . For comn pairs is We measured the goodness of the heuristic algorithm with the standard deviation of the alignment scores: let k was set to 5, the standard deviation varied over a large scale: for sequences of length 100, it was around 9, but for longer sequences, it went up to 268. For larger k values, the imbalances were diminished; for example, in the case of The heuristic approach can be used if there is at least one record in the lookup table that can be utilized for a given pair of sequences. Therefore, it was important to examine that how many times this occurred. We present these results as a percentage.k was set to 15, the size of the sample set had an impact: for size 10, the utilization rate was We also examined how many times the heuristic approach found the optimal alignment. These percentages are presented in the following graphs.k parameter increased, then the exact match rate increased. In the data sets with higher diversity and longer sequences, the optimal alignment was found at a lower rate. The presented results are connected to the standard deviation: when it was low, the exact match rate was high.In the case of the fixed gap penalty , if the k was set to 10 or 15, then the heuristic method found the best alignment in almost every case when it was usable.When the affine gap penalty was used , then, ok values. The size of the sample set had no influence on the run time, and we measured the pre-processing phase separately. In the following tables, timetimediff is the difference between them, and diffdiffA heuristic approach is adequate if it achieves a close-to-optimal result quickly. Therefore, the run times of the algorithms were measured for each data set with different k parameter setting, and on average, the difference was around In the case of the fixed gap penalty , for datFor the affine gap penalty, even the original algorithm took more time (it uses multiple matrices). The results shown in Despite the complexity estimation presented in the previous section, the proposed heuristic approach was faster for each data set.k); therefore, they are presented by sample size. Increasing the sample set size significantly affected the time taken to populate the table: for 10 sequences, 45 pairwise alignments need to be performed, while for 20 sequences, this number is 190.Inserting subalignment records into the lookup table is considered pre-processing, which is not strictly part of the run time of the heuristic algorithm. However, it is important to note how much time this phase takes, since it needs to be performed periodically. If we add the pre-processing time to the run time of the heuristic algorithm, we can calculate how many alignments the method needs for it to be paid off completely. For example, in the case of data set h100 and using 10 sample sequences, if the method can be used for at least 37 pairs of sequences, then the In the case of the affine gap penalty, the pre-processing took more time since the original alignment algorithm is more complex. Another example: for data set h300 and 20 sample sequences to gain time, the heuristic approach needs to be utilized 340 times, and our measurements showed that for pairwiseAlignment function from BioStrings [Using the algorithm inside the database environment has many advantages; however, the question arises of how efficient it is compared to a classical global pairwise algorithm. For the comparison, we chose the Bioconductor  R packagoStrings . This fuThe classical algorithm with file reading can construct the alignments in less time by approximately one and a half orders of magnitude, although its memory usage scaled with the length of the sequences more significantly, while the heuristic approach scaled less than linearly. Reading the sequences from the database added a significant overhead in time compared to reading from a local file. It is important to note that peak memory monitoring also contained the creation of the lookup table, but even so, it used less RAM than the other two methods.Creating data sets allowed us to observe how the characteristics of the input affects the performance of the algorithm, and it has achieved positive results. However, the important question is whether this improvement can be reproduced in a data set that contains real sequences.In the following, we present the measured results for a real data set containing Porcine circovirus sequences. It contained a hundred sequences, and the average length was 1642 bases. The length of the subalignments was set to 300, and the sample set contained 5 sequences. This parameter setting was chosen based on the observation of the previously shown results. The results are the average of 5 runs.Standard deviation: Average utilization rate: Average exact match rate: Average run time for one alignment with the heuristic approach: Average run time for one alignment with the original algorithm: Average pre-processing time: Results for fixed gap penalty:For comparison, the highest alignment score was 1937. The heuristic method took Standard deviation: 0.Average utilization rate: Average exact match rate: Average run time for one alignment with the heuristic approach: Average run time for one alignment with the original algorithm: Average pre-processing time: 319 s.Results for affine gap penalty:In the affine case, if the algorithm found at least one subalignment that it could use, it found the optimal alignment. However, only in about a quarter of the cases was such a match found. Since the heuristic method could not be applied to the majority of the sequence pairs, it is also important to know how long such an unsuccessful search takes, after which it returns to the Needleman–Wunsch algorithm. On average, it took The same search phase with the fixed gap penalty was Even with these extra times, the presented method was faster because the average difference between the two run times was around 2 and 9 h based on the chosen gap penalty.Based on the measured results, this heuristic method takes less time than the original dynamic programming algorithms and, with appropriate parameter setting, it provides a reasonable approximation of the optimal alignment. In the real data set with affine gap penalty, the heuristic approach found the optimal solution for every alignment in less time. The standard deviation of the alignment scores is strongly influenced by the length of the subalignments that have been stored and used. Measurements on a real data set have also shown that this method can be a favourable alternative to pairwise sequence alignment in a database environment. On the generated data sets, it took, on average, k values to find an alignment close to the optimal one, and this approach can be applied to the technique presented in this paper.This heuristic approach can be combined with other heuristics, which can accelerate the global pairwise alignment; therefore, other methods containing this step can be performed in less time. This can increase the use of databases in the bioinformatics community. Among the heuristics described in Different methods can be developed to populate the lookup table. For example, the Smith–Waterman algorithm  can be uTwo possible cases for scoring the alignments are implemented: fixed and affine gap penalty. However, similarity matrices can also be used. The presentation of the method focuses on DNA sequences, but the algorithm can also be applied to RNA sequences. In this variant, similarity matrices play a major role, for example, the ones based on mutation frequencies , since this is the most commonly performed search operation.In this paper, we present a heuristic algorithm for the global sequence alignment task using the MonetDB columnar database management system and for when a large number of similar sequences need to be aligned. The key concept of the method is that relatively high-scoring subalignments generated from the sample set can be stored in the database and reused in subsequent alignments, thus creating a knowledge base. By identifying and inserting these subalignments, the problem size can be significantly reduced. Our approach is optimized for a database environment, which eliminates the need to query the database externally, thus reducing I/O overhead while providing all the benefits of a DBMS.