Word-based or ‘alignment-free’ methods for phylogeny inference have become popular in recent years. These methods are much faster than traditional, alignment-based approaches, but they are generally less accurate. Most alignment-free methods calculate ‘pairwise’ distances between nucleic-acid or protein sequences; these distance values can then be used as input for tree-reconstruction programs such as neighbor-joining. In this paper, we propose the first word-based phylogeny approach that is based on ‘multiple’ sequence comparison and ‘maximum likelihood’. Our algorithm first samples small, gap-free alignments involving four taxa each. For each of these alignments, it then calculates a quartet tree and, finally, the program ‘Quartet MaxCut’ is used to infer a super tree for the full set of input taxa from the calculated quartet trees. Experimental results show that trees produced with our approach are of high quality. Sequence-based phylogeny reconstruction is a fundamental task in computational biology. Standard phylogeny methods rely on ‘sequence alignments’ of either entire genomes or sets of orthologous genes or proteins. ‘Character-based’ methods such as ‘Maximum Parsimony’ ,2 or ‘Ma‘Distance’ methods, by contrast, infer phylogenies by estimating evolutionary distances for all pairs of input taxa. Here, pairwise alignments are sufficient and can be faster calculated than multiple alignments, but still require run time proportional to the product of the lengths of the aligned sequences. However, there is a loss in accuracy compared to character-based approaches, as all information about evolutionary events is reduced to a single number for each pair of taxa, and not more than two sequences are considered simultaneously, as opposed to character-based approaches, where all sequences are examined simultaneously. The final trees are obtained by clustering based on the distance matrices, most commonly with ‘Neighbor Joining (NJ)’  or ‘BIONIn recent years, a large number of fast ‘alignment-free’ methods have been proposed for phylogeny reconstruction, see  for reviAnother advantage of the so-called ‘alignment-free’ methods for genome comparison is that they can circumvent common problems of alignment-based approaches such as genome rearrangements and duplications. Moreover, many alignment-free methods can be applied not only to entire genomes, but also to partially sequenced genomes or even to unassembled reads . A disad‘co-phylog’  is a recSince ‘co-phylog’ and ‘andi’ require a minimum length of the flanking word matches in order to reduce the number of matches that are mere random background matches, they do not perform well on distantly related sequences where fewer exact matches with the required minimum length can be found, if any at all. Moreover, the number of random segment matches grows quadratically with the length of the input sequences while the expected number of homologous matches grows only linearly. Thus, the minimum match length must be increased in these approaches if long sequences are to be compared to limit the number of background matches. This, in turn, reduces the number of homologous segment matches that are found, and therefore the amount of information that is available to estimate phylogenetic distances.Other alignment-free approaches are based on the length of maximal common substrings between sequences that can be rapidly found using suffix trees or related data structures ,27. As aProteome-based Spaced-Word Matches (Prot-SpaM)’ ’ , alignmeMultiple Spaced-Word Matches (Multi-SpaM)’ that combines the ‘speed’ of the so-called ‘alignment-free’ methods with the ‘accuracy’ of the ‘Maximum-Likelihood’ approach. While other alignment-free methods are limited to ‘pairwise’ sequence comparison, we generalize our previous ‘FSWM’ approach to ‘multiple’ sequence comparison. For a binary pattern P representing ‘match’ and ‘don’t-care’ positions, ‘Multi-SpaM’ identifies so-called ‘P-blocks’ consisting of four matching spaced words from four different sequences each. That is, a P-block can be seen as a gap-free ‘micro alignment’ of four different sequences, with matching nucleotides at the ‘match’ positions of the underlying binary pattern and possible mismatches at its ‘don’t-care’ positions, see Figure P-block, an optimal ‘Maximum-Likelihood’ tree topology is calculated with the software ‘RAxML’  = * if and only if P(i) = 0. A spaced word W can be considered as a regular expression where ‘*’ is a ‘wildcard character’. A position i ∈ {1, …, ℓ} is called a ‘match position’ if P(i) = 1 and a ‘don’t-care position’ otherwise. The number of match positions in P is called the ‘weight’ of P. For a DNA Sequence S of length n and a position 1 ≤ i ≤ n − ℓ + 1, we say that a ‘spaced word’ W with respect to P occurs in S at position i – or that  is an ‘occurrence’ of W – if S(i + j − 1) = W(j) for all match positions j of P. This corresponds to the definition previously used in \documentS, i], ) of occurrences of the same spaced word W is called a ‘spaced-word match’. For a substitution matrix assigning a ‘score’ s to every pair  of nucleotides, we define the ‘score’ of a spaced-word match  of length ℓ asW to each other, the score of the spaced-word match is the sum of the scores of the nucleotides aligned to each other. In ‘Multi-SpaM’, we are using the following nucleotide substitution matrix that has been proposed in (A pair ([posed in :(1)\docuP with user-defined length ℓ and weight ‘Multi-SpaM’ starts with generating a binary pattern P with minimal ‘overlap complexity (OC)’ is calculated by running our previously developed software tool ‘rasbhari’ ’ metric , a standAs explained above, both ‘FSWM’ and ‘Multi-SpaM’ rely on binary patterns of ‘match’ and ‘don’t-care’ positions; the results of these programs therefore depend on the underlying patterns. Both programs use the software ‘rasbhari’  to calcuγ-proteobacterial genomes , and three with simulated mammalian genomes .At first, we evaluated ‘Multi-SpaM’ on data sets generated with the ‘Artificial Life Framework (ALF)’ . ‘ALF’ sDNA sequences for roughly 1000 genes per taxon that we then concatenated to full genomes. As in (b1), 0.0018 (b2), 0.0017 (b3), 0.0058 (m1), 0.0068 (m2), 0.011 (m3), respectively. Each data set uses a different guide tree that was sampled from known topologies. The average pairwise distances in our simulated sequence sets are as follows (average number of substitutions per position as estimated with FSWM): m1: 0.11; m2: 0.12; m3: 0.07; b1: 0.30; b2: 0.23; b3: 0.25.We used the base parameter sets for each data set and only slightly modified them to generate s. As in , we usedmb. As shown in Figure Each data generated in this way set contains 30 genomes (taxa) and has a size of around 10 P-blocks that can be used by ‘Multi-SpaM’, but not so much the resulting quartet topologies. Even so, the ‘HGT’ rate in ‘ALF’ had only minor influence on the quality of the resulting trees, compared to the guide tree that is used in the simulation.We also evaluated how the parameters of the genome sequence simulator ‘ALF’ affect the performance of ‘Multi-SpaM’ on the simulated genomes. A figure showing the influence of these parameters is given in the Supplementary Data. In short, the rate of ‘horizontal gene transfer (HGT)’ has a larger influence on the quality of the resulting trees than other parameters of ‘ALF’. This is not surprising, since ‘HGT’ events can lead to false quartet tree topologies, whereas the other program parameters mostly affect the ‘number’ of Escherichia coli and Shigella genomes that are commonly used as a benchmark data set to evaluate alignment-free methods ’  in orderrk study , so usinP-blocks’ is a relatively time-consuming step in ‘Multi-SpaM’. In fact we observed that, for a given set of input sequences, the program run time of ‘Multi-SpaM’ is roughly proportional to the number of P-blocks for which topologies are calculated. However, the maximal number of ‘quartet blocks’ that are sampled is a user-defined parameter. By default we sample up to M = 1 000 000 quartet blocks; in our test runs, the quality of the resulting trees could not be significantly improved by further increasing M . Consequently, our method is relatively fast on large data sets, where only a small fraction of the possible quartet-blocks is sampled. By contrast, on small data sets, ‘Multi-SpaM’ is slower than other alignment-free methods. To further speed-up ‘Multi-SpaM’, we have parallelized our software to run on multiple cores; in Table Calculating optimal tree topologies for the sampled ‘quartet P-blocks, where no single best quartet tree exists, on data sets where sequences are closely related to each other.Apart from the maximum number of sampled quartet blocks, the only relevant parameters of our approach are the ‘length’ and the ‘weight’ (number of ‘match positions’) of the underlying binary pattern. For ‘Multi-SpaM’, we used similar default values as in ‘Filtered Spaced Word Matches (FSWM)’ , namely P-blocks with positive scores can only be expected in sequence regions without insertions or deletions. For real-world data sets, it is difficult to tell how exactly the number of possible homologous P-blocks depends on the pattern length—to find out, one would need either a reliable multiple alignment of the sequences or the full list of homologous P-blocks. But both are impossible to calculate for large genome sequences. As a proxy, to get an idea how the number of homologous ‘quartet’ P-blocks is affected by the pattern length ℓ, we counted the number of ‘pairwise’ spaced-word matches with positive scores for different patterns with a fixed weight and variable length. For various real-world genomes, we found that, with our default length ℓ = 110, the number of spaced-word matches with positive scores is only slightly smaller than with a pattern length of, for example, ℓ = 60. Details are shown in the Supplementary Data.Our default pattern length ℓ = 110 limits, on the other hand, the number of homologous quartet blocks that can be found. Since ‘Multi-SpaM’ is based on ‘gap-free’ four-way alignments of length ℓ, P, to obtain a sufficiently large number of P-blocks. On very large data sets, on the other hand, it is advisable to increase the weight of P, in order to reduce the number of the spaced-word matches that are considered, and thereby the program run time. For the largest data set our study, the set of plant genomes, we increased the pattern weight in our test runs from the default value In ‘Multi-SpaM’, we are using by default a relatively low ‘weight’ of the underlying pattern P-blocks. As detailed above, this is done by iteratively picking a random spaced-word occurrence, and by looking at other random occurrences of the same spaced word in different sequences, until three spaced-word matches with positive scores are found, i.e. until a homologous P-block is found. Since we are using patterns with a low weight, most random spaced-word matches have negative scores. The number of spaced-word matches that have to be evaluated, until a homologous P-block is found, depends on the input sequences and can vary considerably. This may be the reason why the relative run time of ‘Multi-SpaM’, compared to other methods, is rather variable, as can be seen in Table We should mention that it is, in general, not possible to predict the run time of ‘Multi-SpaM’ from the program parameters and the size of the input data alone. A relatively time-consuming step of our algorithm is sampling homologous To distinguish between homologous and background spaced-word matches, we are using a nucleotide substitution matrix that has been published by Webb Miller’s group , the samP-blocks that we have used in the present study.To calculate supertrees from quartet tree topologies, the current implementation of ‘Multi-SpaM’ uses the previously developed software ‘Quartet MaxCut’ ,55. We ahttps://github.com/tdencker/multi-SpaM.The source code of the program is available at Contact:thomas.dencker@stud.uni-goettingen.delqz013_Supplemental_FileClick here for additional data file.


For example, the number of vertebrate genome assemblies currently in the NCBI  database4 increased by more than 50% to 1,485 assemblies in the year from July 2018 to July 2019. In addition to this influx of assemblies from different species, new human de novo assemblies5 are being produced, which enable the analysis of not only small polymorphisms, but also complex, large-scale structural differences between human individuals and haplotypes. This coming era and its unprecedented amount of data offer the opportunity to uncover many insights into genome evolution but also present challenges in how to adapt current analysis methods to meet the increased scale. Cactus6, a reference-free multiple genome alignment program, has been shown to be highly accurate, but the existing implementation scales poorly with increasing numbers of genomes, and struggles in regions of highly duplicated sequences. Here we describe progressive extensions to Cactus to create Progressive Cactus, which enables the reference-free alignment of tens to thousands of large vertebrate genomes while maintaining high alignment quality. We describe results from an alignment of more than 600 amniote genomes, which is to our knowledge the largest multiple vertebrate genome alignment created so far.New genome assemblies have been arriving at a rapidly increasing pace, thanks to decreases in sequencing costs and improvements in third-generation sequencing technologies The Progressive Cactus program can create reference-free alignments of hundreds of large vertebrate genomes efficiently, and is used for the alignment of more than 600 amniote genomes. Multi-species genome alignment involves creating a mapping from each region of each genome to a corresponding region in each other genome, taking into account the possibility of complex rearrangements and copy number changes13. Genome aligners are one of the most fundamental tools used in comparative genomics, but because the problem is difficult, different aligners frequently give different results14, and many intentionally limit the alignments they produce to simplify the problem. Two of the most common limitations are ‘reference bias’, the result of constraining a multiple alignment to only regions present in a single reference genome, and restricting the alignment to be ‘single-copy’, which allows only a single alignment in any column in any given genome, causing the alignment to miss multiple-orthology relationships created by lineage-specific duplications. Cactus6 is a genome alignment program that has neither of these restrictions; it can generate a reference-free multiple alignment that allows the detection of multiple-orthology relationships.Comparative genomics analyses, including species-tree inference14, an evaluation of genome aligners. However, the runtime of that initial iteration of Cactus scaled quadratically with the total number of bases in the alignment problem, making alignment of more than about ten vertebrate genomes completely impractical. To address these difficulties, we present fundamental changes to the Cactus process that incorporate a progressive alignment strategy15, which changes the runtime of the alignment to scale linearly with the number of genomes. We show that the result, which we call Progressive Cactus, is an aligner that retains state-of-the-art accuracy, and continues to lack reference bias, but which is tractable to use on hundreds to thousands of large, vertebrate-sized input genomes. Progressive Cactus has been developed over several years, and has already been successfully used as an integral component of high-profile comparative genomics projects20.The version of Cactus available in 2012 performed very well in the AlignathonThe new Progressive Cactus pipeline is freely available and open source. The only inputs needed are a guide tree and a FASTA file for each genome assembly.21 and TBA/MULTIZ22. Cactus now follows a similar strategy, with the key innovation being that Progressive Cactus implements a progressive-alignment strategy for whole-genome alignment using reconstructed ancestral assemblies as the method for combining sub-alignments. This strategy  not only results in a much faster alignment runtime but also produces ancestral reconstructions.The key innovation of Progressive Cactus is to adapt the classic ‘progressive’ strategy  to a whole-genome alignment setting. Progressive aligners use a ‘guide tree’ to recursively break a multiple alignment problem into many smaller sub-alignments, each of which is solved independently; the resulting sub-alignments are themselves aligned together according to the tree structure to create the final alignment. Progressive alignment has been successfully applied to whole-genome alignment before—for example, by progressiveMauveFigure 24. These pairwise alignments are then filtered and combined into a cactus graph representing an initial multiple alignment using the previously described CAF algorithm6—although we note important changes to the filtering in Methods and Extended Data Fig. 6 to create a more complete alignment. The ancestral assembly is then created by ordering the blocks in this final alignment and establishing a most-likely base call for each column in each block. The resulting ancestral sequence is then fed into later subproblems .Each individual subproblem follows a similar procedure to the original Cactus process. The subproblem procedure begins with a set of pairwise local alignments generated via the sensitive pairwise local-alignment program LASTZ25 workflow framework to organize and distribute its computational tasks. Although genome alignment is a computationally intensive task, using Toil, we can break up the problem into small pieces that can work in heterogeneous compute environments, playing to the advantages of both cheap CPU-rich machines and more expensive memory-rich machines. Because it runs on Toil and supports container execution via Docker and Singularity26, Progressive Cactus can be run on many different environments: single machines , conventional clusters, and commercial clouds.As a practical matter, Progressive Cactus uses the Toil27, therefore supports the addition and removal of genomes from the alignment by taking advantage of the tree structure of the progressive alignments it produces  and mammal (F1 score of 0.795) clades than any aligner that participated in the Alignathon (Supplementary Tables The Alignathon simulated datasetshttps://www.drive5.com/evolver) along a tree of catarrhines. We ran two alignment strategies—one using a fully resolved binary guide tree (which takes full advantage of the new progressive mode), and one using a fully unresolved star guide tree —across variously sized subsets of genomes roughly evenly spaced throughout the catarrhine tree. The alignments using the progressive strategy finished more quickly, with the speed improvement growing larger with the increasing number of species , owing to its linear runtime scaling, as opposed to the quadratic scaling of the star-tree .We further examined whether these small differences in the guide tree affect some species more than others. For any pair of these 48 species, the 30, we find the same level of Jaccard similarity for both the high- and low-quality assembly alignments  or lower quality assemblies in the other alignment   project32, which includes 363 avians, also representing most bird families. The overall topology is shown in Fig. To demonstrate Progressive Cactus, we present results from an alignment of 605 amniote genomes, relating in a reference-free manner to more than 1 trillion bases of DNA across hundreds of millions of years of genome evolution . The amniote-wide alignment combines two smaller alignments: one created for the initial release of the Zoonomia project33.Coverage within the 600-way alignment closely tracks phylogenetic distance and genome size; for example, a median coverage on human of 2.3 gigabases (Gb) from Euarchonta mammalian species, versus 1.2 Gb and 1.0 Gb from more distant Laurasiatheria and Glires mammalian species, respectively Fig. . The anc34, the rate of small indels in any given branch was correlated with the rate of nucleotide substitution , although the relative rates remained lower for insertions (1.2% of the substitution rate for both clades) and for deletions . Notably, we observe similar rates of deletions between eutherian and avian lineages, but evidence of a slightly increased rate of insertions in avians ) insertion and deletion events in the 600-way alignment. As expected from previous studies14. As expected, the alignments more strongly agree in functional regions, such as coding exons, than for the genome as a whole  than either translated method (16–51%)  than MULTIZ (64.9%), for an average increase of 47 Mb. Because, unlike Progressive Cactus, MULTIZ is reference-biased, the difference is starker when looking at the number of bases aligned to a genome not used as the MULTIZ reference   pipeline. The root genome used was derived from 30 megabases selected from the hg19 genome, and is available at http://courtyard.gi.ucsc.edu/~jcarmstr/datastore/progressiveCactusEvolverSim.tar.gz along with the Evolver configuration files that were used. The species tree used for the simulation was obtained from a catarrhine subtree of the 100-way alignment tree available on the UCSC browser.Twenty primate genomes were simulated using Evolver, managed using the evolverSimControl anc0e:0.00122,Bonobo: 0.00784)anc1e:0.003,Gorilla:0.008964)anc2e:0.009693,Orangutan:0.01894)anc3e:0.003471,Gibbon:0.02227)anc4e:0.01204,anc5e:0.001,Sooty_mangabey:0.001)anc6e:0.005,Baboon:0.003042)anc7e:0.01061,anc8e:0.002)anc9e:0.003,anc10e:0.005,anc11e:0.004)anc12e:0.009)anc13e:0.02)anc14e:0.02183,anc15e:0.01065,White-faced_sapajou:0.009)anc16e:0.01,Nancy_Mas_night_monkey:0.01)anc17e:0.01)anc18e;http://courtyard.gi.ucsc.edu/~jcarmstr/datastore/progressiveCactus.EvolverSim.CactusInput.EvenlySpread.tar.gz. A non-default configuration (included in the dataset) was used to change the alignment filtering in both runs to better support the high degree of polytomy in the star-tree runs. Four sets of 2, 6, 10 and 20 genomes were used, each of which were run three times to generate runtime estimates. The sets are as follows: 2 species: rhesus and marmoset; 6 species: rhesus, marmoset, gorilla, drill and black snub-nosed monkey, white-faced sapajou; 10 species: species from 6-species alignment and human, sooty mangabey, proboscis Monkey and Nancy Ma’s night monkey; 20 species: all species.The alignments were generated using Progressive Cactus commit 51eb980b. The input files  are available at 27 package  with the --onlyOrthologs option using the rhesus genome as a reference) and compared with the Evolver MAF using mafComparator .The runtime statistics were gathered using the toil stats command  time spent across all jobs). To generate the recall and precision statistics, multiple alignment format files (MAFs) were exported for each run  to reanalyse the MAFs, with the output of the newest Progressive Cactus version added, to generate the precision/recall statistics (which we extracted from the comparison against the most recent common ancestor (MRCA) truth set). The simulated-mammal results are shown in Supplementary Table For comparison against other genome alignment methods, we aligned data  used in the Alignathon using Progressive Cactus. For comparison, we downloaded all the original Alignathon entries in MAF format. We used the original Alignathon analysis workflow . We used Progressive Cactus commit 36304707 for all alignments in this analysis.The guide-tree analysis was performed on a set of 48 bird genomes previously published28 and Jarvis et al.29, respectively. The ‘permuted’ topology was generated starting from the Jarvis topology, via three randomly chosen subtree-prune-regraft operations followed by three random nearest-neighbour-interchange operations. Each of these three topologies had branch-length estimates performed using phyloFit from the PHAST package  based on fourfold-degenerate sites of BUSCO orthologues. Finally, the ‘consensus’ tree was produced as a strict consensus of the Jarvis and Prum trees  using the ape::consensus method from the APE R package44. The branch-lengths for this tree were generated from the fitted branch lengths for the two input trees, using the consensus.edges function of the phytools R package45. The four final trees that were used in the four Progressive Cactus alignments are shown in Supplementary Fig. The Prum and Jarvis topologies were adapted from Prum et al.29 and Prum28 , and the branch lengths were estimated using the least-squares-fit mode of PHYLIP 48. The distance matrix used was largely based on distances from the 4d site trees from the UCSC browser49. To add those species not present in the UCSC tree, approximate distances estimated by Mash 49 to the closest UCSC species were added to the distance between the two closest UCSC species. We used the hal package to process the HAL file .The Zoonomia alignment was composed of two sets of mammalian genomes: newly assembled DISCOVAR assemblies and GenBank assemblies. The DISCOVAR genomes were masked with RepeatMasker commit 2d947604, using RepbaseThe final guide tree is embedded in the HAL file, and available using the halStats --tree command. The 363 assemblies in the B10K alignment comprised four sets: 236 newly sequenced species for the ‘family’ phase of the project, assembled using SOAPdenovo2 and AllpathsLG, 42 assemblies already sequenced from the ‘order’ phase of the project, 36 assemblies taken from GenBank, and 49 assemblies contributed by other research groups. For the avian guide-tree, we used a tree that the B10K consortium derived as preliminary data from ultraconserved elements.Both alignments were run on the AWS cloud over the course of 3 weeks for the avians and 2 months for the mammals, using a maximum of 240 c3.8xlarge instances and 20 r3.8xlarge instances. Because Toil’s autoscaling mode was used, this capacity was only fully used during the initial phase of the alignment, when the potential for parallelism was at its highest.The 600-way alignment was formed by aligning the two roots of the B10K and Zoonomia alignments, using the xenTro9 (frog), latCha1 (coelacanth), and danRer11 (zebrafish) assemblies as outgroups. This created a ‘linker’ alignment connecting the roots of the two alignments. The B10K and Zoonomia alignments were then added to this linker alignment using the halAppendSubtree command.https://github.com/ComparativeGenomicsToolkit/hal, commit 68db41d) tool on every branch in the 600-way alignment. The ungapped insertion and deletion calls  were filtered so that only calls spanning less than 20 bp  were counted. The rate for each branch was then obtained by dividing the count of these micro-indel events by the total amount of sequence present in the child.We extracted all insertion and deletion events by running the halBranchMutations  on all ancestral assemblies of human within the 600-way alignment . We also ran the same pipeline against human (as existing annotations used the ‘Homo_sapiens’ repeat library). All ancestors up to human-rhesus had over 78% of the human complement of L1PA6 elements  and chicken annotations are Ensembl 85 on galGal4 (GCA_000002315.2). Predicted RNA sequences for each protein-coding transcript are extracted from the genome. Only gene annotations on the primary assemblies were used, those on alternate loci, patches, and assembled sequences were dropped. This results in 84,001 transcripts in 19,695 genes for human and 15,328 transcripts in 14,499 genes for chicken. The human transcripts were then mapped from the human genome to the chicken genome. The steps for each method are outlined below, although the actual execution was done by partitioning the data and using a cluster. Command-line tools from the UCSC Genome Browser group and programs used came from: https://github.com/ucscGenomeBrowser/kent, commit 8a8d921, https://github.com/ComparativeGenomicsToolkit/hal, commit 68db41d, ftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/2.10.0/, version tblastn: 2.10.0, and https://github.com/lastz/lastz, version 1.03.54.Protein-coding transcript annotations were obtained from the UCSC Genome browserThe BLATX alignments were created using protein-translated mode to align the mRNAs to the target genome with BLAT version 36x5. They were then filtered following the same protocol the UCSC Genome Browser uses for creating the other species RefSeq alignments:blat -noHead -q=rnax -t=dnax -mask=lower <dest-genome.2bit> \<src-rna.fa> <dest-rna-raw.psl>We then filter to get near-best in genome. Alignment to chicken uses near best filter of -localNearBest = 0.010 while to human it uses -globalNearBest = 0.010:faPolyASizes <src-rna.fa> <src-rna.polya>pslCDnaFilter <nearBestOption> -minId=0.35 -minCover=0.15 -minQSize=20 \-ignoreIntrons -repsAsMatch -ignoreNs -bestOverlap \-polyASizes=<src-rna.polya> <dest-rna-raw.psl> <dest-rna-mapped.psl>The transMapPslToGenePred command is then used to project the original coding sequence (CDS) onto the alignment.The TBLASTX alignments were created using the protein-translated ‘tblastx’ program to align the mRNAs to the target genome with BLAST+ version 2.10.0+.The database is created using the repeat masking from the UCSC Genome Browser genomes to match what is used within the BLATX methodology above:convert2blastmask -in <dest-genome.fa> -masking_algorithm repeat \-masking_options “repeatmasker, default” -outfmt maskinfo_asn1_bin \-out <dest-genome.mask>makeblastdb -dbtype nucl -in <dest-genome.fa> -mask_data <dest-genome.mask>The mRNAs are aligned and the resulting XML converted to PSL format, filtering to an e-value threshold of 0.01. These are then chained using a program the UCSC group developed for chaining BLAST alignments:tblastx -db <dest-genome.fa> -db_soft_mask 40 -outfmt 5 -query <src-rna.fa> \-out <dest-rna-raw.xml>blastXmlToPsl -eVal=0.01 <dest-rna-raw.xml> <dest-rna-raw.psl>simpleChain -outPsl -maxGap=75000 <dest-rna-raw.psl> <dest-rna-chained.psl>The alignments produced are then filtered in the same manner as the BLATX alignments.50 projection alignment algorithm to project transcript annotation between genomes. The LASTZ alignment chains and nets52 were obtained from the UCSC Genome Browser downloads. These were then filtered to produce a set of syntenic mapping chains using these steps:Both the LASTZ and Cactus transcript mappings use the ‘TransMap’netFilter -syn <genomes.net> <syntenic.net>netChainSubset -wholeChains <syntenic.net> <genome.chain> <mapping.chain>The Cactus alignments are extracted for all primary chromosomes from the HAL file and chained using the same chaining algorithm as the LASTZ chains, with the –noDupes option having a similar effect as the syntenic net filtering:halLiftover --outPSL --noDupes 600way.hal <srcOrganism> \<srcChroms.bed> <destOrganism> <src-dest.psl> <genome.psl>axtChain -psl -linearGap=loose -scoreScheme=HoxD55.q <genome.psl> <mapping.chain>The ‘TransMap’ protocol is used for both the LASTZ and Cactus mapping chains to produce alignments of the transcripts to the other genomes. This used the ‘pslMap’ command to do the mapping and ‘pslRecalcMatch’ to update the statistic in the alignments:pslMap -chainMapFile <src-rna.psl> <mapping.chains> <dest-rna-over.psl>pslRecalcMatch <dest-rna-over.psl> <dest-genome.2bit> <src-rna.fa> <dest-rna-raw.psl>The alignments produced are then filtered in the same manner as the LASTZ alignments.53 to align more UTR bases than translated methods. To facilitate comparisons, CDS alignments from each method were created by trimming the RNA alignments to contain only the CDS regions as defined by the human annotation set.To facilitate the comparative analysis of the alignment methods, we created reduced sets of the alignments using two different approaches. Although both BLATX and TBLASTX will align UTR, the strength of protein-translated methods is in recognizing distant coding sequence relationships. Alignment projection-mapping methods were previously shownAlthough mapping all transcripts is useful, particularly for understanding the utility of the methods in assisting genome annotation, individual transcripts overlap, biasing assessment of transcribed mappings to genes with larger transcript numbers. To remove most of this base multiplicity from comparisons, in addition to showing full transcript results, subsets of the alignments are created using only one representative transcript per gene. For the full RNA alignments, the longest RNA for each gene was chosen, with the CDS alignments choosing the transcript with the longest CDS. The biology of overlapping gene structures and the ambiguities in defining genes cause around 4% of genomic bases to appear in more than one gene in the RNA, and 3% in the CDS gene sets owing to overlap.Individual pairwise alignments were compared at the base-level, consistent with the earlier comparisons reported. In brief, alignment similarity is computed by comparing the set of shared aligned pairs. That is, a pairwise alignment can be viewed as a set of aligned base pairs, each a coordinate from the source (human) and target (chicken) genome. The Jaccard index is, in this context, the number of aligned pairs identical between the two alignments divided by the union of all aligned pairs in the two alignments. It is worth noting that translated alignments are encoded for comparison using their induced base-level alignments. Transcripts or genes that are not aligned by either of the aligners being compared are assigned Jaccard indices of zero.To account for human bases that map to multiple bases in chicken , when comparing the alignments of an mRNA or CDS between two methods, if either or both methods produces multiple alignments, we pick the pair of mappings (one from each method) with highest shared similarity to report. This generally has the effect of removing distant paralogues from the comparison.56  is an algorithm designed to construct a sequence graph from an input set of local alignments (in the Progressive Cactus pipeline computed using LASTZ). We omit a complete definition here, but a sequence graph represents the alignment of a set of nucleotide strings. It can formally be represented using a bi-directed or bi-edged graphBelow we provide updates on the changes made to Cactus to create Progressive Cactus.Progressive Cactus requires input genomes to be soft-masked, but often repetitive sequence goes unmasked due to poor masking or incomplete repeat libraries for newly-sequenced species. This can negatively affect alignment runtimes  and impact quality. For this reason, we mask overabundant sequence before alignment, using a strategy not based on alignment to repeat consensus libraries, but on over-representation of alignments. We first divide each genome into small, mutually overlapping chunks. For each chunk, we align it to itself and a configurable amount of other randomly sampled chunks . To avoid combinatorial explosion due to unmasked repetitive sequence, we use a special mode of LASTZ which stops exploring alignments from any region early if a maximum depth is reached . We then soft-mask any region covered by more than a configurable number of these alignments (currently set to 50). Further details can be found in the src/cactus/preprocessor section of the Progressive Cactus codebase. Although the preprocessing step is automatically run as part of the pipeline, we also provide a cactus_preprocessor utility to run only the preprocessor without producing a full genome alignment.The alignment process for each subproblem begins with a series of local alignments generated using LASTZ. The local alignments fall into two sets: a set of all-against-all alignments among the ingroup genomes, and a set of alignments from ingroup genomes to outgroup genomes. We have found outgroup selection to be absolutely crucial in creating an acceptable ancestral reconstruction: any missing data or misassembly in the outgroup that causes a true deletion in one of the ingroups to be misinterpreted as an insertion in others will mean that the ancestor contains less sequence than it ought to. This missing sequence in turn impacts the alignment between the entire subtree below the reconstructed ancestor and the entire supertree above it: the missing sequence will never be aligned between the subtree and supertree. To avoid this we attempt to use multiple outgroup genomes in each subproblem . Naively aligning each ingroup against multiple outgroups would significantly increase the computation time; to avoid this we note that in general any region already containing an outgroup alignment benefits very little from aligning an additional outgroup. Therefore, we iteratively align each ingroup against one outgroup at a time, pruning away any ingroup sequence already covered by the previous outgroup alignments. In this way, the computational cost is reduced to be far less than naively aligning against the entire outgroup set, while still retaining nearly all of the benefit. In addition, we allow the user to designate certain genomes in the input as being of particularly high quality; these are chosen as outgroups if possible to avoid problems with missing data in regions such as mitochondrial or sex chromosomes that are often missing from some assemblies but not others.KZNF genes, providing an alignment from each gene to the over-40056 homologous KZNF genes in the other genome is nigh-useless; the user is likely interested in only the orthologous copy or copies (in the case of a lineage-specific duplication) in the other genome. For this reason, Progressive Cactus alignments are capable of representing complex orthology/paralogy relationships, with an ability to display the alignment(s) labelled as orthologous, but also the option for a user to request alignments to paralogues at a customizable divergence-time threshold. This is achieved by implicitly producing a gene tree as the alignment is built, albeit with some restrictions, namely that a duplication event is represented by multiple regions in the child(ren) aligned to a single region in the parent species. This forbids the representation of gene-tree-species-tree discordance as would occur in incomplete lineage-sorting or horizontal transfer, as well as the exact ordering of multiple duplication events along a single branch. The restricted problem we solve at each subproblem step is that each alignment block should represent all regions orthologous to a single region of the ancestral sequence, possibly multiple per species; we make no attempt to fully resolve the gene tree when multiple duplications take place along a single branch. However, this still requires resolving the timing of all duplication events to the lineages of the tree: duplicated sequences whose coalescence precedes the speciation event represented in the subproblem should be split, while those following the speciation event should be kept together.Users of a genome alignment are often interested in ‘orthology’ relationships, rather than all ‘homology’ relationships, between a set of sequences. For example, when comparing human and chimpanzee To achieve the desired alignment blocks in each subproblem, in constructing the initial sequence graph during the CAF algorithm Progressive Cactus greedily chooses which pairwise alignments to include in an effort to prevent paralogous alignments between the ingroup species. We developed two algorithms. Both are greedy algorithms designed to rank the pairwise local alignments and then iteratively add the alignments to the graph, at each step choosing to accept or reject the addition of alignments to the graph. Each added alignment ‘glues’ together two alignment blocks, splitting existing alignment blocks as necessary and merging the resulting two alignment blocks into one new block in the graph  orthology relationships within this subtree, refusing these outgroup self-homologies proves useful for assigning orthology between ingroups. Unfortunately, this method is very sensitive to incomplete outgroup assemblies (containing an incorrect number of copies of a duplicated region) or variation in the similarity between closely related paralogues, causing assignment to the wrong copy. As seen in Extended Data Fig. To remedy this problem, we developed an improved duplication-timing method, which we term ‘best-hit filtering’. The method preprocesses the local alignments to define for every base in every input genome a ranking by score of the local alignments that overlap it. The sequence graph is then built by first including the highest-scoring alignment for each base in each genome. We refer to this highest-scoring set as the set of ‘primary’ alignments and the remaining alignments the ‘secondary’ alignments. Note this definition is asymmetric: a pairwise alignment may be primary for one of the substrings it aligns and secondary for the other. All primary alignments are added to the initial graph unconditionally because they represent the most likely orthologue relationship  , using the following tree::0.025291,Rhesus:0.044568):0.07,Tree_shrew:0.19):0.03,:0.11):0.150342):0.02326,:0.087381,:0.104728,Horse:0.05):0.05):0.04);Comparing the best-hit filtering alignment and the one using the single copy outgroup filtering, the amount of human sequence mapping to two or more places in the chimpanzee genome was reduced from 6.1% to 2.6%, while the total amount of human covered by chimpanzee actually increased owing to the removal of ancient homologues, simplifying the initial alignment relationships . To confirm that these improvements were likely caused by the removal of paralogous rather than orthologous alignments, we compared phylogenetic trees implicit in the columns of HAL alignments to independently re-estimated approximately maximum likelihood (ML) trees produced by FastTree 57 for the same regions. The duplication-timing evaluation was performed using a custom pipeline  designed to sample columns from a HAL file and evaluate their trees against an independently re-estimated tree of the same region. For this analysis we used the two 12-boreoeutherian alignments described above, sampling 10,000 columns from the human genome. The comparison trees were built from a context of 1,000 bases around the entries in each sampled column using FastTree 2.1.10 and the -gtr -nt options. Only duplicated columns were counted in the final output (columns containing no duplications did not count in the results). The coalescence pairs were evaluated using the --onlySelf option, meaning that only pairs that included the sampled site were counted in the results. To avoid weighting columns with a high number of copies per genome more than columns with a low number of copies per genome, only a single coalescence was randomly sampled per column.The alignment files are accessible in the URLs listed at Supplementary Table Because HAL does not produce a fully binarized history of duplication events, we compared the MRCA of randomly selected pairs of sites from genomes containing a duplication within the column. If the MRCA species in the HAL tree is a descendant of the MRCA species within the reconciled ML tree, that implies that there are paralogues represented as orthologues within the HAL tree (since a duplication event must have been resolved too early). Similarly, if the MRCA species within the HAL tree is an ancestor of that within the reconciled ML tree, a duplication event must have been resolved too late in the HAL tree, indicating additional false loss or deletion events. The number of paralogous alignments  in the alignment of the 12 boreoeutherian genomes was clearly reduced (46% in the outgroup filtering versus 26% in the best-hit filtering) 9 on identical human, chimpanzee and gorilla assemblies  in two alignments using the outgroup and best-hit filtering methods. We ran using the GENCODE V30 gene set58. We projected the transcripts solely via transMap without the use of the AUGUSTUS modes. Multiple-mapping statistics and the gene composition of the final gene set were taken from the filter_tm_metrics.json file in the CAT output.We separately ran the Comparative Annotation Toolkit   to be left unaligned. To remedy this, we have developed a new alteration to the CAF algorithm, one that now focuses on maximizing the potential size of the alignment graph ‘after’ extension as opposed to ‘before’ extension. We call this addition ‘removing recoverable chains’ because it identifies chains in the cactus graph that represent alignments that could be recovered by the BAR algorithm extension process.59 for additional context. We call a segment ‘a left-adjacent’ to another segment ‘b’ if ‘a’ represents the positive strand and ‘b’ comes before ‘a’ in their sequence and there is no other segment between them. Similarly, we call a left-adjacent to b if a is on the negative strand and a comes before b in their sequence ordering with no other intervening segment. If a is left-adjacent to b, then b is right-adjacent to a.The algorithm is applied as a post-processing step after the CAF process, which proceeds as normal. After the cactus graph is created and filtered, the algorithm identifies ‘recoverable blocks’. Each block is composed of segments, each of which represents a non-overlapping region of a sequence and which strand is being aligned; we briefly review the necessary terminology, but seeA, it is recoverable if all its segments are all left- or right-adjacent to segments from the same block B ≠ A.A block is called ‘recoverable’ if, in the case that the block was removed, all its regions would be contained entirely within a single end alignment in the BAR extension phase. The end alignments are identified by looking at all unaligned sequences between the adjacent segments of a single ‘end’ of a block: in short, two end alignments are created for every block, one for all sequences between each segment and its left-adjacent segment, and similarly for the right-adjacent segments. In practice, this means that for some block Whether a block is recoverable depends only on its immediate neighbouring blocks. However, it is interesting to consider the maximum set of recoverable blocks, and, by contrast, of unrecoverable blocks—these unrecoverable blocks represent a minimal set of anchors that can be extended from to recover the alignment relationships from the original sequence graph as well as potential additional alignment relationships.A’s segments would lie within the end alignment of B if all the recoverable blocks between B and A, including A, were destroyed, we call A recoverable given B. The relationship is transitive: if block A is recoverable given block B, and B is recoverable given C, then A is recoverable given C. All blocks in a chain are recoverable given each other, since all blocks in a chain are collinear with each other, potentially with intervening rearrangements located further down the chain/net hierarchy. Similarly, if any block in a chain is recoverable given another block above the chain in the chain/net hierarchy, the entire chain is recoverable given that block. Owing to this fact, to determine the recoverability status of all blocks, we only have to examine the blocks at the ends of chains and their immediate neighbours, rather than every block.Because the chains and nets within the cactus graph represent a hierarchy of the rearrangements implicit in the alignment, they are helpful for finding a smaller set of anchors to extend from. We consider what anchors could provide recoverability to a block: if a block Although in principle we would need to keep only one block within even unrecoverable chains , to save computational effort in realignment we only destroy or keep entire chains as a unit. In the same spirit, to avoid spending needless effort when the chain is recoverable but very likely is not incomplete, we apply a heuristic and do not remove chains that contain the same number of copies in all ingroups and outgroups.After identifying and removing all recoverable blocks, some blocks previously marked unrecoverable may become recoverable (because adjacent blocks were removed). For this reason, we run the process of identifying and removing recoverable chains multiple times in a loop, until either no recoverable chains are identified or a limit on the number of cycles is reached. The structure of the cactus graph may change after removing recoverable blocks, so we recompute the cactus graph after every removal step. The process that is followed is described in pseudocode as follows:function RemoveRecoverableChainsfor 1 ... n docactusGraph ← CreateCactusGraph(G)RecoverableChains ← ∅for chain C in cactusGraphdoif⊳ A single adjacent end offers the potential for recoverabilityC.leftAdjacencies| = 1 or |C.rightAdjacencies| = 1)(|⊳ Shared adjacencies indicate a non-recoverable rearrangementand C.leftAdjacencies ∩ C.rightAdjacencies = ∅⊳ Links between chain ends indicate a non-recoverable duplicationandC.leftEnd ∉ C.rightAdjacencies thenRecoverableChains ← RecoverableChains ∪ {C}end ifend forif |RecoverableChains| = 0 thenbreakelseRecoverableChainsDestroy each chain in end ifend forend functionTo quantify the effect that the process of removing recoverable chains (described above) had on real alignments, we ran alignments on a set of nine Euarchontoglires genomes with the feature turned on and off. The tree used was::0.009693,orang:0.01894):0.015511,rhesus:0.037601):0.07392,tarsier:0.1114):0.034014,tree_shrew:0.19114):0.002,:0.11015):0.114051)euarchontoglires:0.020593,:0.032898);We used Progressive Cactus commit 56874bde, with the --root euarchontoglires option so that cow and dog were used only as outgroups. Coverage on human increased for all genomes when recoverable chains were removed, especially for those most distant from human  as outgroup to all the existing genomes in the alignment; 2) by being added as a new child of an existing ancestral genome in the alignment; or (3) by splitting a branch in the existing alignment, creating a new internal node and two new branches , at most a single ancestral sequence is re-inferred. This prevents any information from the new genome from propagating to the rest of the tree. Although this saves considerable effort in recomputing other parts of the alignment, it also means that, occasionally, rare stretches of sequence in a newly added genome would not be properly aligned to distant outgroups because they were deleted or missing in the new genome’s close relatives. Re-inferring the ancestral genomes on the path from newly added genomes to the root should address this issue if it appears.https://github.com/ComparativeGenomicsToolkit/hal, commit 68db41d).We tested the effect of adding a new genome to an existing alignment using the same set of simulated catarrhine genomes as described above. To replicate the use-case of an end-user wanting to add a genome to a previously-created alignment, we generated an alignment holding out one of the 20 genomes (the crab-eating macaque), and added that genome back into the alignment by both splitting an existing branch , and by adding the macaque as a new child of an existing ancestor , we ran a single new alignment with the tree anc6e;. The anc6e genome from the original, held-out alignment was used as an unreconstructed ancestral input sequence. We set the ‘runMapQFiltering’ option in the config file to ‘0’ and the ‘alignmentFilter’ option to ‘singleCopyOutgroup’, because these options produce a better alignment of polytomies. We merged the resulting HAL file into a new copy of the existing alignment via the command:halReplaceGenome <copy of held-out alignment> anc6e \--topAlignmentFile <held-out alignment> \--bottomAlignmentFile <add-to-node alignment>.To add the macaque by splitting a branch (the add-to-branch strategy), we ran two separate alignments. We ran the first with the tree  anc5e:0.001, Sooty_mangabey:0.001)anc6e:0.005, Baboon:0.003042)anc7e; (with the --root anc5e option so that only a single subproblem was run), generating a newly reconstructed anc5e ancestor. We then ran a second alignment with the tree anc6e;, again providing the anc6e assembly from the original alignment rather than inferring a new reconstruction.  To merge these two add-to-branch intermediate alignments into a full alignment, we first removed the Rhesus genome from a new copy of the held-out alignment. We then ran  halAddToBranch <held-out alignment> <first add-to-branch alignment> <second add-to-branch alignment> anc6e anc5e  Rhesus Crab_eating_macaque 0.001 0.006.F1 score of 0.926 (Extended Data Table F1 scores.Both methods resulted in alignments that had accuracy deviating less than 0.1% from the full alignment that included the macaque from the start: both addition methods, as well as the full alignment, achieved an Further information on research design is available in the Any methods, additional references, Nature Research reporting summaries, source data, extended data, supplementary information, acknowledgements, peer review information; details of author contributions and competing interests; and statements of data and code availability are available at 10.1038/s41586-020-2871-y.Supplementary InformationA merged PDF containing the individually numbered supplementary figures and supplementary tables referenced in the main text and methods. There are sixteen supplementary figures and fifteen supplementary tables. These supplementary items provide supporting information about the individual experiments described.Reporting SummarySupplementary DataExcel spreadsheet of mammalian genome assemblies used in the 600-way alignment.Supplementary DataExcel spreadsheet of avian genome assemblies used in the 600-way alignment.


Throughout biology, multiple sequence alignments (MSAs) form the basis of much investigation into biological features and relationships. These alignments are at the heart of many bioinformatics analyses. However, sequences in MSAs are often incomplete or very divergent, which can lead to poor alignment and large gaps. This slows down computation and can impact conclusions without being biologically relevant. Cleaning the alignment by removing common issues such as gaps, divergent sequences, large insertions and deletions and poorly aligned sequence ends can substantially improve analyses. Manual editing of MSAs is very widespread but is time-consuming and difficult to reproduce.We present a comprehensive, user-friendly MSA trimming tool with multiple visualisation options. Our highly customisable command line tool aims to give intervention power to the user by offering various options, and outputs graphical representations of the alignment before and after processing to give the user a clear overview of what has been removed. The main functionalities of the tool include removing regions of low coverage due to insertions, removing gaps, cropping poorly aligned sequence ends and removing sequences that are too divergent or too short. The thresholds for each function can be specified by the user and parameters can be adjusted to each individual MSA. CIAlign is designed with an emphasis on solving specific and common alignment problems and on providing transparency to the user.CIAlign effectively removes problematic regions and sequences from MSAs and provides novel visualisation options. This tool can be used to fine-tune alignments for further analysis and processing. The tool is aimed at anyone who wishes to automatically clean up parts of an MSA and those requiring a new, accessible way of visualising large MSAs. Throughout biology, multiple sequence alignments (MSAs) of DNA, RNA or amino acid sequences are often the basis of investigation into biological features and relationships. Applications of MSAs include, but are not limited to, transcriptome analysis, in which transcripts may need to be aligned to genes; RNA structure prediction, in which an MSA improves results significantly compared to predictions based on single sequences; and phylogenetics, where trees are usually created based on MSAs. There are many more applications of MSA at a gene, transcript and genome level, involved in a huge variety of traditional and new approaches to genetics and genomics, many of which could benefit from the tool presented here.An MSA typically represents three or more DNA, RNA or amino acid sequences, which represent partial or complete gene, transcript, protein or genome sequences. These sequences are aligned by inserting gaps between residues to bring more similar residues (either based on simple sequence similarity or an evolutionary model) into the same column, allowing insertions, deletions and differences in sequence length to be taken into account . The firIt has been shown previously that removing divergent regions from an MSA can improve the resulting phylogenetic tree . VariousVarious refinement methods incorporated into alignment software can also improve MSAs . Some trSeveral common issues affect the speed, complexity and reliability of specific downstream analyses but are not addressed by these existing tools. Clean and Interpret Alignments (CIAlign) is primarily intended to address four such issues and to be used (where appropriate) in combination with existing tools which remove unreliable alignment columns. Researchers in many fields regularly edit MSAs by hand to address these issues, however, as well as being extremely time consuming, ensuring reproducibility with this approach is almost impossible and it cannot be incorporated into an automated analysis pipeline. CIAlign automatically removes full columns and full or partial rows from user generated MSAs to address these issues in a fast, reproducible manner and can be easily added to an automated pipeline. The downstream applications of alignments cleaned with CIAlign are not limited to phylogenetic analysis and are too numerous to list, but CIAlign as an alignment cleaning tool is particularly targetted towards users working with complex or highly divergent alignments, partial sequences and problematic assemblies and towards those developing complex pipelines requiring fine-tuning of parameters to meet specific criteria.The first issue we intend to address is that it is common for an MSA to contain more gaps towards either end than in the body of the alignment. This problem occurs at both the sequencing and alignment stage. For example, the ends of de novo assembled transcripts tend to have lower read coverage  and so hStreptococcus agalactiae and Secondly, insertions or other stretches of sequence can be present in a minority of sequences in an MSA, leading to large gaps in the remaining sequences. For example, alignments of sections of bacterial genomes often result in long gaps representing genes which are absent in the majority of species. These gaps can be observed, for example, in multiple genome alignments shown in Thirdly, one or a few highly divergent sequences can heavily disrupt the alignment and therefore complicate downstream analysis. It is very common for an MSA to include one or a few outlier sequences which do not align well with the majority of the alignment. One example of this is in analyses identifying novel sequences in large numbers of datasets. It is common to manually remove phylogenetic outliers which are unlikely to truly represent members of a group of interest  and to interpret the MSA in several ways. CIAlign works on nucleotide or amino acids alignments and will detect which of these is provided. A log file is generated to show exactly which sequences and positions have been removed from the alignment and why they were removed. Users can then adjust the software parameters according to their needs.CIAlign is a command line tool implemented in Python 3. It can be installed either CIAlign takes as its input any pre-computed alignment in FASTA format containing at least two sequences (for some cleaning functions three sequences are required). Most MSAs created with standard alignment software will be of an appropriate scale, for example single or multi-gene alignments and whole genome alignments for many microbial species.all will turn on all the available functions and run them with the default parameters, unless otherwise specified. The clean option will run all cleaning functions, visualise all the visualisation functions and interpret all the interpretation functions, again with the default parameters. Additionally, the user can provide parameters via a configuration file instead of via the command line.The path to the alignment file is the only mandatory parameter. Every function is run only if specified in the parameters and many function-specific parameters allow options to be fine-tuned. Using the parameter option CIAlign has been designed to maximise usability, reproducibility and reliability. The code is written to be as readable as possible and all functions are fully documented. All functions are covered by unit tests. CIAlign is freely available, open source and fully version controlled.CIAlign consists of several functions to clean an MSA by removing commonly encountered alignment issues. All of these functions are optional and can be fine-tuned using user parameters. All parameters have default values. The available functions are presented here in the order they are executed by the program. The order can have a direct impact on the results, the functions removing positions that lead to the greatest disruptions in the MSA should be run first, as they potentially make removing more positions unnecessary and therefore keep processing to a minimum. For example, divergent sequences often contain many insertions compared to the consensus, so removing these sequences first reduces the number of insertions which need to be removed. Sequences can be made shorter during processing with CIAlign and therefore too short sequences are removed last.For each column in the alignment, this function finds the most common nucleotide or amino acid and generates a temporary consensus sequence. Each sequence is then compared individually to this consensus sequence. Sequences which match the consensus at a proportion of positions less than a user-defined threshold (default: 0.65) are excluded from the alignment . It is rIn order for CIAlign to define a region as an insertion, an alignment gap must be present in the majority of sequences and flanked by a minimum number of non-gap positions on either side, which can be defined by the user (default: 5). This pattern can be the result of an insertion in a minority of sequences or a deletion in a majority of sequences. The minimum and maximum size of insertion to be removed can also be defined by the user  . This fuThe crop ends function redefines where each sequence starts and ends, based on the ratio of the numbers of gap and non-gap positions observed up to a given position in the sequence. It then replaces all non-gap positions before and after the redefined start and end, respectively, with gaps. This will be described for redefining the sequence start, however crop ends is also applied to the reverse of the sequence to redefine the sequence end. The number of gap positions separating every two consecutive non-gap positions is compared to a threshold and if that difference is higher than the threshold, the start of the sequence will be reset to that position. This threshold is defined as a proportion of the total sequence length, excluding gaps, and can be defined by the user (default: 0.05) , 2. The The remove short function removes sequences which have less than a specified number of non-gap positions,which can be set by the user (default: 50) .The remove gap only function only removes columns that contain only gaps. These could be introduced by manual editing of the MSA before using CIAlign or by running the functions above . The maiThere are several ways of visualising the alignment, which both allow the user to interpret the alignment and clearly show which positions and sequences CIAlign has removed. CIAlign can also be used simply to visualise an alignment, without running any of the cleaning functions. All visualisations can be output as publication ready image files.x and y axis, with each rectangle representing a single nucleotide or amino acid , calculating a pairwise similarity matrix and generating a consensus sequence with various options.Given the toy example shown in Four example alignments are provided within the software directory to demonstrate the functionality of CIAlign. Examples 1 and 2 use simulated sequences, Examples 3 and 4 use real biological sequences and are designed to resemble the type of complex alignment many researchers encounter.Example 1 is a very short alignment of six sequences which was generated manually by creating arbitrary sequences of nucleotides that would show every cleaning function while being as short as possible. This alignment contains an insertion, gaps at the ends of sequences, a very short sequence and some highly divergent sequences.Example 2 is a larger alignment based on randomly generated amino acid sequences using RandSeq (a tool from ExPASy ) with anNC_012920.1, positions 5,904–7,445, translated to amino acids), querying against 1,565 transcriptomic datasets from the NCBI transcriptome shotgun assembly (TSA) database  under the default settings. 2,855 putative COI transcripts were reverse complemented where required, and those corresponding to the COI gene of the primary host of the TSA dataset were identified using the BOLD online specimen identification engine   sequences were identified by applying TBLASTN v2.9.0  to the h engine    distances  of the rMaterials and methods for benchmarking and for larger scale examples with biological data are provided as Here, an example is presented and the visualisation functions are used to illustrate the functionality of CIAlign. Results will differ when using different parameters and thresholds.CIAlign was applied to the Example 2 alignment with the following options:python3 CIAlign.py infileINFILE outfile_stem OUTFILE_STEMallUsing these settings on the alignment in de novo assembled transcripts, which tend to have a higher error rate at transcript ends, gaps due to difficult to assemble regions and divergent sequences due to chimeric connections between unrelated regions   to determine if cleaning with CIAlign impacts the distance between the true phylogenetic tree and a phylogenetic tree based on a test alignment , Table 1 U test) , Table 1p = 1.89E−67, moderate: p = 2.61E−68, stringent, p = 1.56E−67, Mann–Whitney U test).We also compared the input sequence for our EvolvAGene simulations to consensus sequences based on alignments with and without CIAlign cleaning. For all three stringency levels, CIAlign increased the percentage nucleotide identity between the consensus sequence and the input sequence by between 4% and 5% , Table 1p = 2.22E−35, 1.37E−13, 1.55E−9, respectively, Mann–Whitney U test). For the high quality reads, the reads cleaned with CIAlign generated consensus sequences almost identical to the input sequence, with a mean of 99.2% identity    . The pro U test) ) was als U test) , Table 1CIAlign was also benchmarked using the HomFam  set of bp = 0.928, QD, p = 0.672, Mann–Whitney U test)   , Table 2 U test) , Table 2p = 9.35E−20, Mann–Whitney U test). A linear regression demonstrated that any cleaning with CIAlign increases, on average, the percentage of correct positions in the resulting structure but that the benefit decreases linearly with the amount of material removed by CIAlign , gaps , positions  and correctly aligned pairs  removed  such tools. However, we have calculated the proportion of correctly aligned pairs, gaps and residues removed using the default settings for Gblocks , trimAl Across the EvolvAGene and INDELible alignments, CIAlign removed a median of 0.188% of correctly aligned pairs with the most relaxed settings, 0.749% with moderate settings and 3.76% with stringent settings . To compNone of these tools significantly increased or decreased the distance between trees generated with the test alignments and the true trees except Gblocks, which significantly increased the distance from the true tree with both divergence measures . CleaninThe exact aligned residue pairs removed by CIAlign and the other tools were also compared, to demonstrate the extent to which CIAlign overlaps with and differs from the other tools . As GbloThese results demonstrate that CIAlign is performing a different role to these three tools, as the locations targetted by CIAlign are only removed by other tools at the expense of large sections of the alignment which CIAlign would leave intact.http://github.com/KatyBrown/CIAlign/benchmarking/tables and the data at https://github.com/KatyBrown/benchmarking_data_CIAlign.Full results for Gblocks, trimAl and ZORRO compared to CIAlign are are available in Online Table 6 at As alignment tools take into account all the sequences and columns in the input file, the most scrupulous option will always be to unalign and then realign sequences after running a tool such as CIAlign, rather than using the CIAlign output directly in downstream analysis. To test the extent to which using CIAlign outputs directly without realignment could impact results, we removed gaps from the EvolvAGene alignments cleaned with CIAlign with relaxed, moderate and stringent parameter settings and then reran the original alignment tool on the result. We then calculated the sum-of-pairs score  treatingMemory and runtime measurements were conducted by randomly drawing alignments from the HomFam benchmark set  and measIt should be noted that, besides the size of the MSA and its gap content, the runtime is impacted by which combination of functions is applied. For very long MSAs the size of the final image becomes a limiting factor when creating a sequence logo, as the matplotlib library  has restWe also used CIAlign to clean real biological data from several online databases, in order to test and demonstrate its usefulness in automated processing of different types of sequencing data.p = 0.00, Mann–Whitney U test) after cleaning with CIAlign   . After r U test) . Even foTo demonstrate the ability of CIAlign to remove non-majority indels, we used data for 50 indels across over 150 individuals from the 1000 genomes project , which hCIAlign can also be used to remove clear outliers from an alignment, for example prior to phylogenetic analysis. To illustrate this, we ran the CIAlign cleaning functions on data from the mammalian 10K trees project . Three sWe have demonstrated that CIAlign can successfully mitigate the alignment issues caused by non-majority insertions, poorly aligned sequence ends, highly divergent sequences and short sequences and demonstrated this capability on specific examples, simulated and benchmark datasets and large biological datasets. CIAlign has been shown to significantly improve the accuracy of consensus sequences and secondary structure predictions generated from MSAs  and 7D IIt is important to note that while CIAlign is helpful in mitigating alignment issues, using an appropriate alignment tool and parameters to generate the original alignment is still essential.While the functionality of CIAlign has some overlaps with other software, for example Gblocks , ZORRO  and trimHaving as many parameters as possible to allow as much user control as possible gives greater flexibility. However, this also means that these parameters should be adjusted, which requires a good understanding of the cleaning functions and the MSA in question. CIAlign offers default parameters selected to be often applicable based on our benchmarking simulations and testing with different types of data. However, parameter choice highly depends on MSA divergence and the downstream application. To choose appropriate values it is recommended to first run CIAlign with all default parameters and then adjust these parameters based on the results. Since the mini alignments show what has been removed by which functions it is straightforward to identify the effect of each function and any changes to the parameters which may be required.New features are in progress to be added in the future, such as collapsing very similar sequences, removing divergent columns, and making the colour scheme for the bases or amino acids customisable. CIAlign is currently not parallelised, as the most time limiting function, remove insertions, requires information from the entire alignment. However, a future release will incorporate the ability to process more than one alignment in parallel.CIAlign is a highly customisable tool which can be used to clean multiple sequence alignments and address several common alignment problems. Due to its multiple user options it can be used for many applications. CIAlign provides clear visual output showing which positions have been removed and for what reason, allowing the user to adjust the parameters accordingly. A number of additional visualisation and interpretation options are provided.10.7717/peerj.12983/supp-1Supplemental Information 1CIAlign infile example4.fastaplot_input”. (B) Output alignment after application of CIAlign, generated using the command “CIAlign infile example4.fastaallremove_divergent_minperc 0.5”. Subplots were generated using the “draw mini alignment function. (C) Phylogenetic tree generated manually using the literature to show the current best estimate for the phylogenetic relationships between these 91 families of metazoa. Relationships are based on the literature listed in the Supp. References. (D) PhyML ((A) Input alignment before application of CIAlign, generated using the command “D) PhyML  phylogenD) PhyML  phylogenClick here for additional data file.10.7717/peerj.12983/supp-2Supplemental Information 2m is the slope of the regression line.(A–D) Box plots showing the impact of running CIAlign cleaning functions with relaxed , moderate  and stringent  parameter values on alignments generated with MAFFT global  , MAFFT lClick here for additional data file.10.7717/peerj.12983/supp-3Supplemental Information 3y-axis represents proportion of correctly aligned pairs of residues  Plots showing the mean percentage identity (top) and change in percentage identity (bottom) between consensus sequences generated from EvolvAGene ((A) Plots showing the impact of running CIAlign cleaning functions with relaxed , moderate  and stringent  parameter values on alignments of nucleotide and amino acid sequences simulated using either EvolvAGene  or INDELresidues  removed residues  or INDELresidues , calcularesidues  or INDELresidues . (D) ScaolvAGene  alignmenolvAGene  and the Click here for additional data file.10.7717/peerj.12983/supp-4Supplemental Information 4y-axis represents proportion of correctly aligned pairs of residues (m is the slope of the regression line (C) Violin plots showing the distribution of normalised Robinson-Foulds distances ((A) Box plots showing the impact of running other trimming tools–TrimAl (cyan) , Gblocksresidues  removed residues , MAFFT lresidues , MAFFT gresidues  and Clusresidues . The staistances   * m (number of columns) and the time (A) and memory (B) used by CIAlign to run each of the cleaning and mini alignment functions on the alignment, as the mean of four replicate measurements. 25 input alignments were chosen at random from the HomFam  but with different proportions of gaps. Scatter plots showing the relationship between alignment size, measured as e HomFam  benchmarClick here for additional data file.10.7717/peerj.12983/supp-7Supplemental Information 7APOB gene in Carnivora. Colours represent known monophyletic families of Carnivora. Nodes have been collapsed where multiple sequences from the same family formed a monophyletic clade. Sequences annotated with circles were removed by CIAlign. Top-right, tree based on the same alignment after cleaning with CIAlign, which removed the outlying group. Bottom-right, mini alignments showing the effect of running CIAlign on this alignment. (B) As for (A), but for the RAG1 gene in Carnivora.(A) Left, phylogenetic tree based on an alignment of sequences from the 10k trees project  for the Click here for additional data file.10.7717/peerj.12983/supp-8Supplemental Information 8Relaxed, moderate and stringent parameter settings used for benchmarking with EvolvAGene , INDELibClick here for additional data file.10.7717/peerj.12983/supp-9Supplemental Information 9U tests. Significance is shown as *** if the p-value is less than 0.001, ** if the p-value is less than 0.01, * if the p-value is less than 0.05 and –if the p-value is greater than 0.05. The median percentage correct nucleotides removed is the proportion of positions in the alignment which match the input sequence to BadRead which were removed by CIAlign for each set of simulated reads.Table showing the mean percentage identity between the input sequence to BadRead  and consClick here for additional data file.10.7717/peerj.12983/supp-10Supplemental Information 10U tests comparing the alignment without running CIAlign to the alignment after running CIAlign with the specified parameters. Significance is shown as *** if the p-value is less than 0.001, ** if the p-value is less than 0.01, * if the p-value is less than 0.05 and –if the p-value is greater than 0.05.Results of running CIAlign cleaning functions with relaxed, moderate and stringent parameter values on alignments of nucleotide and amino acid sequences simulated using either EvolvAGene  or INDELClick here for additional data file.10.7717/peerj.12983/supp-11Supplemental Information 11U tests comparing the alignment without running CIAlign to the alignment after running CIAlign with the specified parameters. Significance is shown as *** if the p-value is less than 0.001, ** if the p-value is less than 0.01, * if the p-value is less than 0.05 and –if the p-value is greater than 0.05.Table showing the impact of running other trimming tools –TrimAl , GBlocksClick here for additional data file.10.7717/peerj.12983/supp-12Supplemental Information 12Click here for additional data file.