Motivation: Multiple sequence alignments (MSAs) with large numbers of sequences are now commonplace. However, current multiple alignment benchmarks are ill-suited for testing these types of alignments, as test cases either contain a very small number of sequences or are based purely on simulation rather than empirical data.Results: We take advantage of recent developments in protein structure prediction methods to create a benchmark (ContTest) for protein MSAs containing many thousands of sequences in each test case and which is based on empirical biological data. We rank popular MSA methods using this benchmark and verify a recent result showing that chained guide trees increase the accuracy of progressive alignment packages on datasets with thousands of proteins.Availability and implementation: Benchmark data and scripts are available for download at http://www.bioinf.ucd.ie/download/ContTest.tar.gz.Contact:des.higgins@ucd.ieSupplementary information:Bioinformatics online. Making a multiple sequence alignment (MSA) of nucleotide or amino acid sequences is a crucial step needed in a wide variety of bioinformatics studies. To overcome the extreme computational demands of exact algorithms, MSA software uses heuristics to find near-optimal alignments in a reasonable time. Different software packages, however, use different heuristics and objective functions and explicitly focus on different aspects of the MSA process such as accurate placement of gaps for phylogenetic reconstruction of small numbers of sequences  or sheerAs computational resources and sequence databases grow, and methods for creating MSAs are improved, the practical upper limit for the size of MSAs increases. Currently, alignments of tens of thousands of sequences can be made routinely on a desktop computer. It is therefore important that benchmarks can reflect alignment problems on this scale. Current protein MSA benchmarks can be divided into structure-, phylogeny-, simulation- and consistency-based benchmarks   Kalign 2.04  One test case had too few true positive contacts to be useful for the benchmark. Fourteen test cases were excluded using these criteria, leaving a total of 136 test cases for the benchmark. Details of the test cases are given in i and j in the protein chain, a contact is defined as long range if L, we calculate the precision of the top scoring The PSICOV and Evfold-mfDCA algorithms both assign scores to each predicted contact. We assess the quality of long range contact predictions; for amino acid residues at positions Because of the dependence of the output MSA on input sequence order for most methods tested, where possible we made three alignments for each test case with each alignment method, randomizing the order of the input sequences before each alignment. We calculated the average PSICOV precision and average EVfold-mfDCA precision over the three replicates. Then, we calculated the geometric mean of the average PSICOV and average EVfold-mfDCA precisions to arrive at a single number representing the precision of contact predictions for that test case. We call this the ContTest score for that test case. We rank alignment methods by the mean ContTest score over all test cases. Pfam stores only a single alignment for each family, while Clustal Omega runs extremely slowly when using chained guide trees. Therefore, results for Pfam and Clustal Omega (chained guide tree) are based on a single replicate only.While this scoring method involves only a subset of columns in the alignment, we found that using receiver-operating characteristic curves to score predicted contacts produced almost the same rankings of alignments .We calculate the statistical significance of the difference in two benchmark scores by using the Wilcoxon signed rank test where the ContTest scores for each of the 136 test cases are paired between the two conditions.To validate our benchmark, we use two methods to artificially create alignments which we expect to be worse than some reference alignment. First, given a reasonable starting alignment of a set of sequences, we expect that shifting a random subset of the sequences out of alignment with the others will result in an alignment that is poorer than the original alignment. We shift sequences out of alignment by adding a gap character to the start of the aligned sequence and truncating the last character. For each test case, we take the Pfam alignment and shift 0.5%, 1%, 2%, 3%, 4% or 5% of the sequences out of alignment, creating a series of alignments which we expect to be of decreasing quality and which should therefore result in decreasing benchmark scores.Second, we perform a similar analysis by mutating random residues, which we expect should decrease contact prediction accuracy. Starting with the Pfam alignments, we mutate between 0 and 20% of residues in each alignment to a different residue. We expect that increasing numbers of mutations should again result in a decreasing ContTest score.We found mappings from 80 of the ContTest test cases to HOMSTRAD ] can be calculated as a hypergeometric function 3F2  and an average length of between 31.9 and 370 residues (median 113.6)  see . We used3.6) see , MUSCLE 3.6) see , Clustal3.6) see  and hmmt3.6) see . AdditioAt the top of the rankings are the hmmt, Kalign 1 and Pfam alignments. These results are striking and unexpected. Since the method by which the Pfam alignments are created is not directly applicable to general MSA problems, the quality of Pfam alignments has not previously been assessed in MSA benchmark studies. Similarly, neither Kalign 1 nor hmmt have, to our knowledge, been considered in any recent benchmarking studies, as Kalign 1 is superseded by Kalign 2 and hmmt is not included in recent versions of HMMER.The Pfam alignments are created in two steps. First, a manually curated ‘seed’ alignment is created from a small set of sequences representative of the sequence family. This seed alignment is used to train an HMM which is then used to search a sequence database and align matching sequences to the seed alignment. We note that this process—aligning sequence matches from a database one-by-one to a small seed alignment—is effectively performing progressive alignment with an almost perfectly chained guide tree. Kalign 1 uses a traditional progressive MSA strategy, with pairwise sequence distances estimated by approximate k-tuple matching. However, we found that Kalign 1 with the fast ‘-f’ option skips pairwise distance calculations and performs alignment with a fully chained guide tree, although this is an undocumented feature. Meanwhile, hmmt produces an MSA as a side effect of training an HMM on the input sequences and does not use a guide tree as part of this process. Thus, the sequence-clustering step which is characteristic of progressive MSA is not performed by any of these alignment methods.The next highest scoring method is Kalign 2 . Kalign To verify that the ContTest score correctly ranks alignment methods by accuracy, we created alignments using different parameter sets of the same MSA packages. For a given alignment package, we expect that certain parameter sets produce more accurate alignments than others, on average. These relationships are reflected in the benchmark scores .Both Clustal Omega and MUSCLE are capable of iteratively refining MSAs and using more iterations results in an increase in benchmark score for these methods . In addiAs noted above, the top three highest scoring sets of alignments, those from hmmt, Kalign 1 (fast) and Pfam, were created without the use of a conventional guide tree. To determine the extent to which the lack of sequence clustering contributes to these methods having the highest scoring alignments, we created alignments using MAFFT, MUSCLE and Clustal Omega with fully chained guide trees . Kalign As a further test of benchmark accuracy, we studied how the addition of deliberate errors to a set of alignments affects the benchmark score. First, we selected random subsets of sequences in each Pfam alignment to ‘misalign’ by adding a gap to the start of the aligned sequence, while truncating the final character (see Section 2.5). We observed that greater the percentage of sequences in each alignment that were misaligned, the greater the drop in contract prediction precision from the starting alignments. To compare the ContTest benchmark method with the more traditional structure-superposition benchmark method, we created a hybrid benchmark by adding HOMSTRAD reference sequences to the Pfam sequences in 80 test cases for which we identified a match between the Pfam family and a HOMSTRAD alignment. We then calculated both the ContTest score of each alignment and the SPS for the embedded HOMSTRAD sequences. Scores for the HOMSTRAD sequences are listed in PREFAB  is a strContTest is the first protein MSA benchmark to realistically test alignments of large numbers of sequences and base scores on empirical biological data. Although structure-based benchmarks may contain alignments with many sequences, e.g. HomFam .Conflict of Interest: none declared.Supplementary DataClick here for additional data file.


Alternative sequence alignment algorithms yield different results. It is therefore useful to quantify the similarities and differences between alternative alignments of the same sequences. These measurements can identify regions of consensus that are likely to be most informative in downstream analysis. They can also highlight systematic differences between alignments that relate to differences in the alignment algorithms themselves.Here we present a simple method for aligning two alternative multiple sequence alignments to one another and assessing their similarity. Differences are categorised into merges, splits or shifts in one alignment relative to the other. A set of graphical visualisations allow for intuitive interpretation of the data.AlignStat enables the easy one-off online use of MSA similarity comparisons or into R pipelines. The web-tool is available at AlignStat.Science.LaTrobe.edu.au. The R package, readme and example data are available on CRAN and GitHub.com/TS404/AlignStat.The online version of this article (doi:10.1186/s12859-016-1300-6) contains supplementary material, which is available to authorized users. Multiple sequence alignments (MSAs) aim to organise a set of sequences by placing homologous residues into columns, and their accuracy affects subsequent steps in bioinformatic pipelines such as phylogenetic inference  and protA common method of alignment comparison is though a combination of the sum of pairs score (SPS), and total column score (CS) . The sumHere, we use a complementary method based on a matrix of equivalency functions to allow comparable quantification of both similarity and of alternative sources of dissimilarity. Each position in the matrix corresponds to a residue of the reference MSA, with an equivalency function indicating its relationship to the corresponding residue in the comparison MSA. We present a simple set of quantitative measures and graphical visualisations for interpreting MSA comparisons. An R package generates a standardised set of comparison matrices and scores for analysis pipelines and graphing, and a user-friendly web-tool interface enables easy one-off use.When alignment algorithms make different homology predictions for a set of sequences, the columns of the resulting MSAs will contain different residues. The AlignStat R package contains functions for calculating all MSA comparison statistics and creating plots quantifying differences in a manner that is equivalent for nucleotide or amino acid sequences.n sequences is treated as a matrix of characters (residues plus a gap character) with the same number of rows. The two matrices are therefore defined as P (of dimensions n x p) and Q (of dimensions n x q), where each row represents an aligned sequence. Residues can occur multiple times in a sequence and so are numbered by occurrence such that each character in a row has a unique designation .Each MSA of Sij is the similarity score for each column pair between P and Q, the equivalency function ε is defined in Eq. 2.Where S can be visualised using the plot_similarity_heatmap function of the AlignStat R package. Evaluating S is the most computationally expensive calculation in the AlignStat scoring method and has been implemented in C++ for maximum efficiency.The similarity matrix P we find its “match” in Q by finding the index j at which Sij is maximized. The match between columns, Pi and Qj is then categorised leading to the dissimilarity matrix, D (of dimensions n x p x 5) based on the functions defined in Eq. 3 and Eq. 4. This matrix categorises five types of outcome when the reference and comparison alignments are compared. It is called the dissimilarity matrix because four of the five alternatives correspond to various types of mismatch.For each column in εk is the kth equivalency function as defined in Eq. 4.Where εk are equivalency functions  with the following meanings. The first equivalency (ε1) is a ‘match’, in which the two characters are identical and not gaps. The second equivalency (ε2) is a ‘conserved gap’, when the both characters are gaps. A ‘merge’ is when P contains a gap, but Q contains any other character (ε3). Similarly, a ‘split’ is when Q contains a gap, but P contains any other character (ε4). Finally, a ‘shift’ is when two characters are not identical and neither are gaps (ε5). The D matrix is visualised using the plot_dissimilarity_matrix function of the AlignStat R package.Where the five D are used to describe the sources of dissimilarity between the reference and comparison alignments at each alignment position and each equivalency, k. This leads to the results matrix R (of dimensions 5 x p) defined by Eq. 5.The column averages of R is the results matrix, each row of which is used to summarise a source of dissimilarity from the D matrix.Where R matrix (Ri1) is visualised using the plot_similarity_summary function of the AlignStat R package. The merge, split and shift rows of the R matrix  are referred to collectively as dissimilarities in AlignStat. They are visualised using the plot_dissimilarity_summary function.The match row of the A single, overall similarity score describes the weighted average similarity of the two MSAs, as defined in Eq. 6. The treatment of gaps in MSAs is complex , 10. In Released versions of the R package are available through the comprehensive R archive network (CRAN) and active development versions are available on github (GitHub.com/TS404/AlignStat). In order to allow AlignStat to scale to large MSAs and provide an acceptable run time the core calculation of equivalency functions and scoring statistics was implemented in C++ using the Rcpp framework . A simplAlignStat R package contains a compare_alignments function to calculate the similarity and dissimilarity matrices, and a set of plotting functions to graphically visualise the results. The main compare_alignments function reads input alignments  and outputs a Pairwise Alignment Comparison (PAC) object that contains the matrices and summary information. The example here is a reference MSA of cis-defensin sequences  alignedproteins , compareproteins  Fig. 1)AlignStatplot_similarity_heatmap function generates a heatmap of the similarity matrix S  the overall similarity statistics can be used to compare which alternative alignment methods most accurately recreate the reference MSA, and the columnwise similarity statistics indicate the causes of any discrepancies. In this case, higher scores indicate a higher recapitulation by the comparison alignment of the homologous residues in the reference alignment. When the ‘true’ alignent is unknown, as is often the case for real datasets, then the similarity statistics quantify consensus and uncertainty between the alignments. In this case, columns with higher scores indicate agreement of which residues are agreed upon as homologous between the two MSAs. Low scores indicate significant discrepancies, which may occur due to repeat regions, insertions and deletions, or low conservation.For the defensin example, the highest similarity between the MSAs clusters around the conserved cysteine columns. However, misalignment of non-homologous cysteines and frequent merger of low occupancy inter-cysteine regions by ClustalΩ lead to a similarity score of 45.5 %. The splitting of cysteine columns in the defensin alignment by ClustalΩ indicates which loop insertions and deletions prevent the algorithm from finding true structurally homologous cysteines. In this case, cysteines were split from one column to be merged in with non-homologous cysteines. Similarly, cysteines at the N-terminal end of the proteins are are erroneously split, losing information on their homology. Additionally, an entire set of four sequences was clearly translocated to the right, misaligning all cysteines and inter-cysteine regions. These differences in predicted homology significantly affect any phylogenetics or structure homology modelling using the alignment. By comparison, a ClustalΩ alignment of conserved S1 proteases differs only by minor translocations (similarity score of 81 %) compared to the curated benchmark BALI alignment [AlignStat method and outputs the set of graphs generated by the R script plot functions described above. The matrices and output graphs can then be downloaded. Example data is also provided to perform a test run. The server is capable of performing the method on MSAs of up to 1000 sequences, each with 1000 alignment columns. Additionally, both online and offline versions of AlignStat can compute and visualise sum of pairs analyses of alignments : Platform independentProgramming language: ROther requirements: R 3.1 or higherLicense: Academic Free License 3.0


With sharp increasing in biological sequences, the traditional sequence alignment methods become unsuitable and infeasible. It motivates a surge of fast alignment-free techniques for sequence analysis. Among these methods, many sorts of feature vector methods are established and applied to reconstruction of species phylogeny. The vectors basically consist of some typical numerical features for certain biological problems. The features may come from the primary sequences, secondary or three dimensional structures of macromolecules. In this study, we propose a novel numerical vector based on only primary sequences of organism to build their phylogeny. Three chemical and physical properties of primary sequences: purine, pyrimidine and keto are also incorporated to the vector. Using each property, we convert the nucleotide sequence into a new sequence consisting of only two kinds of letters. Therefore, three sequences are constructed according to the three properties. For each letter of each sequence we calculate the number of the letter, the average position of the letter and the variation of the position of the letter appearing in the sequence. Tested on several datasets related to mammals, viruses and bacteria, this new tool is fast in speed and accurate for inferring the phylogeny of organisms. However, one of the main drawback of these techniques is that they are very time-consuming and expensive in memory usage. As a result, alignment-free approaches have attracted more and more attention and have been applied to biological sequence comparison as well as phylogeny analysis recently7. The noticeable characteristic of these methods is to incorporate some ad hoc numerical feature of sequences. These methods are fast in computational speed when comparing genes and proteins. Generally, alignment-free are categorized into four groups: methods based on k-mer or word frequency, methods based on substrings, methods based on information theory, and methods based on graphical representation. Firstly, methods based k-mer or word frequency are very popular and studied extensively. The classic k-mer method was proposed to compare genetic sequences, in which it counts the frequencies of substrings with k letters appearing in respective sequences8. Afterwards, a lot of k-mer based methods have been developed and applied in sequence analysis and phylogeny for viruses and bacteria. Such as feature frequency profile (FFP)9 which counts the number of substrings with fixed length k occurring in a genome9 and a vector consisting of these numbers is formed for each genome, return time distribution (RTD)10, frequency chaos game representation (FCGR)11, an improved complete composition vector method (ICCV)12 which was proposed by optimizing composition vector (CV)13 and complete composition vector (CCV)14. ICCV is under the assumption of a uniform and independent model to estimate sequence information. Comparing with CCV and CV method, ICCV method is more robust and efficient in performing sequence comparison. Secondly, methods based on substrings employ the similarity of substrings in a pair of sequences. These algorithms are mostly used for string processing in computer science, like average common substring (ACS)15, k-mismatch average common substring approach (kmacs)16. Thirdly, methods based on information theory include global and local characterization of sequences and estimate genome entrop to regions. Among them base-base correlation (BBC)17, information correlation and partial information correlation (IC-PIC)18 are representative articles. Finally, methods based on graphical representation also play important roles in sequence comparison in alignment-free area21.In the past few decades, a basic but crucial task in biology is comparing genes and proteins for predicting the structure and function of these biological sequences. Many methods have been proposed to compare genetic sequences. Traditionally, most of these approaches are the widely used alignment-based methods. In these methods, molecular sequences are optimally aligned based on selected scoring systems. The alignment-based methods often give high accuracy and may reveal the relationships among sequences. Some algorithms have been established and incorporated into softwares for sequence alignmentsAlthough k-mer models are widely used in many biological studies, the position information and the properties of nucleotides is ignored in these methods. The second kind of methods are computationally expensive among alignment-free methods. For the third algorithms, most of them are alignment based. The graph based methods are often used to display some characteristics of sequences directly by curves. Therefore, it is better to use these information for sequences comparison. In this paper, we develop an 18 dimensional feature vector to characterize a DNA sequence. The vector includes the occurrence frequency of each of the four bases, the average position of nucleotides and the biochemical properties of nucleotides. To validate the advantages of our approach, we test it on several data sets and further make comparison with some state-of-the-art sequence analysis techniques. Our method performs as good as alignment methods in accuracy and spends far less time than them. For one viral dataset, it even provides better result than alignment methods. It also works better than k-mer based methods in accuracy.log4(n), i.e. k = floor(log4(n)) (n is the minimum length of sequences studied). From the results we can see the efficiency of our new method is better than FFP on these datasets.To test the effectiveness of our new method, we employ it to different datasets, which include mammalian mitochondrial genomes, virus, and bacteria genomes. Some datasets are small sized and others are medium sized. The length of sequences ranges from ten thousands to several millions base pairs. For each dataset, using our method the multiple encoding vectors of its genetic sequences are calculated. To verify the efficiency of our algorithms, the FFP method is used to compare. We use MEGA to draw the phylogeny of datasets for the two methods. Unlike FFP which applies Jensen-Shannon Divergence to compute the pairwise distance between any two vectors, our new method use Euclidean distance. In the process of using FFP method, we choose k to be the minimum integer of 22. Using our proposed method along with UPGMA algorithm, the phylogenetic tree of 41 mitochondrial genomes is constructed. As displayed in Fig. Primates (red), Cetacea (green), Artiodactyla (pink), Perissodactyla (light green), Rodentia (black), Lagomorpha (dark red), Carnivore (blue), and Erinaceomorpha (grey). According to the FFP method, length of substrings is chosen to be 7 and thus the phylogeny of the 41 species is obtained. The phylogenetic tree is provided in the supplementary file. As displayed in Fig. Perissodactyla are improperly clustered into two clades. Indus River Dolphin from Cetacea is separated from other species from Cetacea. The Carnivore, Primates and Artiodactyla clades are all mistakenly divided into more than one group. Obviously, for this dataset our new method gets a greater result.The first benchmark dataset contains 41 mammalian complete mitochondrial genomes (mtDNA) with about 16500 base pairs. In these mammals, the structure of mtDNA is circular and double-stranded. One strand of each mtDNA has rich guanine referred to as the heavy strand and the other strand includes rich cytosine referred to as the light strand. In our study, the heavy strands of mtDNA are chosen. These sequences are not highly conserved and have a fast mutation rate23. Influenza A viruses are single-stranded, negative-sense, segmented RNA viruses. The genome of flu virus contains around 13500 nucleotides and is divided into 8 segments. The subtypes of Influenza A viruses are classified according to an H number for the type of hemagglutinin (H) and an N number for the type of neuraminidase (N) in viral surfaces24. The viruses have 18 different H serotypes (H1-H18) and 11 different N (N1-N11) serotypes. For instance, the recently appearing H7N3 is an Influenza A strain with a type 7 hemagglutinin (H) and a type 3 neuraminidase (N). The several subtypes of Influenza A viruses are able to infect many species including wild birds, swines, dogs, horses, and even humans. It is known that H1N1, H1N2, and H3N2 subtypes can circulate among humans. In this study, we utilize Segment 6 gene encoding N (neuraminidase) of Influenza A virus to perform phylogenetic analysis. Using our method, the 38 flu viruses are correctly clustered into five groups as show in Fig. 25. According to the FFP method, the choice for length of k-mer of these viruses is 5 and their phylogenetic tree is constructed. As shown in Fig. Influenza A viruses may cause influenza disease in birds and mammalsEnterovirus genus and Picornaviridae family. They form three phylogenetically distinct clusters: HRV-A, HRV-B, HRV-C26. In the previous work, 113 complete genomes composed of the three clusters are used to explore their evolutionary relationships. In addition, 3 HEV-C genomes were also added to the dataset as outgroup. Based on multiple sequence alignment (MSA), the evaluation was very time-consuming although the classification performed well. Unlike the MSA, our new approach is very fast to finish conversion from the genome sequences to 18 dimensional vectors. As shown in Fig. To test our proposed method, we perform the phylogeny of 116 Human rhinoviruses (HRVs). HRVs are the major cause of the common colds and lead to more than one-half of cold-like diseases. HRVs are single-stranded positive sense RNA viruses of about 7200 nucleotides in length. The genome of Human rhinovirus contains a single gene encoding one polyprotein, which is then cleaved to yield 11 proteins. HRVs belong to Ebolavirus genus includes five species: Ebola virus , Sudan virus (SUDV), Bundibugyo virus (BDBV), Tai Forest virus (TAFV), and Reston virus (RESTV)27. Ebola viruses are single-strand negative sense RNA viruses whose genomes consist of about 18900 bases. Each ebolavirus genome encodes seven proteins in which glycoprotein is the only viral protein on the surface of ebolavirus. The first case of human infected by EBOV was reported in 1976 in Zaire (currently the Democratic Republic of the Congo (DRC))28. Recently the epidemic starting in 2014 in Guinea is unprecedent because it is the first outbreak in West Africa and the magnitude is largest among known outbreaks.The Ebolavirus, we apply the novel numerical vector method to construction of the phylogenetic tree of 59 viruses in Ebolavirus genus. As shown in Fig. Ebolavirus. The EBOV strains in the 2014 epidemic in Guinea are clustered together. The EBOV strains in Zaire (DRC) pandemic in 1976–1977 are placed together as a clade. The three EBOV strains from the 1995 outbreak in Zaire (DRC) form a clade. The EBOV strains in DRC pandemic in 2007 are clustered together. The strains from the two epidemics occurred in Gabon are divided into two different clusters according to the outbreak periods. The SUDV branch is clustered with the RESTV branch. BDBV and TAFV viruses are positioned together. Our results are consistent with those in previous work29. Since the viral genomes in this dataset have about 19 kilobases, the k-mer length is thus 7. The phylogenetic relationships among these viruses using FFP method are depicted in Fig. Ebolavirus genus.To understand EBOV epidemics in humans and their relationships with other viruses in genus Coronaviridae. Their genomes in length range from 25,000 to 32,000 nucleotides. Some subtypes of coronaviruses can also infect humans and cause many respiratory and gastrointestinal diseases with highly variable severity. Recently, the emergent Middle East respiratory syndrome coronavirus (MERS-CoV) had high case-fatality rate and caused outbreak of severe respiratory disease in Arabian Peninsula. As a result of pandemics from coronaviruses especially the SARS, the classification and evolutionary relationships among these viruses have been extensively investigated.Coronaviruses are enveloped, single-stranded, positive-sense RNA viruses within the family 30. At the same time, it was positioned away from HCoV-OC43 (group 2) and SARS (group 4). The new isolated HCoV-HKU1 was thought as a distinct member of group 231. However, the HCoV-HKU1 was designated as a member of a new group 532. Using our novel method, the phylogeny of the coronaviruses is reconstructed based on the UPGMA algorithm. As shown in Fig. 33. Group 2, group 4, and new group 5 are separated in the tree, which is consistent with the studies in published papers32. According to the FFP method, the length of substrings chosen for these organisms is 6 and the phylogeny is constructed. As displayed in Fig. We apply our 18 dimensional vector method to a widely studied dataset containing 30 coronaviruses and 4 extra non-coronaviruses as outgroup. The newly discovered human coronavirus NL63 was clustered together with human coronavirus 229E (group 1)Aeromonadaceae, Alcaligenaceae, Bacilleceae, Borreliaceae, Burkholderiaceae, Caulobacteraceae, Clostridiaceae, Desulfovibrionaceae, Enterobacteriaceae, Erwiniaceae, Lactobacillaceae, Mycoplasmataceae, Rhodobacteraceae, Staphylococcaceae, Yersiniaceae. The genome length in this dataset mainly range from 3 to 10 Mb. As displayed in Fig. Lactobacillaceae, Clostridiaceae and Staphylococcaceae from phylum Bacilli are clustered together. In the tree constructed by FFP model , the tradition multiple sequence alignment methods are computationally infeasible to perform phylogeny. To test our model, we analyze a dataset consisting of 59 bacterial species. This set contains 15 families: del Fig. , howeverSequence comparison is crucial to understand the evolutionary relationships among organisms. Methods with alignment usually perform well if the genetic sequences are homologous. However, results obtained with alignment may be unreliable due to genetic recombination and high mutation rates. Besides, these approaches are time-consuming and thus are not suitable to align long sequences and large volume of data. According to the results mentioned above, our proposed alignment-free method shows its advantages on accuracy over alignment-based methods. Compared to a state-of-the-art method based on k-mer, the new technique still has the merit to construct correct evolutionary relationships. Our method is applicable to phylogeny construction of many kinds of species including mammals, viruses, and bacteria.2 which is one of the most popular alignment methods. All programs are implemented on a laptop with 8 GB RAM, Intel (R) Core (TM) i7-4500U CPU, and 1.80 GHz and the running time unit is minute. As listed in Table Additionally, we also compare our tool to ClustalW algorithmPan troglodytes), human (Homo sapiens), monkey (Macaca mulatta), gorilla (Gorilla gorilla), dog (Canis familiaris), horse , mouse (Mus musculus), opossum (Monodelphis domesticus), and platypus (Ornithorhynchus anatinus). The length of these X chromosomes ranges from about 6 to 147 Mb. The accession numbers are listed in the supporting file. Based on the new multiple encoding vector, the UPGMA phylogenetic tree of nine mammals is obtained. As shown in Fig. Our method is very fast and has the potential to construct phylogeny for whole genomes such large as mammalian. Note that mammalian genomes are often divided into several chromosomes. To compute the position of nucleotides, we only choose chromosome X of mammals to do phylogenetic analysis. Our dataset includes the species: chimpanzee  be a DNA sequence of length n, that is, A and G bases are purine denoted by the letter R. The C and T bases are pyrimidine represented by Y. In another way of grouping, the A and C nucleotides are amino denoted by M and the G and T are keto denoted by K. According to the H bound, the G and C have strong H bond and they are represented by the letter S. The A and T bases contain weak H bond and they are represented by W. We define three numerical values for each of the letters R, Y, M, K, S and W. To characterize the distribution of R and Y in the sequence Q, we first replace the letters A and G by R and replace the C and T by Y in the sequence Q. Then the sequence only contains two kinds of letters: R and Y. For R, define wR(⋅):{R, Y} → {0, 1} such that wR(si) = 1 if si = R and 0 otherwise. For R, we define the R, the average position of R and the variation of the position of R appearing in the sequence Q.R occurring in Q.Let R appears.Let R.Let Let Y we define wY(⋅):{R, Y} → {0, 1} such that wY(si) = 1 if si = Y, and 0 otherwise. Then we get three characteristics for Y: nY, μY and M, K, S, and W. Thus an 18 dimensional vector of the DNA sequence Q is defined by 38.Similarly, for The datasets analysed during the current study are available in the “supplementary file” and the R source code in this paper is freely available to the public upon request.Supplementary PDF File


Evolutionary constraint for insertions and deletions (indels) is not necessarily equal to constraint for nucleotide substitutions for any given region of a genome. Knowing the variation in indel-specific evolutionary rates across the sequence will aid our understanding of evolutionary constraints on indels, and help us infer how indels have contributed to the evolution of the sequence. However, unlike for nucleotide substitutions, there has been no phylogenetic method that can statistically infer significantly different rates of indels across the sequence space independent of substitution rates. Here, we have developed a software that will find sites with accelerated evolutionary rates specific to indels, by introducing a scaling parameter that only applies to the indel rates and not to the nucleotide substitution rates. Using the software, we show that we can find regions of accelerated rates of indels in the protein alignments of primate genomes. We also confirm that the sites that have high rates of indels are different from the sites that have high rates of nucleotide substitutions within the protein sequences. By identifying regions with accelerated rates of indels independent of nucleotide substitutions, we will be able to better understand the impact of indel mutations on protein sequence evolution.The online version of this article (doi:10.1007/s00239-016-9761-9) contains supplementary material, which is available to authorized users. Different types of mutations are under different mutation and selection dynamics. In terms of protein structure, there can be regions of the protein that are unconstrained for the amino acid compositions, but constrained for their overall length Grishin . Also thWith nucleotide substitutions, we can identify sites in the genome, which are unusually conserved, indicative of evolutionary constraint, or sites that are changing at an unusually fast rate. Several methods are available that utilize the comparative data from multiple species and take into account the phylogeny to estimate different rates across the genome assuming well-developed models of sequence evolution  that only scales the insertion (λ) and deletion (μ) rate parameters  . This requires removing the extra bookkeeping in the generative model.So, instead of Eq. (19) in Rivas and Eddy , we alloi includes all possible residues as well as gaps.Instead of Eqs. (20) and (21) in Rivas and Eddy , we justThus, in addition to Eqs. (22), (23), and (24) in Rivas and Eddy , we needF84ε model, called F84ε-relaxed, in the PHAST software, including the three modifications except for the extra bookkeeping. The program dnaml-ε by Rivas and Eddy also does several extra steps when calculating the likelihood that is different from PHAST. For example, it divides the branch lengths by a constant calculated from the background frequencies, and it performs a midpoint rerooting of the tree. To verify our program was correct we had to get exact results for the likelihood calculations as dnaml-ε. Thus, several options were added to perform the same procedures that dnaml-ε uses. With these options turned on, we confirmed that dnaml-ε and our program computes the same values. In contrast, F84 does not use a special pruning algorithm, and can be implemented using the native pruning algorithm of PHAST. We also confirmed that when presented with alignments without any gaps, F84ε produces the same result as F84.We implemented this relaxed version of the F84, F84ε, and F84ε-relaxed. Since we were mostly interested in estimating indel rates on codon sequences of the protein-coding regions instead of the whole genome, we modified the software so that it can estimate the model on data consisting of many multiple alignments of different number of sequences instead of one whole-genome alignment.We extended the program phyloFit  The indel model F84ε-relaxed is compared against the model F84ε-relaxed + ρindel which allows a scaling of the insertion deletion rates (λ and μ) for each site. The latter comparison will identify the sites in the alignment that have significantly higher or significantly lower indel rates than the rates estimated on the whole data.We extended the program phyloP , we did not use PHAST’s original scaling algorithm, and instead optimized the scaling parameter by multiplying it directly to the parameters λ and μ, before they are plugged into the Eqs. (6), (7), and (9) of Rivas and Eddy  or acceleration (ρindel > 1) depending on the estimated value of ρindel. We also report a score, following the convention of PHAST, which is a log transformation of the p-value multiplied by ±1 depending on conservation (+1) versus acceleration (−1). The score from the first model comparison is called “phyloP score” in PHAST. We call the score from the second model comparison “indelP score” . We retrieved 21,124 gene trees in newick format and 21,124 cDNA alignments corresponding to those gene trees in fasta format. We only retained alignments that have a Homo Sapiens sequence and have at least five sequences. Because we are interested in short indel events and not long composite insertion or deletion of whole domains, we cleaned the alignment with GBLOCK  within PHAST to see if we could apply the scaling parameter (ρindel) to HKY + G and get appropriate results. Below is the substitution matrix for HKY + G, and the model with the scaling parameter multiplies the rates in the ‘–’ row and ‘–’ column with ρindelBefore implementing the HKY + G substitution rate matrix estimated for the primate alignments is as follows:π  = , which are in turn used as prior distribution in the likelihood calculation. Because gaps are rarer than residues in our dataset, the rate of gap turning into a residue  =  and a parameter for the geometric distribution of ancestral sequences p = 0.998058. These parameters correspond to the substitution matrix as follows:The parameters estimated based on the primate alignments are F84ε did not suffer from the problem of large influence of prior distributions on the column-likelihood that we experienced with HKY + G. The main reason is that the probability of a site under F84ε only relies on the prior distribution of residues (non-gaps) and does not take into account the prior distribution of gaps into the equation . The leλ = 0.008509; μ = 0.061660; α = 0.513467; β = 0.486533, with a prior residue distribution π  =  and a parameter for the geometric distribution of ancestral sequences p = 0.998058. These parameters correspond to the substitution matrix as follows:The parameters estimated based on the primate alignments are The difference in the parameter estimates is minor. But, the difference in the model assumption, and the corresponding changes in the formulas , 2) and and2) anF84ε-relaxed with the scaling parameter ρindel across 1162 alignments containing 942,411 sites. Likelihood ratio test on each site comparing the models F84ε-relaxed versus F84ε-relaxed + ρindel identified around 2 % of sites that are under indel rates significantly different from the null model with a significance level of α = 0.05. Using a Bonferroni-corrected significance level of α = 5.3e−8, there were 102 sites that had significantly different indel rates , but sites with small p-values with scaling parameter ρindel < 1 (conserved rates) are non-existent. This is because for the dataset we constructed, a majority of the columns in the alignment were columns full of residues and no gaps, so the null model estimated on the whole dataset was mostly representing sites with no indel events. Since columns full of residues with no gaps are as conserved as sites can be for indel events, we could not identify any sites that are more conserved than the null.The sites that have significantly different rates were all found to have accelerated rates and not conserved rates as shown in the volcano plot in Fig. We visualize the indelP score and the phyloP score for an example gene family of podoplanin (PDPN) to show how the scores are distributed across the alignment as shown in Fig. Those sites that were identified to have significantly different indel rates did not overlap with the sites that had significantly different nucleotide substitution rates Table , confirmλ) and deletions (μ) are also increased. Rates of nucleotide substitutions (α and β) are not affected by the amount of gaps, as expected.To understand the effect of the amount of gaps in the data on the estimated parameters, we ran the software with three different alignment filtering criteria: with at most 15 contiguous gaps, at most 30, and at most 45 contiguous gaps. In Table As is the case with any sequence models, our method relies on accuracy in sequence alignment. But, since we are trying to estimate accelerated indel rates, and regions containing gaps are especially prone to alignment errors, the negative effect of misalignment is more severe for our problem. Currently, we do not have a solution to automatically take care of this problem. We suggest that the user has to manually check for any sites that are identified to be accelerated in indel rates to make sure the regions are not misaligned. Figure Here, we developed a new software that can estimate accelerated rates of indels independent of rates of nucleotide substitution. Before, evolutionary constraint in molecular evolution was mainly studied in context of nucleotide substitutions. But evolutionary constraint for indel mutations is not necessarily equal to constraint for nucleotide substitutions. Using the software, we can disentangle the two concepts and measure the rates separately. We can estimate the global rate of insertion and deletions for multiple sets of alignments, and afterwards identify sites that have significantly higher rates of insertions and deletions compared to the global rate estimated. Conservation metrics such as phyloP scores have proven to be useful in identifying functional elements in the genome, and predicting deleterious effects of mutations. Whether indel-specific rate metrics can be utilized in a similar way will need to be explored in the future. One difference with indels compared to nucleotide substitutions is that while it is relatively easy to identify regions accelerated for indels, practically it is difficult to identify regions conserved for indels as shown with our results. This may limit the utility of the metric in predicting the deleterious effect of indel mutations in the region. Another difference between indels and nucleotide substitutions is that indels have larger variance in their mutation rate across the genome, such that regions with accelerated rate of indels may be difficult to interpret. Regardless, the ability to measure accelerated indel rates independent of nucleotide substitution rates will contribute to our understanding of how proteins evolve through indel mutations.https://github.com/HanLabUNLV/Phasterate. The data associated with the study can be found at https://github.com/HanLabUNLV/PhasterateData.The software can be found at Supplementary material 1 (PDF 493 kb)Below is the link to the electronic supplementary material.


Frameshift translation is an important phenomenon that contributes to the appearance of novel coding DNA sequences (CDS) and functions in gene evolution, by allowing alternative amino acid translations of gene coding regions. Frameshift translations can be identified by aligning two CDS, from a same gene or from homologous genes, while accounting for their codon structure. Two main classes of algorithms have been proposed to solve the problem of aligning CDS, either by amino acid sequence alignment back-translation, or by simultaneously accounting for the nucleotide and amino acid levels. The former does not allow to account for frameshift translations and up to now, the latter exclusively accounts for frameshift translation initiation, not considering the length of the translation disruption caused by a frameshift.We introduce a new scoring scheme with an algorithm for the pairwise alignment of CDS accounting for frameshift translation initiation and length, while simultaneously considering nucleotide and amino acid sequences. The main specificity of the scoring scheme is the introduction of a penalty cost accounting for frameshift extension length to compute an adequate similarity score for a CDS alignment. The second specificity of the model is that the search space of the problem solved is the set of all feasible alignments between two CDS. Previous approaches have considered restricted search space or additional constraints on the decomposition of an alignment into length-3 sub-alignments. The algorithm described in this paper has the same asymptotic time complexity as the classical Needleman–Wunsch algorithm.human, mouse and cow genes of ten mammalian gene families from the Ensembl-Compara database. The results show that our method is particularly robust to parameter changes as compared to existing methods. It also appears to be a good compromise, performing well both in the presence and absence of frameshift translations. An implementation of the method is available at https://github.com/UdeS-CoBIUS/FsePSA.We compare the method to other CDS alignment methods based on an application to the comparison of pairs of CDS from homologous The online version of this article (doi:10.1186/s13015-017-0101-4) contains supplementary material, which is available to authorized users. The accuracy of the sequence alignments and similarity measures are directly related to the accuracy of subsequent analysis. CDS alignment methods have many important applications for gene tree and protein tree reconstruction. In fact, they are useful to cluster homologous CDS into groups of orthologous splicing isoforms , 2 and cFS translation consists in alternative AA translations of a coding region of DNA using different translation frames . It is aHere, we consider the problem of aligning two CDS while accounting for FS translation, by simultaneously accounting for their nucleotide and AA sequences. The problem has recently regained attention due to the increasing evidence for alternative protein production through FS translation by eukaryotic gene families , 19.n and m is a combination of its score at the nucleotide level and its score at the AA level. They described a O(n2m2) algorithm in  A coding DNA sequence (CDS) is a DNA sequence on the alphabet of nucleotides n is a multiple of 3. A coding sequence is composed of a concatenation of i,  is computed before The couples  the algorithm computes the maximum score of an alignment between k inserted (resp. deleted) codons is We implemented the present CDS alignment algorithm with an affine gap penalty scheme  such thahuman, mouse and cow genes belonging to the family and satisfying Definition We evaluated the algorithm through applications on a mammalian dataset containing CDS sequences from ten gene families obtained from the database Ensembl-Compara version 83 . The firWe compared the accuracy of five pairwise global alignment methods, including the present method, for computing CDS alignments in the presence or absence of FS translation between the compared CDS. The five methods vary according to the alignment algorithm used, either the present CDS alignment algorithm called FsePSA allowing to penalize both FS translation initiation and extension, or the CDS alignment algorithm called MACSE  penaliziThe present CDS alignment algorithm is used in two of the five methods, namely fse and fse0. These two methods differ according to the value given to the parameter of MACSE  is used A levels .We used the five methods to compute pairwise alignments between the pairs of CDS within each of the ten gene families of our dataset, yielding 4011 alignments in total for each of five methods. In the absence of available benchmarks for the direct evaluation of the accuracy of CDS alignments, we base our evaluation on four indirect strategies.In the first strategy, we consider the CDS multiple alignment of each gene family obtained using MACSE  as a benBased on the results of the large-scale experiments discussed in the following, the best compromise for default values of FsePSA parameters are N CDS, we consider the MACSE  was usedTable c in the alignment that are aligned with a triplet of nucleotides yielding the same amino acid as c. The third criterion Gap_init is the number of gap-containing columns in the alignment, either insertion or deletion columns that are preceded by a different type of column. The fourth criterion Gap_length is the overall number of gap-containing columns in the alignment. The fifth criterion FS_init is the number of FS translation segments found in the alignment. The last criterion FS_length is the overall number of columns in the alignment intersecting a FSext codon.Six criteria were defined and used to compare the five pairwise alignment methods. Given a pairwise CDS alignment, the first criterion Identity_NT counts the number of gap-free columns in the alignment containing a nucleotide match. The second criterion Identity_AA counts the number of IM and FSext codons K and N, and the FSext codon AAT ending at position 42 yielding the amino acid N different from the amino acid K yielded by the triplet AAG. Note that the definitions of the six criteria exploit the definitions of codon sets used in Definition For each of the nine cases obtained by combining the values of the parameters fs_open_cost , G2 (containing CDS B1-001) and G3 (containing CDS B2-202).We computed the normalized pairwise similarity scores between the CDS, using the five alignment methods ,G3) between the groups, confirming the results of the third evaluation strategy. For all methods, the CDS phylogeny reconstructed inside the group G2 is ,,))) inducing an evolution of the seven genes with a speciation event at the root of the gene tree. The phylogeny reconstructed for the groups G1 and G3 is ,,))), inducing an evolution of the genes with a duplication event at the root of the phylogeny.Table n and m of two compared CDS are 12.55× nm for fse and fse0  and 6 (for needlenuc) of the “Additional file 4: Pairwise alignments for the 3-CDS benchmark. Zip file containing the sequence file and the pairwise alignment files at the fasta format for the manually-built3-CDS benchmark considered in the “Additional file 5: Pairwise alignments for the 21-CDS dataset. Zip file containing the sequence file and the pairwise alignment files at the fasta format for the 21-CDS benchmarkconsidered in the “


The next-generation of sequencing technologies, along with the development of bioinformatics, are generating a growing number of reads every day. For the convenience of further research, these reads should be aligned to the reference genome by read alignment tools. Despite the diversity of read alignment tools, most have no comprehensive advantage in both accuracy and speed. For example, BWA has comparatively high accuracy, but its speed leaves much to be desired, becoming a bottleneck while an increasing number of reads need to be aligned every day. We believe that the speed of read alignment tools still has huge room for improvement, while maintaining little to no loss in accuracy.Here we implement a new read alignment tool, Fast Seed-and-Vote Aligner (FSVA), which is based on seeding and voting. FSVA achieves a high accuracy close to BWA and simultaneously has a very high speed. It only requires ~10–15 CPU hours to run a whole genome read alignment, which is ~5–7 times faster than BWA.In some cases, reads have to be aligned in a short time. Where requirement of accuracy is not very stringent, FSVA would be a promising option.https://github.com/Topwood91/FSVAFSVA is available at  Next-generation sequencing technologies have developed rapidly in recent years mainly in two regards. On the one hand, the throughput potential is tremendous. For example, a system consisting of a set of 10 HiSeq X ultra-high-throughput instruments (HiSeq X10) can deliver over 18,000 human genomes per year. On the other hand, the cost of whole genome sequencing is decreasing steadily. Currently the cost of whole genome sequencing for an individual or patient stands at roughly $1,000. It seems likely that this trend will continue, and sequencing costs will continue to fall. This allows access to genome sequencing for a large percentage of the population. All of these changes have led to a sharp increase in the amount of sequence data and pose a new challenge to sequence analyzers.Usually, the data produced by a sequencing platform is not a single sequence with all DNA information, but consists instead of a large number of short subsequences, called reads, with partial DNA information. Read alignment is then required to map reads to a reference genome and identify the coordinate of each individual read on the reference. The past few years have witnessed the appearance of diverse read alignment tools, which can be roughly divided into two categories: tools based on hash table and tools based on prefix/suffix trie . A tool Read alignment is usually the first and most time consuming step of genome sequence analysis. Although some existing tools are widely used with great success, their speeds cannot keep up with data increases. The very widely-used BWA definitely has many advantages and achieves relatively accurate results, but its speed is not as fast as could be desired. Several new versions of BWA such as BWA-SW  and BWA-To accelerate the speed, researchers have tried many methods, such as seeking assistance from GPU , cloud cIn this study, we propose a new read alignment tool, Fast Seed-and-Vote Aligner (FSVA), which is ~5–7 times faster in running time than BWA-MEM while keeping a similar mapping accuracy as BWA-MEM. In practice, for a whole genome read alignment (library size ~200 GB), FSVA costs ~10–15 CPU hours on a single core and ~4–6 CPU hours on four cores. The respective time cost of BWA-MEM in the same scenario is about ~70 CPU hours and ~20 CPU hours. This advantage of speed makes FSVA a promising read alignment tool for big data.FSVA, borrowing the seed-and-vote strategy, builds a hash table for a reference genome and extracts seeds from the read to vote the coordinate. Compared with Subread, the main improvement of FSVA lies in the longer seed, which allows improved running speed and accuracy. While a longer seed cannot be represented as an integer in many programming languages, we avoid this problem by expressing the seed as a large prime number, guaranteeing a seed can be represented as an integer, and the size of hash table is not too big. This specific method is introduced in detail in the METHODS section. Experiments on simulated data and real data illustrate the great advantages of FSVA on time saving and present the alignment accuracy of FSVA as close to that of BWA-MEM, which is shown in the RESULTS section.FSVA, based on a seed-and-vote strategy, extracts seeds from a read and makes them vote the coordinate of the read. This method includes two steps: indexing and voting. The detailed methods are described in following sections. One point to note is that in the METHODS section, our algorithm is introduced based on 150 bp reads, which is the read length of HiSeq X10. In the actual situation, our tool FSVA can automatically adjust its parameters to fit various read lengths.Building the index refers to the building of a hash table for a reference genome sequence. In our hash table, the key is a 32bit unsigned integer converted from a subsequence of the reference genome. The value is a vector of 32bit unsigned integer, representing the location of the reference which a seed can be exactly mapped to.A DNA sequence, which usually contains only 4 characters , can be converted to a quaternary number. Thus an n-long DNA sequence can be converted to an unsigned quaternary integer with n digits, or an unsigned binary integer with 2n digits, namely, a 2n bit binary integer. Here we extract 31 bp subsequences from the beginning of a reference genome as keys, and the size of a sliding window of each pair of neighbor subsequences is set as 8 bp. To store a 31 bp subsequence, a 62bit unsigned binary integer is needed. For the sake of memory saving, we designate the 62bit unsigned integer modulo a large prime number M. Herein M is no bigger than the maximum of a 32bit unsigned integer. Thus the 62bit unsigned integer is converted to a 32bit one, which is the final key utilized in the hash table. If a 31 bp subsequence extracted from the reference genome consists of other characters , it is dropped without key calculation. The process of key calculation is shown in Fig. The hash table has M pairs of key and value, with the key coming from the modulo operation. Since the key is calculated from converting an unsigned 62bit integer into a 32 bit one by modulo M operation, different subsequences may be given the same key. Therefore, a vector is utilized to store coordinates of the subsequences with the same key, and the value of the key is this vector. If no value exists for a key, we mark NULL. Figure We treat each one 31 bp subsequence extracted from a read as a seed. Thus an n-long read can generate a total of n-30 different seeds. The process of key calculation of hash table building is utilized also to get the key of a seed. Then, by querying the hash table with the key of a seed, the location of the seed can be located. The seed set with size n-30 of an n-long read can search out its corresponding coordinate set with at most n-30 vectors. Coordinates recorded in these vectors vote the coordinate of the read, and the one with most votes is selected. Herein, the vote counting is based on a block. A block is an interval of the genome reference with same length of the read. Figure In practice, the final mapping block should have at least two votes; otherwise, we believe this read cannot be mapped to the reference genome. Second, if more than one block is tied for most votes, we choose one randomly and set its mapping quality as 0. Third, if the alignment has more than two mismatches, we perform a Smith-Waterman dynamic programming between the read and the extending block. The extending block is the block with the most votes extending towards upstream and downstream with 36 bp Fig. . Fourth,In our experiment, we set the length of seed as 31 bp, and for hash table also 31 bp subsequences are extracted from the genome reference to calculate keys. If seed is shorter, many seeds will be generated from a read and much more coordinates could be selected as candidates waiting to be voted upon. Consequently, the time cost is higher. On the other hand, if the seed length is larger than 32 bp, it cannot be converted to a 64bit integer and cannot be represented in most programming languages, which will introduce trouble on the programming side.Besides, in our algorithm, 31 bp and not 32 bp is selected as seed length is to avoid an unwanted situation where a perfect match block gets less votes than a block with mismatches. In general, we should ideally prefer the perfect match block. We give an example in Fig. In this way if the seed length is set as 31 bp, the abnormal voting results caused by 32 bp seed length could be avoided. Each case of exact match will be given 15 votes. If the read length is not 150 bp, FSVA can automatically adjust the seed length to fit the read length by default. Users can manually set seed length also in our tool configuration.For each alignment, FSVA calculates a mapping quality score by comparing the votes of the optimal and the suboptimal blocks. Specifically, the mapping quality score is calculated as following:Where optimal and suboptimal represent the number of votes of the block with the most votes and second most votes, respectively. Obviously our mapping quality score is a multiple of 6 and no more than 60.To study the performance of FSVA, we compared FSVA with Subread, BWA and Bowtie2 . BWA is In our experiments, when running BWA-MEM, Subread and Bowtie2, all options are set as default. Although FSVA can run on multiple threads, to simplify the comparison of time cost, we ran the test only on single thread for both the simulated data and the real data.Our simulated data is produced by wgsim, a tool provided by SAMtools . With thHere we use wgsim to fetch 1 million simulated reads from the whole genome sequence hs37d5. Some arguments in Wgsim are set to simulate the properties of reads. To simulate the real situation, we allowed the base error rate be 0.4% and the mutation rate be 0.1%, in which the rate of SNP mutations is 0.085%, and the rate of indel mutation is 0.015%. To study the effect of read length, we generate 125 bp and 150 bp reads respectively in the simulation test.As the read is taken from the reference, we know its exact coordinate. If the distance between the real read and the predicted one from a tool is no more than 30 bp, we treat it as a correct alignment.First, to evaluate the influence of seed length on the final alignment results in FSVA, we did a test on 150 bp reads and 125 bp reads using seeds with different length, and Table The overall performances of BWA-MEM, Subread, Bowtie2 and FSVA on simulated data are shown in Table Consistent with intuitions, longer reads lead to better confident mapping percent, error rate and higher time cost for all tools. For FSVA, the time increase caused by long reads could almost be ignored, which means FSVA will be more competitive with the trend of reads becoming longer and longer.Figure Although FSVA is not superior to BWA-MEM in terms of mapping accuracy, its advantage of time saving is extremely significant. In some cases, read data needs to be processed in a short time and the requirement of accuracy is not very stringent, and for this FSVA is undoubtedly the best choice. In next section, tests on real data proves that the difference of accuracy between FSVA and BWA-MEM does not have much influence on downstream variant calling.As for storage memory, BWA-MEM, FSVA, Subread and Bowtie2 need 5.2, 7.1, 6.7 and 3.2GB respectively, on both 125 bp and 150 bp reads. This level of memory cost can be tolerated by a modern personal computer, let alone a server. Thus, memory cost is not a major concern of read mapping tools.Five real whole genome sequenced datasets from Illumina HiSeq X10 were used to evaluate these four tools. All the reads were 150 bp and the number of reads varied from ~300 to ~500 million. The library sizes of these five datasets are shown in Table Table To study the influence of alignment results on downstream variant calling, we rely on SAMtools. SAMtools is a suite of utilities for interacting with high-throughput sequencing data. One of its utilities is taking output generated by short read aligners like FSVA and BWA-MEM, and calling variants. For each of these four tools, FSVA, BWA-MEM, Bowtie2 and Subread, we consistently utilized SAMtools as variant caller. Then, with some statistical factors of the called variants starting from each aligner, we compared the performance of the four tools. These statistical factors included the number of variants and Ti/Tv ratio. For variants, Ti/Tv is a ratio of the number of transition to transversion substitutions. Recent human studies particularly from the 1000 Genomes Project have been showing that for whole human genome, this ratio should be around 2-2.1. Since Subread did not complete output in the big sequencing datasets (as seen in Table In Fig. Figure According to the results of the real data, it is reasonable to say that the read alignment performance of FSVA can be used to get a high-quality variant set. The tests on real data supplement evidence from the simulation test, and show the higher performance of FSVA. Considering both the simulation test and real data test, we believe FSVA is a very competitive read alignment tool.FSVA utilizes the seed-and-vote strategy. Like most read alignment methods, it builds a hash table for a reference genome first. Then it extracts seeds from each read and searches the hash table to find the location of the seed in the reference genome. The coordinate of a read is voted by the seeds of the read.The most significant advantage of FSVA is its time saving potential. For a whole set of human genome sequencing data, the time cost of FSVA is one sixth or one seventh of BWA-MEM or Bowtie2. For example, for a sequencing library with 200G size, time cost of FSVA is 13.5 h while BWA-MEM costs 74 h, and Bowtie2 requires 86.5 h on a single CPU core. This impressive feature makes FSVA very competitive in short read alignment with large size, especially for cohort study.The accuracy of FSVA is illustrated here both on simulation data and real sequencing data. Experiments on simulation data show that the alignment accuracy of FSVA is almost good as BWA-MEM, especially when the mapping quality is selected as over 18, the difference on error rates between BWA-MEM and FSVA is very small, which can basically be ignored. On real sequencing data, since we do not know the correct coordinate of short reads, and usually the main focus of a pipeline for whole genome sequencing data analysis lies on variant calling, read alignment is just the first step. We explored the influence of four mapping tools, FSVA, BWA-MEM, Bowtie2 and Subread, on variant calling. In most cases, variants called based on the results of BWA-MEM are highest, 0.3–0.4 million more than that of FSVA and Bowtie2. For variant calling, BWA-MEM may be the most sensitive, while FSVA appears to have the best specificity. About 99.8% of variants called base on FSVA also could be found based on other short read alignment tools, and 85.67% of variants called based on BWA-MEM could be identified based on FSVA. For a cohort study, where the data involved is almost a tsunami and the accuracy for an individual is not critical, FSVA is a good choice.FSVA is not suitable for very short reads. As FSVA uses the seed-and-vote strategy, if the read is too short, the extracted seeds should be shortened, otherwise the number of seeds is not enough to vote. But very short seeds are unrepresentative and will introduce too much noise. We suggest FSVA being applied to any library in which read length is over 100 bp. Fortunately, the trend in biotechnology development is towards reads becoming longer and longer.FSVA is sensitive to SNP and error base. For FSVA, the location of an SNP or an error base will affect the voted output. It is clear enough that an SNP or a base error appearing in the head or tail of a read exists in less seeds than one appearing in the middle of the read. This means an SNP or a base error in the middle of a read will introduce more wrong-voting than if it were at the end of a read. If an indel exists in the middle of a read, the wrong-voting is again worsened. For whole genome sequencing or whole exon sequencing, this problem is alleviated since the reads are randomly cut and an SNP or an indel will be in the middle of some reads and will be at the end of other reads. For amplicon sequencing, since there are a lot of replicate reads, FSVA should be considered more before use.In this paper, we proposed a new short read alignment algorithm, named FSVA. FSVA adopts the seed-and-vote strategy, achieving a significant improvement on speed over existing methods. In some cases, reads have to be aligned in a short time and requirements of accuracy are not very stringent. In these incidences, FSVA would be a good choice.


Enterococcus faecium strain compared to seven closely related genomes. The analysis took less than two minutes on a single 2.1 GHz processor. The output is available in fasta format and can be visualized using an accessory program, gmosDraw (freely available with gmos).Prokaryotic and viral genomes are often altered by recombination and horizontal gene transfer. The existing methods for detecting recombination are primarily aimed at viral genomes or sets of loci, since the expensive computation of underlying statistical models often hinders the comparison of complete prokaryotic genomes. As an alternative, alignment-free solutions are more efficient, but cannot map  a query to subject genomes. To address this problem, we have developed gmos (Genome MOsaic Structure), a new program that determines the mosaic structure of query genomes when compared to a set of closely related subject genomes. The program first computes local alignments between query and subject genomes and then reconstructs the query mosaic structure by choosing the best local alignment for each query region. To accomplish the analysis quickly, the program mostly relies on pairwise alignments and constructs multiple sequence alignments over short overlapping subject regions only when necessary. This fine-tuned implementation achieves an efficiency comparable to an alignment-free tool. The program performs well for simulated and real data sets of closely related genomes and can be used for fast recombination detection; for instance, when a new prokaryotic pathogen is discovered. As an example, gmos was used to detect genome mosaicism in a pathogenic  Similar to viruses, prokaryotes often exchange homologous genetic material by horizontal gene transfer (HGT). Such sequences newly acquired from other organisms may help the host to adapt to environmental changes –3, or beEscherichia coli and Staphylococcus aureus genomes affected by horizontal gene transfer [As an efficient alternative, alignment-free programs can be used for annotating genome mosaic structure in whole bacterial genomes. An example is alfy, which was applied to transfer ,19. Howen subject genomes. The query mosaic structure is determined directly from the efficient pairwise and, only when necessary, multiple sequence alignment of the query and subject genomes.To address this problem of efficient and informative detection of genome mosaicism in closely related prokaryotes, we have developed a new program, gmos. This program analyzes one or more query genomes (either complete or draft) when compared to a set of Enterococcus faecium. gmos was written in C programming language and is available under the terms of the GNU General Public License from http://www.zpr.fer.hr/osobe/mirjana/gmos/. The software package includes documentation and test data.We show that gmos is a very efficient tool for detecting genome mosaic structure; its speed and memory performance are comparable to alignment-free solutions. We also demonstrate its accuracy on simulated sequences by comparing it to a statistically based tool that models recombination events, and by analyzing the recombinant structure of the pathogen m query genomes and the second file comprising n subject genomes . The program compares each query sequence to all subject sequences and outputs its mosaic structure. For the sake of simplicity, the implementation details are further presented only for the case when a single query genome is compared to a set of n subject genomes.The input to gmos are two files: the first file comprising Q be a query DNA sequence and S be a set of DNA subject sequences: S = {S1, S2,.., Sn} represented as strings over the alphabet Σ = {A, C, G, T}. gmos takes Q and S as input and computes the mosaic structure of Q, denoted M. M represents a list of best local alignments between regions of Q and regions from any subject of S. In the internal representation of Q and S, the characters that are not elements of the alphabet Σ are masked and treated as mismatches in the alignment computation.Let L, which comprises n lists of local alignments between Q and each Si, i = 1,.., n, and (ii) computation of the mosaic structure of the query, M. The program returns M, a list of local alignments between query regions and best locally aligned subject regions, each at least f characters long (default: 200 bp).There are two main phases of the program : (i) theQ) and each subject Si, i = 1,.., n, are constructed using their exact matches as alignment seeds. The minimal length of an exact match, considered to be an alignment seed, is computed for a pair  from the length of the shortest unique substring , i = 1,.., n, are determined using an enhanced suffix array [Q and Si, longer than expected by chance alone [Q and Sj, j ≠ i, which starts at the same query position, is stored as an interval in the corresponding interval tree of a  pair. The interval trees are constructed for each pair , similarly to the procedure used in [A bp apart . The first phase results in the set of n lists of local alignments (the set L).In the first phase, local alignments between a query (hustring ) for eacix array –23 constce alone  and long used in . The reg used in  and thenM, is constructed from n lists of local alignments (the set L). For each query region locally aligned to more than one subject region, the most similar subject region is chosen by comparing the pairwise alignment scores of each query-subject pair. In addition, efficient multiple sequence alignment is applied to detect the best local alignment(s) when multiple subjects are locally aligned to overlapping query regions. We apply a version of multiple sequence alignment called the central star approach [M finally comprises a list of local alignments at least f bp long (default: 200), where each local alignment is the best alignment between a query region and the overlapping subject regions. Also, the E-value of each local alignment is computed as in BLAST [λ (lambda) and K are the Karlin-Altschul parameters [m is the query length, n is the total length of all subject sequences and S is the new score of a local alignment re-computed using the refined DNA substitution scoring matrix (GTR model) [L. The parameters λ and K are analytically computed as previously described for ungapped alignment [λ and K for gapped alignments are expected to be similar to λ and K for ungapped alignments [In the second phase, the mosaic structure of a query genome, approach ,26, wherin BLAST ,28:E=Kmrameters ,28, m isR model) . This DNlignment ,28,30, gignments .n subject sequences of equal length: the sequence lengths ranged from 10 kb to 5 Mb and n ∈ {10, 50, 100}. Both programs were run with their default settings and the running time for each graph point was calculated as the average over 10 runs (each on a different data set with the same characteristics). The running time and memory analysis of gmos was performed on closely related sequences simulated using Dawg  and compS1, S2, S3). All sequences were of equal length, L, where L ∈ {10 kb, 25 kb}. The data sets were simulated using Dawg [L/5). The closest subject sequence of each query segment switched between S1 (SS1- S2- S1- S2- S1 (as previously described in [s) between a query segment and the immediate common ancestor between a query and its most closely related subject sequence (either S1 or S2) ranged from 0.001 to 0.11  to the common ancestor was also s. Accuracy was measured as the fraction of query nucleotides correctly assigned to the true closest relative (either S1 or S2). Each graph point represents the average accuracy over 10 runs (S1 or S2 were returned as best hits). The highest accuracy was obtained for s = 0.1 and L = 10 kb (0.94 ± 0.02) and for s = 0.05 and L = 25 kb (0.96 ± 0.03) . The ev.11 Figs . The evo 10 runs . We used ± 0.03) . As expeQ) was compared to a set of 10 subject sequences , each 10 kb long. In the experiment, sequences S1 and S2 were identical and shared the immediate common ancestor with a query sequence , s, ranging from 0.001 to 0.1 . Again, since the program has been developed for closely related sequences, accuracy of the program declined for s > 0.1 (In the next experiment using simulated data (generated using Dawg ), a quersequence . In this1 to 0.1 . Accurac 10 runs . The ana s > 0.1 , similar s > 0.1 .We also compared gmos to a statistically based software, ClonalOrigin , on simut), the rate of true positives (TPR), and the rate of false positives (FPR) of both programs were calculated as the average over 10 runs (each run on a different data set with the same characteristics) , each 10 kb long. A query was constructed to be most closely related to S1, with its second and fourth segment acquired from S2, as described in the section above , and the false positive rate (FPR) were measured for each s value  . The simon above , thus si s value . TPR reps = 0.001 to 0.0025 , with ClonalOrigin surpassing gmos over moderate evolutionary distances (s = 0.005 to 0.025) and gmos outperforming ClonalOrigin over larger evolutionary distances (s ≥ 0.05).Both programs had lower sensitivity over very small evolutionary distances, o 0.0025 . At thesTPR increased and FPR decreased with evolutionary distance and the program performed best for the highest evolutionary distance, s = 0.1 . In comparison, ClonalOrigin achieved the best result for s = 0.01 . However, in the case of ClonalOrigin, FPR and the number of incorrectly detected recombination events increased with larger evolutionary distances, s > 0.025. The program was not tested for s > 0.05 due to the substantial decreases in performance and computational burden at these values. In addition, gmos outperformed ClonalOrigin by orders of magnitude in terms of speed for all data sets; analysis of a single data set by gmos took around 0.04 seconds across all evolutionary distances, while the running time of ClonalOrigin increased with larger values of s , which was compared to seven other Enterococcus faecium genomes  was treated as a query and the other seven genomes were considered subjects. In accordance with the original result [Enterococcus faecium 1,231,408; gmos analysis showed that the strain’s supercontig 1.3 is most similar to the supercontigs 1.2 of two clade B strains, Enterococcus faecium Com12 and 1,141,733 . The recombinant strain , were returned for some query regions as their closest relatives. Among the returned local alignments, the lowest identity was 82%, which is in agreement with the previous test , and does not require pretreatment of input sequences. These properties make it an ideal tool for situations where rapid screens for recombination are needed and as a complement to more precise, statistically based programs. Its output is the list of best locally aligned regions in multi-fasta format. Additionally, a sequence mosaic structure can be visualized using the accessory program, gmosDraw, which enables easy interpretation of the results.S1 FigQ and the subject set {S1, S2, …, S10} (simulated data). Q is most closely related to sequences S1 and S2 along its entire genome.Genealogy of a query sequence (TIF)Click here for additional data file.S2 FigS1 and S2 using the simulated genealogies Click here for additional data file.S1 FileThe results of accuracy and speed comparison of gmos to ClonalOrigin on simulated data sets.(PDF)Click here for additional data file.S2 FileEnterococcus faecium 1,231,408. Tables B and C also include the ClonalOrigin results from the same data set. Table A) Genes of clade B origin along the supercontig 1.2. Table B) Genes of clade B origin along the supercontig 1.3. Table C) Genes of clade B origin along the supercontig 1.5.Tables listing the genes of clade B origin along the supercontigs 1.2, 1.3, and 1.5 of (XLS)Click here for additional data file.


Multiple sequence alignment (MSA) is commonly used to analyze sets of homologous protein or DNA sequences. This has lead to the development of many methods and packages for MSA over the past 30 years. Being able to compare different methods has been problematic and has relied on gold standard benchmark datasets of ‘true’ alignments or on MSA simulations. A number of protein benchmark datasets have been produced which rely on a combination of manual alignment and/or automated superposition of protein structures. These are either restricted to very small MSAs with few sequences or require manual alignment which can be subjective. In both cases, it remains very difficult to properly test MSAs of more than a few dozen sequences. PREFAB and HomFam both rely on using a small subset of sequences of known structure and do not fairly test the quality of a full MSA.In this paper we describe QuanTest, a fully automated and highly scalable test system for protein MSAs which is based on using secondary structure prediction accuracy (SSPA) to measure alignment quality. This is based on the assumption that better MSAs will give more accurate secondary structure predictions when we include sequences of known structure. SSPA measures the quality of an entire alignment however, not just the accuracy on a handful of selected sequences. It can be scaled to alignments of any size but here we demonstrate its use on alignments of either 200 or 1000 sequences. This allows the testing of slow accurate programs as well as faster, less accurate ones. We show that the scores from QuanTest are highly correlated with existing benchmark scores. We also validate the method by comparing a wide range of MSA alignment options and by including different levels of mis-alignment into MSA, and examining the effects on the scores.http://www.bioinf.ucd.ie/download/QuanTest.tgzQuanTest is available from Bioinformatics online. Multiple sequence alignment (MSA) is a very common first step in the analysis of sets of homologous protein sequences  each amino acid of a protein sequence belongs to . Since tWe set up two sets of test sequences, one with a small number of sequences (200) in each test case to test low throughput accurate aligners, and the second with more sequences in each test case (1000) to test high throughput aligners. We validate the method by comparing the results with SPS scores from structural alignment comparisons and by creating alignments with increasing amounts of random alignment error.To build the dataset for our benchmark we first choose all homologous Pfam family—Homstrad pairs where each Homstrad family has at least three reference sequences. We furthermore filter all duplicate sequences from all the Pfam families using the Cd-hit program .There are two benchmark configurations in our experiments. In the scalability configuration, we choose all Pfam family + Homstrad pairs with more than 6000 unique sequences, resulting in 151 Pfam + Homstrad pairs. In the accuracy benchmark, we choose all Pfam family + Homstrad pairs with more than 1000 unique sequences, resulting in 238 Pfam—Homstrad pairs.cn. For each Pfam + Homstrad pair firstly all hn available Homstrad sequences are selected. Three of these sequences are randomly assigned as reference sequences. Next, hn Homstrad sequences to make up test sets of cn sequences. For both, the scalability and the accuracy configurations, this last step is repeated five times, however, the selection of Homstrad sequences and the assignment of references is kept constant across the resamplings, and across the different cn. In the scalability configuration initial test sets of cn = 4000 sequences are generated. Smaller test sets with cn sequences from the initial test sets of 4000 sequences, such that larger test sets are always super-sets of smaller test sets. For the accuracy configuration sequences are sampled independently, such that non-reference sequences from the cn = 200 test sets might not be included in the cn = 1000 test sets.The next step is to generate the test sets of sequences of a chosen size We report here some statistics for the accuracy benchmark of 238 Pfam + Homstrad pairs. For the reference alignments, their lengths go from 39 to 887. The average sequence identity between the reference secondary structure sequences in the reference alignments go from 59.73% to 98.54%. The average sequence identities of the reference alignments go from 9.3% to 86.7%, with the median of 33.66%. For the 200 sequence test cases, the average lengths of the Pfam sequences go from 23.5 to 548.7.We use each aligner setting to align all the test sequence sets in the benchmark (200 sequence setting or 1000 sequence settings). For each chosen reference sequence from an alignment, the program generates its corresponding filtered alignment by deleting all columns from the alignment which correspond to a gap in the reference sequence. Three filtered alignments are generated for each alignment, corresponding to three chosen reference sequences. We then submit all the filtered alignments to the Jpred server, using its RESTful API (v.1.5), to predict the secondary structure of their corresponding reference sequences. For each predicted secondary structure sequence, we calculate its confusion matrix—the 3 × 3 contingency table calibrating the reference secondary structure states  versus the predicted states—by comparing it with the reference secondary structure extracted from the Homstrad database. The accuracy of the secondary structure prediction for one reference sequence of each filtered alignment is calculated by summing the diagonal elements of the confusion matrix and dividing it by the sum of all of the elements of the confusion matrix.To make a fair comparison, we only keep the sets of Pfam family + reference sequence + sample index where the filtered alignments of all aligner settings generate a predicted secondary structure from the Jpred server.The program then calculates the average prediction accuracy of one aligner setting as the average prediction accuracy of all of its chosen filtered alignments.The programs we use to test QuanTest are: T-Coffee Version t_coffee -output = fasta_aln -n_core = 1 -quiet -no_warning -in = in_file -outfile = out_fileClustal W2 Version 2.1 . For this scalability configuration, we used a test set of Pfam + Homstrad pairs, with at least 6000 unique sequences. 151 families satisfy this criterion. Each test case was resampled to produce smaller test cases of between 30 and 4000 sequences. At each test case size, the subsets were resampled 5 times. This produced 755 alignments to be tested and the results are shown in For the secondary structure prediction accuracy (SSPA) benchmark itself, we generated test cases of either 200 or 1000 sequences, sampled from the bigger datasets. This allowed us to compare two broad categories of alignment program: high accuracy and low capacity or lower accuracy and higher capacity. Both have 238 test cases, randomly sampled five times from the full Pfam families, to give a total of 1190 test alignments in each case. Each test alignment consists of 197 non-reference and 3 Homstrad reference sequences or 997 non-reference and three Homstrad reference sequences.http://www.bioinf.ucd.ie/download/QuanTestScl.tgz). We chose subsets of different numbers of sequences in the alignments according to a log scale: We first tested how the secondary structure prediction accuracy depends on the number of sequences in the alignments. We used our first scalability configuration of 151 Pfam families, each with at least 6000 unique sequences  for us to get stable and significant results.The prediction accuracy could depend on the selection of reference and/or non-reference sequences. A reference sequence in this context is a sequence for which the true secondary structure is known and is going to be predicted. For this analysis we always selected three Homstrad reference sequences per family. A non-reference sequence is either a Pfam sequence for which no reliable secondary structure is known or a Homstrad sequence whose secondary structure is not to be predicted. For 238 families, we selected 197 non-reference sequences, resampled five times from the larger datasets. The reference sequences were kept fixed for each re-sample of non-reference sequences. All alignments were therefore comprised of 200 sequences. This gave 5 × 238 = 1190 alignments and 3 × 238 = 714 reference sequences. For each of the 1190 sets of sequences, we generated a MSA using Clustal Omega in default mode.For each of the 1190 alignments we calculated the average prediction accuracy over the three reference sequences. To quantify the effect of reference sequence selection we determined, for each sequence, the absolute difference of its prediction accuracy from the average of three. The distribution of these differences is shown in Next we calculated for each of the 714 reference sequences the average prediction accuracy over the five re-samples. To quantify the effect of non-reference sequence selection we determined, for each sequence, the absolute difference of its prediction accuracy from the average over the re-samples. This distribution is depicted in P-value of 0.0005 according to the Mann-Whitney U test. For this reason we did not rely on just one reference sequence but take the average over three reference sequences for the subsequent analyses.One can see in We compared the prediction accuracy of the alignments with the traditional alignment quality score: the Sum-of-Pairs score (SPS score) (For both numbers of sequences, We also observe that increasing the number of sequences increases the SSPA. This is consistent with the results in We tested how adding errors to the alignments changes the prediction accuracy. We followed the same protocol as in (The results in This experiment used the accuracy configuration of 238 Pfam families, each with 5 samples of 200 sequences. We benchmarked the alignments of a wide range of aligners and options: Clustal W2 in default mode , and PASp < 0.01. T-Coffee performs on a par with Clustal Omega in default mode, while the difference in average SSPA scores between MUSCLE—2 iterations and MUSCLE—default mode with 16 iterations is not significant.Here we used the accuracy configuration of 238 Pfam families, with 5 samples of 1000 sequences for each Pfam family. The experiment considered only aligner options which can build an alignment from a bigger number of sequences in reasonable time: Clustal W2 in default mode, Clustal Omega in default mode, Clustal Omega with 1 iteration, Clustal Omega with 2 iterations, MUSCLE with 1 iteration, MUSCLE with 2 iterations, MAFFT fast mode, MAFFT in default mode, Kalign 2, HMMER in default mode, and PASTA in default mode.QuanTest is based on using SSPA as a proxy for multiple sequence alignment accuracy or quality. It behaves well as benchmark for several reasons. First it correlates well with the main existing measure of alignment quality: SPS score of a small set of reference sequences against a structure based alignment of the same sequences. Secondly, when we add alignment errors in a controlled fashion, increasing levels of error cause a monotonic and regular decrease in QuanTest score. Finally, when we compare different alignment options of various programs, we mainly see the ‘higher accuracy’ options, giving better scores. This latter point is potentially circular but we do see that the use of iterations or the use of consistency, for example, giving higher scores.The question then is, if QuanTest gives basically the same answer as SPS score, why use QuanTest? QuanTest is much more easily automated to give benchmark test sets of almost any size and for any subtype of proteins, as long as one or more sequences have known secondary structure. In all of the cases here, this is based on using 3D structures to infer secondary structure. This process is much easier to automate completely reliably, than multiple structure superposition  and therA further reason for using QuanTest is that it is based on all of the sequences in an alignment. SPS scores are based only on the subset of sequences of known structure. This subset can be tiny and unrepresentative so that testing very large alignments can be problematic (Compared to the ContTest score, the QuanTest SSPA score is less dependent on the lengths of the reference protein sequences, and it can be used for alignments comprised of fewer sequences than ContTest. In this study we demonstrate the usefulness of QuanTest for 200 sequences, whereas ContTest would typically require more than 1000 sequences to give an accurate scoring.Clearly, the method presented here will only work for protein sequences. A further restriction is that it will not work for sequences whose 3D structure is hard to determine. It will be heavily biased towards single domain globular proteins, but this bias is true of most structure based protein benchmarks. BALiBASE is a notable exception, in that it does include representative alignments from outside of this range but it is hard to scale it to many test cases or very big alignments. In the current test sets, we have been quite restrictive regarding the number and type of test examples. These numbers can easily be expanded to include literally thousands of test cases of almost any size. Other protein structure prediction measures —such as Supplementary DataClick here for additional data file.