Genome sequencing provides a powerful tool for pathogen detection and can help resolve outbreaks that pose public safety and health risks. Mapping of DNA reads to genomes plays a fundamental role in this approach, where accurate alignment and classification of sequencing data is crucial. Standard mapping methods crudely treat bases as independent from their neighbors. Accuracy might be improved by using higher order paired hidden Markov models (HMMs), which model neighbor effects, but introduce design and implementation issues that have typically made them impractical for read mapping applications. We present a variable-order paired HMM that we term VarHMM, which addresses central issues involved with higher order modeling for sequence alignment.Compared with existing alignment methods, VarHMM is able to model higher order distributions and quantify alignment probabilities with greater detail and accuracy. In a series of comparison tests, in which Ion Torrent sequenced DNA was mapped to similar bacterial strains, VarHMM consistently provided better strain discrimination than any of the other alignment methods that we compared with.Our results demonstrate the advantages of higher ordered probability distribution modeling and also suggest that further development of such models would benefit read mapping in a range of other applications as well.The online version of this article (doi:10.1186/s12859-017-1710-0) contains supplementary material, which is available to authorized users. Escherichia coli spread in Germany )Efforts have been made to develop FSAs that use higher order likelihoods, and Crooks et al.  for examAlthough extended FSAs have shown promising improvements, they have primarily been designed for multiple protein sequence alignment and do not model probability distributions directly. They are also limited to dipeptides or post-alignment processing, and use standard gap penalties for indels. Thus for nucleotide sequence alignment and classification, a different model may be more suitable.Paired hidden Markov models (HMMs) are very similar to FSAs but use probabilities instead of scores, and are thus well suited if we want to directly model the probability distributions for different alignment events. Higher order HMMs in particular, can provide a more detailed representation of alignment distributions than FSAs and 0th order HMMs, but their application in sequence alignment has so far been very limited. This is partly because paired HMMs can be slow, but also due to additional complexities that are introduced by higher order models.Attempts to move beyond a standard 0th order paired HMM for sequence alignment has hence been more limited than the case of FSAs, though Nánási et al.  recentlyIn addition to the Nánási et al.  models, Although it is common to use sequence alignment when performing pathogen classification with sequenced DNA, alternative approaches have been developed for classifying guanosine triphosphate proteins (GTPases) and protein folds, which use techniques such as neural networks , random We here present a variable order paired HMM for sequence alignment and read classification that we term VarHMM, which addresses higher order modeling issues and exploits similarities between reads to improve processing time. In contrast to previous methods, we directly model probability distributions for higher order alignment and indel likelihoods, and our method has been developed for nucleotide sequence alignment and classification. VarHMM uses the same states as a standard paired HMM, and is not mutually exclusive with HMMs such as the Nánási et al.  models, Our alignment method VarHMM, is based on a higher order paired HMM implementation. We implemented VarHMM as well as 0th, 1st, and 2nd order models for comparison . In the following, we describe how seeding, alignment, and training are performed, and how VarHMM and the higher order paired HMMs were implemented.In this section, we briefly describe how the probability of a single alignment is calculated, and in the next section we will explain how this can be used to find the ‘best’ alignment.All of the paired HMMs that we implemented use five states as shown in Fig. p0(S1:S2) is: If we use a 0th order paired HMM to align Sequence1 and Sequence2 in Fig. a values denote the state transion probabilities (e.g. aMX is the probability of transitioning from M to X), and for example p(A:A) is the probability that A matches with A and qx(T) is the probability of outputting a T nucleotide in Sequence1 .where the In the case of a 1st order model, the alignment probability of Sequence1 and Sequence2 is instead: p(T:T|A:A) for example is the probability that T matches T given the prior alignment between A and A, and qy(C|G) is the probability of outputting a C with a G occurring immediately before in Sequence2. When a gap occurs immediately before a match between two nucleotides (or a gap occurs immediately prior in the sequence where there is a nucleotide output), then a 0th order model is used to calculate the probability as can be seen in the case of the last match which uses p(T:T). As will be described in further detail in the next section, all of the paired HMM models use the highest order possible after previous gaps in the alignment have been taken into account, while VarHMM also adjusts the order based on which probabilities have sufficient training data to be considered reliable.where alignment matrix and then use them to find the alignment we think is best. For example, we can use the alignment matrices to store the probability that we align the first nucleotide in Sequence1 in Fig. In order to find what we think is the best alignment between two sequences, we compare the probabilities of different alignments. The general idea is to store the different alignment probabilities in an There are different ways to implement the alignment matrices and select a final alignment, but one approach is to use the Viterbi algorithm , which ui and j to refer to the nucleotide positions in Sequence1 and Sequence2 respectively.One difficulty with higher order paired HMMs is that they need to look back at the preceding nucleotides at each position in an alignment. HMMs are designed with the assumption that probabilities in a state are independent however, and so using higher ordered probabilities with a paired HMM and an algorithm such as Viterbi introduces difficulties that must be addressed. In particular, a higher order probability is dependent on the preceding alignment, but matches between the preceding nucleotides are not decided until we have calculated the alignment matrices as seen in Fig. In the higher order models, the Markov order that is used for the alignment probability is dependent on what occurred in the preceding states, which can be seen if we consider the three following examples:i,j in the alignment matrix . At each position i,j similar to how the Viterbi matrices are updated . If we for example enter a match state, then the number of times that we had a nucleotide output in both Sequence1 and Sequence2 increases by one, and we therefore increase the xc and yc values that are chosen from the Viterbi path when we use the equations in Fig. i=0 and j=0 all counts are set to zero.As shown in Fig. Xc and Yc counts are then used to help determine the order when we calculate the probability of an alignment i,j position for the alignment matrices as shown in Fig. The E. coli K12 substrain DH10B for training and testing. Two different sequencing sets from an Ion Torrent PGM 314v2 were used (datasets from https://figshare.com/s/6101b0605da5e9a6f72e): 1) C24-698, mode read length >200bp (average read length 244bp) and 35.7X average coverage, 2) B22-730, mode read length of 400bp (average read length 325bp) and 33.9X average coverage (494921 reads). We refer to the first dataset as R200 and the second dataset as R400. Half of the R200 dataset was used for training and the other half testing, which we term R200-train and R200-test respectively.We used sequencing data from E. coli DH10B with Last using its default parameters ,p(C|A),...etc.). If the counts for a given alignment or output is below a count of cmin, then a lower-order conditional probability is used (highest possible is then selected). We used a maximum order of 3 for both the M and the X and Y states, and a value of 100 for cmin.When training higher order HMMs, the training data set might not be sufficiently large to properly capture the higher order distributions in which case the model may perform worse than a lower order HMM. This was noticeable in the training set, where a 3rd order HMM appeared to yield worse results than a 2nd order model. To address this issue, we implemented VarHMM, which is a variable order paired HMM. VarHMM uses the same implementation as the higher order paired HMMs, but for each order (0th to 3rd order), it also counts the number of samples for all of the alignment and nucleotide output probabilities . We evaluated each method by aligning reads from E. coli K12 substrain DH10B, and focused on classification between substrains as this can be particularly difficult due to the high degree of similarity.To evaluate VarHMM and the different paired HMMs, we compared their performance with commonly used alignment software packages: Bowtie2 , Last 99, SegemeE. coli DH10B (NC_010473.1) and either 1) E. coli MG1655 (U00096.3), or 2) E. coli C3026 (NZ_CP014272.1). All of the alignment software parameters except for Last were set according to Caboche et al. [Reads from the R200 and R400 datasets were aligned to a sequence database comprising two genomes:  NZ_CP0142.1. All E. coli DH10B genome it was marked as a correct mapping. If a read was mapped to more than one position it was classified to a genome according to two criteria: 1) If all the aligned positions were in the same genome, then the read was classified to that genome, 2) If the first criterion were not true, then the highest scoring alignment was selected if it was greater than the second best alignment by a given threshold value .In each test, if a read was classified to the E. coli DH10B genome for a set of reads . We therefore derived two sets of results for TMAP, with one set based on both criteria that we label ‘TMAP’ in the plots, and an additional set where only the second criterion was used which we label ‘TMAP*’ .In the case of TMAP, the first criterion caused the results to decrease significantly in one of the test sets because it was unable to find any matches in the E. coli DH10B and MG1655. For both the R200 and R400 sets, VarHMM showed the best performance, followed by Last which at most points outperformed the 0th, 1st, and 2nd order models. The 1st order model sometimes performed better than the 2nd order model, and illustrates how the performance of a higher order paired HMM can decrease due to insufficient training data as described in the Methods section. Bowtie2 performed slightly better than TMAP, while TMAP scored higher than Segemehl for the R400 set in Fig. For each alignment method, we compared the percent of wrongly mapped reads with the percent that was correctly mapped. Figure DH10B which the other aligners likely matched to both DH10B and MG1655. Although this benefited TMAP in the first test, in the second test with E. coli DH10B and C3026, TMAP aligned a set of reads to only C3026 and not DH10B, resulting in a significant number of incorrectly mapped reads.When using both criteria 1 and 2 for classifying reads, TMAP was able to align a small number of extra reads compared with the other alignment methods when the percent of wrongly mapped reads was relatively high in Fig. E. coli DH10B and C3026. For the sake of visual clarity, the results for TMAP are not included but shown in Additional file Figure As VarHMM was trained using Last alignments, the HMM models may have achieved higher accuracy than the other aligners in part due to Last’s performance. To rule out this possibility, and also ensure that our estimated parameters . The results are shown in Additional file Unlike the other HMMs, VarHMM uses In addition to the alignment tests performed in Fig. E. coli DH10B and MG1655 and can be seen in Table A comparison between the run times of the different alignment methods was made using the R200 dataset with VarHMM achieved the best performance amongst all of the alignment methods that we compared with. Although we focused on Ion Torrent reads, accurately modeling nucleotide distributions is fundamental to alignment of any sequenced data. As such, a higher order based model such as VarHMM should also provide advantages with other types of sequencing.All of the paired HMMs were slower than the other alignment methods that we compared with, and future work will be focused on improving algorithm design and implementation to increase speeds. We also limited VarHMM to a maximum 3rd order model, but it would be possible to make a more dynamic design that uses higher orders for the output states to further improve performance.VarHMM is complementary rather than competitive to fast aligners such as Bowtie2. Its relative slowness makes it inappropriate for huge datasets, and its niche is smaller datasets where best-possible accuracy is desired. An obvious example is bacterial strain discrimination, where our results suggest that VarHMM can provide better sensitivity and accuracy for more difficult scenarios as described in the Results section. In addition to smaller datasets, VarHMM could also be applied alongside a fast alignment method for mapping shorter reads as they tend to be harder to align. Ultimately, standard zero-order methods will hit an accuracy ceiling that can be broken only by higher-order sequence modelling.Pathogen detection and biosurveillance is crucial for public safety, and accurate classification of reads is fundamental when using epidemiological methodologies. Our results show that VarHMM can achieve both high accuracy and sensitivity when classifying reads, providing a powerful tool that can be used for pathogen detection. These findings also have important implications going forward, where our results demonstrate the advantages of using higher ordered probability distribution modeling, and suggest that further development of such models would benefit read mapping in a range of other applications as well.Project name: VarHMMProject home page:https://github.com/gitvarhmm/varhmmOperating system(s): Windows, Mac, LinuxProgramming language: C++Other requirements: NoneLicence: GNU General Public License v3.0


Constructing alignments and phylogenies for a given locus from large genome sequencing studies with relevant outgroups allow novel evolutionary and anthropological insights. However, no user-friendly tool has been developed to integrate thousands of recently available and anthropologically relevant genome sequences to construct complete sequence alignments and phylogenies.VCFtoTree, a user friendly tool with a graphical user interface that directly accesses online databases to download, parse and analyze genome variation data for regions of interest. Our pipeline combines popular sequence datasets and tree building algorithms with custom data parsing to generate accurate alignments and phylogenies using all the individuals from the 1000 Genomes Project, Neanderthal and Denisovan genomes, as well as reference genomes of Chimpanzee and Rhesus Macaque. It can also be applied to other phased human genomes, as well as genomes from other species. The output of our pipeline includes an alignment in FASTA format and a tree file in newick format.Here, we provide VCFtoTree fulfills the increasing demand for constructing alignments and phylogenies for a given loci from thousands of available genomes. Our software provides a user friendly interface for a wider audience without prerequisite knowledge in programming. VCFtoTree can be accessed from https://github.com/duoduoo/VCFtoTree_3.0.0.The online version of this article (10.1186/s12859-017-1844-0) contains supplementary material, which is available to authorized users. The developments in next-generation sequencing technologies have now allowed us to study human genomic variation at the population scale. For example, 1000 Genomes Project alone sequenced more than 2500 individuals from diverse populations, uncovering more than 88 million variants including single nucleotide variants (SNVs), insertion-deletion variants (INDELs) (1–50 bp), and larger structural variants , rather VCFtoTree, a user friendly tool that extracts variants from 5008 haplotypes available from 1000 Genomes Project, ancient genomes from Altai Neanderthal [https://github.com/duoduoo/VCFtoTree_3.0.0. Those scripts can be easily modified to add other species or populations. The resulting alignments from our pipeline can also be integrated into other applications that require alignments, such as calculation of population genetics summary statistics or genome-wide applications, such as phylogenetic analyses of windows across the entire chromosomes.To address this need in the community, we present nderthal  and Deninderthal , and Rhenderthal  to this VCFtoTree are from 1000 Genomes Phase 3 final release. This dataset contains single nucleotide, INDEL, and structural variants (SVs) from 2504 individuals from 26 worldwide populations [http://cdna.eva.mpg.de/neandertal/altai/AltaiNeandertal/VCF/) [http://cdna.eva.mpg.de/neandertal/altai/Denisovan/) [The modern human variants used in ulations . Our proulations  to our pChimpanzee and Rhesus Macaque are often used as outgroups in human evolutionary genetics studies . Thus, oVCFtoTree to transform the variant calls into complete, aligned sequences as we describe below.Once our program fetches and sorts all the variant calls from designated sources as described above, our pipeline transforms these variant calls to complete sequences for alignment. There are computational tools to manipulate VCF files from 1000 Genomes Project , while the second haplotype carries a third allele (“T”). Based on this information, our script generates two sequences based on the reference genome to represent these individual haplotypes. For the first haplotype, the script leaves that position as it is (“a”), but for the second haplotype, the script replaces “a” with a “T” to represent the variation in this haplotype  vary and often not definitive. Moreover, the insertion sites of most duplications and mobile element insertions are not known. As such, our current pipeline is not equipped to reliably integrate structural variants in the alignment files. We ignore the structural variants for constructing our alignment. However, for the researchers to be able to assess the variation in the region fully, we provide the structural variants in the user-designated regions into the log file (log.txt). It is important to note that structural variants often resides in “complex”, repeat-rich regions of the genome where there are other alignment issues . It is aLocus with multiple variant types: There are some loci that can have an INDEL and a single nucleotide variant for different haplotypes. These sites are designated in the VCF file as “VT = SNP,INDEL”. In such cases, we convert the single nucleotide variant call to an INDEL format and treat this particular VCF line as a multiallelic INDEL as described above and as exemplified in Fig. b)Locus with multiple entries: There are some cases where 1000 Genomes VCF files report different alleles affecting the same locus in different lines, rather than designating them in a single line as multiallelic variants. In these cases, our pipeline combines these variants, creating a multiallelic variant line and then treats them as such Fig. .c)Complex regions with overlapping INDELs: Some highly repetitive sequences may vary in the number of repeats and they are designated as overlapping INDELs. We were able to integrate a subset of those where the multiple haplotypes are missing different sizes of sequences that are present in the reference genome (overlapping deletion INDELs). Briefly, our pipeline combines these events as multiallelic INDEL variants and then treats them as such. However, our pipeline cannot handle overlapping INDELs with sequences that are not present in the reference genome. If the region specified harbors such novel insertions overlapping with other INDELs, our program will not run and instead return an error message. The overall impact of this shortcoming is small given that there are only 892 distinct cases of such overlapping insertion INDELs reported in 1000 Genomes Project, excluding X/Y chromosomes Additional file In 1000 Genomes VCF files, there are several loci where variant calls are complex: i.e., they harbor more than one kind of variant type or overlapping INDELs, or multiple entries were made for the same locus. Here, we list the approaches that we took to integrate these complex variants in our pipeline:After generating the alignment for 5008 haplotypes from the 1000 Genomes Project, our pipeline can add variation data from Altai Neanderthal and Denisovan genomes, as well as Chimpanzee and Rhesus Macaque reference sequences to the alignment. Since, the archaic hominin variant calls were directly made from human reference genome, the alignment is automatic. That is we treat the VCF files from these ancient genomes similar to the 1000 Genomes VCF files, with the exception that we only consider homozygous variants as described above. For the nonhuman primates, we use existing pairwise alignment files for chimpanzee and rhesus macaque reference genomes to human reference genomes for a given region to construct the alignments. The challenge here is to incorporate all length changes that we introduced to the alignments while we integrate insertion INDELs. To do this, we use a custom python script in our pipeline to add these additional sequences to the archaic hominin and nonhuman primate genomes as gaps (“-”) before integrating these sequences to our alignment.Even though we primarily intend our application to be used for human genomes, we also implemented two options to broaden its scope. First, we allow researchers to load their own vcf file for phased genomes. Second, for nonhuman species, the researchers can also load any given reference genome to the program to work along with vcf files from that species. In the first options screen, it is possible to choose “other” and in the next screen locations of the reference input file (.fa.gz) and the desired vcf file (.vcf.gz) can be designated. Other than the input reference and variation files, all the algorithms, corrections and exceptions that we outlined above remain the same. There are three considerations that need to be noted here. First, most nonhuman reference genomes are not very high quality and may cause problems given that our analysis pipeline depends on the accuracy of the reference genome for constructing alignment output. We have not tested our approach extensively with nonhuman reference genomes. Second, we assume that the variation annotations in the custom vcf files will be identical to those used in 1000 Genomes Project. Third, it is important to remind that our approach only works with phased genomes. Even though there are not many phased nonhuman genomes currently available, we foresee that in the near future they will be. Our tool will be ideal to analyze such data.The initial alignment constructed is in FASTA format. Our pipeline also uses a python script to transform the format of FASTA to Phylip format. We provide both alignment formats as output files.The last step for our pipeline is to build the phylogeny is to run RAxML  or FastTVCFtoTree emerges from our own needs in our laboratory and we used previous versions of this pipeline in our recent publications [ications –21. We aications , ERAP2 [ications , NE1 [24ications   to output an alignment file for a 10,000 bp region for 2504 individuals running on a ISO system with 2.6 GHz Intel Core i5 processor and 8 GB 1600 MHz DDR3 memory. The second major bottleneck in timing is the tree building step. Especially RAxML requires rather long run times for large regions. The specifications and their run-time specifications can be found in Stamatakis 2006 [The run time of kis 2006 . FastTrehttp://popgen.uchicago.edu/ggv/), Galaxy [VCFtoTree complements such tools by providing a graphic user interface to investigate the haplotype structure of a locus at the population level while generating alignments for further analyses in software such as MEGA [Next-generation sequencing platforms increased the amount of genomic data tremendously. The critical bottleneck in anthropological genetics research has consequently shifted from production of data to analyses of data. For now, most of the available computational tools  a, Galaxy  and 1000, Galaxy , among o as MEGA  and DNAs as MEGA .VCFtoTree can also be useful in cross-species analysis. By using VCFtoTree, users can resolve the haplotype structure for the given region, finding the haplotype groups that compose the population. Then by choosing 1–2 representative haplotypes from each haplogroup, users can use commonly used multiple sequence alignment tools such as MEGA [MUC7 [FLG [LCE3BC [VCFtoTree as they become available. Overall, we believe that our pipeline will be useful for researchers in anthropological and evolutionary genomics, who are interested in locus-specific analyses.In addition to within species analysis,  as MEGA , Seaview as MEGA , to realGA [MUC7 , FLG [19 [LCE3BC . We contProject name: VCFtoTree.Project home page:https://github.com/duoduoo/VCFtoTree_3.0.0.Operating system: Mac OS El Capitan V10.11.5 or later.Programs required: samtools, tabix, wget.Programming languages: Python, Unix.License: Not applicable.Additional file 1: Table S1.Complex regions with multiple overlapped INDELs. (TXT 25 kb)Additional file 2: Figure S1.Phylogenies generated by VCFtoTree. a) EDAR ; b) NE1


BioMed Research International, 2015). The supervised models were built and tested using a Saccharomycete yeast benchmark dataset proposed by Salichos and Rokas (2011). Despite several pairwise protein features being combined in a supervised big data approach; they all, to some extent were alignment-based features and the proposed algorithms were evaluated on a unique test set. Here, we aim to evaluate the impact of alignment-free features on the performance of supervised models implemented in the Spark big data platform for pairwise ortholog detection in several related yeast proteomes.The development of new ortholog detection algorithms and the improvement of existing ones are of major importance in functional genomics. We have previously introduced a successful supervised pairwise ortholog classification approach implemented in a big data platform that considered several pairwise protein features and the low ortholog pair ratios found between two annotated proteomes  within the twilight zone could be achieved for a yeast proteome pair that underwent a whole genome duplication. The feature selection study showed that alignment-based features were top-ranked for the best classifiers while the runners-up were alignment-free features related to amino acid composition.The incorporation of alignment-free features in supervised big data models did not significantly improve ortholog detection in yeast proteomes regarding the classification qualities achieved with just alignment-based similarity measures. However, the similarity of their classification performance to that of traditional ortholog detection methods encourages the evaluation of other alignment-free protein pair descriptors in future research.The online version of this article (10.1186/s12859-018-2148-8) contains supplementary material, which is available to authorized users. Therefore such sequences are similar but not homologous [Homology between DNA or protein sequences is defined in terms of shared ancestry. Sequence regions that are homologous in species groups are referred to as conserved. Although useful as an aid in diagnosing homology, similarity is ill-suited as a defining criterion . High semologous . Though mologous . The menmologous , 5.In homology regions, two segments of DNA may share ancestry because of either a speciation event (orthologs) or a duplication event  . The disMany graph-oriented –12, treeSaccharomycete yeast dataset [Some benchmark papers , 26 eval dataset  built fr dataset . The YGO dataset  and the  dataset  as well  dataset  and Orth dataset  were eva dataset .On the other hand, the massive growth of genomic data has required big data frameworks for high-performance processing of huge and varied data volumes . ConsequSaccharomycete yeast proteomes reported by Salichos et al. (2011). In this paper, we intend to improve our previously reported big data supervised pairwise ortholog detection approach [Evaluating the influence of alignment-free pairwise similarity measures on the classification performance of several supervised classifiers that consider data imbalance under the Spark platform .Saccharomycete yeast proteomes that constitute benchmark datasets with “traps” for ortholog detection algorithms.Extending the test set to other related In 2015, our group proposed a novel pairwise ortholog detection approach based on pairwise alignment-based feature combinations in a big data supervised classification scenario that manage the low ratio of ortholog pairs to non-ortholog pairs (millions of instances) in two yeast proteomes . We builapproach  as follok-mers counts were considered as a first step in the ortholog and co-ortholog assignment pipeline proposed by [https://sourceforge.net/projects/ti2biop/. Generally, these protein features have been used to characterize functionally proteins at low sequence similarity using machine learning algorithms [Four color map descriptors: topological descriptors  derived from protein four-colour maps .Nandy’s descriptors: topological descriptors  derived from Cartesian protein maps (Nandy’s DNA representation extended to proteins) .k-mers or k-words: frequency of each subsequence or word of a fixed length k in a set of sequences [equences .k-mers or spaced words: contiguous k-mers with “don’t care characters” at fixed or pre-defined positions in a set of sequences [Spaced equences .Amino acid composition: the fraction of each amino acid within a protein , 49.i, Rj placed at the topological distance λ from each other within the protein sequence. Further information can be found at http://www.csbio.sjtu.edu.cn/bioinf/PseAAC/type1.htmChou’s pseudo amino acid composition descriptor: It is an improvement of the amino acid composition descriptor by adding information about the sequence order . The seq: square autocorrelation of amino acid properties along the sequence [Geary’s auto correlationsequence .Moran’s auto correlation: autocorrelation of amino acid properties along the sequence .Total auto correlation: autocorrelation descriptors  based on given amino acid properties are normalized all together .Composition (C), Transition (T) and Distribution (D) (CTD) descriptors: information from the division of amino acid into three classes according to the value of its attributes e.g. hydrophobicity, normalized van der Waals volume, polarity, etc. So, each amino acid is classified by each one of the indices into class 1, 2 and 3. C descriptors: the global percent for each encoded class  in the sequence, T descriptors: the percentage frequency to which class 1 is followed by class 2 or 2 is followed by 1 in the encoded sequence. D descriptors: the distribution of each attribute in the encoded sequence , 55.Quasi-Sequence-Order (QSO) descriptors: combination of sequence composition and correlation of amino acid properties defined by Chou KC (2000) .Alignment-free similarity measures have shown several advantages over the alignment-based ones: (i) not sensitive to genome rearrangements, (ii) detection of functional signals at low sequence similarity and (iii) often less computationally complex and time consuming , 38. In posed by  while foposed by  availablgorithms , 43.FourIn order to evaluate the influence of the alignment-free features on ortholog detection, we build three kinds of supervised pairwise ortholog detection models (i) one based on previously reported alignment-based pairwise protein features  (ii) a new one incorporating only the alignment-free features listed above and (iii) another one resulting from the combination of alignment-based and alignment-free protein features. For model building we are using different machine learning algorithms  implemented in the Spark big data architecture as well the gold-groups reported by Salichos and Rokas in 2011. Each supervised approach was evaluated in several benchmark yeast proteome pairs containing “traps” for ortholog detection . The evaSpark classifiers are introduced here since they manage complete datasets instead of the ensemble of classifiers built with the corresponding data in partitions as in Hadoop MapReduce implementations. The Spark random oversampling may also speed up the pre-processing while the resampling size parameter value over 100% may improve the classification of the minority class in extremely high imbalanced datasets  like paiAs a result of the experiments in this study, the advantages of the Spark big data architecture over MapReduce implementations in terms of classification performance and execution time for supervised pairwise ortholog detection have been confirmed, conversely, the introduction of alignment-free features into several supervised classifiers that use alignment-based similarity measures did not significantly improve the pairwise ortholog detection. In fact, the feature selection study showed that alignment-based similarity measures are more relevant for the supervised ortholog detection than alignment-free features. However, many of the supervised big data classifiers built with both alignment-based and alignment-free features surpassed the traditional methods like RBH, RSD and OMA in three pairs of yeast proteomes. Precisely, some of these tree-based supervised classifiers could detect more ortholog pairs at the twilight zone (< 30% of protein identity) in two whole-duplicated genomes. These findings encourage us to keep on working on improving our alignment-free protein features in order to fill the gap of the alignment algorithms when genetic events blur the ortholog detection.P1 = {x1, x2, …, xn} and P2 = {y1, y2, …, ym} in [S1: Similarity based on global alignment scores.S2: Similarity based on local alignment scores.S3: Similarity based on the physicochemical profile from matching regions (with no gaps) of aligned sequences at different window sizes .and S4: Similarity based on the pairwise differences of protein lengths. Despite S4 being included with the previous (S1…S3), it is not an alignment-dependent measure. All these similarity measures were normalized by the maximum value.We have previously defined the following alignment-based similarity measures for protein pairs found in two yeast proteomes , ym} in :S1: SimiAAX and AAY represent the numerical vectors of proteins xi and yj, respectively.Protein sequences from yeast proteomes are turned into numerical vectors using the alignment-free methods listed in the background section. The Pearson correlation coefficient was selected as an alignment-free similarity measure between two numerical vectors. The selection is based on the valuable information obtained with the significance value of the Pearson coefficient . Each alS5-S26) are listed below. Each pairwise similarity measure is labelled by its corresponding alignment-free method and the main parameters used.S5: Similarity based on amino acid composition.S6: Similarity based on pseudo-amino acid composition with λ = 4. The parameter λ is the topological distance between two amino acids in the sequence pseudo-amino acid composition concept where the sequence order effect is integrated to the amino acid composition, λ < protein length.S7: Similarity based on pseudo amino acid composition with λ = 3.S8: Similarity based on pseudo amino acid composition with λ = 10.S9: Similarity based on k-mers composition with k = 3 where k represents the size of contiguous words (matching positions).S10: Similarity based on k-mers composition with k = 2.S11: Similarity based on Geary’s auto correlation.S12: Similarity based on Moran’s auto correlation.S13: Similarity based on Total auto correlation.S14: Similarity based on Composition, Distribution and Transition (Composition).S15: Similarity based on Composition, Distribution and Transition (Distributions).S16: Similarity based on Composition, Distribution and Transition (Transition).S17: Similarity based on Composition, Distribution and Transition .S18: Similarity based on four-color maps.S19: Similarity based on spaced k-mers/spaced words composition with k = 2 (matching positions (1)) and one “don’t care positions” (0); patterns: “101”.S20: Similarity based on k-mers/spaced words composition with k = 2 and two “don’t care positions”; patterns: “1001”.S21: Similarity based on spaced k-mers/spaced words composition with k = 2 and three “don’t care positions”; patterns: “10,001”.S22: Similarity based on spaced k-mers/spaced words composition with k = 3 and one “don’t care positions”; patterns: “1101”, “1011”.S23: Similarity based on spaced k-mers/spaced words composition with k = 3 and two “don’t care positions”; patterns: patterns: “10,011”, “10,101”, “11,001”.S24: Similarity based on spaced k-mers/spaced words composition with k = 3 and three “don’t care positions”; patterns: “100,011”, “110,001”, “101,001”, “100,101”.S25: Similarity based on Nandy’s descriptor.S26: Similarity based on Quasi-Sequence-Order with maxlag = 30.The alignment-free pairwise similarity measures evaluated in this study (As the same measure or function (Pearson correlation) is used to quantify the previously-mentioned alignment-free pairwise similarities; thus, we are definitely evaluating the corresponding alignment-free protein features giving rise to them.The general classification scheme for pairwise ortholog detection using supervised big data algorithms managing the ortholog/non-ortholog imbalance found in yeast proteome pairs is represented in Fig. https://spark-packages.org/package/saradelrio/Imb-sampling-ROS_and_RUS [The training step involves the ortholog/non-ortholog imbalance management, and the testing step includes the selection of the adequate quality measures for imbalance datasets. The main pre-processing algorithms proposed to cope with data imbalance are labelled as ROS (Random Oversampling) and RUS (Random Undersampling). The Spark implementation of these algorithms are available at a spark-packages site _and_RUS . The new_and_RUS .The performance of the big data supervised models shown in Table In order to evaluate the performance of pairwise ortholog detection algorithms we use the gold-groups  retrieved by Salichos and Rokas  from theData splitting into two training and testing sets. The training process is carried out by using curated ortholog pairs (positive set) found either in pre-WGD species or in track A/B of post-WGD species. Similarly, a curated negative set is made up of all possible non-ortholog pairs found between two yeast proteomes deprived of paralogs (gold-groups).The testing step is carried out on entire proteome pairs excluding the pairs used in learning steps. Test sets are made up of all possible annotated protein pairs  found between pre-pre WGD or pre-post WGD or post-post WGD yeast species pairs. Three of the traditional unsupervised algorithms  for pairwise ortholog detection were also comparatively evaluated on the test sets.TP) and “true negatives” (TN), respectively. Paralogs are considered as “traps” for ortholog detection algorithms because they can be easily misclassified as “orthologs”. The selected evaluation metrics AUC, G-Mean, TPRate (TPR) and TNRate (TNR) are suitable for imbalanced datasets [The performance evaluation of both methods (supervised vs. unsupervised ortholog detection) is based on previously curated classifications; so, curated orthologs and non-orthologs are considered as “true positives”  are selected in order to analyze the quality of our approach. Table S. cerevisiae - K. lactis, S. cerevisiae - C. glabrata, C. glabrata - K. lactis, and K. lactis - K. waltii). We include the total number of pairwise features, the total of protein pairs per class and the imbalance ratio (IR).Annotated proteome pairs from related yeast species of the Protein sequences of the previously listed proteomes can be found in Additional file S1..3 with S3 calculated by using windows sizes 3, 5 and 7. In the alignment-free case we use S4..27 and then, in the alignment-based + alignment-free case we use all the similarity measures. The different models to be compared are built with ScerKlac and tested in ScerCgla, CglaKlac and KlacKwal datasets. The algorithms in Table Three study cases were designed to inspect the influence of the alignment-free features on the supervised classification for ortholog detection. Thus, big data supervised classifiers are compared considering three study cases: alignment-based features, alignment-free features and alignment-based + alignment-free features. Specifically, in the alignment-based case we use similarity measures G-Mean and AUC for Decision Trees, Random Forest, Logistic Regression, Naive Bayes and Support Vector Machines for the study cases with alignment-based, alignment-free and alignment-based + alignment-free features are shown in Table AUC and G-Mean values (0.9977) correspond to the ROS (130% resampling) and RUS pre-processed Spark Random Forests in the ScerCgla and KlacKwal datasets with the alignment-based features as well as to the ROS (100% resampling) Spark Decision Trees in the ScerCgla dataset with the alignment-based + alignment-free feature combinations. These G-Mean results outperformed the best value of 0.9941 reported in our previous paper [ScerCgla with a version of Hadoop MapReduce Random Forest. The best values (AUC = 0.9486) of the unsupervised classifiers correspond to RSD 0.8 1e-05 .The classification quality measures us paper  for Scerended in ). This tTable ScerCgla protein pairs, 294 out of 27,775,380 CglaKlac pairs and 356 out of 27,770,047 KlacKwal pairs. The highest true positive percentage (99.16%) corresponded to the RUS pre-processed Spark Decision Trees in the KlacKwal dataset with alignment-based features. On the other datasets, the best true positive percentages were also obtained with the alignment-based features; 99.04% and 96.94% that corresponded to the RUS pre-processed Spark Random Forest in ScerCgla, and to the ROS (130% resampling) Spark Random Forest in CglaKlac, respectively. In total, alignment-based features by themselves and alignment-based + alignment-free feature combinations surpassed the alignment-free and the classical unsupervised approaches. Generally, the alignment-free feature-based classifiers with imbalance management outperformed the unsupervised classifiers in each dataset, with the exceptions of the best RSD classifiers (RSD 0.8 1e-05) and (RSD 0.5 1e-10) in CglaKlac and KlacKwal, and the RBH classifier in KlacKwal. The Spark Decision Trees improved their performance with the combination of alignment-based and alignment-free features in ScerCgla, two yeast species that underwent a single round of whole genome duplications with subsequent gene losses. Specifically, the ROS (130% resampling) Decision Trees equalled the second best result (98.71%) of the ROS (130% resampling) Spark Random Forest in such a complex dataset.The ortholog pairs placed in the twilight zone are: 311 out of 30,558,738 Run time is presented in Table ScerCgla dataset is summarized in Table The feature importance study carried out in the sw and nw) besides the physicochemical profile with window size 3 (profile3). On the other hand, among the alignment-free features, the amino acid and pseudo amino acid composition of λ = 3 and 4, the compositional descriptor (CTD_C) along with the length of the sequences turned out to be the most important. When analyzing the alignment-based + alignment-free case, the relevant features are sw, nw, profile3, profile5, profile7, amino acid composition (acc) and CTD_C.In the alignment-based case, the most important features are those derived from local and global alignments (S. cerevisiae - C. glabrata), even in the twilight zone. The alignment-based features and the alignment-based + alignment-free combinations along with the Spark imbalanced classification of Random Forest and Decision Trees achieved better effectiveness, as well as faster ortholog pair detection times even in such a complex dataset. The inclusion of different pairwise similarity measures in the decision system may prevent the algorithm from missing gene/protein pair relationships during the classification process.In the Algorithm Performance experiment, the classification results achieved by our supervised pairwise ortholog detection approach changed slightly when using alignment-based, alignment-free and alignment-based + alignment-free feature combinations. The excellent performance of the alignment-based combinations could be caused by the appropriate selection of the substitution matrixes and gap penalties in relation to the sequence diversity of yeast proteomes . In contScerCgla dataset; which contains “traps” for ortholog prediction algorithms. Such “traps” consist of the paralogs originated from the whole genome duplication event that the genomes S. cerevisiae and C. glabrata underwent, subsequently the loss of many of these paralogs provides confusion to the ortholog prediction algorithms [This study corroborates the results of our previous paper  in the sgorithms .When dealing with Spark Random Forest classifiers, small differences were shown when applying different Impurity metrics, namely entropy and Gini. Although entropy led to better results in the alignment-free and the alignment-based + alignment-free cases, Gini could be preferable as due to its efficiency, i.e. there is no need to compute the logarithmic expression. In this sense, future studies should be oriented towards the tuning of other parameters’ values, namely pre-processing policies, number of trees, or maximum depth, as these may allow significant differences to be obtained in terms of both efficiency and accuracy.Regarding the comparison between Spark and Hadoop Random Forest implementations, the former showed a significant reduction in execution time while increasing the quality of classification. Efficiency is due to the capabilities of Spark when embedding data into memory to avoid disc overhead, whereas classification performance is achieved by a smart design of the learning procedure where the model is built iteratively using all available data .. However, when alignment-based features were combined with alignment-free features selected in the study the results slightly improved in some datasets, similarly in the twilight zone. This is a new motivation for further research aimed at analysing the inclusion of new alignment-free features or the tuning of parameter values.From the feature importance evaluation in supervised classifiers we can conclude that alignment-based features should continue to be of much importance for ortholog detection, mainly when the local alignment is combined with the global alignment and the physicochemical profile features are derived from matching regions of pairwise aligned proteins. The exclusion of synteny ) among the alignment-based similarity measures did not affect the classification performance in relation to our previous report . Howeverk-letter words [k-mers between two sequences, the higher may be their functional similarity. In a similar way, the compositional alignment-free features such as amino acid composition, pseudo-amino acid composition and composition, transition and distribution also seem to be highly related with the ortholog concept since they have been useful for functional similarity detection.The results obtained so far emphasize the importance of the local sequence similarity to detect protein functional similarity so that most of the classical ortholog detection methods start with BLAST alignments intrinsically based on the identification of query substrings or er words , 65, 66.er words  works onScerCgla dataset, which contains “traps” for ortholog detection. Conversely, the lesser classification quality values of RBH, RSD and OMA, mostly of RBH, can be explained by their only sequence similarity approach [CglaKlac and KlacKwal) where it could detect a higher number of orthologs than our supervised proposals with just alignment-free features. This achievement might be the result of the RSD intrinsic combination of similarity and evolution distance information [The success of the supervised algorithms might have been obtained from the combination of several alignment-based pairwise features like global and local alignment scores and the physicochemical profiles at different window sizes, as well as from the recent incorporation of alignment-free measures together with the training from curated datasets. By combining global and local alignment similarities, we have considered structural and functional protein similarities, respectively. These protein features have been complemented with physicochemical and alignment-free information in order to cope with homology detection pitfalls caused by significant matches of short sequences, remote homology, convergent evolution and other evolutionary and genetic events. Precisely, the alignment-based + alignment-free feature combination together with all the Spark and the pre-processing benefits allowed Decision Trees to detect remote orthologs at higher success rate in the complex approach , althougapproach . Howeverormation .ScerCgla) which underwent a whole genome duplication and subsequent differential losses of gene duplicates.It is worthy to mention that pure alignment-free supervised classifiers showed similar performance than the RSD algorithm for ortholog detection; and when alignment-free features are incorporated into Spark Decision Trees with imbalance management, a higher success rate (98.71%) was achieved within the twilight zone of the more complex yeast proteome pair  managing the imbalanced rate between orthologs and non-orthologs have surpassed the Hadoop MapReduce Random Forest classifiers with the best results in our previous work, considering both the quality of classification and the execution time. Although the introduction of alignment-free pairwise features into tree-based supervised models did not significantly improve the classification rates achieved with several alignment-based similarity measures, it was shown that some compositional alignment-free features might have positively contributed to ortholog detection, especially to identify orthologs at the twilight zone. The introduction of the alignment-free features in ortholog detection is an open field that we will keep exploring in future research.The extension of previous experiments to different yeast species from the Additional file 1:S. cerevisiae, C. glabrata, K. waltii and K lactis. (ZIP 5844 kb)Proteome fasta files for the following yeast species:


Read alignment is the first step in most sequencing data analyses. Because a read’s point of origin can be ambiguous, aligners report a mapping quality, which is the probability that the reported alignment is incorrect. Despite its importance, there is no established and general method for calculating mapping quality. I describe a framework for predicting mapping qualities that works by simulating a set of tandem reads. These are like the input reads in important ways, but the true point of origin is known. I implement this method in an accurate and low-overhead tool called Qtip, which is compatible with popular aligners.The online version of this article (doi:10.1186/s13059-017-1290-3) contains supplementary material, which is available to authorized users. Read alignment is often the first task when analyzing sequencing data. This is the process of determining each read’s point of origin with respect to a reference genome. Much prior work is concerned with making read aligners computationally efficient . That satandem simulation framework for predicting mapping qualities for all the alignments in a dataset in a manner that is agnostic to the aligner and parameters used. We also introduce Qtip, a tool implementing the framework. Qtip operates alongside and in cooperation with an aligner like Bowtie 2 [We introduce the Bowtie 2 ; the terBowtie 2 , BWA-MEMBowtie 2 , and SNABowtie 2  alignersWe demonstrate that Qtip’s predictions are superior to those made by the read aligners themselves, both on average and for most specific MAPQ thresholds tested. We use simulation experiments to show this for various read aligners , alignment settings , and accuracy criteria. We also perform a variant-calling experiment to show the improved mapping qualities can benefit downstream analysis. To our knowledge, this is the first description of a general technique for characterizing alignment uncertainty that is applicable across software tools and alignment settings.incorrect, and we call this an alignment error. Incorrect alignments lead to interpretation problems later [Given a sequencing read and reference genome, a read aligner like Bowtie 2 , BWA-MEMms later , 8.Aligners use heuristics – computational shortcuts – to limit the effort expended. Heuristics affect which alignments can and cannot be found, shaping what errors the aligner might make. Additional file The read is reported to have originated from a locus in the reference genome, but actually originates from a sequence not included in the reference.No alignment to the reference is found, but the read actually originates from some locus in the reference.Lr in the reference is reported, but the read actually originates from a different locus in the reference, Lt.An alignment to locus We can divide alignment errors into three categories, as suggested in the MAQ study : The reLt falls below the minimum similarity threshold (Smin), or when the alignment at Lt is missed due to alignment heuristics. Category 3 errors are caused by a combination of repetitive DNA, sequencing errors, genetic differences, and alignment heuristics. Category 3 errors and the related idea of multi mappers, reads that align equally well to many loci, are discussed in prior studies [Category 1 errors might be caused by contaminating DNA, or by an inappropriate or incomplete reference genome sequence. Category 2 errors can occur when the alignment at  studies , 9. CateHere we focus on the task of predicting mapping qualities for aligned reads in light of category 3 errors. Category 2 errors are not considered, since no mapping-quality prediction is needed in those cases. Although category 1 errors affect mapping-quality prediction, we assume they are rare enough to be ignored. In principle, category 1 errors could be included in our model, e.g. by assuming a global prevalence of category 1 errors and scaling predictions accordingly, or by including contamination in the simulation.p that the reported alignment is correct. Let the mapping qualityp (or q) indicate higher confidence. The SAM/BAM format [q, rounded to the nearest integer, be reported in the MAPQ field of each alignment. We, therefore, seek a method that predicts q  accurately across a range of alignment scenarios.While searching for alignments, aligners uncover information that can be used to predict whether a given alignment is correct. For instance, if the aligner discovers that a read aligns equally well to several copies of a repeat, its confidence that the selected alignment is correct will be low. If the aligner discovers that a read aligns perfectly to one locus and very poorly to a few others, confidence will be higher. Confidence is measured as the probability M format  requiresq≤3). Other measures that do not take genomic repeats into account, such as E values [Mapping quality measures something distinct from alignment score. A high alignment score indicates high sequence similarity (few mismatches and gaps) between read and reference. It does not imply high mapping quality. For instance, consider a read that aligns with no gaps or mismatches to two distinct loci in the reference. The alignment score is high because there are no gaps or mismatches, but there is only a 50% chance of choosing the correct alignment =1 if ai is correct and 0 otherwise. Let incorrect(ai)=1−correct(ai). An alignment is considered correct if the leftmost base involved in the alignment is within 30 nucleotides (nt) of the leftmost base in the simulated substring, with appropriate adjustments for soft clipping. Let Q=q0,q1,…,qn−1 be mapping qualities corresponding to a0,a1,…,an−1, as predicted by the read aligner, and let P=p0,p1,…,pn−1 be the corresponding correctness probabilities, using the relationship that Q′ and P′ are defined similarly, but for the mapping qualities predicted by Qtip.Let Q′) compare to the aligner’s (Q). CID and RCA capture how well Q′ranks alignments from most to least likely to be correct relative to Q. CID and RCA are invariant under monotonic transformations of P and P′; they are concerned only with how well alignments are ranked, not with probabilities per se. CSED and RCE capture how closely P′ matches the the true correctness relative to P; i.e., CSED and RCE are concerned with how well P′ and P fit their probabilistic interpretation.We define plots (cumulative incorrect difference or CID and cumulative squared-error difference or CSED) and measures  that characterize how Qtip’s predictions  for i=0 to n−1 , and we judge Qtip’s efficacy according to the line’s tendency to stay below y=0.Let A and P sorted in descending order by P, and likewise for cumulative squared error vectorE is the vector e0,e1,…,en−1 such that E′ defined similarly for 2 Let S be the element-wise difference E′−E. When si<0, Qtip’s mapping qualities yield a lower squared error up to the ith alignment.Let si’s  for i=0 to n−1 . Like for the CID plot, we judge Qtip’s efficacy according to the line’s tendency to stay below y=0.The CSED plot draws a line representing the RCA is defined as P′)−SSE(P))/SSE(P), where RCE is defined as (SSE(The distinction between the rank-based (CID and RCA) and probabilistic (CSED and RCE) metrics relates to how downstream tools, e.g. variant callers, use mapping qualities. Freebayes  and the We note that the problem of evaluating and plotting the relative quality of two sets of mapping-quality predictions is not specifically addressed in past studies. Receiver operating curve (ROC)-like plots are used for the related task of comparing aligners , 5, wherWe conducted simulation experiments to show how Qtip’s mapping-quality predictions compare to those made by the read aligners. We vary several experimental conditions, including (a) read length, (b) aligner parameterization, (c) reference genome, (d) read alignment tool, and (e) read simulator. The simulator encodes the read’s true point of origin in the read name, allowing Qtip to check later whether an alignment is correct.L and 4L nt, where L is the read length. We simulated 4 million reads/pairs for each sample. Aligners were configured to consider fragment lengths in the 2L– 4L range as concordant. Thus, most simulated pairs aligned concordantly (consistent with paired-end constraints) whereas some aligned discordantly. Simulator commands, and implications for fragment lengths, are discussed in Additional file We used Mason to simulate five Illumina-like samples with unpaired reads of length 50, 100, 150, 250, and 500, respectively. We simulated five paired-end samples with the same lengths, with most fragment lengths being between 2y values were scaled with We used Qtip together with Bowtie 2 to align and predict mapping qualities for each Mason-simulated sample. We rounded Qtip’s predictions to the nearest integer per the SAM/BAM format. For each alignment, we parsed aligner-predicted and Qtip-predicted mapping qualities, as well as the read’s true point of origin as provided by Mason. We calculated RCA and RCE Table  and plotTo measure the variability of Qtip’s predictions, we repeated each experiment ten times starting from step 2 onward, seeding the pseudo-random number generator differently in each trial. RCA and RCE tables describe all ten trials whereas, for clarity, the CSED plot describes only the first trial.Qtip’s mapping qualities are, overall, superior to those predicted by Bowtie 2, as indicated by the negative RCAs and RCEs Table . This isy=0, indicating the aligner-reported mapping qualities exhibit a lower cumulative squared error at those thresholds. This is most prominent in the unpaired experiments, particularly for 50 nt reads. However, Qtip’s superior predictions at other q thresholds – especially low ones – help bring the overall RCE below zero in all cases. For paired-end samples, CSEDs show Qtip’s predictions are superior at nearly all q thresholds.There are portions of the CSED plots Fig.  where thZea mays AGPv4. The human GRCh38 primary assembly is 3.10 Gbp long (2.95 Gbp excluding N’s) with 50% of the genome annotated as repetitive according to RepeatMasker [To study how genomes of varying length and repetitiveness influence Qtip’s performance, we experimented with four reference genome assemblies spanning three species: human GRCh37, human GRCh38, mouse GRCm38, and atMasker . GRCh37 atMasker , making We tested three aligners – Bowtie 2, BWA-MEM, and SNAP – with each genome. The changes made to each aligner for it to work with Qtip are detailed in Additional file Qtip-predicted mapping qualities are superior in nearly all scenarios, as indicated by negative RCAs and RCEs Table . The excy=0. Qtip’s mapping qualities seem to perform worse for many thresholds in the BWA-MEM unpaired experiments, especially for Zea mays. However, Qtip’s qualities consistently perform better at very low thresholds. Qtip’s mapping qualities perform particularly well for the Bowtie 2 Zea mays experiments, and for all the paired-end experiments.CSED curves Fig.  again shTo assess how the greater incidence of category 1 errors affects the results, we repeated the human experiments, expanding the simulation to include reads both from the reference genome and from sequences in the CHM1 hydatidiform mole assembly not present in the reference. We used Assemblytics  to obtaiWe also conducted simulation experiments varying the sensitivity level of the aligner  is the typical F1 score, related to the harmonic mean of precision and recall. Setting β>1 gives recall more weight than precision and setting β<1 gives precision more weight than recall. We tried values of β ranging from 0.25 to 4 to cover a range of precision–recall tradeoffs. Further details on alignment and variant calling are given in Additional file We used Freebayes v1.1.0  to call Q) to eliminate some alignments prior to variant calling, eliminating alignments with Q<1 by default. Since we are concerned with the overall accuracy of mapping qualities and not with any particular threshold, we reran Freebayes with various integer Q thresholds: 0–12, 15, 20, and 30. Freebayes also associates a genotype quality value with each called variant, given in the VCF file’s QUAL field. We used the vcfroc tool from vcflib (https://github.com/vcflib/vcflib) to evaluate all possible QUAL thresholds for all possible Q thresholds, ultimately selecting Q and QUAL thresholds maximizing Fβ.Like other variant callers and downstream tools, Freebayes uses thresholds for mapping quality (β’s examined except the lowest (β=0.25), Qtip-predicted mapping qualities yielded superior Fβ. For β≥1, Qtip’s predictions yielded more true positives and fewer false positives than the original predictions. For 0.25≤β≤0.5, Qtip’s predictions yielded around 15,000–20,000 more true positives at the cost of around 300–1,500 more false positives. Notably, these improvements were achieved simply by changing the mapping qualities; the alignments are the same and the variant caller has not been modified or tuned in any way. We also note that Qtip’s improved performance is obtained using a smaller range of mapping-quality values. Qtip-predicted mapping qualities in this experiment ranged from 0 to 36, whereas Bowtie 2 mapping qualities ranged from 0 to 42.The results are presented in Table The tandem simulation framework adds an overhead to the alignment process. We measured Qtip’s overhead when analyzing public datasets ERR050082 and ERR050083. Specifically, we measured how the running time and the peak memory footprint grew when Qtip ran alongside the aligner, versus when the aligner ran by itself. The running-time overhead is modest for Bowtie 2 and BWA-MEM, ranging from 5 to 10% Table . For SNAR=r0,r1,…,rn−1), a read aligner, alignment parameters, a reference genome in FASTA format, and any other files required, such as a genome index. The tandem simulation framework , paired-end and concordantly aligned (conc), paired-end and discordantly aligned (disc), or paired-end with the opposite end having failed to align (bad-end). Qtip trains each model with alignments of the same category. Qtip parameters control the minimum number of tandem reads or pairs of each category to generate. The default number for each category is x is the number of input alignments of that category. Both the scaling factor and the function are configurable via Qtip’s –sim-function and –sim-factor parameters. Qtip enforces a minimum of 30,000 tandem reads for the conc and unp categories and 10,000 for the disc and bad-end categories. The formula for the number of training records is discussed further in Additional file When predicting mapping quality, Qtip uses a different model depending on whether the alignment is unpaired  drawing an appropriate-length substring from the reference genome uniformly at random, (b) possibly reverse-complementing it, according to the template strand, (c) mutating the extracted sequence according to the template pattern of mismatches and gaps, and (d) setting the new read’s quality string equal to the template’s. The simulated read’s point of origin is encoded in the read name, allowing later steps to check whether an alignment is correct. The process for simulating a paired tandem read is similar, with fragment length determined by the template. More details are given in Additional file Importantly, some aspects of the input data are hard to mimic. For example, errors made by 454 and Ion Torrent sequencing technologies can manifest as spurious extensions or retractions of homopolymers. Since genome substrings are matched with templates randomly, homopolymer errors in the template will often fail to line up with homopolymers in the substring. Other aspects of the input data are not as difficult to mimic, but happen not to be captured in Qtip’s simulation. For example, if a dataset is enriched or depleted for reads drawn from a particular genomic feature , Qtip’s simulation, which draws reads uniformly at random from across the genome, will not exhibit that pattern. While we demonstrate Qtip performs well despite these deficiencies, they nonetheless illustrate that it is difficult to construct tandem reads that truly mimic input reads in all ways. We return to this in the “q’s for the input alignments. Qtip trains separate models for each alignment category: unp, conc, disc, and bad-end. The particular features used to train a model vary depending on the alignment category and read aligner. We briefly summarize these here, but more details are provided in Additional file Given training records derived from tandem reads aligned in step 4, we train a model in steps 5 and 6 that is later used to predict These features are included regardless of aligner or alignment category: (a) the alignment score of the best alignment, (b) the difference between the alignment score of the best alignment and that of the second-best alignment if one was found, (c) the length of the aligned read, (d) the sum of the base qualities of the aligned bases, and (e) the sum of the base qualities of the soft-clipped bases. For a concordantly aligned pair, the inferred fragment length (from the SAM TLEN field) is also included as a feature.By default, Qtip uses an implementation of random forests  from theQtip’s predictions are accurate in various scenarios: various read lengths, unpaired or paired reads, various alignment tools and parameters, etc. We defined novel measures (RCA and RCE) and plots (CID and CSED) for evaluating and plotting mapping-quality predictions. The framework is easy to adapt to other aligners; the aligner must be modified to output feature data in an extra SAM field. Nor is it difficult to add new features to an already-adapted read aligner. Since Qtip’s ensemble tree model is scale-agnostic, scaling guesswork it not necessary when adding a feature.This framework is also applicable to specialized alignment settings, such as spliced RNA-seq alignment. In that case, a nuanced notion of correctness is needed; we care not only where an alignment lands on the reference but also whether it includes the correct splice junctions. There is room for improvement in predicting mapping qualities for spliced alignments. Popular tools use simplistic prediction functions drawing quality values from a small range of possibilities. TopHat  and STARTandem simulation works to the degree that tandem reads can be sampled from the same distribution as input reads. In reality, sampling from the same distribution is not possible. Qtip mimics some aspects of the input data but not others. Homopolymer extensions and retractions are not captured, for example, creating a fundamental difference between tandem and input reads. A tradeoff exists here: Qtip’s simple model mimics some aspects of the input without sacrificing efficiency, whereas a more complex and less efficient model could improve accuracy by mimicking more aspects. A task for future work is to measure various points in this tradeoff space, and to define measures for characterizing how and to what extent a set of tandem reads differs from the input reads.A question for future work is whether Qtip’s sampling strategy can be improved. A strategy using importance sampling, for example, might favor tandem reads originating from more difficult-to-predict portions of the sample space. Importance might originate from repetitive elements, or from certain patterns of mismatches and gaps. Together with appropriate weighting during model training, this could achieve comparable accuracy while reducing the number of tandem reads required. It could also reduce the prediction variability we see in experiments involving longer reads and more repetitive genomes.We presented the tandem simulation framework and the Qtip software tool implementing the framework. To date, strategies for predicting mapping qualities have either been ad hoc or required the user to prepare training data tailored to the scenario at hand. Qtip runs alongside a read aligner and builds an input model, simulates tandem reads, aligns those using the same aligner and parameters, then uses the trained model to predict mapping qualities. The model and training data are produced automatically and are tailored to the scenario at hand. While Qtip adds an overhead to the read alignment process, it is reasonable, with the time overhead in the 6–28% range and the memory overhead in the 1–10% range. This framework, its improved predictions, and the evaluation performed here should make authors of downstream software tools more confident that mapping qualities can be treated as the probabilities they claim to be, and to integrate those probabilities into their models rather than simply thresholding.1 For a group of alignments sharing the same Q, the penalty is averaged across the group’s elements in C and C′. That is, if k≤i≤l.2 For a group of alignments sharing the same Q, the corresponding elements of E and E′ equal the mean squared error of the group. That is, if k≤i≤l.


Accurate reconstruction of ancestral states is a critical evolutionary analysis when studying ancient proteins and comparing biochemical properties between parental or extinct species and their extant relatives. It relies on multiple sequence alignment (MSA) which may introduce biases, and it remains unknown how MSA methodological approaches impact ancestral sequence reconstruction (ASR). Here, we investigate how MSA methodology modulates ASR using a simulation study of various evolutionary scenarios. We evaluate the accuracy of ancestral protein sequence reconstruction for simulated data and compare reconstruction outcomes using different alignment methods. Our results reveal biases introduced not only by aligner algorithms and assumptions, but also tree topology and the rate of insertions and deletions. Under many conditions we find no substantial differences between the MSAs. However, increasing the difficulty for the aligners can significantly impact ASR. The MAFFT consistency aligners and PRANK variants exhibit the best performance, whereas FSA displays limited performance. We also discover a bias towards reconstructed sequences longer than the true ancestors, deriving from a preference for inferring insertions, in almost all MSA methodological approaches. In addition, we find measures of MSA quality generally correlate highly with reconstruction accuracy. Thus, we show MSA methodological differences can affect the quality of reconstructions and propose MSA methods should be selected with care to accurately determine ancestral states with confidence. Given an ensemble of known sequences, ancestral sequence reconstruction (ASR) refers to methods used to recover the genetic sequence character states of their common ancestors. It has been used to study molecular evolution of photo-reactive proteins , thermalAncestral reconstruction begins with a hypothesis of how taxa descend from common ancestors in a tree-based structure or phylogeny. The taxa are represented as tips of the tree, progressively connected by branches to their common ancestors represented by the internal nodes of the tree. The common ancestor sequence of the entire sample of taxa is the root of the tree. Protocols for ASR usually involve four steps : 1) seleTwo main paradigms for ancestral state reconstruction exist: maximum parsimony (MP) and probabilistic methods, which include maximum likelihood (ML) and Bayesian reconstructions. Probabilistic methods use an explicit model of substitution, unlike the implicit model embedded in MP. These methods can also estimate confidence in each inferred ancestor, often expressed as the posterior probability of the data . AlthougEstablished ML reconstruction methods can be divided into two types: marginal reconstruction and joint reconstruction. Marginal reconstruction assigns a character state to a single node on the tree and averages over all possible ancestral states at each other node, whereas joint reconstruction assigns a set of character states to all ancestral inferred nodes on the tree simultaneously. Marginal reconstruction can be considered an approximation for joint reconstruction . For “seMultiple sequence alignment is a crucial step in ASR and can yield two different outcomes: structural or evolutionary homology . For evoHere, we investigate how several popular MSA tools  impact AWe simulated protein sequence data sets under a variety of realistic evolutionary scenarios using a combination of several simulation parameters. We first generated an ensemble of phylogenetic trees under the birth-process varying 1) tree height, 2) sampling fraction, and 3) taxon count. Following On each tree, protein sequence evolution was simulated under the WAG model  using twWe aligned the tip sequences from the simulated data sets above using each MSA tool listed in The character states at ancestral nodes were reconstructed from each aligner’s MSA using FastML . The truThe accuracy of an internal node’s reconstructed sequence to its corresponding true sequence was measured using a score based on the method of x-axis) and few differences between methods. At sampling fraction 0.99, all methods have excellent and virtually equal performance  compared with reconstruction using the true alignment (baseline). The accuracies decrease moving along the tree —deeper ancestors are harder to reconstruct accurately—but tend to increase again near the root . This increase is explained by the influence of the information conveyed by the denser-sampled nodes concentrated in the root region, which is a consequence of the sampling fraction of 0.01 . In contrast, MUSCLE and MUSCLE* gave virtually the same results, showing no considerable differences when using the guide tree. For Clustal and PAGAN the use of the guide tree improved accuracies in almost all regions, but worsened performance considerably for nodes close to the root.Using the true tree as guide tree for MSA led to intriguing results. For PRANK variants, using the guide tree consistently improved the accuracy along all trees  were low for most of the tools, with PRANK variants showing worst results. PRANK + F had slightly higher percentage of error attributed to deletions compared with PRANK, and using the guide tree resulted in similar distributions. PAGAN* also showed deletion errors marginally higher than other tools, but lower than PRANK.We analyzed the contribution of insertion and deletion errors to the accuracy measure to discover specific biases in the MSA tools. Insertion and deletion errors are included in the accuracy measure  and represent the percentage of residues present (insertion) or not present (deletion) in the reconstructed ancestral node compared with the true sequence. Recall that correct ASR would result in insertion and deletion error scores of 0 (see above). Again, concentrating on simulation conditions where the MSA methods had contrasting results , we discovered biases in all tools, including reconstructions based on the true alignment . Deletiox-axis), we observed considerable biases in some tools. By this measure, PRANK + F*, PRANK*, PRANK + F, MUSCLE variants, and MAFFT’s INS-i variants showed best results, all with overall insertions errors below 0.2 (with differences in dispersion). Other MSA tools displayed a strong bias towards insertions, particularly FSA, which yielded insertion errors of >0.8 .For insertion errors , x-axis,The bias towards insertions results in longer reconstructed sequences A. HoweveAlthough the alignment length may give some insights into the performance and utility for downstream analyses of different MSA methods, its accurate estimation has no particular value itself. Rather, the ability of the MAFFT INS-i, PRANK, and MUSCLE variants to give individual inferred ancestral sequences with lengths most closely resembling true values is an important measure of their superior performance.devol measure from MetAl , values were subtracted from 1 to produce an accuracy measure, more-readily related to the other metrics. 2r) typically higher than 0.75 for most of MSA tools and quality measures. An exception was the TC score, which showed lower correlation (2r around 0.60) when compared with other quality measures.We compared reconstruction accuracy with measures of MSA quality. MSA quality measures were calculated using the om MetAl  and the om MetAl : DevelopOnly small differences were observed for specific aligners. The most notable of these is the Modeler score, which yielded anomalously high values for FSA when compared with other measures and aligners .MetAl (1 − devol) and SP-score measures performed best (correlation > 0.85).Despite the generally good overall correlations between MSA quality measures and reconstruction accuracy within specific MSA tools, the comparison between metrics over different alignment tools, especially in contrasting scenarios, shows some alignment quality metrics orthogonal to reliable reconstruction. In our primary analysis, we simulated sequences using indel rate parameters of 0.01 and 0.05. However, analyses of mammalian and bacterial orthologs from the OrthoMam  and COG We tested the impact of MSA tools on ancestral state reconstruction accuracy using amino acid sequences simulated under various realistic conditions  and 2. WHowever, we found the choice of MSA method can impact on ASR under more demanding conditions. Our analyses reveal that some factors, like tree topologies and indel rates, have a more significant impact on ASR than others . Altering tree shape by lowering the sampling fraction, increasing the number of substitutions per site and increasing indel rates all reduce reconstruction accuracy; in these difficult cases, differences between MSA tools were revealed 2. MAFFT figs. 3 andThere was a notable bias towards insertions by all aligners . A slighThe problem between overalignment and underalignments has been extensively discussed . We obseMetAl devol and SP-score achieved good correlation overall . In more-challenging situations, MSA methods must be chosen with caution to provide reliable reconstructions of ancestral states.evolver from the PAML suite (Ultrametric phylogenetic trees were simulated using ML suite  under a ML suite . Trees wML suite . In totaFor each tree, ten sets of amino acid sequences were simulated using INDELible  with gamWe evaluated aligners that use a variety of different approaches, comprising the progressive aligners Clustal Omega , MUSCLE We calculated ASRs using FastML . MarginaInsertion error: the number of residues present in the reconstructed sequence but absent in the true sequence, divided by the length of the alignment.Deletion error: the number of residues present in the true sequence but absent in the reconstructed sequence, divided by the length of the alignment.Substitution error: the number of mismatched residues divided by the length of alignment.Reconstruction accuracy was evaluated as in All error measures range from 0 to 1, with error equal to 0 being the ideal case with no differences between reconstructed and true sequences according to the metric. Subtracting the sum of these error scores from 1 provides a measure of overall accuracy, representing the proportion of the pairwise alignment sites at which a correctly aligned residue appears (e.g., see Supplementary DataClick here for additional data file.