We are interested in the problem of predicting secondary structure for small sets of homologous RNAs, by incorporating limited comparative sequence information into an RNA folding model. The Sankoff algorithm for simultaneous RNA folding and alignment is a basis for approaches to this problem. There are two open problems in applying a Sankoff algorithm: development of a good unified scoring system for alignment and folding and development of practical heuristics for dealing with the computational complexity of the algorithm.We use probabilistic models  as a unifying framework for scoring pairwise alignment and folding. A constrained version of the pairSCFG structural alignment algorithm was developed which assumes knowledge of a few confidently aligned positions (pins). These pins are selected based on the posterior probabilities of a probabilistic pairwise sequence alignment.Pairwise RNA structural alignment improves on structure prediction accuracy relative to single sequence folding. Constraining on alignment is a straightforward method of reducing the runtime and memory requirements of the algorithm. Five practical implementations of the pairwise Sankoff algorithm – this work (Consan), David Mathews' Dynalign, Ian Holmes' Stemloc, Ivo Hofacker's PMcomp, and Jan Gorodkin's FOLDALIGN – have comparable overall performance with different strengths and weaknesses. RNA secondary structure can be predicted accurately from sequence data alone. For example, the predicted secondary structure of ribosomal RNA has been essentially confirmed by recent crystal structures; 97–98% of the predicted base pairs are confirmed by experimental structures . The trocomparative analysis 39]. SCFGunambiguous with respect to both secondary structure and alignment [parse tree π that describes how the grammar aligns and scores the two sequences. There must be a strict one-to-one relationship between parse trees and alignments, as well as parse trees and base-paired secondary structures, in order for the optimal parse tree to be interpretable as an optimal alignment and structure.Ideally the SCFG design should be formally lignment ,38. ThatWe consider one alignment to be a set of aligned residue pairs. Any two alignments that yield the same set of aligned residue pairs are considered identical. That is,a-bdef-gandab-de-fgalignment-ambiguous if there exists an alignment that can be generated by more than one possible parse tree.are the same alignment ,. A grammar is structurally ambiguous if there exists a secondary structure that can be generated by more than one possible parse tree.Similarly, we consider one secondary structure to be a set of base pairs. Any two structures that yield the same set of base pairs are the same structure. A grammar is The consequence of using an ambiguous grammar is that the probability of a single alignment or structure may be spread across many parse trees that describe the same set of aligned residues or base pairs, therefore an optimal parse tree could represent a less optimal alignment or structure that merely has fewer alternative parse tree representations. Our previous work  indicateIn our previous work  we demons     S → aS | T | εGT → Ta | aPa' | TaPa'P → aPa' | NN → aS | Ta | TaPa'T → aPa' implies a basepairs with a', '|' denotes 'or' between production rules, and ε is the null string used to represent an ending production. The symbols a is used generically to represent any nucleotide of RNA . All 16 possible base pairs are permitted, including non-canonical pairs with low probability.was found to give good secondary structure performance for a reasonably small number of parameters. The notation G6") was found to give slightly better single sequence structure prediction performance, but appears to be difficult to extend to an alignment-unambiguous pair grammar.We can extend this grammar to the problem of pairwise structural alignment, the simultaneous sequence alignment and structure prediction of two sequences, simply by making it generate two correlated sequences instead of one. A different grammar from  ("G6") wx and y. As we are interested in identifying shared structure, we first extend base pairing states to the pairwise case, a basepairs with a' in sequence x and b basepairs with b' in sequence y. This rule effectively captures basepair correlations observed in evolutionarily conserved secondary structures. In unpaired regions, the grammar reverts to an alignment algorithm so we logically replace each unpaired region of the SCFG with a typical pairHMM model of alignment[To handle structural alignment, the SCFG is extended to emit a correlated pair of sequences, alignment,41. The which has eight nonterminals and 35 production rules. We postulate that this grammar is both structurally unambiguous and alignment-unambiguous (evidence is given in the Appendix).S → aS, which imply a geometric length distribution. Unambiguous SCFG designs that capture a more biologically realistic model of RNA are more complex, and we have deferred them to future work.This grammar lacks at least two features that are thought to be biologically important in RNA folding. It does not have stacking rules; base pair emissions are statistically independent of each other. It also does not have explicit length distributions for hairpin loops, bulge loops, interior loops, and multiloops. All length distributions are modeled in this grammar by rules of the general form Given a parameterized pairSCFG model and two input sequences, we find the maximum likelihood parse tree  by a pairSCFG CYK algorithm . More foπ P in memory and O(N3M3) in time for two sequences of lengths N and M. As we are focusing on the global alignment of homologous RNAs, we assume that the lengths of the two sequences are roughly comparable and the algorithm is O(N4) in memory and O(N6) in time.From this example, the correspondence between grammar rules and the CYK algorithm should be clear. An example parse tree from the pairSCFG is shown in Figure transition parameters for the probability of using a production rule from emission parameters that generate any nucleotide(s) from the rule.Given the set of pairSCFG production rules above, we need to determine all the necessary probability parameters. As is common for many stochastic models ,31, we dx, y and y, x. Second, we tie several additional "equivalent" parameters together, as follows.We use two types of parameter tying to reduce the number of free parameters in the model. First, we assume that the model should be symmetric, assigning the same probability to An untied version of the pairSCFG would require 2234 emission parameters: there are 12 rules that generate single nucleotides (4 parameters apiece), 8 aligned nucleotide rules (16 parameters apiece), and 8 aligned base pair nucleotide rules (256 parameters apiece). We reduce these to one emission distribution for single  nucleotides, one symmetric 4 × 4 matrix for emitting unpaired aligned residues, and one symmetric 16 × 16 matrix for aligned base pairs. The tied model has 150 emission parameters, of which 129 are free.x and y by tying together the sets of parameters utilized for "gap open", "gap extend", and "gap closure". For example Lx are tied to the rules of Ly as they both represent "gap extend" and "gap closure" rules. The tied model has 22 transition parameters, of which 16 are free.The untied pairSCFG has 35 production rules, hence 35 transition probabilities. We group transition parameters such that gaps are treated equivalently in The 172 parameters of the pairSCFG were then estimated from the frequencies observed in annotated ribosomal RNA secondary structures from multiple alignments in the European Ribosomal Database ,43. SequBecause the training dataset consisted of many sequences but only two different RNA structures , we had some concern that the model might fail to generalize to other, shorter, structural RNAs. To address this concern, we compared two models trained on two independent training sets. The first is the aforementioned rRNA dataset. The second is composed of all seed Rfam v7.0 families marked as published. Each family was filtered at 80% identity and those with fewer than two sequences remaining were subsequently removed. The SSU Rfam family was removed to avoid overlap with the rRNA training set. The resulting dataset consisted of 96 families, a total of 2054 sequences with an average length of 137 nucleotides, and a total of 302,993 nucleotides.Figure O(N4) memory and O(N6) time, the full (unconstrained) pairSCFG CYK algorithm is barely practical. On current computers, it becomes unreasonable for RNAs of more than about 100 nucleotides in length. Our heuristic strategy is to constrain the algorithm by a small number of primary sequence alignment pins representing confidently aligned pairs of residues. Using pins (or anchors) to fix portions of the alignment is used by a number of sequence alignment programs to allow very long alignments [At ignments . Given ai, a, and j are indices in x and k, b, and l are indices in y. Sequence x is length M and y is length N. The subsequence i...j aligns to k...l. For each nonterminal in the grammar we keep a four dimensional matrix, i, j, k, l. The indices a and b locate the split points in a bifurcation rule. With bifurcation rules, we must consider all possible split points, therefore 1 ≤ i <a <j ≤ M and 1 ≤ k <b <l ≤ N.By convention, qz as a coordinate pair (qz (x), qz (y)) where the nucleotide qz (x) aligns to the nucleotide qz (y). The ordered set of alignment pins Z pins, numbered from the 5' end of sequence x, i.e. z = 1..Z. We always define two pins (Z ≥ 2) as boundary conditions: one which comes before the 5' nucleotide of each sequence q1 =  and one which follows the 3' nucleotide of each sequence qz = .We define a pin We seek to calculateπ P , qz (y)}) ∈ π P  as the range of index k in y which must be considered for a particular i in x. A position i between pins qz and qz+1 has a segment i) which implies qz (y) <k <qz+1 (y). We refer to edges of the range, in this case qz (y) and qz+1 (y) as L (i) and R (i) respectively. Refer to Figure q1 and qZ), i) is the full N nucleotides of y, and the algorithm computes the full unconstrained structural alignment algorithm.Given i, a, and j) of the first sequence, x. However, for a particular instance of each of these indices, the corresponding range of analogous positions in y  is reduced. If we were using an ungapped alignment approach, a single midpoint pin would reduce the range of indices in y by 1/2, effectively reducing the memory by 1/4 and the runtime by 1/8.Each constraint provided is an additional pin in i is a pin, the precise alignment partner is known. It might seem that the segment i)should therefore be one nucleotide, but this is only true in ungapped alignments. In gapped alignments, we must consider that the pin may end or begin a gap. An indel aligns a nucleotide to nothing and in the dynamic programming algorithm one index k progresses while the other i is fixed. Consequently, to consider all possible gap states requires i = qz (x)) to imply qz-1 (y) <k <qz+1 (y). In this case the segment i) overlaps the segments i - 1) and i + 1), shown in Figure When a position of interest Our work on single sequence grammars indicated that prediction accuracy can vary significantly between RNA families . The recWe also obtained published test sets from David Mathews,35 and IWe utilize base pair sensitivity and base pair positive predictive value (PPV) as structure prediction accuracy measures . We compWe calculate the alignment identity between the trusted pairwise structural alignment and our predicted alignment, defined as the number of alignment columns correctly determined relative to the trusted (given) alignment. As with structure comparisons, we compute alignment identity cumulatively over all columns in all alignment pairs.We determine the standard deviation of all three measures by bootstrapping using 10,000 samples.Gs. The constrained CYK algorithm was implemented for the pairSCFG in ANSI C. The source code for the package, called Consan, is freely available under the GNU General Public License (GPL) from [Parameter estimation (training) and the CYK algorithm was written for both the pairSCFG and its corresponding single sequence grammar PL) from . The tradpswalign program, a pairHMM implementation from his Dart package v0.2[To determine pins, we utilize Ian Holmes' kage v0.2. The promfold v3.1.2, clustalw v1.83, Dynalign (Sept 2004), and Stemloc from the Dart software package v0.2 (Oct 2004), RNAfold from the ViennaRNA v1.6 package, the PMcomp.pl script (Nov 2004), and FOLDALIGN v2.0.3. Dynalign is used with the parameters suggested in [For benchmarking experiments, we use ested in , Stemlocested in , PMcomp ested in . Benchmas have only slightly worse performance than standard minimum energy methods. In order to verify that the new Gs single-sequence grammar design performs similarly to the previously tested small grammars, we trained it using the rRNA training set and compare its performance to mfold for single sequence folding on a test set of 100 tRNA and 100 5S sequence randomly chosen from the Rfam v7.0 seed alignments. As shown in Table s grammar shows the expected performance lag relative to mfold, a necessary trade-off for a simpler grammar[In earlier work , we showr grammar.s.As a control experiment to verify that the extension to a pairSCFG does not significantly alter single sequence prediction accuracy (and that we had not made any glaring implementation errors), we evaluated the pairSCFG on "pairwise alignments" of the test sequences to themselves, with the expectation that it should show similar performance to the single sequence grammar GWe then tested whether the corresponding pairSCFG improved structure prediction accuracy on the same test set, using the unconstrained pairSCFG CYK algorithm. We trained the pairSCFG on all pairs of the rRNA training set. Using the same 5S and tRNA test set, we randomly selected an "informant" sequence from the 5S and tRNA seed alignments within the 55–80% identity range emphasized by the rRNA training set. In evaluation, however, we only consider the ability to predict the original test sequence's structure.mfold or Gs. On self vs. self identical alignments, the pairSCFG gives similar basepair sensitivity as the single sequence grammar Gs on which it was based, as expected. PPV performance drops, because the pairSCFG overcalls base pairs in these identical alignments. Its parameterization has learned that base pairs are often more conserved than single stranded residues in homologous alignments.Table We expect that the performance of any pairwise structural alignment algorithm will depend on the similarity of the sequence pair. Closely related pairs will lack sufficient covariation to differ from single sequence structure prediction. Pairs at too great a distance may be difficult to align even when secondary structure information is taken into account.To test the effects of sequence identity on both structure and alignment performance, we built a series of test sets binned by pairwise identity. The Rfam v7.0 seed alignment sequence pairs for 5S and tRNA are divided into 20 bins, each representing a 5% identity interval for the known structural alignment. Within each bin 10 sequence pairs are selected at random for both 5S and tRNA. Because sequence pairs were unavailable at the lowest percent identities, the resulting test set contains 184 tRNA pairs between 9% and 99% identical and 140 are 5S pairs between 32% and 99% identical.We then utilize these binned sets to compare the pairwise alignment accuracy of the pairSCFG to ClustalW, shown in Figure mfold and the single sequence grammar Gs, we extract all individual sequences and determined the base pair sensitivity on this collective set. These results are shown in Figure We also examined the structure prediction accuracy of the pairSCFG over a range of sequence identities. For comparison to mfold or Gs. For sequence pairs between 40 and 90% identity, the pairSCFG improves structure prediction with respect to mfold by about 10% on average for 5S and about 20% for tRNA.For structure prediction, the accuracy of the pairSCFG at low (<25%) and high (>90% identity) is not strikingly different than single sequence prediction by Although the alignment and structure prediction results above are promising, the runtime and memory requirements of the unconstrained pairSCFG algorithm are extreme. For two representative tRNAs, lengths 76 and 77, the algorithm requires 290 MB of memory and 910 seconds. For two representative 5S sequences, lengths 116 and 117, the algorithm requires 1556 MB of memory and 22,902 seconds.pins , and then use those pins in a constrained Sankoff pairSCFG CYK algorithm.Figure We think we can afford to focus on comparisons within a particular broad range of similarity because comparative genome sequencing has advanced so that there are often many sequence homologs available at different levels of similarity. We usually have some freedom to pick and choose "optimal" homologs for assisting an RNA secondary structure prediction.dpswalign program in his Dart package, and we calculate an "optimal accuracy" alignment that maximizes the sum of these posteriors [To obtain a set of pins consistent with a single alignment, and to rank them by reliability, we use a pairHMM to calculate a posterior probability for each aligned pair of residues in a pairwise sequence alignment, using Ian Holmes' steriors . In prinWe then select a subset of "quality" pins from the set of aligned residues in the optimal accuracy alignment. Pin selection is a trade-off. With each additional pin, the constrained algorithm will take less memory and time; however, because pins are fixed, an incorrect pin choice cannot be rectified later by the constrained algorithm. Additionally, pins that are evenly spaced provide more of a performance gain, as opposed to pins that are tightly clustered. One method of enforcing pin spacing is to define a window around each pin in which no other pins can be selected.We examined a number of different combinations of posterior probability cutoffs between 0.80 and 1.0 and "protection windows" between 0 and 30 nucleotides using a pin selection strategy which is greedy, always selecting the best available pin given the protection window. Regardless of the posterior threshold, with short protection windows (≤ 15 nucleotides) incorrect pins are detected in many pairs, including those with high percent identity . The average compute resource requirements (runtime and memory) increase with window length, as this is the primary determinant of the number of pins possible. For example, at a protection window of 20 nucleotides, the average number of pins found for the set of 5S pairs containing at least one pin is 4.3 and the average runtime is approximately 219 seconds. Increasing the window to 25 nucleotides decreases the average pins found to 3.6 which results in a average runtime of 259 seconds. At a particular window length, the posterior probability threshold affects both the number of sequence pairs for which any pin can be found and the ability of the constrained algorithm to reproduce the unconstrained results.The settings for the posterior threshold and window length parameters trade off compute performance against alignment and structure prediction accuracy. Table Having decided on this strategy, we next evaluated how many pins are generated for sequences of different levels of identity. The availability of at least one quality pin correlates roughly with the percent identity of the sequence pair. All pairs > 45% identical have at least 2 quality pins whereas only 4 pairs < 30% identical have 1 quality pin apiece. Furthermore, 82/324 pairs find the maximum number of pins (6 for 5S and 4 for tRNA) permissible given the length of the sequences and a protection window of 20 nucleotides. We then asked how many of these selected pins are correct relative to the structural alignment . For 8/112 5S pairs and 106/781 tRNA pairs, no pins met our selection criteria and Consan falls back to an unconstrained Sankoff algorithm. With these criteria, our algorithm obtains sensitivity measures of 74 ± 2 on 5S, 79 ± 5 on SRP, and 87 ± 2 on tRNA. Thus, Dynalign performs slightly better on 5S RNAs, and on the other two sets the methods perform about the same.Stemloc is comparable in several respects to our work. Stemloc is also based on a pairSCFG, though its grammar is quite different from the grammar in Consan. Holmes' constrained algorithm is based on the general concept of fold and alignment envelopes [Ian Holmes' nvelopes ,31. Our Holmes' test dataset consists of 22 pairs of sequences from Rfam seed alignments. He reports a single basepair sensitivity and alignment accuracy value for each pair, using default parameters of 100 alignments and 1000 folds. Because Stemloc is not scoring symmetric, we examine all pairs, excluding self-to-self. Using our benchmark procedure, Stemloc obtains 65 ± 4 sensitivity, 61 ± 4 PPV and 71 ± 3 alignment accuracy. Using our constrained pairSCFG, (comparing each pair only once) we achieve 67 ± 4 sensitivity, 61 ± 5 PPV and 71 ± 4 alignment accuracy on the same dataset. Thus the two pairSCFG implementations show comparable performance on Stemloc's test dataset.We compared the five constrained Sankoff implementations on our test set binned by percent identity. We compare each pair only once, with the sequence order being determined randomly. For Dynalign, we used the parameters described in , excludiFigure Despite both using pairSCFG approaches, Consan appears to produce better alignments and better structures over a wider identity range than Stemloc. We can think of at least two hypotheses for this effect. First, because Stemloc currently calculates alignment envelopes defined by complete pairwise alignments, its performance might suffer at low percent identities where obtaining any accurate complete primary sequence alignment is difficult, even in a large sample of suboptimal alignments. We may be more robust to this effect by constraining only on a subset of confident pins. Second, although the Stemloc grammar was said to be both structurally unambiguous and alignment unambiguous, we think it is in fact alignment ambiguous, meaning that multiple parse trees can correspond to the same set of aligned residue pairs. This might have a negative impact on accuracy at low identities, because an ambiguous grammar does not rank suboptimal alignments correctly by their probabilities.O(N33) in time and O(N22) in memory where O(N2) for alignment envelopes and O(N3) for fold envelopes in time, and O(N2) in memory for both. The final structural alignment phase remains O(N6) and O(N4) in time and space respectively. PMcomp uses precomputed pair probability matrices, computed in O(N3) in time. The structural alignment phase uses a structural banding approach, restricting the difference in the span of matching base pairs to Δ which reduces the computational effort to O(N3) in memory and O(N4) in time. FOLDALIGN uses both alignment banding (δ) and structural banding (λ) to reduce the time complexity to O(N2λ2δ2) and the memory complexity to O(λ3δ). Our pinned approach calculates its pins in O(N2) time and space and the structural alignment remains in general O(N6)and O(N4) in time and space; in the limit of an alignment where all positions are pins, our constrained algorithm reduces to O(N3) in time and O(N2) in space.In terms of computational resources, the Dynalign algorithm utilizes a banding approach that reduces the algorithm to  distance,35 betweTable Stochastic context-free grammars provide a unifying framework for simultaneously scoring of alignment and secondary structure folding, providing a strong formal basis for scoring systems in comparative RNA secondary structure applications . Holmes'Stemloc's concept of alignment and fold envelopes is a general one, and the concept includes our pins as a special case. As Holmes notes , there aGrammar design issues are of great interest to us. Earlier work  showed tBe that as it may, we do not think that either the Stemloc or the Consan grammar design will prove to be the best for this problem. Both implementations use relatively simple grammars that do not approach the descriptive power of the current thermodynamic rules for RNA folding. For example, neither grammar has a model of explicit loop lengths akin to the hairpin, bulge, and interior loop length tables of RNA folding thermodynamic rules; nor do they model nearest-neighbor base pair stacking correlations as the energy tables do. Between our work and Holmes', the demonstration that two different pairSCFG implementations can fold RNAs as well as the best current thermodynamic approach , despite the fact that our pairSCFGs clearly lack a treatment of some more complex statistical features known to be important in RNA structure, indicates that it will be promising to explore this direction further, with more biologically realistic grammars.Parameterization of these grammars is also an area of future work. Thus far, we have used a single point estimate for our parameters, based on maximum a posteriori training using a mixed ribosomal RNA dataset. The most glaring problem with this is that since we are comparing RNA sequences of different levels of evolutionary divergence, we would prefer not to use single point estimates, but to instead use a rate-based model that allows our parameters to be conditional on evolutionary time. Preliminary data (not shown) shows that evolutionary models akin to those described by Knudsen and Hein ,23 improRDD developed the constrained Sankoff algorithm, wrote the code, carried out the experiments, and drafted the manuscript. Both authors collaborated closely in writing the final version.The proposed grammar is both structurally unambiguous and alignment unambiguous. Reeder et. al.  suggesteannotation of the consensus structural alignment, rather than an aligned sequence pair, such that each possible structural alignment has one and only one annotation string. Then, for this alternative representation of the grammar, we show that we can generate a deterministic parser.We start from the observation that if a deterministic parser can be generated for a grammar, the grammar must be unambiguous by definition . This obFor a single sequence, a nested (non-pseudoknotted) secondary structure is uniquely described by an annotation of unpaired ('.') and paired  bases. Extending this to structural alignment, we must also introduce gap symbols to indicate when an unpaired residue is only present in X ('-') or only present in Y ('_'). In our grammars, a consensus base pair must be present in both sequences, so gaps may only be placed in unpaired columns. Using these symbols, each structural alignment is uniquely described by a single annotation string. Figure Because the rules of our pairSCFG directly map to the symbols (and meanings) of the annotation alphabet, we can transform our pairSCFG grammar into the following annotation grammar:S → .S | -Lx | _Ly | T | εT → T. | Rx- | Ry_ | <P > | T <P >Lx → .S | -Lx | T | εLy → .S | _Ly | T | εRx → T. | Rx- | <P > | T <P >Ry → T. | Ry_ | <P > | T <P >P → <P > | NN → .S | -Lx | _Ly | T. | Rx- | Ry_ | T <P >Now, if a parser can be generated for this annotation grammar, then our pairSCFG must be unambiguous. Note that a failure to identify a parser would not prove that the grammar is ambiguous, but rather that no conclusion can be made with respect to the grammar's ambiguity status. The popular yacc and bison parse generators are only capable of handling a subset of context-free grammars and fail to generate a parser for the annotation grammar. A more general parse generator is the MSTA parse generator of the COCOM compiler construction package. The MSTA parse generator produces a parser for the annotation grammar, which demonstrates that our pairSCFG is structurally unambiguous and alignment unambiguous.


Multiple sequence alignment is the foundation of many important applications in bioinformatics that aim at detecting functionally important regions, predicting protein structures, building phylogenetic trees etc. Although the automatic construction of a multiple sequence alignment for a set of remotely related sequences cause a very challenging and error-prone task, many downstream analyses still rely heavily on the accuracy of the alignments.β-lactamase families demonstrate that the score can distinguish sequence patterns with different degrees of conservation. Secondly, we evaluate the quality of the alignments produced by several widely used multiple sequence alignment programs using a novel alignment quality score and a commonly used sum of pairs method. According to these results, the Mafft strategy L-INS-i outperforms the other methods, although the difference between the Probcons, TCoffee and Muscle is mostly insignificant. The novel alignment quality score provides similar results than the sum of pairs method.To address the need for an objective evaluation framework, we introduce a statistical score that assesses the quality of a given multiple sequence alignment. The quality assessment is based on counting the number of significantly conserved positions in the alignment using importance sampling method in conjunction with statistical profile analysis framework. We first evaluate a novel objective function used in the alignment quality score for measuring the positional conservation. The results for the Src homology 2 (SH2) domain, Ras-like proteins, peptidase M13, subtilase and The results indicate that the proposed statistical score is useful in assessing the quality of multiple sequence alignments. A wealth of molecular data concerning the linear structure of proteins and nucleic acids is available in the form of DNA, RNA and protein sequences. Multiple sequence alignment has become an essential and widely used tool for understanding the structure and function of these molecules. The results of annotation of gene/protein sequences, prediction of protein structures or building of phylogenetic trees, for instance, are critically dependent on the quality of the given alignment. It has been recognized that the automatic construction of a multiple sequence alignment for a set of remotely related sequences can be a very demanding task. Therefore, there is a need for an objective approach to evaluate the alignments produced by alignment programs.Two popular measures for scoring entire multiple alignments are the sum of pairs (SP) score and the column score (CS) . These sZ-score that is based on profile analysis  * 100.     (12)AQ score addresses to the question how many percent the test ConsAA is from the reference ConsAA. It is presumed that the higher the AQ value the better is the quality of the alignment. Note that the AQ score does not require the conserved positions to be the same in the test and reference alignments, only the number of the conserved residues counts.The We compared the alignment quality of six multiple sequence alignment programs which have been widely used in bioinformatics: Clustal , TCoffeeWe used the BAliBASE 3.0 database to test the alignment quality of the alignment programs. The BAliBASE is built for comparing alignment programs . The datEach alignment in the BAliBASE has two versions: one with full-length sequences and another with truncated sequences containing the sequences corresponding to the homologous regions only. We used only the truncated sequences, except in reference set 4, which only contains the full-length sequences. The BAliBASE annotates reliably aligned regions as core blocks. As in most of the studies using BAliBASE, we compared the alignment programs using the core block sequences only.The BALiBASE provides a program called bali_score for calculating the SP and CS quality measures for the test alignment . We usedConsAA was then calculated using 15 different FDRs varying from 0.01 to 0.15. The AQ score was calculated using the core blocks of each alignment. The core blocks of the new alignments were determined to consist of positions including one or more residues in the core block of the reference alignment. Additionally, the SP and CS scores were calculated for the core blocks using bali_score program.In order to compare the quality of the alignments, the alignment programs were used to align each family in the BAliBASE database. The significance of the observed maxZ score was calculated for each alignment position. The proportion of conserved residues The AQ, SP and CS scores were calculated for the 7 test alignments and the BAliBASE reference alignments for each set of sequences in the five reference sets. The results are presented as medians within each reference because the distributions of the AQ score values in the references were skewed. The statistical significance of the differences between the alignment programs were tested at FDR = 0.05 using Wilcoxon signed rank test. This test statistic was chosen because the same sequences were used in each method, and hence the scores could not be considered to be independent. The Bonferroni correction was used within each reference set to correct the effect of making multiple tests simultaneously. In the results section, the corrected p-values less than 0.05 were considered as statistically significant. For comparison of the relationship between the individual AQ and SP scores, we calculated the Spearman rank correlation coefficient separately for each 7 alignment methods.All authors participated in conceiving and designing the manuscript. VA, TA and EU carried out the theoretical considerations. VA performed the computational experiments and statistical analysis. VA, TA and MV drafted the manuscript. All authors read and approved the final manuscript.MultiDisp visualization and conservation scores for the Ras-like protein positions 1–31. PNG formatted figure includes MultiDisp visualization of the Ras-like protein positions 1–31 (upper) and the corresponding conservation scores (lower). The curves show (red) the scaled -log(p)-values with Blosum62 scoring matrix, (magenta) the scaled -log(p)-values with grouping of amino acids, (blue) Mean Distance and (green) Information content scores for the alignment.Click here for fileMultiDisp visualization and conservation scores for the Ras-like protein positions 32–62. As Click here for fileMultiDisp visualization and conservation scores for the Ras-like protein positions 63–92. As Click here for fileMultiDisp visualization and conservation scores for the Ras-like protein positions 93–122. As Click here for fileMultiDisp visualization and conservation scores for the Ras-like protein positions 123–152. As Click here for fileMultiDisp visualization and conservation scores for the SH2 domain positions 1–27. PNG formatted figure includes MultiDisp visualization of the SH2 domain positions 1–27 (upper) and the corresponding conservation scores (lower). The curves show (red) the scaled -log(p)-values with Blosum62 scoring matrix, (magenta) the scaled -log(p)-values with grouping of amino acids, (blue) Mean Distance and (green) Information content scores for the alignment.Click here for fileMultiDisp visualization and conservation scores for the SH2 domain positions 28–55. As Click here for fileMultiDisp visualization and conservation scores for the SH2 domain positions 56–82. As Click here for fileMultiDisp visualization and conservation scores for the SH2 domain positions 83–109. As Click here for fileTuning the importance sampling procedure. PDF file describes the simulation procedure for finding the appropriate parameter values of the importance sampling procedure.Click here for file


DIALIGN. This previous version is superior to Needleman-Wunsch-based multi-alignment programs on locally related sequence sets. However, it is often outperformed by these methods on data sets with global but weak similarity at the primary-sequence level.We present a complete re-implementation of the segment-based approach to multiple protein alignment that contains a number of improvements compared to the previous version 2.2 of objective function. Based on these results, we propose several heuristics to improve the segment-based alignment approach. For pairwise alignment, we implemented a fragment-chaining algorithm that favours chains of low-scoring local alignments over isolated high-scoring fragments. For multiple alignment, we use an improved greedy procedure that is less sensitive to spurious local sequence similarities. To evaluate our method on globally related protein families, we used the well-known database BAliBASE. For benchmarking tests on locally related sequences, we created a new reference database called IRMBASE which consists of simulated conserved motifs implanted into non-related random sequences.In the present paper, we discuss strengths and weaknesses of DIALIGN in view of the underlying On BAliBASE, our new program performs significantly better than the previous version of DIALIGN and is comparable to the standard global aligner CLUSTAL W, though it is outperformed by some newly developed programs that focus on global alignment. On the locally related test sets in IRMBASE, our method outperforms all other programs that we evaluated. Global methods align sequences from the beginning to the end  is calculated which is defined as the sum of similarity values of aligned nucleotides or amino acid residues . To define the weight score w(f) of f, we consider the probability P(f) of finding a fragment f' of the same length as f and with a Needleman-Wunsch score NW [f'] ≥ NW [f] in random sequences of the same length as the input sequences. w(f) is then defined as the negative logarithm of this probability; see , we denote the k-th character in sequence Si. As in the original DIALIGN algorithm, we traverse the comparison matrix for S1 and S2, and at every position , we consider fragments starting at this position; suitable fragments are then added to a growing set F of candidate fragments from which the algorithm selects a fragment chain with maximum total score with respect to the underlying objective function  < 0, we actually detected a low-quality sub-fragment. If k >T, the procedure stops and no further increasing of k is being considered, otherwise we set M = T.6. If k is incremented by 1 and we continue with the step 3.7. The length L = 4 and the maximum length of fragments containing such regions of low quality is T = 40. These values have been determined based on systematic test runs on BAliBASE. At this point, we want to mention the impact of the parameters L and T on the quality of the produced output alignments. For example, with values L = 3 or L = 5 the alignment quality is dramatically worsened compared with the default value L = 4.By default, our program uses a length threshold for low-quality sub-fragments of worst-case time complexity is O(l1·l2·M) where l1 and l2 are the lengths of the input sequences. By excluding long fragments with low-scoring sub-fragments, we ignore a large number of fragments that would have been considered for alignment in previous program versions. Therefore, our new heuristics allows us to increase the maximum possible fragment length from M = 40 to M = 100 without excessively increasing the total number of fragments that are to be looked at. A further extension of M is prohibited due to numerical instabilities. Altogether, the resulting alignments can reflect the extension of existing homologies more realistically than the previous version of DIALIGN with only a moderate increase in program running time.Our stop criterion for low-scoring sub-fragments not only improves the quality of the resulting alignments but also reduces the program running time. The runtime of our pairwise algorithm is proportional to the number of fragments that are considered for alignment. Thus, the weight scores. A general problem with this greedy approach is that if a wrong fragment is accepted for multiple alignment, it cannot be removed later on. Note that even a single wrong choice in the greedy procedure can impair the quality of the resulting alignment dramatically. Thus, special care has to be taken to prioritise fragments for the greedy algorithm. We observed that in many cases spurious but high scoring fragments from pairwise alignments are inconsistent with a good overall multiple alignment. Due to their weight scores, however, such fragments may be incorporated into the multiple alignment by the original DIALIGN, thereby leading to output alignments of lower quality.As mentioned above, DIALIGN uses a greedy optimisation procedure for multiple alignment. The order in which fragments are included into the multiple alignment is determined based on their random occurrence in sequences the length of the input sequences. Thus, weight scores are purely based on intrinsic properties of fragments – and on the length of the input sequences – but they do not take into account the context of a fragment within the pairwise alignment. In reality, however, the context of a fragment is crucial to assess its reliability. If a fragment f is part of a high-scoring pairwise alignment, then f is, of course, far more likely to be biologically significant than if the same fragment f would be found isolated in otherwise un-related sequences. Therefore the overall similarity among two sequences should be taken into account if fragments are ranked prior to the greedy procedure.As explained in the previous section, the weight score of a fragment depends on the probability of its S1,..., Sn be the input sequences and let f be a fragment involving sequences Si and Sj. Next, let w denote the total weight score of the pairwise alignment for Si and Sj – i.e. the sum of weight scores of an optimal chain of fragments – and let W be the total sum of weight scores of all pairwise alignments. That is, we defineIn our new program, we adopt the following approach: we multiply the weight score of each fragment by the square of the total weight score of the respective sequence pair divided by the overall weight score of all pairwise alignments. Let adjusted weight scoreWe then define the w'(f). This way, we prefer fragments belonging to sequence pairs of high similarity over those from weakly related sequence pairs. Altogether, this weight adjustment respects the similarity of the sequence pairs better than the previous method and hence may keep the greedy procedure from adding isolated spurious fragments that would have led to a lower-scoring and biologically less meaningful output alignment. The sorted list of fragments from the optimal pairwise alignments are kept in a binary heap structure that can be updated efficiently when inconsistent fragments are to be removed or modified as explained in the next section.and in our greedy algorithm, fragments are sorted according to their adjusted scores f is completely discarded in the greedy procedure – even if just a few residue pairs are inconsistent with the current alignment. In such a situation, it would be of course more sensible to remove only those inconsistent residue pairs from f and to give the remaining sub-fragments a second chance in the greedy selection process. It is easy to see that a fragment f is consistent with an existing alignment A if and only if each pair of aligned residues in f is consistent with A. In our new implementation, we use the following procedure for non-consistent fragments. An inconsistent fragment f is processed from left to right. Starting with the left-most residue pair, we remove all inconsistent residue pairs until we find the first consistent pair p. Next, we consider all consistent residue pairs starting with p until we find again an inconsistent residue pair. This way, we obtain a consistent sub-fragment f' of f for which we calculate the weight score w(f'). By construction, f' is consistent with the existing alignment and could, in principle, be added to the list of accepted fragments.In the original DIALIGN approach, an inconsistent fragment f' into the growing multiple alignment since the score w(f') might be smaller than the original score w(f). Instead, we insert f' at the appropriate position in our sorted list of fragments depending on its adjusted weight score w'(f'). With the binary heap structure mentioned in the previous section, consistent sub-fragments of inconsistent fragments can be efficiently re-positioned according to their newly calculated adjusted weights. The remainder of f is treated accordingly, i.e. inconsistent residue pairs are removed and the remaining consistent sub-fragments are inserted at appropriate positions in the list of candidate fragments. Note that with our weighting function w, the weight score w(f') of a sub-fragment f' contained in a fragment f can in general be larger than the weight w(f). In the above situation, however, we have necessarily w(f') ≤ w(f') [and therefore w'(f') ≤ w'(f)] since we assumed that f is part of the optimal pairwise alignment of two sequences. If the score w(f') of a sub-fragment of f exceeded w(f), then f' would have been selected for the optimal pairwise alignment instead of f.However, we do not immediately include The previous implementation DIALIGN 2.2 uses pre-calculated probability tables to calculate fragment weight scores; these tables are based on the BLOSUM 62 substitution matrix. They have been calculated years ago and are difficult to re-calculate if a user wants to employ another similarity matrix. It is therefore not possible to run DIALIGN 2.2 with substitution matrices other than BLOSUM 62. In our new implementation, we use a rather efficient way to estimate the probabilities that are used for our weight score calculations. We pre-calculated probability tables for a variety of substitution matrices. In addition, the user can re-calculate these tables 'on the fly' for arbitrary matrices with a moderate increase in program running time.Objective functions for sequence alignment, we define the weight score of a fragment f involving sequences Si and Sj asAs explained in section w(f) = - logP(f)P(f) denotes the the probability for the occurence of a fragment f' of the same length as f and with Needleman-Wunsch score NW [f'] ≥ NW [f] in random sequences of the same length as Si and Sj. By random sequences we mean independent identically distributed (iid) sequences where each residue occurs at any position with probability 1/4 for nucleic acid sequences and 1/20 for protein sequences, respectively. In the following, we outline how our program approximates the probabilities P(f).where  of finding a fragment f' of length n and with Needleman-Wunsch score NW [f'] ≥ s in random sequences of length 2·n. Note that  depends on the underlying substitution matrix but not on the length or composition of the input sequences Si and Sj. The numerical values  are estimated as follows:In a first step, we estimate the probability  for . The experimental values  should approximate  with sufficient accuracy for values of n and s where enough experimental data are avail able. This is the case if  is not too small.1. Random experiments are performed to obtain a preliminary estimates , we first compute the probabilitys P1 for a single random fragment f' of length n to have a Needleman-Wunsch score NW [f'] ≥ s. P1 can be easily calculated as a sum of convolution products. Similar as in  ≥ s in sequences the length of the input sequences. This step is computationally less expensive and can therefore be carried out during the program run. Let li and lj be the lengths of the input sequences Si and Sj, respectively. Similar as in [P asIn a second step, we use ar as in , we compPT is a threshold parameter. During a program run, the values P are calculated for all possible values of n and s before the pairwise alignment of sequences Si and Sj is carried out. The negative logarithms -log P are stored in a look-up table where they are retrieved during the pairwise alignment to define the fragment scores.where  for several substitution matrices of the BLOSUM family. To determine the experimental probability values Pexp, we carried out 106 random experiments for each relevant pair of parameters . Here, we considered values for n betwen 1 and a maximum fragment length M = 100. Files with the resulting values of  values are distributed together with our software package. To calculate P(f), we use a threshold probability PT = 10-8. Our program can also be used to calculate the values  for arbitrary user-defined substitution matrices. Calculating these values using 105 random experiments for each value of n and s takes around 20 minutes on a Linux workstation (RedHat 8.0) with an 1.5 Ghz Pentium 4 processor and 512 MB Ram. In our experience, 105 random experiments are sufficient to obtain high-quality probability estimates.We pre-calculated the probabilities globally related protein families. All BAliBASE sequences contain homologous core blocks with verified 3D structure; alignment programs are evaluated according to their ability to correctly align these blocks. According to the BAliBASE authors, these core blocks cover 58 % of the residues in the database. However, sequence similarity is clearly not restricted to those regions of verified 3D structure so, in reality, far more than 58 % of the total sequence length are homologous to other sequences in the respective sequence families. Also, the sequences in BAliBASE are not realistic full-length sequences, but they have been truncated by the BAliBASE developers in order to remove non-related parts of the sequences. As a result, BAliBASE consists almost entirely of globally related sequence sets; this is why global alignment programs such as CLUSTAL W perform best on these benchmark data.We evaluated the performance of our program and compared it to alternative multi-alignment software tools using a wide variety of benchmark sequences. As a first set of reference data, we used the well-known BAliBASE 2.1 . BAliBASlocally related sequence sets, Lassmann and Sonnhammer used artificial random sequences with implanted conserved motifs [specificity of a tool, see e.g. [IRMBASE (Implanted Rose Motifs Base).To study the performance of alignment programs on d motifs . Random see e.g. ,10,20. Urandom sequences. This way, we produced three reference sets ref1, ref2 and ref3, of artificial protein sequences. Sequences from ref1 contain one motif each and sequences from ref2 and ref3 contain two and three motifs each, respectively. Each reference set consists of 60 sequence families, 30 of which contain ROSE motifs of length 30 while the remaining 30 families contain motifs of length 60. 20 sequence families in each of the reference sets consist of 4 sequences each, another 20 families consist of 8 sequences while the remaining 20 families consist of 16 sequences. In ref1, random sequences of length 400 are added to the conserved ROSE motif while for ref2 and ref3, random seqences of length 500 are added.As Lassmann and Sonnhammer did in their previous study, we produced groups of artificial conserved sequence motifs using the ROSE software tool . ROSE sisum-of-pair score where the percentage of correctly aligned pairs of residues is taken as a quality measure for alignments. In addition, we used the column score where the percentage of correct columns in an alignment is the criterion for alignment quality. Both scoring schemes were restricted to core blocks within the reference sequences where the 'true' alignment is known. For IRMBASE, the core blocks are defined as the conserved ROSE motifs. In general, the sum-of-pairs score is more appropriate than the column score because this latter score ignores all correctly aligned residues in an alignment column if one single residue in this column is mis-aligned. However, there are situations where the column score is more meaningful than the sum-of-pairs score. This is the case, for example, for BAliBASE reference sets containing 'orphan sequences'.For both BAliBASE and IRMBASE, we used two different criteria to evaluate multi-alignment software tools. We used the simulated data only roughly reflects their performance on real data. Nevertheless, in the absence of real-world benchmark data for local multiple alignment, the results on IRMBASE can give us an idea of how different algorithms deal with locally conserved motifs.To compare the output of different programs to the respective benchmark alignments, we used C. Notredame's program aln_compare . Tables statistically significant according to the Wilcoxon Matched Pairs Signed Rank Test. On BAliBASE reference sets ref1, ref2 and ref3 where sequences contain only small insertions and deletions, the performance of DIALIGN-T is roughly comparable to CLUSTAL W, but yet still significantly worse than T-COFFEE, PROBCONS or MUSCLE. Our program is statistically significantly superior or equal to all tested methods, except MUSCLE and PROBCONS, on the sequence sets with larger insertions or deletions (ref4 and ref5 of BAliBASE).For globally related sequence families, Tables relative performance of the different alignment tools is similar under the two alternative evaluation criteria that we used (sum of pairs and column score) – although, the absolute values of the column scores are, of course, lower than the sum-of-pairs scores. Maybe surprisingly, both versions of DIALIGN are superior to all other programs in our study on the locally related sequences from IRMBASE – while on the other hand, DIALIGN was outperformed by alternative methods on reference sets 4 and 5 of BAliBASE. These sequence sets are also considered locally related because they contain larger insertions and deletions then other BAliBASE sequences. The reason for this apparent discrepancy is that the ref4 and ref5 sequence sets in BAliBASE are not truly locally related, but they still show some similarity outside the conserved core blocks. In IRMBASE, by contrast, sequence similarity is strictly limited to the conserved motifs.Overall, the from scratch and used a variety of novel program features, it is not possible to tell exactly to what extend each of these features contributed to the improved program performance. Systematic test runs with varying parameters indicate, however, that the superiority of our new program compared to the previous program DIALGIN 2.2 on locally as well as on globaly related sequence families is mainly due to the program features explained in the third section. The improvement that we achieved with these heuristics is statistically significant. The features explained in section Further program features also improved the program accuracy, though here the improvement was not statistically significant.Since we re-implemented the DIALIGN algorithm pairwise alignments that are performed before fragments are included into the multiple alignment. As explained in section Excluding low-scoring sub-fragments, the runtime for pairwise alignment is roughly proportional to the number of fragments that are considered for alignment and, for sequences of length l1 and l2 and a maximum fragment length M, up to l1 × l2 × M fragments are to be considered. In our new program DIALIGN-T, the maximum fragment length M is increased to 100 compared to 40 for the original DIALIGN program. Nevertheless, the program running time is only slightly increased for the globally related protein families from BAliBASE and considerably decreased for the locally conserved sequences from IRMBASE. This is due to the heuristic stop criterion for fragments introduced above. The slowest program in our comparison was T-COFFEE which is more than eleven times slower than DIALIGN-T on IRMBASE and more than five times slower on BAliBASE. POA was the fastest method. On BAliBASE, the program PROBCONS produces the best results in terms of alignment accuracy. The program is, however, the second slowest program after T-COFFEE on both BAliBASE and IRMBASE. MUSCLE provides so far the best tradeoff between running time and quality on globally related sequence families, but when it comes to local alignments both running time and alignment quality decrease drastically. The memory consumption of our method has been improved compared to DIALIGN 2.2.Table pairwise fragment alignments, we will develop heuristics that are able to integrate multiple local alignments into the final multiple alignment. This approach should further improve the sensitivity of our methods for locally conserved motifs.With the development of DIALIGN-T, we significantly improved the segment-based approach to multiple protein alignment on both local and global benchmark data. The new heuristics that we introduced, generally favour consistent groups of low-scoring fragments over isolated higher-scoring fragments. This way, we improved the program performance on globally related sequence sets where the segment approach was previously inferior to programs such as CLUSTAL W and POA. On these data sets, our new method is significantly more accurate but only slightly slower than DIALIGN 2.2. On BAliBASE, the performance of our approach is now comparable to the popular global alignment program CLUSTAL W. For locally related protein families, DIALIGN-T performs significantly better and is also considerably faster than the previous DIALIGN 2.2 which was, so far, the best available method on locally related protein families. In addition to these improvements, it is now possible to use arbitrary user-defined substitution matrices which was not possible for the original DIALIGN program. To further enhance the performance of our method, we are planning to improve the greedy algorithm that DIALIGN uses for multiple alignment. Rather than focusing on T and L based on test runs with BAliBASE, and we assume that this is how the program parameters for most multiple protein aligners have been tuned during the last years. Therefore, the question has been raised if current protein alignment programs are overfitted with respect to BAliBASE. Parameter overfitting is a serious problem for many Bioinformatics algorithms. For example, many gene-prediction programs have a large number of parameters to adjust, so it is easy to tune these programs to perform well on a given set of training data. For such programs it is therefore absolutely necessary to clearly separate training data that are used for parameter tuning from test data that are used to evaluate the program. The situation is totally different in multiple alignment. Most multi aligners have only a very small number of parameters to adjust. For our algorithm, for example, the only important parameters to tune are T and L. BAliBASE, on the other hand, comprises a large variety of test sequences for global multiple alignment. It consists of 139 sequence sets, each of which contains several core blocks, so there is a total of several hundred core blocks that are used to test alignment quality. It is absolutely impossible to tune a small number of parameters in such a way that they work well only on BAliBASE but not on other globally related protein sequences. Thus, if an alignment program performs well on BAliBASE, one can safely assume that it also works well on other globally related protein sequences, even if BAliBASE has been used to adjust its parameter values. In fact, it turned out that the parameters that we tuned on BAliBASE work not only well for these global test data but also on the totally different artificial local test sequences from IRMBASE.Finally, we would like to make some general remarks on parameter tuning and program evaluation in multiple sequence alignment. As mentioned above, we identified suitable values for our parameters globally related sequence sets. This does not only refer to the selection of protein families that are included into BAliBASE. As mentioned above, many protein sequences in the current release of BAliBASE are not real-world protein sequences, but have been artificially truncated by the developers of BAliBASE in order to make them globally related. With these non-realistic global test sequences, the BAliBASE authors carried out a systematic program evaluation and – not surprisingly – found out that Global alignment programs generally performed better than local methods [The real problem with BAliBASE is its heavy bias towards  methods . The pic methods ,28, Pref methods .relative order within the input sequences. There are two major reasons for making this assumption. First, an order-preserving multiple alignment that represents homologies by inserting gap characters into the input sequences provides a convenient visualisation of existing homologies. Second – and more importantly -, the order-preservation constraint greatly reduces the noise created by random similarities. A program that would return all detectable local or global similarities among the input sequences without the above ordering constraints would necessarily return many spurious random similarities. To reduce this noise, arbitrary threshold parameters would have to be applied which, in turn, could prevent a program from detecting some of the real homologies. With the ordering constraint that is implicitly imposed by most alignment programs, weak homologies can be detected, provided they are order-consistent with other detected similarities, i.e. if they fit into one single output alignment. Many evolutionary events such as insertions, deletions and substitutions preserve the relative ordering among sequence homologies. In this situation order-respecting alignment methods are, in principle, able to represent all true biological homologies in one multiple alignment. Nevertheless, for distantly related protein families, non-order-preserving events such as duplications or translocations need to be taken into account. Such events play an important role in comparative analysis of genomic sequences which became an important area of research during the last years [As a concluding remark, we would like to address a fundamental limitation of most multi-alignment methods, including the one presented in this paper: these methods implicitly assume that homologies and conserved motifs occur in the same st years . Recentlst years ,3. FurthGöttingen Bioinformatics Compute Server (GOB-ICS):DIALIGN-T is available at Project name: DIALIGN-T• Project home page: • Operating system(s): UNIX and LINUX• Programming language: C• Other requirements: none• License: GNU LGPL• Any restrictions to use by non-academics: none• A program package with functionalities to compute alignments of nucleic acid sequences will be available soon.ARS conceived the new heuristics, implemented the program, constructed IRMBASE, did most of the evaluation and wrote minor parts of the manuscript; JWM participated in program evaluation; MK provided resources; BM supervised the work, provided resources and wrote most of the manuscript.


One of the most powerful methods for the prediction of protein structure from sequence information alone is the iterative construction of profile-type models. Because profiles are built from sequence alignments, the sequences included in the alignment and the method used to align them will be important to the sensitivity of the resulting profile. The inclusion of highly diverse sequences will presumably produce a more powerful profile, but distantly related sequences can be difficult to align accurately using only sequence information. Therefore, it would be expected that the use of protein structure alignments to improve the selection and alignment of diverse sequence homologs might yield improved profiles. However, the actual utility of such an approach has remained unclear.We explored several iterative protocols for the generation of profile hidden Markov models. These protocols were tailored to allow the inclusion of protein structure alignments in the process, and were used for large-scale creation and benchmarking of structure alignment-enhanced models. We found that models using structure alignments did not provide an overall improvement over sequence-only models for superfamily-level structure predictions. However, the results also revealed that the structure alignment-enhanced models were complimentary to the sequence-only models, particularly at the edge of the "twilight zone". When the two sets of models were combined, they provided improved results over sequence-only models alone. In addition, we found that the beneficial effects of the structure alignment-enhanced models could not be realized if the structure-based alignments were replaced with sequence-based alignments. Our experiments with different iterative protocols for sequence-only models also suggested that simple protocol modifications were unable to yield equivalent improvements to those provided by the structure alignment-enhanced models. Finally, we found that models using structure alignments provided fold-level structure assignments that were superior to those produced by sequence-only models.When attempting to predict the structure of remote homologs, we advocate a combined approach in which both traditional models and models incorporating structure alignments are used. The current stream of genome sequence data has lead to a bottleneck between DNA sequencing and elucidation of protein function . TherefoHomology-based methods for structure prediction rely on the observation that proteins that share a common ancestor will usually have similar sequences, and that, in turn, similar protein sequences produce proteins with similar fold and (often) similar functions . HoweverThe profile-type methods address this problem by incorporating family-specific information inherent in multiple sequence alignments into sequence searches -15, and Because profile HMMs model the information present in a sequence alignment, they are affected by quality of the input alignment , but theGiven the limitations inherent in sequence alignments as inputs for profile HMMs, researchers have explored the use of sequence alignments derived directly from structural alignments of proteins, particularly in cases where superfamily-level assignments are desired ,26-31. SOne limitation in using structure alignments for profile HMMs is the weak level of structure representation in many superfamilies. In addition, coverage of a given superfamily is often strong in one area and weak in others. Hence, models made from alignments of these sequences, even if supplemented with additional homologous sequence, may not optimally describe the entire superfamily in question. It has also been suggested that there is an optimal range of sequence similarity for training of profile HMMs, and that using a very broad range of diverse sequence can lead to "profile dilution", reducing model quality ,32.et al. and Sillitoe et al. have suggested that pools of models, each built iteratively from a single structural representative (referred to here as the "master" sequence) provide better performance at the superfamily level than models incorporating structure alignments of multiple masters [et al. also reported that combining the two types of models yielded essentially no benefit [Accordingly, the literature record for the use of structure alignments for profile HMMs yields mixed results. Several researchers, using a variety of experimental arrangements, have reported that HMMs based on structure alignments do not provide a benefit over standard HMMs. Gough  masters ,30. Sill benefit . Howeveret al. used structure alignments to augment iterative profile construction, and reported improved results when attempting to predict very distant homologous relationships. However, some of the improvement reported was based on the combination of the structure alignments with additional structure-derived information (secondary structure and solvent accessibility) [et al. explored the inclusion of structure alignments in the construction of profiles in a profile-to-profile approach based on PSI-BLAST [et al., they found that profiles utilizing structure alignments provided some improvement over standard models, and that this increase in performance could be furthered through the addition of other structural data into the models. Casbon and Saqi tested a profile-to-profile approach relying on a hybrid PSI-BLAST/HMM protocol [vice versa. This suggests that the two types of models might be complementary if used in combined searches.Other researchers have provided qualified support for the use of structure alignments in the construction of profiles  . Other gSI-BLAST . Similarprotocol . They foWe undertook a large-scale assessment of the utility of structure alignments for the generation of profile HMMs, using the traditional sequence-to-profile method. First, we tested several iterative protocols to determine which method generated the most sensitive profile HMMs with HMMER , using oIn order to determine the best iterative methodology for building both single-master HMMs and SLAHMMs with HMMER, performance was compared for four different parameter sets . In all cases, an alignment was built through repetitive HMM searches against a sequence database (with re-training of the HMM after each cycle), but the parameter sets tested different cutoffs for sequence inclusion into the growing alignment, use of heuristics to improve the alignment, and the method for aligning sequences to the model to create the alignment. Test sequences (or "probes") were then searched against the resulting models and the correct or incorrect structure assignments recorded. The superfamily assignments provided in SCOP ,34 were When single-master HMMs were compared to each other for all four parameter sets PS1-PS4), the difference in performance was extremely small  also resulted in relatively similar performance between the various iterative parameters Figure . HoweverHowever, the results also suggest that the use of the heuristics (to select sequences for addition to the model) applied in PS1 and PS2 had a beneficial effect on the quality of the models. Both PS1 and PS2 outperformed counterparts that did not use the heuristics. In addition to the small performance improvement, the heuristics provided the practical benefit of smaller alignment sizes (because they are more selective with respect to added sequences), and therefore lower computational overhead.It was possible to use both sets of models together in a combined search, simply by placing both in a single database and searching the probes against this database. When performance of the combined models was compared for each parameter set, the difference again narrowed, much as it did for the single-master HMMs Figure . AlthougWhen directly compared, single-master HMMs clearly outperformed SLAHMMs Figure . This reAssessment of the specific probe assignments made by SLAHMMs vs. single-master HMMs confirmed that SLAHMMs made unique assignments. At a strict cutoff of 80 incorrect assignments (EPQ ~0.05), the two types of HMMs shared 839 correct assignments of the same probes . However, while single-master HMMs correctly assigned an additional set of 224 probes uniquely, SLAHMMs also correctly assigned 42 probes uniquely  were made prior to the first incorrect assignment. The rest were made after errors had been recorded. In contrast, of the unique assignments made by the single-master HMMs, 91 (~41%) were made prior to the first recorded error. This suggests that, as expected, SLAHMMs are very generalized models that capture more difficult assignments made right at the edge of "the twilight zone", but can miss easier matches.If the single master HMMs are allowed a looser cutoff of up to 1575 errors , they will correctly assign 31 of the 42 sequences that SLAHMMs capture at a cutoff of 80  but did not actually require the explicit incorporation of structural information in the form of a structure-based alignment. Further, it has been suggested that highly accurate alignments are not essential to the production of useful profile HMMs .To see if structural alignments were required, we aligned our test protein domain sequences with ClustalW . These sTo make the iterative HMM searches computationally tractable during initial model building, BLAST was used to pre-filter a large pool of possible homologs from the sequence database for each SCOP master domain (see methods). Though it provides practical benefits, the BLAST pre-filter could also limit the possible sequence space available for training each single-master HMM. Thus, it could be argued that the primary benefit of SLAHMMs might be primarily practical rather than theoretical: they allow the HMM to sample a broader range of sequence space, without the computationally intensive requirement of iterating directly against the entire sequence database.To help determine the practical vs. theoretical benefits of SLAHMMs, the 42 probes that SLAHMMs uniquely assigned at an EPQ ~0.05 , making the additional HMM training requirements minimal. Since the alignments used to build SLAHMMs are derived from sequence alignments already available from the generation of standard, single-master HMMs, they can be conveniently created as an adjunct set of models to improve overall performance.The primary limitation of SLAHMMs is that they require at least two structural representatives per superfamily to be modeled, and some superfamilies still lack even a single structural representative. However, with the gradual growth in structural data likely to be generated from the various structural genomics projects ,39, mult-3. The intent of the chosen E-value cutoff was to provide a balance between broad coverage of the resulting models  and challenging potential assignments with which to test the methods. The chains included the ASTRAL "genetic domains", and were filtered to be at least 80 residues in length. Domains from only the first five classes in the SCOP hierarchy were used, as these represent the "typical" globular proteins to which this work aims to successfully assign structures.The SCOP structural classification database ,34 has bThese domains were used both for model building and benchmarking. The filtered ASTRAL set from above was additionally filtered to only retain superfamilies with three or more members, so that models could be constructed based on the structure alignment of two sequences, while still retaining a sequence to test the resulting model (leave-one-out test).The selected SCOP/ASTRAL superfamily representatives are distant homologs, and it is difficult to obtain an accurate alignments of them using sequence information alone -25. Ther17 structures and one superfamily were removed from the set because of problems caused by inconsistencies between the ASTRAL sequence records and their corresponding PDB files. Also, our CE variant was in still in early development, and failed to align 14 superfamilies. Therefore, these were also removed from the test set. These removals reduced the set of initial structures from 257 superfamilies/1995 chains to 242 superfamiles/1575 chains. However, all five classes of SCOP were still well represented. A breakdown of the representation of SCOP fold classes in this sequence set is avaliable in The Genbank non-redundant protein sequence database ("nrprot") was utilized for the collection of sequence homologous to the SCOP domains. Two releases were used, an older release (downloaded 10/18/02) for early development and testing of the method . All BLAST hits to each SCOP domain with an E-value of < 500 were stored in a miniature database as possible homologs to that domain. HMM searches against the entire nr60 database were prohibitively slow, particularly for purposes of iteration, so pre-filtering the database was required. This technique was first described for the SAM-T98 protocol .hmmcalibrate; this provided additional scoring accuracy. Reported E-values were based on the default theoretical background database size in HMMER . We fouwissprot  was usedall of the collected sequences were aligned back to the model, including the SCOP domain. The domains collected from BLAST can be somewhat shorter than ideal for representing the sequence space around the SCOP domain (even with filtration for coverage), and the aim of this alignment step was to lengthen the model such that it encompassed all conserved sections of the SCOP domain.The ClustalW-created sequence alignment was used to train an initial HMM with HMMER. This model was then used to search the pre-filtered sequence database (matching the SCOP domain) for additional homologous sequences. This model was built as a global/local searching model, meaning that it would attempt to capture sequences that span the length of the model. In the first iteration, After the first iteration was complete, the alignment produced was again used to build and calibrate another HMM using HMMER. The pre-filtered database was again searched and the sequences scoring below a set E-value threshold (determined by the parameter set as discussed below) were collected and aligned back to the model. The model was then re-built using this new alignment, and another iterative cycle initiated, etc.We tested 4 different iterative strategies in order to determine which would be the most successful at later structure assignment Table . We nameFrom the second iteration onward in PS1, only the newly collected sequences were freshly aligned to the model, while the prior alignment was retained unchanged. This arrangement allowed sequences at each level of similarity to align to a model most suited to their relationship to the SCOP domain. It also guarded against iteration and alignment drift by anchoring both the alignment and the model to the initial SCOP domain.The iteration was run for a total of six cycles. With each cycle, the E-value threshold was raised to allow more distant sequences into the alignment as detailed in Table In addition to the E-value criterion, sequences were filtered prior to addition to the alignment at each iteration based on heuristics that aimed to improve the quality and relevance of the alignments. First, sequences were filtered based on their coverage of the HMM model, calculated as the percentage of match states available in the model aligned to by the sequence. It was determined empirically that a reasonable cutoff level was 70% model coverage in the first three iterations and 50% model coverage in the last three iterations. Sequences that aligned to the model below these cutoffs were almost always short fragments or poor alignments.In addition to the filtration for coverage, a heuristic was applied to screen out poor alignments. The measure used was the ratio:is is the number of insertion states in the model caused by the sequence and ms is the match states the sequence matched to. The point of this measure was to detect poor alignments in which a sequence only matched a few highly conserved match states, but otherwise aligned to insert states in the model . These sorts of alignments were sometimes seen at the higher E-value  thresholds, and were presumably not helpful to the construction of a good model. Through the use of Rim, large insertions in the model were still possible, provided the sequence also aligned to a large amount of match states. This prevented the exclusion of sequences simply because they produced a large but valid insertion in the model. Rim was required to be less than 10 for the initial iteration and less than 3 for all other iterations. Rim was set to a very high level in the first iteration because alignments at the low E-value used will almost always provide a very good alignment without any heuristic.where Rim progression, but continually re-aligned all sequences back to the new model after every iteration. PS3 and PS4 tested a much less controlled iteration, in which the heuristic parameters were essentially turned off, as well as a more rapid E-value progression .At the end of the iterative cycle each SCOP domain had a matching sequence alignment corresponding to the sequence space surrounding it, as well as a profile HMM that provided a model of the alignment for purposes of sequence searches. At this point, the model was stored as a single-master HMM for later use in the benchmarking experiment, while the sequence alignment was used in the creation of SLAHMMS.Structure alignments were used to generate a high-quality multiple sequence alignment of the SCOP master domains for each superfamily. The sequence alignments to each SCOP master were then grafted on to the structure alignment scaffold by using the one-to-one correspondence of the SCOP master sequence in both alignments. In most cases, the SCOP master did not participate in some columns of the sequence alignment; these columns were removed prior to the merging procedure. In large superfamilies, the resulting alignment could contain thousands of sequences.The superfamily alignments generated by this merging process often had redundant copies of sequences from the nr60 database. These redundancies occurred because several SCOP masters detected and incorporated the same sequences as they iteratively build alignments. This was expected and desirable, as it meant the HMMs were reaching far into the available sequence space. The redundant sequences were removed such that only the best instance of each sequence was retained.The best instance of a sequence in an alignment was chosen based on its relationship to the SCOP master that retrieved it during the initial iterative alignment construction. The aim was to select the instance that was likely to be in the highest quality alignment with its SCOP master. This was determined using a set of cascading tests using the data stored during the iteration runs. In order, the tests were: iteration cycle at which the sequence was added to its alignment, the E-value it matched the model with at the time of its addition, and the length of the sequence fragment (a longer length implies a better SCOP master partner for the sequence). The sequence instance which scored best in these tests was retained in the superfamily alignment, and all other instances of the same sequence were removed. Once the redundancy had been removed from the alignment, it was suitable for submission to HMMER to generate a SLAHMM representative of the entire superfamily.prior to the redundancy removal described above, so that the maximum number of sequences collected without the use of the removed domain could be retained. The net effect of the procedure was to create alignments that essentially were constructed as if the removed SCOP master was never present. The SLAHMMs could then be directly compared with the single-master HMMs.In normal use, it is expected that a single SLAHMM model would be generated and stored. However, for purposes of benchmarking it was necessary to remove a SCOP master and all of the sequence it detected from the alignment, so that that domain could be tested against the (now uncontaminated) model in a leave-one-out test. A version of the alignment for each SCOP master was generated, which was missing that master and its detected sequences. This was done a posteriori method for match state generation [hmmbuild. The use of 25 match states as a cutoff was arbitrary, but seemed reasonable based on reviews of the alignments.When producing SLAHMMs with HMMER, difficulties were encountered in a few superfamilies. It appeared that the cause was the input alignments, as these problems were not encountered with the single-master alignments. In some superfamilies, the superfamily-wide alignments were very sparse, with large sections filled mostly with gaps, and areas of conserved structure with little or no recognizable sequence conservation. This type of alignment occasionally caused severe model shortening with HMMER where, despite the input of a large alignment, HMMER would generate a model containing less than 25 match states. In such cases, this problem was addressed by turning off the more sophisticated maximum neration  and usinhmmbuild for three alignments in both PS3 and PS4. Therefore, in order to provide PS3 and PS4 with the same amount of models as PS1 and PS2, models from PS1 and PS2 were copied over to their closest partners  for these cases. In three cases, the models did not make any assignments in the subsequent benchmarking studies. Thus, the added models should have no notable effect on the results for PS3 and PS4. The net result of the above construction steps was 1575 single-master HMMs and 1575 SLAHMMs  for each parameter set.HMMER crashed when attempting to build a model with During collection of the final results Figures , 3, 4, Hone possible correct SLAHMM hit . Probe-centric scoring allowed for direct comparison of the number of correct assignments between the two methods. All incorrect assignments for a given probe where counted when determining coverage vs. error [The experiments were run by using each SCOP master sequence as a probe against all of the models of each type, as well as a combined database containing all the models. Structure assignments were scored in a "probe-centric" fashion, in which each SCOP domain could only be correctly assigned once. This was done because for each of the 1575 domains, there were at least two correct superfamily-level hits available in the case of single-master HMMs, but only s. error ,40, withdifferent superfamily but the same fold were counted as correct hits, and matches of a probe to a model in the same superfamily were ignored . The requirement for a cross-superfamily match meant that some of the probes from the superfamily tests could not possibly be correctly assigned in the experiments (because no models were available from another superfamily in the same fold); these were removed, leaving 675 probes (and possible correct assignments). To expand the size of the test set, we added additional probes from smaller superfamilies which had not been used to make any models because they did not have sufficient structural representatives to meet our initial superfamily filtration criteria. Inclusion of these additional test sequences increased the size of the test set substantially, from 675 sequences from 88 superfamilies to 933 sequences from 291 superfamilies. Thus, the reported theoretical EPQ values for fold recognition .Underpredictions refer to cases where the SCOP authors did not group similar structures into the same fold grouping, even though structural comparison indicates a similar overall fold (and perhaps even an evolutionary relationship). Hence, the structures are grouped only into the same vice versa). In both directions, an E-value less than 1 was required. Second, the structures providing the sequence match (both from a model and probe standpoint) were required to align with a Z-score of 4 or greater using CE [To determine which fold-level pairings would be provided such special treatment, we adopted a standard based on both sequence and structural characteristics. First, the HMM match between domains at the sequence level was required to be bi-directional . It seems likely that the regular α/β alternation in these domains produced a distinctive generalized sequence signal that our HMMs detected. It is unclear if these matches were simply the result of analogy or a true homologous relationship. To our knowledge, detectable sequence similarity between these folds has not been previously reported.Artifact generation refers to cases where, because of a SCOP domain's architecture, models built using that SCOP domain incorporated sequences related to other SCOP domains. Then, when the experiment was run, these models correctly recognized these other SCOP domains, but this recognition was treated as an "error" because these domains were not part of the same SCOP superfamily as the domain used to initiate the model.Specifically, such artifact generation occurred where a protein consisted of two interacting fragments that flank a central fragment. In some cases, SCOP split such a protein into two compact domains, with one domain consisting of the central fragment, and the other consisting of the two flanking regions joined together at the break points into a single "composite domain". When this flanking domain was used to build an initial ClustalW alignment in the protocol described above, the central fragment was incorporated via the homologous sequences brought into the initial alignment . The resulting models then recognized the central fragment as a homolog.Only two superfamily pairings displayed these types of errors for multiple domain and model matches. In one pair, the domains of b.92.1  flank those of c.1.9 . In the other, some  domains of c.4.1 (nucleotide binding domain) flank those of c.3.1 (FAD/NAD(P)-binding domain). These matches were therefore ignored in the collection of the results.Cases involving other superfamilies were also detected, but in every one of these cases, the errors did not occur for multiple probes and models across the superfamily. They only occurred for a probe and its corresponding flanking domains with the same PDB ID and chain designation. Therefore, all of these cases could easily be dealt with by adding a simple prohibition to the collection of results: any errors where a probe and matching model shared the both the same PDB ID and chain were ignored. This prohibition automatically removed most artifactual matches, while having a minimal effect on the collection of legitimate errors .superfamilies scored higher?Although the methods above removed all obvious cases where correct hits were mislabeled as errors, it was important to institute a generalized method to help suppress the effect of these sorts of errors, in the event that some were missed. This was done by limiting the amount of errors counted for a given probe to one per incorrect superfamily match. This had the effect of allowing artifacts and underpredictions to only add one incorrect match to the total for each probe, as opposed to several. At the same time, it would be expected that limiting counted errors in this fashion would only have a small effect on the counting of legitimate errors, as these sorts of errors tend to be random and so come from multiple different superfamilies for a given probe. As described above, only the first correct match of probe to its superfamily was counted. By only counting each incorrect superfamily once per probe, the experiment effectively asked the question: at a given score cutoff, has this probe been assigned to the correct superfamily, and how many incorrect ES conceived of the study, performed the experiments and drafted the manuscript. PB helped conceive the study and edited the manuscript. Both authors read and approved the final manuscript.Click here for file


The performance of alignment programs is traditionally tested on sets of protein sequences, of which a reference alignment is known. Conclusions drawn from such protein benchmarks do not necessarily hold for the RNA alignment problem, as was demonstrated in the first RNA alignment benchmark published so far. For example, the twilight zone – the similarity range where alignment quality drops drastically – starts at 60 % for RNAs in comparison to 20 % for proteins. In this study we enhance the previous benchmark.The RNA sequence sets in the benchmark database are taken from an increased number of RNA families to avoid unintended impact by using only a few families. The size of sets varies from 2 to 15 sequences to assess the influence of the number of sequences on program performance. Alignment quality is scored by two measures: one takes into account only nucleotide matches, the other measures structural conservation. The performance order of parameters – like nucleotide substitution matrices and gap-costs – as well as of programs is rated by rank tests.Most sequence alignment programs perform equally well on RNA sequence sets with high sequence identity, that is with an average pairwise sequence identity (APSI) above 75 %. Parameters for gap-open and gap-extension have a large influence on alignment quality lower than APSI ≤ 75 %; optimal parameter combinations are shown for several programs. The use of different 4 × 4 substitution matrices improved program performance only in some cases. The performance of iterative programs drastically increases with increasing sequence numbers and/or decreasing sequence identity, which makes them clearly superior to programs using a purely non-iterative, progressive approach. The best sequence alignment programs produce alignments of high quality down to APSI > 55 %; at lower APSI the use of sequence+structure alignment programs is recommended. Correctly aligning RNAs in terms of sequence and structure is a notoriously difficult problem.O(nm3) in time, and O(nm2) in space, for m sequences of length n. Thus, most structure alignment programs  implemeDYNALIGN , JPHYDITDYNALIGN , RALEE O MathTyBased upon these results we now provide recommendations to users on the current state of the art for aligning homologous sets of RNAs:1. Align the sequence set with a (fast) program of your choice.2. Check the sequence identity in the preliminary alignment:• if APSI ≥ 75 %, the preliminary alignment is already of high quality;• if 55 % < APSI < 75 %, realign with a good sequence alignment program; at present we recommend MAFFT (G-INS-i) , We hope that the BRAliBase 2.1 reference alignments constitute a testing platform for developers, similarly as the BRAliBase II was already used for parameter optimization/training of MAFFT , MUSCLE k ∈ 2, 3, 5, 7, 10, 15) and for all programs given in Table The database, which consists of 18,990 sequence set files plus their reference alignments, and scripts used for benchmarking are available . Plots sFor the construction of reference alignments we used "seed" alignments from the Rfam database version 7.0 ,23. In mk ∈ {2, 3, 5, 7, 10, 15} sequences for a given average pairwise sequence identity range . Theref used in  consistsJust as in the previous BRAliBase II benchmark  we used In  we used As both scores complement each other and are correlated, we use the product of both throughout this work and term this new score BRALISCORE.The software package R  offers nFor statistical analyses we computed the BRALISCORE for each alignment. To rate the alignment programs or program options, we ranked these scores after averaging over all datasets. Because the score distributions cannot be assumed to be either normal or symmetric, we used as non-parametric tests the Friedman rank sum and the Wilcoxon signed rank test. R's Friedman test was accommodated to calculate the ranking. Afterwards the Wilcoxon test determined which programs or options pairwisely differ significantly. As already shown in  programsThe following program versions and options were used:ClustalW : version 1.83[sion 1.83default: -type=dna -alignGO -gapopen=GO  -pwgapext=GE -gapext=GE gap-opt: -type=dna -align -pwgapopen=MATRIX -pwdnamatrix=MATRIXsubst-mat.: -type=dna -align -dnamatrix=MAFFT : version 5.667[ion 5.667default: fftnsdefault: ginsiold: fftns --op 0.51 --ep 0.041old: ginsi --op 0.51 --ep 0.041MUSCLE : version 3.6[rsion 3.6,26-seqtype rnaPCMA : version 2.0[rsion 2.0POA : version 2[version 2MATRIX-do_global -do_progressive prank : version 270705b – 1508b[b – 1508bGR -gapext=GE-gaprate=ProAlign : version 0.5a3[ion 0.5a3java -Xmx256m -bwidth = 400 -jar ProAlign_0.5a3.jarProbConsRNA : version 1.10[sion 1.10Prrn : version 3.0 (package scc)[kage scc)The author(s) declare that they have no competing interests.A.W. developed the BRAliBase 2.1 and performed the benchmark; I.M. developed the ranking tests. All authors participated in writing the manuscript.