Alignment-free methods are alternatives to alignment-based methods when searching sequence data sets. The output from an alignment-free sequence comparison is a similarity score, the interpretation of which is not straightforward. We propose objective functions to interpret and calibrate outputs from alignment-free searches, noting that different objective functions are necessary for different biological contexts. This leads to advantages: visualising and comparing score distributions, including those from true positives, may be a relatively simple method to gain insight into the performance of different metrics. Using an empirical approach with both DNA and protein sequences, we characterise different similarity score distributions generated under different parameters. In particular, we demonstrate how sequence length can affect the scores. We show that scores of true positive sequence pairs may correlate significantly with their mean length; and even if the correlation is weak, the relative difference in length of the sequence pair may significantly reduce the effectiveness of alignment-free metrics. Importantly, we show how objective functions can be used with test data to accurately estimate the probability of true positives. This can significantly increase the utility of alignment-free approaches. Finally, we have developed a general-purpose software tool called KAST for use in high-throughput workflows on Linux clusters. The rapid growth in genomic data sets poses problems for sequence search and comparison. The most frequently used methods for sequence comparison are based on sequence alignment. They include very well established methods such as BLAST , and algMany different alignment-free scoring methods are described in the literature and they have been the subject of a number of recent reviews. In , a numbek-mer frequencies, as we explain below. We have been motivated to understand how to interpret the outputs of these methods, i.e. their scores; to gain insight into why they might fail, and the situations when they are most successful. Here we provide a brief overview of the rationale behind the alignment-free metrics we have explored in this paper, and mention some of the previous analytical studies that have interpreted and characterised their score distributions.In this paper, we have attempted to restrict our focus to methods that are based on r and query q sequence that can be decomposed into DNA words typically called k-mers of a specified length k in base pairs. For example, if k = 4 then a DNA sequence of 4 letters A, C, G, and T can be decomposed into 4k = 44 = 256 possible words or 4-mers (GGCT or AAGA are examples of 4-mers). The D2 statistic is a sum over all possible k-mers, applied to the product of counts for a k-mer $w$ in the two sequences i.e.:One of the best characterised alignment-free scoring approach is the D2 metric or D2 statistic . The D2 Cr and Cq are the counts (number of occurrences) of k-mers in the sequences r and q, respectively. Because this statistic is based on counts, it can have an arbitrary value that will tend to increase with the length of the sequences under consideration. Various modifications have been proposed to the D2 metric to give it a more practical value, e.g. to scale the scores output from the metric so that they lie between 0 and 1, where lower scores represent more similar sequence pairs. For instance, the vector cosine distance can be written as:Here, D2 metric rather than giving a value between –1.0 and 1.0, it may be adjusted to give a number between 0.0 and 1.0.Further modifications may then be applied to the vector cosine metric by different authors to obtain more convenient scores. For instance, in the k-mer is often used instead of the counts. K-mer frequencies have been shown to generate signatures unique to different organisms . The metrics we have investigated in this paper are based on k-mer frequencies. Once the frequency vectors have been calculated, various distance measures can be used to estimate similarity, which include the Euclidian, Manhattan, Canberra or Chebyshev distances   ,25. Thesd2Shepp) . For exai and j, as:Ci, j is a sum of the species (i.e. k-mers) common to both sites (i.e. sequences), with the sum using whichever species count is the lesser between the two sites; the Si and Sj are the total counts of species at each site. An important assumption here is that the two sites are of a similar size. The Google similarity distance is used to find semantic similarity in web pages. It implements the idea that related key words with similar meanings should be ‘close’ according to this measure  and min are the greater value between Si and Sj, or lesser, respectively. When formulated this way it performs in a similar manner to the Bray-Curtis dissimilarity.Here, the max, thus preserving biological context; but this is not true for alignment-free methods.On one hand, sequence alignments are often very large and complex data sets. They may incorporate thousands or even millions of sequences and may be used for a range of sophisticated applications including predicting protein structure or elucidating evolutionary patterns .On the other hand, when using an alignment-free metric, the result of comparing two sequences is nothing more than a number, i.e. the similarity score. It is important to realise that at this stage all knowledge of the input sequences has been lost: for a single pairwise comparison the similarity score has no biological meaning at all. It is only when data sets consisting of three or more sequences are processed that the alignment-free scores can provide some biological insight.E-value from BLAST – instead they only measure relative similarity within the set of sequences being searched. The principle output from alignment-free search is therefore the ranking of the similarity scores . As we highlight in this paper, each metric measures ‘similarity’ in different ways, sometimes with significantly different assumptions and biases.For example, for a set of three sequences , the relative difference in the scores might indicate that sequence pair A and B are more similar than sequence pair A and C. Alignment-free metrics, however, do not provide an objective measure of similarity such as, for instance, the We propose that an objective  function should be applied to replace the biological context that has been lost in alignment-free search. This function can be used to map the pair of sequence identifiers associated with a score onto external knowledge bases that provide biological context. Using the knowledge-bases, it will encode a particular set of criteria relevant to the specific search problem performed by the researcher, thus allowing sequence pairs that are true positives to be identified within that domain. Without an objective function, it is still possible to generate a distribution of scores, but there would be no basis to identify true positives within that distribution and so no basis to evaluate how effective the search algorithm is. We demonstrate how an objective function can be used to calibrate alignment-free metrics for different applications using test data. As a result of this process, similarity scores can be mapped onto likelihood values, indicating the probability of a correct prediction.We have performed empirical studies to characterise the properties of the score distributions generated by a number of different alignment-free scoring metrics that use k-mer frequencies. We have used objective functions to identify true positives within those score distributions. The advantage of our empirical approach is that it is relatively simple to implement and it does not require the specialised statistical background that may be required to engage with some of the analytical approaches described above. Moreover, it may be readily applied to both nucleotide and protein sequence.To aid our investigations we developed the software KAST. This was needed due to the lack of other suitable alignment-free software that could process both DNA and protein sequences. KAST was designed for efficient execution and to provide suitably flexible input/output formats. This allowed us to embed it in data analysis pipelines so that we could explore a variety of practical investigations. It is designed to be a general-purpose tool for alignment-free sequence comparison. While a number of software packages exist , they teHere we describe the data sets, objective functions, and analyses that we have performed. We have split this into two sections, one for amino acid (i.e. protein) sequence and the other for nucleotide sequence. The list of metrics we have investigated is given in Table Schizosaccharomyces pombe or fission yeast with 5138 proteins . Tables of orthologs could then be retrieved from DIOPT. Finally, the DIOPT data sets were adjusted so that the identifiers used by DIOPT matched those in the FASTA headers of the protein sequences. Thus, it was possible to map between the FASTA headers and the ortholog data from DIOPT.The evaluation process involved running the alignment-free sequence comparison with sets of protein sequences from two different species (e.g. yeast species Y1 and Y2). A protein from species Y1 was compared against all the proteins from species Y2 using a single alignment free metric. The scores were ranked. Then, using the objective function, a positive prediction was called if the top ranked pair of proteins was present in the set of DIOPT orthologs, otherwise it was a false prediction. The process was repeated for every protein in species Y1 so that the number of correctly predicted orthologs could be counted. This was done in two stages: first we perform the alignment-free comparison on all the proteins from species Y1 using the KAST software (presented later in this paper), then we evaluate using an objective function that counted the correct predictions using Python scripts. True orthologs were counted if the DIOPT evidence was one of ‘high’, ‘moderate’ or ‘low’.k-mer lengths. We therefore used subsets of this data, including 1/10th and 1/30th of the data for D. melanogaster. For instance when the 1/30th data set is used (1107 proteins) with all the data from C. elegans (28 400 proteins) this results in 28 882 800 comparisons, a similar number to the yeast system.The following investigations of predictive accuracy were performed. In each of these tests, all the sequences from one FASTA file are compared against all sequences in another FASTA file. This results in 5138 × 6713 = 34, 491, 394 comparisons for the fission yeast versus brewer’s yeast case. For the fly-worm case, the numbers get large and time-consuming, especially for the larger We investigated this by evaluating the different metrics in two different ways:By using the top hit only, i.e. the optimal ranked pair of proteins based on the scores.By generating the receiver-operator curves (ROC) and calculating the area under the curve or AUC.The frequency of occurrence of different scores for a variety of metrics and k-mers was explored. If the maximum and minimum possible scores fall between 0 and 1 (which it does for most of the metrics) then that score range might be divided into 100 bins and the number of scores falling into each bin counted. The score frequency for a histogram bin is calculated by dividing each of the 100 bin counts by the total number of scores . The histograms indicate how the scores distribute over the range of allowed scores. Two different types of score frequency histograms were calculated:All-scores distribution: this is when all sequences in the two species are compared to each other and consists of both positive and negative scores. Negative scores dominate the frequency values in this histogram because they vastly outnumber the positive scores. For example for the 34 491 394 comparisons made in the two yeast case, there are 7090 pairs that are classified as true orthologs by DIOPT .Positive-only distribution: this distribution is based on the set of sequence pairs corresponding to true positive scores according to DIOPT. Only unique pairs were included (i.e. one-to-one mappings), meaning that if a protein from the first species of the pair mapped to multiple proteins in the second species, then only the first such ortholog mapping was included. It consists of 4409 pairs for the yeasts system. Note that this distribution is a subset of the all-scores distribution.A and B, the overlapping index η is given by the sum, over all scores s, of whichever distribution has the minimum score-frequency f, i.e.We have used the histograms of score frequencies as a basis to empirically investigate and compare the behaviour of various scoring metrics and parameters. To summarise these results into a single figure, we calculated the fraction of overlap between two score histograms. This has previously been described as the overlapping index : for twoWe have calculated correlations between the length of the protein sequences (the mean of the pair) and the score, to investigate if these are independent from each other. This was performed in R version 3.1.2  using thD. melanogaster data.We calculate the cumulative difference in score frequency between two score histograms, i.e. that of the positive-only and the all-scores. This can be used with an objective function to identify a specific score that maximises positive scores whilst allowing most negative scores to be filtered out. For instance, the difference between the curves is calculated by iterating over each of the bins in the two histograms that correspond to the same scores, and subtracting the positive-only score frequencies from those from the all-scores. The sum of these differences gives the cumulative difference curve and the maximum of this curve gives a score that may be used as a cut-off score. To map scores from pairwise comparisons onto fractional values indicating the likelihood of a prediction being correct, we have taken the output from KAST and stored the highest ranked sequence pairs and their scores. By iterating over the range of possible scores, usually from 0.0 to 1.0, it is possible to quickly evaluate predictive accuracy at a range of different scores. This allows the selection of a score that gives a desired level of predictive accuracy – assuming that only the top ranked score is of interest. We performed this using the fly-worm system with 1/10th of the To develop an objective function for taxonomic assignment with DNA sequences, we used the data set generated by Genometa of whole (or nearly whole) genome sequences . A numbeIn order to evaluate taxonomic comparisons between these two data sets taxonomic information was downloaded from the NCBI taxonomy database  and an iThe evaluation process involved using alignment-free metrics to score similarity between sequences from the strain data set when compared to sequences from the species data set. Similar to the protein case, one sequence at a time from the strain data set was compared against all sequences in the species data set. The scores were ranked. The objective function called a positive prediction if the sequence identifiers belonging to the top ranked pair of sequences mapped to the same taxonomic level, e.g. the species or genus level. After performing all pairwise sequence comparisons using the KAST software, the correct taxonomic predictions were counted using the objective function.L were created by dividing the genome sequence into ∼10 equal sized intervals. To avoid overlaps between reference and query fragments, and to sample along the length of the genome, the five query and reference fragments were taken from alternating intervals. This was to reduce biases in the data that may arise because many genome sequences are recorded from the origin of replication: they may therefore be considered to be semi-aligned, and this can significantly inflate the accuracy achieved with the shortest fragments. Each fragment was created by taking the first L bases from each of those intervals, where L was initially set to 100 bp, and then doubled until reaching 102,400 bp. In order to reduce the complexity of our results, only fragments of length L equal to 0.1, 0.4, 1.6, 6.4, 25.6 and 102.4 kb were included in the figures. Thus for a fragment size of 0.1 kb there are 5 × 1052 = 5260 sequences in the species data set, and 5 × 653 = 3265 in the strain data set; and when these data sets are compared to each other 17 173 900 comparisons are made. There may be slightly less sequences in the data sets for longer L because a few of the genome sequences were incomplete and of insufficient length to generate the required fragments.We investigated a number of different parameters using the following data sets. Fragments of different sequence lengths were extracted from each of the species (reference) and strain (query) data sets. Five fragments of the length We investigated how the following parameters affect predictive accuracy.and how this may change with different sequence lengths, for different metrics.This was investigated in the following ways:L-equal: here the reference and query sequences are of equal length. There are six different data sets corresponding to the six different fragment lengths L.L-unequal: here the query sequences are allowed to vary over the six different fragment lengths but the reference query is fixed at the first 250 000 bp. The original idea was to include the whole reference genome sequence, but this led to cumbersome data sets and little difference in performance. There are six different data sets corresponding to the six different query lengths L.The calculation of this histogram is similar to that described under the equivalent paragraph for proteins. Two different types of score frequency histograms were calculated:L-equal and L-unequal): this is when all sequences in the species and strain data sets are compared to each other and consists of both positive and negative scores. There is a different distribution for each of the 6 different fragment lengths L. Negative scores dominate the frequency values in this histogram because they vastly outnumber the positive scores. For example if 5 × 1052 species sequences of length 0.1 kb are compared against 5 × 653 strain sequences of length 0.1 kb this gives a total of 17 173 900 scores, of which 5 × 5 × 493 = 12 325 sequences would be true positive scoresAll-scores distribution : this consists of the strain fragments of a particular fragment length  matched to the whole genome sequences in the species data set. Only the first 250 kb of the whole genome sequence was used to reduce file sizes and compute speed. A positive pair shared the same taxonomic identifier: using ‘species’ as the taxonomic identifier resulted in 5 × 493 = 2465 sequence pairs (consisting of 187 species). For comparison, using the ‘genera’ taxonomic identifier gives 22 300 matched sequence pairs (consisting of 121 genera that now encompass approx. 440 species). There is a different file for each of the 6 different fragment lengths L.Positive-only distribution  as well as gzipped versions. Output can be provided as a distance matrix (for use with phylogeny applications) or in a number of different tabular formats. The scores are ranked in the KAST outputs.Currently over 10 different scoring metrics are implemented in KAST. These include the cosine metric; the d2 variant of the cosine metric; distance measures such as Canberra, Manhattan, Chebyshev, and Euclidian; the Bray-Curtis  dissimilEach of the metrics can be used with binary masks to implemented gapped or spaced k-mers . For exaWe have also added parameters to filter sequence pairs with dissimilar lengths, and to apply a cut-off score.For the KAST distribution, please see the Data Availability section.We first consider the performance of various metrics on protein ortholog prediction, then we consider their performance for the taxonomic classification of DNA fragments.S. pombe) and (S. cerevisiae); and comparisons between two much more distantly related species, a fly (D. melanogaster) and a worm (C. elegans). These data are described in the materials and methods. In Figure k-mer lengths 1 to 4. Using k-mers of length 5 does not improve on the results shown here. We have measured the accuracy of ortholog prediction in two different ways: firstly, by considering whether or not the pair of sequences giving the highest ranked score is a true ortholog; and secondly by calculating the AUC or area under the curve for the receiver-operator curves (ROC curves). The data for this figure are given in In Figure k-mer length 4 the fraction of correct predictions is 65% for NGD in the yeasts, and this falls to 52% for NGD in the fly-worm system. It is also worth noting that the fly-worm system involves identifying the correct ortholog out of a data set of 28 400 C. elegans proteins, whereas the this number is considerably smaller at 6713 for the yeasts.The yeasts comparison tends to give more accurate results than the fly-worm comparisons, as might be expected due to the smaller evolutionary distance in the yeast system. With k-mers , while the other four metrics tend to show improved accuracy with longer k-mers. According to Figure k-mer length. This is in contrast to the Manhattan, NGD, BC and d2 metrics, and these four metrics behave in a similar manner to each other. In the yeasts, ortholog predictions based on the fraction correct increase from 16% to 25% for k-mer length 1, up to 54% to 65% for k-mer length 4 (with D2 giving slightly worse performance here).Three of the metrics clearly show decreased accuracy for longer k = 1: here the AUC has a value of 0.911 in the yeast system and it compares favourably to the best metric with these parameters, which is BC with 0.913. Performance, however drops off significantly for the Canberra metric with larger k-mers, with the AUC falling to 0.480 and 0.463 (slightly worse than random) for k = 4, which compares to 0.928 and 0.822 for BC, in the yeast and fly-worm systems respectively.The Canberra metric gives good performance for D2 and Manhattan metrics in the yeasts comparison. This minimum of the AUC curve can be at least partially explained using factors that we explore in the subsequent subsections. This includes the fact that scores for the metrics are not independent of the length of sequences being compared, but instead may correlate with this parameter.There is a minimum in the AUC curve for most metrics at k-mer length 2. This is most pronounced for the k-mers 1 to 4. These metrics are shown because, of the metrics represented in Figure K = 2, with the smallest dip in the curve at this point. When comparing the distributions for Euclidian, it is clear from Figure k-mer length increases there is much less difference between the ‘positives’ and ‘all-scores’ distributions. The overlapping index between the two distributions indicates how similar these distributions are. The more similar they are, the less successful the metric is at identifying signal from noise. Ideally, the true orthologs or ‘positives’ would have a distribution with minimal overlap to those of the ‘all-scores’.In Figure k-mer length 1 tend to be distributed closer to 0.0, the bottom of the range of possible scores; while for k-mer length 4 the scores tend to be distributed close to 1.0, the top of the scale. There is also a relatively clear separation  between frequency of score distributions for ‘all-scores’ and those for ‘positives’, across all k-mer lengths, and the separation increases slightly for longer k-mers. This corresponds to better predictive performance being achieved with longer k-mers.In contrast to the Euclidian metric, the distributions for NGD have greater spread over the range of possible scores and the ‘positives’ and ‘all-scores’ distributions are more clearly distinguishable. Scores for Distributions for the other metrics are available as figures in the supplementary information, k-mer length 2, which corresponds to the minimum of the AUC curve in Figure k-mer length 2, leading to distributions centred on the mid-point of the range of scores and which are spread over the full range of scores, e.g. from 0.0 to 1.0. These distributions have relatively large overlaps. As k-mer lengths increase, the score distributions tend to be pushed towards the extremes of the range of allowed scores, the distributions are relatively compressed, and the overlaps are smaller. The smallest overlap for these four metrics is for k-mer length 4, shown in purple in Figure In Figure k-mers. Figure In contrast, the Euclidian, Chebyshev and Canberra metrics tend to have worse performance with larger k-mer lengths 1 to 4 we have used a dot plot to indicate the relationship between the mean sequence length and the score, for each protein pair. We have calculated the correlation between mean length and score using Spearman’s rank correlation. The correlation values are given in the figure legend and are weakly anti-correlated for k=1 and strongly anti-correlated with values below –0.8 for k-mers 3 and 4. Thus for the Euclidian metric, which gives poorer performance as the k-mer length increases, the scores for longer k-mers are dependent on protein sequence length. For longer k-mers, if the mean length of the protein pair is relatively short, then the score is likely to be high, predicting a relatively dissimilar pair; while if the mean length of the protein pair is relatively long, then the proteins are predicted to be relatively similar. Ideally, there should be zero correlation between score and sequence length.In Figure k-mer lengths remains the same.These results are for the fly-worm system and figures for other metrics are given in k-mer lengths and scoring metrics. For the D2, BC, NGD and Manhattan metrics the highest correlation is given for k-mer length 2 . As mentioned previously, the Euclidian metric shows significant correlation for all k-mer lengths, with k-mer length 1 giving the lowest correlation magnitude, while the Canberra metric gives very high correlation for k-mers lengths 3 and 4. This metric’s accuracy is very poor  they are likely to have a very small number of possible 4-mers in common; (b) the number of 4-mers present in one sequence but not present in the other is likely to be higher; while (c) the vast majority of 4-mers will be absent in both sequences. The number of shared 4-mers will tend to increase with longer sequences and this may be basis of the observed correlation, although other factors may be involved. The selection of the distance metric used needs to be considered carefully in these circumstances because the usual assumption is that the majority of counts are non-zero. In these cases, where the majority of counts are zero, the Euclidian distance behaves poorly, the Manhattan distance behaves considerably better, and the Canberra distance only suffers from correlation and poor performance with the longer k-mers.Note that for k-mer length 2 . The BC and NGD metrics operate in a similar manner. They consider sequence length to be an important component of sequence similarity. This can be a reasonable assumption for protein ortholog prediction, where sequences of protein orthologs are likely  to have similar lengths.The best options for the protein ortholog prediction problem in terms of accuracy are the BC and NGD metrics without any length filters . Manhattan is just as accurate as these two metrics for It is desirable for alignment-free search to be applied with a measure of the probability of success, which may be provided as an input parameter for the search. For example, in the context of protein ortholog prediction, by applying a particular cut-off score we should be able to limit search results to protein pairs with, for instance, at least an 80% chance of being a correct prediction.First, consider that an optimal score threshold can be derived from the score frequency distribution plots shown in Figure k-mers get longer, the peak of the difference curve shown in Figure k-mer length 4, and here almost all the positive scores (over 99%) are equal to, or below, the cut-off score of 0.98.A way of identifying this cut-off score is shown in Figure k-mer length 4, without a cut-off score applied, all correct predictions are included (1081 out of 2147 predictions) and 0 are excluded. This corresponds to a predictive accuracy of 50% (based on the top hits). When applying a cut-off score filter of 0.915 . An issue with species prediction in these data sets is that only 493 of the 653 query sequences have a shared species classification with the reference data set, meaning that the maximum accuracy is 74%. This is higher for genera classification at nearly 99%.Note that for the related problem of predicting the genera (rather the species) the predictive accuracy is significantly higher i.e. around 85% for L-equal: here it is 0.2% and 0.3% for k = 2 and k = 5 respectively with 100 bp. Accuracy increases with fragment length, tending to level off for the larger fragments .Increasing the fragment length leads to greater predictive accuracy. Accuracy is very low for the shortest fragments, especially for k-mer length from 2 to 5 bp may improve predictive accuracy. For shorter sequence lengths, however, longer k-mers may lead to reduced accuracy. Indeed, Figure k = 5 is worse than k = 4 and no better than k = 3 for the smaller fragment lengths in the L-equal case.Increasing the k where k is the k-mer length, and four derives from the number of DNA bases. For example, a k-mer length of 3 bp results in 43 = 64 words and a k-mer length of 5 bp results in 1024 words. Often a k-mer length of 4 bp is a reasonable choice  appear to perform differently to each other. Although they have similar performance in the L-equal case, in the L-unequal case, however, d2Star performs much better than d2S, but the difference is less for the shortest and longest fragments. The d2Star metric performed the best of all the metrics with the L-unequal data set, for k-mers of length 3–5 bp . The D2, Manhattan and Euclidian metrics are similar to each other in terms of predictive accuracy, with the Chebyshev method giving the weakest performance for these predictions.For the metrics that do not compute the background, there are some inconsistencies but they generally behave slightly better in the k-mer length, the sequence fragment lengths, the relative lengths of the sequence pair being compared, and the scoring metric being deployed.The distribution of score frequencies can vary significantly for key parameters used in alignment free sequence comparison. These include the L-equal case for two metrics: d2S and NGD. For the shortest k-mer length, i.e. 2 bp, with both metrics the score distributions are relatively similar for all sequence fragment sizes. For the longest k-mer, i.e. 5 bp, the score distributions are much more dissimilar and distinct, except perhaps for the two longest fragments (L = 25.6 kb and L = 102.4 kb). These results highlight the differences between score distributions that arise from using different sequence lengths . Figures with the distributions for other metrics, and for the L-unequal case, are given in In Figure D2 distribution have only considered cases where the pairs of sequences have equal or similar lengths. There is a risk that an erroneous assumption might arise, i.e. that alignment-free methods operate independently of sequence length. There is some theoretical reasons why this could be true. An influential review . The reference sequences are given as for the L-equal and L-unequal cases (see methods for further detail). Results are given for four different k-mer lengths, 2, 3, 4 and 5. The closer the overlapping index between the score distributions is to 1.0, the more similar the distributions.Figure k = 4 , shows that d2S  there is a significant increase in accuracy between L = 25.6 kb and L = 102.4 kb (from 45% to 53%), even though the overlap index between these score distributions is relatively high at 0.96. Other metrics show similar behaviour. The issue arises because true positives are most likely to be found in the tail of the distribution: relatively small changes in the tails can influence performance and the overlap index does not reflect the similarity between the tails well. It is also worth noting that these sequences differ in length by a factor of 4. Sequences that are more similar in length may still allow for a shared interpretation of the scores. Figures It is tempting to consider that, as the score distributions become more similar, the interpretation of their scores will also become more similar – for instance, in terms of the fraction of correct predictions as given in Figures K = 3 and K = 4 for instance).In certain circumstances, it is advantageous to cut the sequence fragments so that they are all of equal lengths. Then the sequence pairs will generate scores from the same score-frequency distribution, with the same degree of statistical noise, and this can greatly simplify interpretation. This is particularly relevant for the query fragments – if the reference fragments are sufficiently long (e.g. a large chromosome fragment) they may not require any adjustment. The important point is that the sequence lengths influence the distributions of similarity scores for the alignment-free approaches we have explored in this section, with some metrics being more sensitive to this effect than others are. Although in most of our experiments the Chebyshev metric gives poor performance, it is worth noting that it does display the most consistent scoring with varying sequence length, as shown in Figure L-unequal) will sample the k-mers relatively well, potentially compensating for the lesser sampling of the query sequence. In general, this gives more accurate results in our tests, and the overlaps given in Figure L-unequal case than the L-equal case. The overlaps for the BC, NGD and Canberra metrics, however, are very small. For these metrics, using the L-unequal reference sequence length (250 kb) with the much shorter query lengths gives very poor results. There is an assumption in the design of these metrics that sequences will be of similar length. It is therefore important to consider this when using these metrics because they do not function with sequence pairs of very dissimilar lengths.There is no need for the reference and query sequences to be of equal lengths, so long as the difference in length is consistent throughout the data set. A long reference sequence , and comprises 34,491,394 protein sequence comparisons. The speed-ups available when using multiple cores are given in Table k-mer length 4 amino acids can be particularly demanding on computational resources, both for time and RAM (running with 8-cores required 47 Gb of RAM).For reference, we have provided some example KAST timings for protein comparison, using k-mers in the range of 1 to 4 amino-acids, the NGD metric, and outputting the scores for all comparisons (-n 0 option in KAST). These results are based on the two yeasts system We have compared execution times of KAST against ALF and CAFE, with the results shown in Table KAST is similar but slightly slower in execution speed when compared to ALF with 1 core, but just over 7 times faster when 16 cores are used. It is also has a slightly lower RAM requirement when used with multiple cores as compared to ALF. CAFE is more than 10 times slower than ALF, and more than 70 times slower than KAST when used with 16 cores. It does, however, use about 25% of the RAM of KAST and ALF.k-mer counts converted to frequencies. We therefore iterate over all possible k-mers. When using k-mers of 4 amino acids, 160 000 k-mers are possible, and the vast majority of these will have counts (and therefore frequencies) of zero. It would be possible to modify metrics to take advantage of this. It is also possible to take advantage of the strategies implemented in other codes  consider sequence pairs that differ in length to be significantly dissimilar, not because of the content of those sequences, but simply because the sequences are of different lengths. While this is well known for the BC metric due to its established use in ecology , it is nWhen predicting protein orthologs this sensitivity to relative sequence length may not be a problem, because reported orthologs often, but not always, have similar lengths: indeed, NGD and BC do very well in these cases. However, this behaviour is certainly problematic when considering the taxonomic classification of fragments of whole genome DNA, for instance from a genome or metagenome assembly, that may differ in length by orders of magnitude.k-mer frequencies in alignment-free analysis applied to both DNA and protein sequence. Some care is required concerning how frequencies are implemented in practice. Strictly speaking, a frequency should be a continuous rather than a binary variable. Ideally, this means that each k-mer should have counts much greater than zero in each sequence—and this requires relatively short k-mers and relatively long sequences. While this is often achievable for DNA sequences, it is much more difficult with protein sequence due to its larger alphabet and shorter sequences.We have attempted to focus on the concept of k-mer frequencies, it is possible, through a combination of the k-mer length and sequence length parameters, to inadvertently create a situation where the vast majority of k-mers are absent, those that are present have very low counts, and the numbers of shared k-mers between two sequences could be minimal. In these cases, the k-mer frequencies would be represented as binary values of presence or absence, rather than as more continuous variables. This may not be obvious to the user and can result in unexpected behaviour that significantly impairs the performance of the metric.Although a metric may be designed for The situation is likely related to what is known in machine learning as the ‘curse of high dimensionality’. With such high dimensional data, the concept of distance may be unclear because metrics behave differently to what seems intuitive based on 3D space . In suchk-mer length 1, when k-mer frequencies are almost all above zero and most are shared between the sequence pair; it is at best random with k-mers of 3 or more amino acids, when few k-mers are shared between sequences, and those that are shared will rarely have counts above 1. We have shown that in this situation its true positive scores are highly correlated (negatively) with sequence length. The Euclidian and Chebyshev metrics also do poorly in these high-dimensional spaces, while some metrics do well . Due to the smaller alphabet in DNA sequence, k-mers of length 4 or 5 bp do not usually suffer from these problems (unless perhaps the sequences are extremely short in length).For example, with the longer k-mers we have used with protein sequences, the Canberra metric undergoes an unexpected drop in performance as the k-mer length increases, and the dimensionality of the space increases. While Canberra is a competitive metric for k-mers should not be classified as alignment-free methods at all.Some authors have classified methods that use the binary, presence or absence of k-mers differently from those based on frequency-based distances ,68. IndeWe have calibrated alignment-free metrics using an empirical approach with objective functions and test data. This provides a relatively convenient way to generate score distributions corresponding to different parameter sets. In particular it allows us to isolate true positives and their score distributions. An important outcome is that these distributions may be used to calculate probability values for predictions made on unseen data sets with similar characteristics to the test data.Calibrating alignment-free metrics in this way is highly advantageous for downstream processing. It provides a direct indication about how effective the search or sequence comparison procedure will be. However, this may not be straightforward: high quality test data can be time-consuming to curate, and developing an objective function to identify true positives may also require some time and technical skill. Nonetheless, without calibration there is a significant risk that the alignment-free metric and parameters chosen may be sub-optimal or even inappropriate for the purposes of the researcher.Finally, due to the lack of alternatives, we have developed the KAST software as a general-purpose tool for fast and efficient alignment-free sequence comparison. This is freely available, along with various scripts, data sets and analyses that we have used in this paper (see the Data Availability section).https://pure.aber.ac.uk/portal/en/ (search with the title of this paper). There we have made available the scripts we used to make figures in this paper, including additional figures with different parameter sets to those shown in the main body of the paper. Alternatively, the DOI of the data set is: https://doi.org/10.20391/3e9573af-363b-4ff7-9f27-72eb84440b68.An example set of Linux and Python scripts, including data and the KAST executable, that perform the main parts of our analysis, are available at The data we have made available for the protein use-case includes (a) FASTA files containing the protein sequences for the yeasts system and the fly-worm system; (b) FASTA files with the true orthologs presented as pairs of sequence for use with the KAST interleaved format; and (c) the associated DIOPT files with the ortholog mappings.The data we have made available for the DNA use-case includes (a) FASTA files containing the DNA sequences for the strain (query) and species (reference) data sets; (b) FASTA files with positive pairs for use with the KAST interleaved format; and (c) the associated files with the NCBI taxonomic mappings.For both these use-cases, scripts are available that run KAST, evaluate the output with an objective function, make score-frequency histograms, generate probability scores from the histograms, and annotate KAST output with the probability scores.https://github.com/martinjvickers/KAST. A manual is also provided: https://github.com/martinjvickers/KAST/wiki.KAST source code is freely available via GitHub: https://hub.docker.com/r/biocontainers/kast. Instructions for the commands to run this are found in the KAST documentation.A docker container with KAST and a ‘lite’ version of the scripts, so-called because they are missing some of the larger data sets available from the DOI given above. This can be accessed at lqac062_Supplemental_FilesClick here for additional data file.


Improving the prediction of structures, especially those containing pseudoknots (structures with crossing base pairs) is an ongoing challenge. Homology-based methods utilize structural similarities within a family to predict the structure. However, their prediction is limited to the consensus structure, and by the quality of the alignment. Minimum free energy (MFE) based methods, on the other hand, do not rely on familial information and can predict structures of novel RNA molecules. Their prediction normally suffers from inaccuracies due to their underlying energy parameters.We present a new method for prediction of RNA pseudoknotted secondary structures that combines the strengths of MFE prediction and alignment-based methods. KnotAli takes a multiple RNA sequence alignment as input and uses covariation and thermodynamic energy minimization to predict possibly pseudoknotted secondary structures for each individual sequence in the alignment. We compared KnotAli’s performance to that of three other alignment-based programs, two that can handle pseudoknotted structures and one control, on a large data set of 3034 RNA sequences with varying lengths and levels of sequence conservation from 10 families with pseudoknotted and pseudoknot-free reference structures. We produced sequence alignments for each family using two well-known sequence aligners (MUSCLE and MAFFT).We found KnotAli’s performance to be superior in 6 of the 10 families for MUSCLE and 7 of the 10 for MAFFT. While both KnotAli and Cacofold use background noise correction strategies, we found KnotAli’s predictions to be less dependent on the alignment quality. KnotAli can be found online at the Zenodo image: 10.5281/zenodo.5794719 Understanding RNA structure is essential to understanding its function. RNA plays an active role in many processes that occur within the cell, such as in transcription , translacovariation. Given enough sequences from a related family and an alignment of high structural consistency, comparative sequence analysis (CSA) has been shown to accurately predict secondary structures [The majority of computational methods focus on secondary structures—the two dimensional structure of an RNA molecule. Due to similar functions, homologous RNA molecules conserve their common structure. Conservation takes the form of compensatory mutations in response to point mutations that would otherwise cause a change in the structure , 9. Compructures . Despiteructures . These pCurrent alignment-based methods couple their covariation with another metric for determining structure and fall into two categories: (1) those that take an unaligned set of sequences and solve the structure and alignment problem concurrently through iterative refinement, and (2) those that take a pre-aligned set of sequences and predict the structure given alignment. Examples of category (1) are algorithms such as locARNA –14, FoldExamples of category (2) are algorithms such as RNAalifold , HxmatchDespite their coupling, these programs still heavily rely on the quality of the alignment to make accurate predictions. In addition, they only predict the consensus structure rather than the structures for all input sequences. Within alignment-based programs, there is an opportunity to address these shortcomings.restricted unpaired bases and define them as unfavorable bases toward the final structure. We force these bases to be unpaired in our predicted structures.In this work we focus on category (2) algorithms and present KnotAli, a novel RNA pseudoknotted secondary structure prediction algorithm which enhances its minimum-free-energy prediction using conserved structural information. Given a sequence alignment of functionally similar RNA molecules, KnotAli finds their individual structures. KnotAli combines two types of information into the prediction. It first uses covariation to find a guide structure and then uses this guide structure to guide the energy minimization step for each sequence that makes up the alignment. We introduce KnotAli’s prediction accuracy was benchmarked against other existing alignment-based prediction algorithms, two that can handle pseudoknotted structures , Cytosine (C), Guanine (G), and Uracil (U). When referring to an alignment of multiple RNA sequences, in addition to the four bases we sometimes observe a “-” (gap) which holds the position of an insertion/deletion (indel) in the alignment. Note that due to indels an alignment might be longer than the RNA sequences—we denote this length as We represent an RNA molecule with its sequence, canonical base pairs. We refer to bases by their position in S. A base pair is then defined as the pairing of two bases i and j where nested if disjoint if When an RNA sequence forms a structure, its complementary bases pair together and form hydrogen bonds. ‘A’ pairs with ‘U’ and ‘G’ pairs with either ‘C’ or ‘U’—termed mentclass2pt{minimmentclass2pt{minimpseudoknotted when at least two of its base pairs, pseudoknot-free structures—see Fig. An RNA structure is considered entclass1pt{minimaentclass1pt{minimaAlgorithms. We start with a high level definition of how the different types of algorithms work and their complexities.Alignment based algorithms such as RNAalifold , HxmatchThermodynamics-based algorithms –29 find MFE pseudoknotted secondary structure prediction is NP-hard , 31 and udoknots , 34 giveIt has been shown that the accuracy of MFE RNA secondary structure prediction decreases with sequence length both for pseudoknot-free  and pseuIn this work we aim to address the mentioned shortcomings. Using the coupling of covariation and thermodynamics, KnotAli is capable of finding possibly pseudoknotted structures in hairpins  and H-tyhairpins  with arbenergy models. KnotAli uses the energy parameters of HotKnots V2.0 [Many algorithms for the prediction of RNA secondary structures, use a set of parameters to calculate the free energy of the structure. These sets of free energy parameters are called ots V2.0 , as theymentclass2pt{minimIn this section we provide the description of our algorithm, KnotAli, in “Mutual Information or x, y at columns a, b of the alignment respectively; similarly, let x at column a and y at column b. We define the mutual information between column a and column b of an alignment, denoted MI, as follows:Our mutual information function is adapted from the MIToolbox . In a stAdjusted mutual information or  sources . The red sources . Averageediction .a and b of a given sequence alignment is defined as:The average product correction for columns ediction .To determine at what point the mentclass2pt{minimmentclass2pt{minimKnotAli’s algorithm incorporates base pair information obtained from conserved structure of homologous RNA sequences into an MFE-based method to predict individual RNA secondary structures for each RNA sequence in the input alignment. In doing so, we bring together three main ideas: 1) selecting a set of intermediary base pairs based on adjusted mutual information, 2) identifying restricted unpaired bases, and 3) relaxed free energy minimization based on a guide structure.KnotAli uses the average column and alignment mutual information, adjusted mutual information . Path 6 first aims to identify only non-nested (i.e. crossing) base pairs given the guide structure. If found, these crossing base pairs are then provided to the “restricted pseudoknotted energy minimization” to predict the output structure. This is to allow for formation of competing crossing base pairs. Following Path 7, first the MFE pseudoknot-free structure given the guide structure is predicted. Then relaxed stable stems  are identified and passed for further modification as in the second method following Path 12. This path aims to allow for formation of competing nested base pairs. Path 8 also aims to allow formation of competing nested base pairs, this time by first opening the loops of the guide structure from outside for each disjoint substructure in the guide structure. Then following the same steps as the third method through Path 10  in the dynamic programming algorithm. In case of restricted pseudoknot-free energy minimization, W is decomposed into three cases: (1) j is unpaired, (2) i and j pair together to form a loop ) and the minimum free energy between i and j that can be handled separately.The energy of the MFE structure between indices W has a fourth case which handles pseudoknotted structures ), as shown in Fig. WMB uses other pseudoknot-specific recurrences  approach uses the base pair scores found before and builds a set of vertices and edges where vertices are positions from 1 to n and the edges are all pairings with a A positive and negative base pairs.Cacofold is an alignment-based method that can handle pseudoknotted structures. Cacofold uses probabilistic folding methods and positive and negative covariation scores to find a consensus structure. Cacofold is part of the R-scape package , 49, 50.E-value is an expectation value signifying the expected number of false positives [E is defined as N is the number of column pairs and x.An ositives . E is deCovariation power is an estimate of the expected ability to detect covariations [riations , 50. Covpositive base pair is a base pair which reports high covariation . In contrast, a negative base pair is a base pair which reports low covariation but high covariation power. Negative base pairs are forbidden to appear in the final structure.A Cacofold groups positive base pairs into nested subsets. The first subset is made up of the maximal number of positive pairings such that there are no crossing base pairs or triplets, and succeeding sets are made up of the remaining positive base pairs. The subsets are used as constraints for the secondary structure prediction algorithms. RNA basic grammar  is used In this section we provide the details of our experiment design.http://doi.org/10.5281/zenodo.5794719.We tested all algorithms on a large dataset with 3034 (pseudoknotted and pseudoknot-free) RNA sequences. This dataset was compiled from the dataset of , 55, by ument}<3 , 46. ThiThe RNA sequences in our database are from ten (pseudoknotted and pseudoknot-free) RNA families with reference structures previously determined through comparative analysis . The pseTo evaluate the structural similarities within differently-sized sequences, the sequences first have to be aligned and gaps placed such that they all have the same length. The strength of a sequence aligner, therefore, plays a fundamental role in the quality of the predicted alignments. In a previous benchmark study , 10 diffMUSCLE tends to reduce the number of gaps within the alignment, whereas MAFFT tends to add an increased number of gaps, especially in instances where there is higher variation within the alignment. Both programs only require a FASTA file as input. No additional parameters were used.true positives (TP) is defined as the number of correctly predicted base pairings within the structure; the number of false positives (FP), similarly, is the number of predicted base pairs that do not exist in the reference structure; and any base missed in the prediction that corresponds to a pairing in the reference structure is a false negative (FN).We evaluate the performance of algorithms based on three measures: sensitivity (Sen), positive predictive value (PPV), and their harmonic mean (F-measure):Sen, PPV, and F-measure are unitless measures that range between 0 and 1. When the predicted structure is the same as the reference structure, their value is 1. In contrast, when PPV and/or sensitivity is 0, there are no common base pairs between the reference and predicted structure and F-measure is set to 0. High PPV describes an algorithm which predicts a small number of false positives.In contrast, high sensitivity shows an algorithm’s ability to overall find base pairs from a sequence. Algorithms seek to maximize both. Therefore, combining both sensitivity and PPV helps to better describe the different strengths of algorithms.We consider the performance of an algorithm to be superior or inferior to another one if the difference in their accuracy is considered significant based on a two-sided permutation test , 63. Thep-value is then the proportion of We take samples (with replacement) from vectors p-value is less than 0.05, we reject the null hypothesis, concluding that the difference in performance of If the calculated p-value of 0.05. As evident in Table The default settings of Hxmatch and Cacofold were used when testing. As mentioned in “We note the difference in output between our algorithm and the others—namely individual structures versus a consensus structure. For comparison, the consensus structure, from the other three algorithms, is applied to all individual structures. When comparing the results of all algorithms to the reference structure, we did not consider non-canonical base pairs as well as loops of size Recall that KnotAli receives a multiple sequence alignment as input and predicts individual structures for each of the sequences in the alignment.To reduce the effect of sequence alignment on performance of KnotAli, we selected two of the best performing sequence aligners, MUSCLE and MAFFT, as explained in “KnotAli’s predictions are guided by the intermediary base pairs—base pair information obtained from covariation in the sequence alignment. To capture covariation we used mutual information as well as adjusted mutual information as it was shown previously that the adjusted mutual information improved accuracy in protein structure prediction . To asseOur second contribution in designing KnotAli is identification of restricted unpaired bases from the sequence alignment. These bases are forced as unpaired in the relaxed energy minimization step of the algorithm. Figure We compared performance of KnotAli . Prediction accuracy of all algorithms significantly decreased on these families. In particular, RNAalifold and Cacofold could not find any of the reference structure base pairs  resulting in 0 accuracy in case of Group II Intron, while Hxmatch found few base pairs of the reference structure (accuracy of 0.018). KnotAli’s performance on the same test case was not ideal but was significantly better than the rest, improving Hxmatch’s by over 8-folds. All algorithms performed better on Group II Intron when MAFFT was used as the sequence aligner and KnotAli stayed in the top spot. KnotAli’s performance accuracy was over 5-folds better than the others on Group I Intron with MUSCLE and over 4-folds better than the rest on Group I Intron with MAFFT. On telomerase, however, RNAalifold and Cacofold outperformed KnotAli using MUSCLE and MAFFT respectively.For the remaining 5 families with moderate sequence conservation level, namely 16s, 23s, RNaseP, SRP, and tmRNA, regardless of the aligner used, RNAalifold performed significantly better on 16s and 23s  while KnotAli performed significantly better than the rest on RNaseP, SRP and tmRNA . We note that all families in this group except SRP had at least one pseudoknotted reference structure in their family. A major difference in performance of RNAalifold and KnotAli on the 16s family is their PPV value . This indicates that KnotAli identifies more base pairs that are not in the reference structure . In case of 16s, we observe that KnotAli’s sensitivity is similar to that of RNAalifold but its PPV is significantly lower than that of RNAalifold, contributing to its lower F-measure.We further assessed the effect of varying sequence lengths in an alignment on performance of all algorithms. We restricted sequence lengths for two families in our dataset with the largest range of lengths. For SRP family with original length range of 28–533 we only compared sequences within 200–350 lengths, and for Group I Intron with original length range of 210–736, we only considered sequences of length between 325 and 450 resulting in 285 and 33 sequences, respectively. The restricted subfamilies were re-aligned using MAFFT and MUSCLE. Results were then compared to the F-measure of structures from the previous prediction . As expected all algorithms saw an improvement on their accuracy of prediction on both families when length range of sequences comprising the alignment was tighter , we observe a sharp decline in accuracy for Cacofold (from .277 in 16s to .093 in RF00177). We therefore, conclude that alignment quality has an observable effect of the performance of both algorithms while KnotAli is less sensitive to it than Cacofold.compatible base pairs are not considered as false positives in calculation of F-measure.We recognize that the reference structures for the sequences within the dataset were determined through comparative sequence analysis . We noteinconsistent if the reference structure includes either compatible if they are not part of the reference structure. Table More specifically, we consider a predicted base pair Comparing the results to Table For the three families with low sequence conservation level , KnotAli performs significantly better than the others on Group I Intron and Group II Intron families regardless of the aligner used. While there is a significant improvement in accuracy of KnotAli on telomerase family compared to its previous F-measure, it is not yet able to beat RNAalifold or Cacofold.In the remaining five families with medium sequence conservation , KnotAli outperforms the rest on RNaseP, SRP  and tmRNA regardless of the aligner used. It is now on par with RNAalifold on 16s family (the previous winner) but still underperforms on 23s when compared to RNAalifold.In this work we present KnotAli, a novel algorithm that given a multiple sequence alignment as input predicts the possibly pseudoknotted secondary structure of each RNA sequence within the alignment. KnotAli first identifies a set of intermediary base pairs utilizing a noise adjusted mutual information metric (We further compared KnotAli’s performance to Cacofold (that similar to KnotAli utilizes background noise correction strategies and predicts possibly pseudoknotted structures), on a family of sequences with known spurious alignment (RF00177). We found KnotAli to be more resilient to changes in alignment quality compared to Cacofold. While both Cacofold and KnotAli use To adjust for inaccuracy in comparative analysis-based reference structures (such as the ones used in this work) caused by large unstructured segments in these structures, we further analyzed performance of all algorithms using adjusted F-measure in which compatible base pairs (those that do not contradict the reference structure) are not penalized as false positive. We observed that KnotAli’s performance further improved compared to other algorithms (only performing worse on the 23s and telomerase families).Overall, we find KnotAli to provide an improvement over existing methods for prediction of possible pseudoknotted structures from families of functionally related RNAs. We showed that KnotAli performs better that the compared methods in majority of RNA families in our dataset, and is less sensitive to quality of multiple sequence alignment when compared to Cacofold. There is, however, room for improvement. We showed the positive effect of using a better scoring model in the case of RNAalifold see “” sectionAdditional file 1. Recurrences. We provide the recurrences which make up the thermodynamic MFE prediction within KnotAli.Additional file 2. Energy Table. The table shows the free energy parameters which determine the free energy of the structures.Additional file 3. P-tables. We provide all non-included tables from our paper. Section 1 gives the MI vs MIp comparison table and the associated p-values. Section 2 gives the Restricted vs Non-restricted table and its associated p-values. Section 3 gives the non-included heatmaps for MUSCLE. Section 4 gives the p-values for the comparison with existing algorithms tables included in the text. Section 5 gives the p-values for varying sequence length table. Section 6 gives the p-values for when compatible bases are not considered false positives. Section compares the results between the other algorithms and against themselves.Additional file 4. Cross-validation. We provide our validation results for the heatmap-based threshold pick. Validation results were done through a 70-30 split over 1000 iterations where the threshold was picked based on the 70% training set and assessed on the 30% test set.


Secondary structure prediction accuracy (SSPA) in the QuanTest benchmark can be used to measure accuracy of a multiple sequence alignment. SSPA correlates well with the sum-of-pairs score, if the results are averaged over many alignments but not on an alignment-by-alignment basis. This is due to a sub-optimal selection of reference and non-reference sequences in QuanTest.We develop an improved strategy for selecting reference and non-reference sequences for a new benchmark, QuanTest2. In QuanTest2, SSPA and SP correlate better on an alignment-by-alignment basis than in QuanTest. Guide-trees for QuanTest2 are more balanced with respect to reference sequences than in QuanTest. QuanTest2 scores correlate well with other well-established benchmarks.http://bioinf.ucd.ie/quantest2.tar, comprises of reference and non-reference sequence sets and a scoring script.QuanTest2 is available at Bioinformatics online Multiple sequence alignment (MSA) is an important step in many bioinformatics analyses. The choice of MSA algorithm may be informed by its performance on one or many MSA benchmarks. MSA benchmarks are collections of nucleotide or amino acid sequences, for which the perfect reference alignment is assumed to be known. Construction of protein reference alignments may make use of the proteins’ three-dimensional (3D) structure and/or manual adjustment. It may be difficult to obtain large number of homologous 3D structures, so that the number of reference sequences in benchmark alignments tends to be small. For example, the largest alignment in BAliBASE3 . In PA, sequences are not all aligned simultaneously but as pairs of individual sequences or smaller alignments to form larger alignments. The order in which these pairwise alignments are performed affects the final alignment and can be encoded as a ‘guide-tree’ . Also, the error, as measured by the residuals, is large, that is, in the top 2% of values encountered. This plot says that there is little or no agreement between the SP and SSPA scores of the QuanTest data for this family, when we use 14 different alignment options. The residuals and regression line slopes are recorded for all 151 protein families in However, this relationship does not hold on an alignment-by-alignment basis . This isdo not compare the same alignments. As pointed out in effective alignment with respect to the reference sequences and is in general smaller than the total alignment. This effective alignment is produced by the smallest section of the guide-tree that subtends all the embedded reference sequences; this is called the effective guide-tree with respect to the embedded reference sequences  select sequences from the immediate neighbourhood of the references, (ii) sequences that are close to one reference but far from the other two and (iii) sequences that lie ‘between’ two or three references, as measured by the deviation from the triangle inequality. Of the two steps, the reference selection is the more important. This had been demonstrated in ––threads=1 -i <in> -o <out> ––guidetree-out=<dnd> , ––threads=1 -i <in> -o <out> ––guidetree-out=<dnd> ––full , Decipher (Nov. 2018), R script, generated in time, see -fa <in> , -fa <in> -t 1 -gt_export <dnd> <in> <out> (Famsa (v1.2.5), <in> > <out> (FSA (v1.15.9), <in> -format fasta -quiet -printtree <dnd> > <out> ,  > <out> ––anysymbol ––quiet ––thread 1 ––treeout <in> > <out> (Mafft (v7.407),  > <out> ––localpair ––anysymbol ––quiet ––thread 1 –treeout <in> > <out>Mafft L-INS-i, ––parttree ––anysymbol ––quiet ––thread 1 –treeout <in> > <out>Mafft PartTree, ––dpparttree ––anysymbol ––quiet ––thread 1 ––treeout <in> > <out>Mafft PartTreeDP, -in <in> -out <out> -quiet -tree2 <dnd> (Muscle (v3.8.31), e2 <dnd> -in <in> -out <out> -quiet -tree2 <dnd> -maxiters 2Muscle2, -o <out> -num_threads 1 <in> (MSAProbs (v0.9.7), s 1 <in> ––mem 10G ––protein ––treeout dnd ––in <in> ––out <out> , ut <out> ––input=<in> ––datatype=Protein ––num-cpus=1 (Pasta (v1.6.3), ––input=<in> ––datatype=Protein ––num-cpus=1 ––aligner=musclePastaM, ––input=<in> ––datatype=Protein ––num-cpus=1 –merger=musclePastMM, ––input=<in> –datatype=Protein ––num-cpus=1 ––aligner=muscle ––merger=musclePastaMMM, -read_fasta <in> -clustal <out> poaV2/blosum80.mat (POA (V2), ––threads 1 ––output-format fasta ––quiet <in> <out> , n> <out> -d=<in> -o=<out> -f=fasta -seed=1 -quiet -showtree -uselogs (Prank (v.170427), -uselogs <in> > <out> (Probcons (v1.12),  > <out> -in <in> -output fasta -n_core 1 -tree_mode upgma (TCoffee (v11.00.8cbe486), -s <in> -m amino ––cpu 1 (UPP (v2.0), The aligners used in this study are:Using this new selection procedure, we re-generate the 1200 alignments, comprised of 1000 sequences, for each of the 12 example families. The results are shown in We now present SP versus SSPA results for a selection of 30 MSA programmes, which are listed above with their respective command lines. However, 30 alignments are still a relatively small number for determining reliable regression lines. One way to increase the number of alignments without increasing the number of aligners/options is to use alignment instability . DependiWhile the correlation of the SP and SSPA scores has been improved, the average SP and SSPA scores themselves did not change appreciably; this can be seen in The ultimate purpose of an alignment benchmark is to rank aligners. This can be seen in effective size.Embedded benchmarks are an imperfect solution to the problem of compiling large MSA benchmark sets, using real data. In exceeds that of default MUSCLE, suggesting that Pasta’s way of merging profiles (referred to as ‘transitivity’) has a protective effect as the alignment size grows.We found that both the SP and the SSPA scores for the aligners that we evaluated gave a sensible ordering, which has been suggested by other benchmarks. Decipher and MAFFT L-INS-i were the most accurate, both in terms of SP and SSPA scores. We found that SSPA gave a more benevolent assessment of Prank’s performance, whose gap placement strategy is usually severely penalized by the SP score for non-simulated benchmarks. We also found that Pasta . One then calculates the SSPA for each of these alignments and selects the alignment which attains the highest SSPA as ‘the best’.http://bioinf.ucd.ie/quantest2.tar (QuanTest2 is a reconfiguration of the original QuanTest dataset (est2.tar .btz552_Supplementary_MaterialClick here for additional data file.


Low complexity regions are fragments of protein sequences composed of only a few types of amino acids. These regions frequently occur in proteins and can play an important role in their functions. However, scientists are mainly focused on regions characterized by high diversity of amino acid composition. Similarity between regions of protein sequences frequently reflect functional similarity between them. In this article, we discuss strengths and weaknesses of the similarity analysis of low complexity regions using BLAST, HHblits and CD-HIT. These methods are considered to be the gold standard in protein similarity analysis and were designed for comparison of high complexity regions. However, we lack specialized methods that could be used to compare the similarity of low complexity regions. Therefore, we investigated the existing methods in order to understand how they can be applied to compare such regions. Our results are supported by exploratory study, discussion of amino acid composition and biological roles of selected examples. We show that existing methods need improvements to efficiently search for similar low complexity regions. We suggest features that have to be re-designed specifically for comparing low complexity regions: scoring matrix, multiple sequence alignment, e-value, local alignment and clustering based on a set of representative sequences. Results of this analysis can either be used to improve existing methods or to create new methods for the similarity analysis of low complexity regions. Protein sequences are composed of amino acid fragments of varying diversity. Fragments with low diversity in the amino acid composition are called low complexity regions (LCRs). Due to their frequent occurrence and capacity to expand through replication slippage, they can easily increase protein sequence space and contribute to novel protein functions. They are known to play a key role in protein functions and may be relevant to protein structure . For exadroplets .Methods and algorithms for searching for similarities among protein sequences have always been important tools in biology that allowed researchers to predict protein functions from the sequence data alone. Many approaches are known from the literature that are suitable for searching for similar protein sequences. However, these methods are based on statistical models that are optimized to compare high complexity fragments. Due to that, for many years, protein regions that are characterized by low complexity of amino acids had been ignored and excluded from such type of analysis.Recently, the research community became more interested in the so-called Dark Proteome that is mostly composed of intrinsically disordered proteins or proteins that contain intrinsically disordered regions . TherefoThe community has already made some efforts to develop tools that are capable of automatically assigning functional roles of LCRs. One such example is the web server LCR hound  which idHere, we present a study that compares the performance of three state-of-the art methods for searching for similarities among protein sequences: BLAST , HHblitshttps://doi.org/10.5281/zenodo.6759535).The workflow of our experimental approach is shown in The definition of LCR in a protein sequence is not well specified. General agreement is that LCRs in proteins should have an excess of one or a few types of amino acid residues, but still, there is no consensus which metric is most appropriate. LCRs can be composed of homopolymers, short tandem repeats and irregular regions with low entropy. The scientific literature provides different terminology and definitions of these regions that are typically based on the sequence composition and periodicity . CAST 1 and fLPSHere, we consider LCRs as sequences characterised by low entropy in amino acids composition. This assumption is based on the definition provided by authors of the SEG method which is one of the most popular tools for searching for LCRs .To identify LCRs, we used SEG strict parameters  of sequences.BLAST is the first method we used to search for similar sequences. It uses Smith–Waterman algorithm to calculate local alignment based on a query and database sequences . We convmax_target_seqs to maximal possible value (1 073 741 798). Additionally, to improve searching for LCRs using available BLAST options, we changed task and comp_based_stats parameters. A study of selected BLAST parameter settings that can be applied for LCRs analysis can be found in [Task option is responsible for the default parameter set adjusted for specific types of sequences. Possible options are: blastp, blastp-fast and blastp-short. Blastp-short makes the following modifications to the default options: sets scoring matrix to PAM30, sets gap opening cost to 9, sets gap extension cost to 1, sets word size to 2, clears filter options and changes e-value (which is not applicable in our case because we explicitly set it). We left the scoring matrix parameter unchanged with its default value of PAM30 since it is recommended for short sequences [Comp_based_stats option is responsible for composition-based statistics. This option changes the scoring matrix by recalculating score values of frequently occurring amino acids in the query sequence, which is mainly caused by LCRs. It simply decreases the significance of LCRs while searching [Below, we enumerate modified parameters. We set e-value to 0.0001 and found in . Task opequences . Comp_baearching , and theHHblits is able to search for distantly related proteins that share a common ancestor. It is a part of the HH-Suite package and it uses Hidden Markov Model profiles to find similar sequences using HMM-HMM comparison . Therefomask which is responsible for choosing a masking strategy and the second is comp-bias-corr which changes correction for locally biased composition of amino acids. While analysing LCRs, it is recommended to turn off both of these parameters by setting their values to 0. We also removed the max-seqs parameter as it is deprecated and not available in the newest version of MMseqs. The result of running uniclust-pipeline is several databases created with different identity thresholds: 10, 20 and 30%. From these results, we selected Uniboost30, a database with highest sequence identity that is equal to 30%. However, the threshold is still low, and therefore, the results obtained using this dataset contain more distant similarities.Uniclust-pipeline uses MMseqs to cluster similar sequences and to create their Hidden Markov Model profiles , 27. We id, diff, norealign, sc and noprefilt. These parameters control a number of results that are relevant to query sequences and their detailed analysis is provided in Id parameter changes maximal pairwise sequence identity. We set this option to 100 (default is 90) because there are LCR families that are highly similar in their amino acid patterns. By default, HHblits selects the most diverse set of sequences, but here, we wanted to analyse the most similar ones, and therefore, we turned this setting off (diff parameter). Setting the norealign parameter disables MAC algorithm which significantly increases the number of matches [Sc option changes the method which is used to calculate amino acid score. We examined all available parameters, and based on the empirical analyses, we chose the value of 0 which uses background frequencies. Default value for this parameter is optimized for compositional bias correction [noprefilt parameter was introduced to speed up searches by filtering out cases that are ‘obviously’ not similar. However, we noticed that this parameter is filtering too many similar LCRs, so finally we disabled prefiltering.We performed the analyses of both HCRs and LCRs using the same e-value as in the case of BLAST (0.0001). For LCR analysis, we modified the following parameters:  matches . Sc optirrection . noprefiHHblits database contains profiles of HMMs which are a condensed form of MSA. Here, we notice an issue related to the MSA creation. In The third method we analysed in this work is CD-HIT which uses a greedy incremental algorithm. In a nutshell, first all sequences are sorted by length in descendent order. Then, the longest sequence creates the first cluster. The sequence that creates a cluster is called a representative and all the following sequences are being aligned to it to determine whether they should be included into this cluster or not. If the similarity score of a particular aligned sequence is higher than a threshold, the sequence is added to the corresponding cluster. Otherwise, the sequence creates a new cluster and becomes its representative . To speeWe performed exploratory data analysis by investigating: (i) how the results from the selected methods overlap with each other, (ii) what is the average sequence similarity and (iii) what is the number of alignments for a given length? Finally, we present selected cases to show how BLAST, HHblits and CD-HIT analyse LCRs. We also analysed amino acid composition and biological roles of selected examples. To find more interesting cases, we filtered out results (hits) that include sequences from the same families for BLAST and HHblits. We assumed that minimal length of LCR is 6 amino acids.To investigate biological features of the selected proteins, we performed a three layer functional analysis composed of the following stages: (i) We scanned popular DNA/protein databases  , 35–37, To quantitatively compare the results from the three methods, we created Venn diagrams . In addiAs shown in In For BLAST and HHblits results, we sorted alignments by e-value and divided them into 10 groups where each group contains about 10% of sequences. Therefore, the group size of BLAST is approximately 888 thousand pairs each and the group size of HHblits is approximately 222 thousand pairs. Within each group, we compared lengths of alignments to their similarity and the number of alignments. The similarity in an alignment between two sequences is denoted by percentage of similar amino acids where similar amino acids are these which have positive score in a scoring matrix.For BLAST, the results for three best percentiles  differ from the results for other percentiles A. It sughttp://bib.oxfordjournals.org/) and comparison of lengths of alignments in different e-value percentiles for LCRs and HCRs are provided in Supplementary materials.Corresponding figure for HCRs  obtained with the different methods. Here, we show data that indicate the following features: (i) some of the methods’ features are more appropriate for HCRs than LCRs and (ii) similarity of LCR sequences may but does not have to indicate a similar function of these sequences.For BLAST, we selected 5 representative alignments with their e-values that illustrate different problematic cases. At least, one of the proteins from alignments belongs to the krueppel C2H2-type zinc-finger protein family. Selected alignments are presented in Table 2The first alignment presents homopolymers of serine that consist of 25 amino acids with e-value equal to ocument} . The secM matrix . MatchesThe third, the fourth and the fifth examples show other issues that we can observe if we use BLAST to analyse similarities between LCRs. All of these alignments are 12 aa long. The third and the fifth alignments consist of homopolymers of proline. The third one is a perfect match over the total length of LCRs. On the other hand, the fifth alignment consists of two LCRs with different lengths resulting in an alignment with a shorter LCR length. As a result, both alignments have similar lengths and scores. However, the length of homopolymer in an alignment may be crucial for its function . TherefoWe used this opportunity to analyze biological properties of the LCR pairs presented in Moreover, a rare function can be contributed to the protein from three last examples. The polyproline stretch of ZFHX4, the zinc finger homeobox protein 4, serves as an assembly element for the human butyrylcholinesterase into a tetramer . We coulFor HHblits, alignments with a lower e-value are less similar than alignments with higher e-value found by BLAST, even if the input protein dataset is the same in both cases. Perfect alignment of serine of the same length in case of BLAST has an e-value equal to ocument} . AdditioThe first alignment has a better score and e-value than the second. Both of them have 100% similarity. Additionally, the second alignment has 2 mismatches and worse e-value than the first one with 3 mismatches. The third, the fourth and the fifth alignments have three mismatches and are assigned slightly different e-value, score and similarity. The third and the fifth hit sequences are 19 amino acids in length. Furthermore, the biggest difference among e-values is between the third and the fifth alignments. The fourth example is in the middle but has a different hit sequence length (16 aa).et al. [For most of the LCRs from et al. . This seAnother example of similarity to the same serine-rich fragment of the chloroplast sensor kinase is the nucleolar and coiled-body phosphoprotein 1  which acts as a platform to connect RNA polymerase I with enzymes responsible for ribosomal processing and modification . ExperimOur analysis of LCR results from CD-HIT is slightly different in comparison to BLAST and HHblits as results of CD-HIT are in the form of sequence clusters. Since the number of the sequences returned by CD-HIT was significantly smaller in comparison to BLAST and HHblits , we deciIn the Another disadvantage is that CD-HIT creates two different clusters from two highly similar sequences from composition and length perspectives. The fifth cluster from Our analysis confirms that selected methods are most efficient for comparing high complexity protein sequences as they rely on statistical models designed for sequences with diverse amino acid composition. This is why masking low complexity parts improves searching for homologous proteins . An obviThe methods analysed in this paper use general purpose scoring matrices. They are efficient to align typical protein sequences but fail while applying them for non-standard amino acid composition of sequences such as LCRs . For exaMSA and consequently a dataset of profiles of Hidden Markov Models for LCRs may be created in many different ways. One problematic case is shown in E-value is a statistic useful while searching for similarities that have biological meaning as it provide the estimate if a particular similar sequence may occur in a database by a chance . HoweverZinc finger homeobox protein 4 protein to only a part of poly-P region from Glyceraldehyde-3-phosphate dehydrogenase,testis-specific protein. Therefore, local alignment based tools lose information about difference in LCR lengths. However, current knowledge says that homopolymer length may affect protein function. For instance, poly-Q region may cause neurological disease if it reach a given length [Local alignment causes loss of information about length of longer LCRs than query sequence. Such a case is presented in n length . Therefon length . In thisSelection of representative sequences in CD-HIT may result in wrong assignment of sequences of different types of LCRs. This is especially visible in LCRs that consist of two different types of LCRs. As mentioned in Results section, if a representative sequence consists of two types of LCRs, then three types of LCRs may join this cluster: LCRs of the first type, LCRs of the second type and the combination of both LCRs. As a result, a single cluster contains different types of LCRs. Such a scenario is present in examples 1 and 2 in Even if we showed that standard methods are not appropriate for the similarity analysis of LCRs we could identify some examples that are statistically sound (see Results section for BLAST and HHblits). In this work, we collected a few examples of LCR similarity/identity that may follow the rule of transitivity. If the function of one LCR is known and we show its similarity to another LCR, then we may hypothesize that the role of the second LCR may be identical or similar, e.g. if LCR1 binds a protein, we may argue for an analogous binding mechanism of a similar LCR2.Our work focuses on three state-of-the-art methods for protein sequence comparison that are based on the most popular approaches for protein similarity analysis such as local alignment or profiles of Hidden Markov Models. There are other tools for protein sequence comparison that can be found in the literature, but they usually rely on similar approaches. In this paper, we point out why these methods are not suitable for LCR analysis. However, since there are no tools specifically designed for LCRs, we suggest to use BLAST. It seems to be the best choice to search for similar LCRs. HHblits may also be useful in LCR analysis, but for more distant similarities. Finally, we conclude that CD-HIT cannot be used for analysing LCRs as the local comparison to representative sequences results in badly clustered sequences that are not similar to each other. Scientists should be aware of these drawbacks while using these methods for searching for similar LCRs. On the other hand, our results may be used to improve existing methods or to design new ones especially crafted for LCR comparison.We would like to alert the community that similarity searches reported by canonical tools may result with false positive hits, even if they use the optimal parameter set for these methods.We indicate which design assumptions of the selected methods are not suitable for the analysis of low complexity regions (LCRs).In the article, we advise on how to adjust HHblits and CD-HIT parameters to find similar LCRs more efficiently.This knowledge can be used to improve existing methods or to create new methods specifically designed to analyze the similarity between low complexity regions.canonical_methods_supplementary_bbac299Click here for additional data file.