O(n log n). Here we evaluate the ICZT algorithm for chirp contours on the unit circle and show that it is numerically accurate for large areas of the parameter space. The numerical error in this case depends on the polar angle between two adjacent contour points. More specifically, the error profile for a transform of size n is determined by the elements of the Farey sequence of order n − 1. Furthermore, this generalization allows the use of non-orthogonal frequency components, thus lifting one of the main restrictions of the IFFT.This paper shows that the inverse chirp z-transform (ICZT), which generalizes the inverse fast Fourier transform (IFFT) off the unit circle in the complex plane, can also be used with chirp contours that perform partial or multiple revolutions on the unit circle. This is done as a special case of the ICZT, which in algorithmic form has the same computational complexity as the IFFT, i.e.,  Inverse Chirp Z-Transform (ICZT) is a generalization of the Inverse Fast Fourier Transform (IFFT), which is one of the most popular and useful algorithms2. The sampling points used by the ICZT are distributed along a logarithmic spiral contour in the complex plane. The shape of this contour is determined by the complex parameters A and W such that the k-th contour point is equal to AWk−, where k is a zero-based index  time, which makes them fast and practical. The Chirp Z-Transform (CZT), which generalizes the Fast Fourier Transform (FFT) and also runs in O(n log n) time, was discovered in 19699. The inverse algorithm, however, remained elusive for the next 50 years. The ICZT algorithm also runs in O(n log n) time10, where n is the size of the transform. This enables applications in which the CZT is paired with the ICZT similarly to how the FFT is often paired with the IFFT. Application domains that could benefit from this include signal processing, electronics, medical imaging, radar, sonar, wireless communications, and others.Many applications require both   Figure W. The red points indicate the numerical error for the three contours from Fig. W above 22.5° eventually wraps the chirp contour over the unit circle more than once  algorithm. Most of these attempts have focused on the special case of inverting the CZT for chirp contours on the unit circle. In one case13, a modified version of the forward transform, in which the circular chirp contour was traversed in the opposite direction, was presented as the ICZT transform. This approach, however, does not really invert the forward CZT in the general case, i.e., for logarithmic spiral contours. This approach also doesn’t work for all chirp contours that lie on the unit circle, because the inverse of the Vandermonde matrix used by the forward transform is not, in general, a Vandermonde matrix. Traversing the contour in reverse order is equivalent to using another Vandermonde matrix as the transformation matrix. Our paper provides experimental results that show that this method does not work.There have been several unsuccessful attemptsChirp Transform Algorithm (CTA)15 and the Fractional Fourier Transform (FRFT)17. Both algorithms are special cases of the CZT for chirp contours that lie on the unit circle, but the FRFT contours always start at the complex point . It has been estimated18 that the inverse chirp transform can be performed in 2M(n) + O(n), where M(n) can be taken in The literature describes two algorithms that generalize the FFT on the unit circle: the O(n) memory.As described in Supplementary Section A and W.The FFT and IFFT are two very similar algorithms. They are also very stable numerically. The reason for this is that the harmonically-spaced frequency components that they use are orthogonal. This condition doesn’t hold for the CZT and the ICZT, which explains why the ICZT algorithm is substantially different from the CZT algorithm. The numerical accuracy of the ICZT depends on the values of its parameters 19. This problem is solved with approximate or iterative algorithms22. The accuracy and speed of approximate algorithms depend on the desired precision20, which is often controlled by an oversampling parameter22. The computational complexity of iterative algorithms22 depends on the condition number of the problem, i.e., they may require many iterations to converge. In contrast, the CZT and ICZT algorithms are exact and their computational complexity depends only on the problem size.A different, but related, problem is generalizing the FFT or the IFFT to nonequispaced sampling points on the unit circle7 as follows: x is the input vector of length N and X is the output vector of length M. Using matrix notation this can be stated as: A is the following diagonal matrix W is the following Vandermonde matrix: The chirp z-transform (CZT) is defined9, i.e., W can be expressed as P and Q are the following two diagonal matrices: Using Bluestein’s substitutionX can be computed in O(n log n) time, where Thus, the CZT algorithm can be viewed as an efficient implementation of the following matrix equation: Algorithm M = N, the ICZT can be stated10 by inverting Eq. , i.e., k for each k ∈ {0, 1, …, n − 1} is given by:Because O(n log n) time by exploiting the structure of the matrices. See reference10 for more details, proofs, and the pseudo-code for ToeplitzMultiplyE and all of its dependencies.Combining these results leads to the following closed-form solution for the ICZT: n − 1, where n is the size of the transform.This section proves that the singularities of the ICZT on the unit circle are related to the elements of the Farey sequence of order Definition 1.  A Farey sequence of ordernis denoted by  Fn. It consists of all rational numbers in the interval  with an irreducible fraction representation  p/q  that satisfies the inequality  q ≤ n,  where n is a positive integer.p/q is irreducible if there is no other fraction a/b for which ∣a∣ < ∣p∣ or ∣b∣ < ∣q∣ such that p/q = a/b. For example, 0/1 is irreducible, but 0/5 is not. To give another example, both 2 ∕4 and 3 ∕6 are reducible to 1/2.A fraction By mathematical convention, the numbers in each Farey sequence are sorted in increasing order. For example, the first five Farey sequences are equal to: n contains all elements of the Farey sequence of order n − 1 and some new elements that are unique to Fn. Because Fn includes all elements of Fn−1 and Fn−1 includes all elements of Fn−2, it follows that Fn includes all elements of Fn−2 as well. In fact, Fn includes all elements of F1, F2, …, Fn−1. This property becomes more clear if the Farey sequences shown above are rewritten with extra space between some of the terms as shown below: p/q ∈ Fn appears in the sequences F1, F2, …, Fn−1 is equal to n − q. For example, the fraction 1/3 appears 5 − 3 = 2 times in F1, F2, F3, and F4.The Farey sequence of order The next definition links Farey fractions to polar angles.Definition 2.  Each angle  θ  that can be expressed in radians as  θ = 2πp/q  is a Farey angle of order n if p/q is an irreducible fraction that is an element of the Farey sequence of order n.π are equivalent.Note that each Farey sequence includes two distinct elements 0/1 and 1/1. However, these two fractions map to the same Farey angle, i.e., 0 and 2i2πp/q is a root of unity of order q. The converse is also true: each root of unity corresponds to a Farey angle and the order of the root of unity determines the order of the Farey sequence in which the fraction p/q first appears.The following theorem proves that for each Farey angle the corresponding complex exponential eTheorem 1.  Let  be an angle expressed in radians. Then,  θ  is a Farey angle of order n if and only if there is an integer  q ∈ {1, 2, …, n} such that the complex exponential  eiθ  is a root of unity of order  q  for someq ≤ n.Proof. (⇒) Suppose that θ is a Farey angle of order n. Then, θ = 2πp/q, where p/q ∈ Fn. This implies that the complex exponential eiθ is a root of unity of order q. That is, p is an integer.iθ is a root of unity of order q ≤ n, i.e., θq is an integer multiple of 2π. That is, without loss of generality, θq = 2πp, where p is a non-negative integer between 0 and q − 1. Therefore, θ ∈ Fq ⊆ Fn, as required.□(⇐) Suppose that the complex exponential eFn−1 to the singularities of Eq.  Suppose that p/q ∉ Fn−1. Then, q ≥ n. Because p and q are coprime, this implies that s ∈ {1, 2, …, n − 1}. Therefore, each term in the denominator of each element uk is non-zero.□(⇐) Conversely, suppose that W in Eq. . This discretization hits many Farey angles of different orders, which leads to the stratified appearance of the plot. This layering effect is explained by the link between Farey sequences and the prime factorization of the number of regularly-discretized angles. See Supplementary Section Figure Figure Figure W when the number of samples is about 4 times greater than the number of contour points. More specifically, the plots were drawn with 4096, 4098, and 4099 samples, respectively. Both 4096 and 4098 share integer factors with 1024, which explains the presence of layers in Fig. The bottom row of Fig. n.In other words, a plot in which the horizontal axis is discretized using a fixed step can lead to a misleading picture of the error. The behavior of the error may be completely different between the discretized points. The shape of the plot depends both on the sampling procedure and on the distribution of Farey angles. As shown below, the number of Farey angles where the transform is singular grows quadratically with the size of the transform, which makes it difficult to draw a complete plot of the error function for large values of n on the unit circle are related to the elements of the Farey sequence of order n − 1. Table n increases the length of the Farey sequence Fn−1 also increases26. These values imply that the number of singularities grows faster than the transform size. This growth is roughly quadratic, which follows from the popular approximation formula Fn .As proven in Theorem 2, the singularities of the ICZT of size N = 1024 there are 318453 singularities. In other words, the number of singularities is approximately two orders of magnitude larger than the number of all points plotted in Fig. The large number of singularities makes it more difficult to draw and interpret the error plots. For example, when Some discretizations select more Farey angles of small orders than others. For example, the discretization used in Fig. 10. This is consistent with previous observations that Vandermonde systems can be ill-conditioned and should be solved with double precision or higher29. Interestingly, it was also shown that some ill-conditioned Vandermonde systems can be solved with high precision32. In our previous work, however, the chirp contours were off the unit circle, i.e., they were expanding or contracting logarithmic spiral contours. We wanted to check if additional numerical precision is still needed for contours with thousands of points that are restricted to lie on the unit circle.The next experiment studied the behavior of the error function for contours with 2048 points. Our previous experience with the ICZT indicated that the numerical accuracy decreases for contours with large number of pointslibquadmath library33 through an interface provided by Boost Multiprecision34, which makes it easier to use various floating point formats in C++. The plots show that the numerical accuracy can be boosted by increasing the precision of the floating-point numbers. Switching from double precision to quadruple precision reduces the error by approximately 16 orders of magnitude. The overall shape, however, remains mostly the same.Figure N = 2048 using only double precision. Unless the polar angle of W is close to a Farey angle, the error is usually small when the chirp contour is on the unit circle. Nevertheless, if additional numerical accuracy is needed, then one can switch from double to quadruple floating-point precision, which is becoming more popular and easier to use.The results also indicate that the transforms are computable for M = N and for chirp contours that lie on the unit circle. The error formulas for a more general case with logarithmic spiral contours that span a 360° arc off the unit circle are given in ref. 10.This subsection states formulas that approximate the absolute numerical error for the sequential applications of the forward and inverse transforms, i.e., CZT followed by ICZT and vice versa. The formulas are given for the square case in which U1, U2, U3, T, and B. The first three terms are derived from the elements of the generating vector u . For coU1 is equal to the logarithm of the Euclidean norm of the first row of the matrix u except u0  . The ted by Eq. . It is smentclasspt{minimaT, is the sum of three simpler terms10. In this special case, however, they are all equal to A and W have a magnitude of 1, i.e., ∣A∣ = ∣W∣ = 1. Thus, the term T is equal to: The fourth term, B, that depends on the size of the transform, N, and the number of precision bits, p, used in the calculations. This offset determines the base level of the error function. It is defined as follows: C1 and C2 are implementation-dependent constants. For 64-bit and 128-bit IEEE-754 floating-point numbers25, the value of p is set to 53 and 113, respectively.The formulas also include an offset term, x and the computed vector The absolute numerical error for a sequential application of the CZT followed by the ICZT is equal to the Euclidean distance between the original input vector The error values for the CZT–ICZT procedure can be analytically approximated using the following formula: X is the true output vector and The absolute error of the ICZT–CZT procedure is R2 coefficient (see Methods). Table R2 coefficients for the two experimental procedures and for transform sizes between 16 and 2048. In all cases, the results were computed by averaging the R2 values from 10 independent runs of the corresponding procedure, where each run used 10 random input vectors. For each value of N, the fits used 4099 regularly-discretized polar angles of W. The average R2 value increases with the size of the transform and approaches 1, which indicates that the formulas predict the numerical error very well. The results also show that there is no substantial difference between the R2 values for the two experimental procedures.The accuracies of Eqs.  and 23)23) were Additional results for the accuracy of the error prediction formulas are given in Supplementary Sections 13 that is reproduced below: X(k) by conjugating all elements of the CZT of the vector X(k)*, where * denotes elementwise conjugation. In matrix form, Eq. , optimal scaling for this approach can be achieved by normalizing the vector y, i.e., Ref. that Eq.  needs to.e., Eq.  with theA was equal to 1, the value of M was equal to N, and the polar angle of W was sampled using regular discretization as described in Methods. The accuracies were measured with the CZT–ICZT procedure, also described in Methods, but with Algorithm 2 replacing the ICZT implementation in the second condition.The rest of this section describes three experiments that compared the accuracy of Algorithm 1 to the accuracy of Algorithm 2. In all experiments, the value of W. In contrast, Fig. x. In both plots, the three red points correspond to the chirp contours shown in Fig.  Figure  Figure W. In contrast to Figs. x∣∣. Because this discretization did not hit any primitive roots of unity of order 2048, there were no angles for which the reverse-as-inverse approach is accurate. The increased floating-point precision also did not help to reduce its error.Finally, Fig. N, i.e., only when the ICZT reduces to a permutation of the elements of the IFFT output vector. Algorithm 2 assumes that there is no interference between the frequency components, which is true only when they are orthogonal or, equivalently, when the value of W is a primitive root of unity of order N. For all other values of W the reverse-as-inverse approach fails to compute the ICZT.To summarize, the experiments showed that Algorithm 1 is accurate for most chirp contours and the numerical error of the CZT followed by the ICZT can be predicted using Eq. . In contThis paper generalized the inverse fast Fourier transform (IFFT) to work with contours that perform partial or multiple revolutions on the unit circle. This was accomplished by analyzing the numerical error properties of the ICZT algorithm for a special case with circular chirp contours on the unit circle. The paper also proved that the ICZT singularities are tied to elements of Farey sequences, where the Farey order is smaller than the transform size. Formulas for predicting the numerical error were derived as well. The experiments showed that these formulas fit the empirically-observed numerical errors very well.W the ICZT could be very inaccurate. The error prediction formulas derived in this paper allow the practitioners to avoid these singularities. For most values of W, however, the ICZT is accurate and can be computed for large transform sizes.The FFT and the IFFT are restricted to use orthogonal, harmonically-spaced frequency components that are generated by the integer powers of the complex roots of unity. The CZT and the ICZT can use non-orthogonal frequency components, even for the special case with chirp contours on the unit circle. This additional flexibility, however, comes at a cost. For some values of the transform parameter Chirp Transform Algorithm (CTA) and the Fractional Fourier Transform (FRFT) are two popular algorithms16 that can be viewed as special cases of the CZT for chirp contours on the unit circle. The corresponding inverse algorithms, however, have not been described until now. Supplementary Section Inverse Chirp Transform Algorithm (ICTA) and the Inverse Fractional Fourier Transform (IFRFT).The 10. Future work could combine the insights from these two studies to derive error prediction formulas for logarithmic spiral contours off the unit circle that perform partial or multiple revolutions. In that case, the frequency components are not orthogonal and can also decay or grow exponentially. Future work could also try to derive bounds for the condition number of the transformation matrix and relate them to the error formulas derived in this paper.This paper analyzed the numerical error of the ICZT algorithm for chirp contours on the unit circle that can perform partial or multiple revolutions. The numerical error for logarithmic spiral contours off the unit circle that span 360° was analyzed in our previous paperW. The transform parameter A was always set to 1 because changing its polar angle is equivalent to changing the polar angles of the elements of the ICZT output vector10, which does not affect the expected numerical error over all possible input vectors.This section describes the methods that were used to evaluate the ICZT algorithm for chirp contours that lie on the unit circle. The experiments systematically varied the size of the transform and the sampling procedure for the polar angle of the parameter W: (1) generate a complex input vector x by sampling its real and complex parts from a uniform distribution on the interval , where q ≤ n. All Farey sequences share the mediant property. That is, if a/b, p/q, and c/d are any three consecutive elements of a Farey sequence, then p/q = (a + c)/(b + d).Farey sequences38. Algorithm 3 shows the pseudo-code for an efficient procedure38 that generates a Farey sequence of order n. This procedure was used for some of the figures that explicitly included all ICZT singularities in a given subset of the parameter space.The mediant property enables the enumeration of the elements of a Farey sequence by an algorithmR2 coefficient. That is, given two vectors a =  and b = , the R2 value was computed as follows: a and b, i.e., The accuracy of the numerical error prediction formulas was evaluated using the R2 coefficient is equal to 1 minus the fraction of the variance of the vector b that is unexplained by the vector a. Our approach centers the vectors  before computing R2. Thus, the constant offset R2 value.In other words, the a was set to the predicted logarithms of the numerical errors. The vector b was set to the logarithms of the numerical errors that were computed by either the CZT–ICZT or the ICZT–CZT procedure.In the experiments, the vector 9 expresses the Vandermonde matrix as the product of two diagonal matrices and a Toeplitz matrix. Thus, the key to the ICZT algorithm is finding a computationally efficient way to invert this Toeplitz matrix.The CZT matrix is equal to the product of a Vandermonde matrix and a diagonal matrix. Bluestein’s substitution24 expresses the inverse of a Toeplitz matrix as a difference between two products of upper-triangular and lower-triangular Toeplitz matrices. These four matrices can be described by two generating vectors u and v, but the formula does not specify how to find these vectors. The matrix used by the CZT, however, is a special case of a symmetric Toeplitz matrix. For this matrix there is a special case of the Gohberg–Semencul formula with just one generating vector10. Furthermore, we were able to express the elements of this vector in terms of the transform parameter W. This led to an efficient ICZT algorithm that runs in O(n log n) time.The Gohberg–Semencul formula10. There are at least two different approaches for computing these products in O(n log n) time: (1) embedding the Toeplitz matrix into a larger circulant matrix  and (2) expressing the Toeplitz matrix as a sum of a circulant matrix and a skew-circulant matrix of the same shape using Pustylnikov’s decomposition41, see also ref. 42, p. 40 and ref. 43, p. 66.The ICZT algorithm uses fast FFT-based subroutines for multiplying a Toeplitz matrix by a vector44 by a vector in O(n log n) time . The term f-circulant with f = −1 is also used in the literature to refer to skew-circulant matrices45. The required FFT sizes and the order of the operations performed by these two approaches are different. Their numerical accuracy, however, is similar. In all experiments described in this paper we used the embedding approach.Both subroutines multiply a circulant matrix or a skew-circulant matrixSupplementary Information.


The miRNAture pipeline implements a workflow specific to animal microRNAs that automatizes homology search and validation steps. The miRNAture pipeline yields very good results for a large number of “typical” miRBase families. However, it also highlights difficulties with atypical cases, in particular microRNAs deriving from repetitive elements and microRNAs with unusual, branched precursor structures and atypical locations of the mature product, which require specific curation by domain experts.Homology-based annotation of short RNAs, including microRNAs, is a difficult problem because their inherently small size limits the available information. Highly sensitive methods, including parameter optimized  Originae (22.1)  lists 19e (22.1) . These ne (22.1)  counts oe (22.1) . The disexample, ,9,10.canonical miRNAs that are processed by Drosha and Dicer and thus share several common features [blast-based homology searches alone tend to produce false positives that require extensive curation, which largely relies on the properties expected for a miRNA [Despite their small size of only about 80–100 nt, miRNA precursors are readily detected by sequence comparison due to the often extreme level of sequence conservation . Homologfeatures : (a) evi a miRNA . The fea a miRNA ,19.Infernal [Infernal, however, depends critically on the quality of the sequence alignment and the annotated consensus structure that is used to build the CM. Although the Rfam database [Rfam miRNA alignments is that family-membership is much more loosely defined than miRNA families in miRBase [Rfam miRNA alignments are thus composites of miRBase-families. miRBase, on the other hand, contains more sequences, but does not provide curated alignments and consensus structures. The precursor sequences available for a given family often have inconsistently defined ends, and the alignments for different families tend to differ in which species they include. Recent work also identified a moderate number of erroneous entries [Homology search for pre-miRNAs, as for other families of structured RNAs benefits substantially from including information on the consensus structure. This is achieved naturally using covariance models (CMs) ,21 as imInfernal . The accdatabase  already  miRBase , where s entries , see als entries .MIRfix [miRNAture is specifically designed to identify and annotate metazoan miRNAs in a homology-based setting. The miRNAture pipeline thus is complementary to the tools and pipelines that extract miRNA candidates from small RNA-seq data, such as mirDeep2 [miRTRAP [miRRim2 [blastn [nhmmer [cmsearch [tRNAscan-SE [miRNAture pipeline follows the same philosophy for canonical miRNAs—the initial acquisition of candidates using generic homology search tools operates at increased sensitivity and returns large numbers of false-positives candidates, which are then weeded out by very stringent, miRNA-specific filters.These observations call for an integrated workflow to perform homology search and to evaluate the search results in a consistent manner. In this contribution we improve on ideas from MIRfix  and intemirDeep2 , miRTRAP[miRTRAP , or miRR[miRRim2 . Homolog[miRRim2 . The bal [blastn , nhmmer  [nhmmer  and cmsecmsearch ) can be Ascan-SE , which amiRNAture pipeline is outlined in let-7 family across vertebrates, before testing the pipeline on the human miRNA complement after excluding all human sequences from the queries. Both show-case applications are concerned with data that have been analysed extensively in the past and thus allow us to point out both strengths and weaknesses of miRNAture and the published miRNA annotations. The source code of miRNAture, together with the technical description are available at: https://github.com/Bierinformatik/miRNAture (accessed on 26 February 2021).This contribution is organized as follows—the strategy and organization of the miRNAture pipeline is composed of three modules: (1) Homology search operating on miRNA precursors; (2) prediction of the positioning of mature miRNAs within the precursor (Mature annotation); and (3) an Evaluation scheme designed to identify false positive miRNA annotations. The pipeline is distributed with pre-computed CMs for the miRNAs in Rfam v.14.4 [miRNAture produces annotation files in GFF3/BED format and FASTA files for validated candidates as well as summary reports that highlight possibly problematic cases, tagging these for manual inspection. The architecture of miRNAture is summarized in The m v.14.4 , which amiRNAture can use either individual miRNA sequences or pre-computed/user provided CMs. In sequence mode, miRNA-specific strategies based on blastn [blastn hits are aggregated into extended regions as described in Rfam v.14.3 stockholm alignments, or user defined ones, for example, inferred from miRBase [nhmmer [cmsearch [miRNAture also offers the option to search the target genome with user-defined CMs. These can be obtained, for example, from alignments of miRBase sequences, directly from Rfam, or from the user’s own alignments.In the initial step, n blastn  are empln blastn ,34, full [nhmmer  to detercmsearch . miRNAtuE-value, coverage and if available, bitscore ). Then miRNAture disambiguates the reading directions in case overlapping candidate loci at both strands have been detected. To this end, quality measures provided by Infernal  are used. The final candidate lists are merged. For each candidate, coordinates and the supporting initial hits are reported. This three-fold search might seem redundant at first glance as, not surprisingly, there is a large overlap between the search results. However, as can be seen in Independent of the strategy chosen for the initial step, the candidate sequences are then filtered based on specific threshold values: miRNAture attempts to identify the location of the mature miR and miR* within the preliminary precursor sequences. To this end, we adapted MIRfix [https://github.com/Bierinformatik/MIRfix/releases/tag/v2.0.0, accessed on 26 February 2021). Mature miR/miR* sequences were obtained from miRBase. For each family, the result of this step is an alignment of corrected and trimmed miRNA precursor sequences annotated with the placement of the mature sequences and finally a structure-annotated sequence alignment. Corrected alignments were pre-calculated for miRBase families only and are available together with the corresponding CMs here: http://www.bioinf.uni-leipzig.de/publications/supplements/21-001 (accessed on 26 February 2021).In the next step, d MIRfix  and releIn the final stage, the corrected, structure-annotated alignments are used to evaluate homology search results. Since miRNA hairpins form extremely stable RNA secondary structures , we can miRNAture in terms of measures like sensitivity or precision requires a dataset with a reliable ground truth of positive and negative instances. Such a dataset, however, is currently not available for miRNAs despite extensive efforts of the curators of miRBase [independent method to generate artificial data, in our case alignments of miRNA families. To our knowledge, no such tool is available. We shall return to this issue in the discussion section. We will, however, use a simple simulation to asses some properties of miRNAture’s filters. Instead of a quantitative evaluation of miRNAture, we therefore consider two scenarios in which a semblance of the ground truth is known from extensive manual curation—the history of the let-7 family and the human miRNA complement. In both cases the discussion will focus on the differences between current annotation on the findings of miRNAture.Testing the performance of  miRBase . On the  miRBase  and uncl miRBase , on the miRNAture pipeline can be downloaded from https://github.com/Bierinformatik/miRNAture (accessed on 26 February 2021). It is provided as Conda package for installation, which resolves all dependencies and includes a detailed user manual, a tutorial and extensive example data.The let-7 family [let-7 family was studied extensively in the past [miRNAture, we performed re-annotation experiments with several primate genomes, the mouse genome and the Pacific transparent sea squirt, Ciona savignyi, as targets. In each case, all miRNAs annotated for the target genome were removed from the alignments and CMs of the query, see miRBase, we intersect miRNAture derived let-7 loci also with the manual annotation of Hertel et al. [miRBase annotation (MIPF0000002), are considered our gold standard annotation. homology stage and the final stage of miRNAture, respectively. In summary ≥91% of all annotated let-7 loci in all species were recovered by miRNAture, while in all cases, except the solitary tunicate, one of the annotated loci was not identified. Furthermore, between 1 and 11 additional loci per genome are considered valid, novel let-7 candidates. In the following paragraphs we discuss these results in detail.The 7 family  is one o7 family ,13,37,387 family ,39. Sincthe past ,41,42 itl et al. . The latK-let-7 in all primates, in the nomenclature of Hertel et al. [K-let-7 sequences shows a multi-loop structure where the typical miRNA hairpin is expected, while G-let-7-1 was detected in the mouse genome considering only the homology stage, but discarded by structural filters in the evaluation stage for similar reasons. See The missing candidates correspond to locus l et al. , which cl et al.  reportedhomology stage of miRNAture is optimized for sensitivity and only let-7 was used a query, bona fide miRNAs that share some similarity with let-7 are expected to have passed filters. The additional let-7 loci found by miRNAture were therefore compared to the annotation of other miRNA families. We indeed found overlaps with miRBase annotation for the human specific hsa-mir-4699 (MI0017332), and the families mir-3596 (MIPF0001194), and mir-625 (MIPF0000534). A mir-3596 was annotated in Rattus norvegicus and identified by miRNAture also in mouse. The mir-625 family was know in human and macaque only. These cases account for a third of the additional matches. Almost all of the remaining loci overlap with regions annotated as repeats. Only three loci  do not overlap with available annotation. The similarity of mir-625 and let-7 was noted before e.g., by Rfam, which includes mir-625 in their let-7 miRNA precursor family RF00027. In [mir-3596 is treated as a member of the let-7 family, highlighting that miRNAture presumably classified them correctly as novel candidates.Since the miRNAture in the presence of large sequence divergence we artificially mutated two of the human let-7 genes (chr21:16539829-16539913:+ and chr3:52268269-52268368:−) with increasing number of mismatches. For up to 10 point mutations, the loci were recovered at the homology stage, 4 candidates passed the homology filters and 2, overlapping the original loci, also survived the structural filters. At higher artificial mutation rates none of the initial candidates satisfied the structural constraints. As expected, at even higher mutation rates eventually also the initial homology search fails.To check the behavior of miRNAture is the annotation of a genome of interest with a set of available miRNA family CMs. To simulate such a use case and simultaneously further benchmark miRNAture, we used a set of 350 miRNA families with a human entry in miRBase v.22 to construct query alignments from which all human sequences were removed, see A typical use case for homology stage, miRNAture detected miRNA candidates for all but a single family. Considering the annotation of mature sequences and curation at structural level with MIRfix [We find that, at the h MIRfix  in the vmiRNAture at the homology stage we distinguish families where all candidates overlap exactly with annotation , with 100 initial candidates, of which 69 passed the filtering steps. However, none of them matched the annotated loci. In mouse, six precursor loci have read support at mature sequence loci of which 4 are annotated by miRBase as high confidence miRNAs. Additional homologs at a single locus without read support are annotated in rat and some primate genomes: Macaca mulatta, H. sapiens (very weak read support) and P. troglodytes. Input for miRNAture was a CM model built from mouse validated sequences, which in comparison to the known human locus contain 20% more nucleotides and 10% additional consensus positions.In order to better understand the performance of miRNAture comprises 284  There is no mature sequence alignment available that allows the correct annotation of detected candidates; this is in particular the case for species-specific families. (b) The location of the mature sequences was determined based on similarity to human loci alone, without additional information. With the artificial removal of the human data this information is unavailable in our benchmark. (c) The miRNAture pipeline favors the opposite strand. Details can be found in The 11 families with candidates disjoint from human annotation are Filtered out even though they show a perfect match with the current annotation. These cases can be traced back to miRNAs which belong to either species specific families, thus lacking homology information, or have only been found in a small set of other species, consequently restricting the available dataset of mature loci, or folding into invalid secondary structure. In total, this lead to rejection of 33 loci, see For 27 families, all candidates where miRNAture did not cover in the human genome fall into two broad classes: (1) members of repetitive families, for which we consider it uncertain whether they should be considered as canonical miRNAs; (2) precursors with deviant secondary structure or unusual placement of the mature sequences within the predicted secondary structure. These families deserve a closer look whether they are canonical miRNAs in the stringent sense used here. If so, they may prompt a future adjustment of the filtering criteria; (3) Families for which the query alignment and secondary structure contains an insufficient number of precursor sequences or contains undetected errors in alignment, positioning of the mature sequences, or consensus structure annotation. We regard them as border line cases that deserve further investigation into the underlying evidence, preferably from multiple species—a task that goes beyond the scope of this contribution.In summary, the loci that additional loci overlap with one or more annotated element(s)  additional candidates were predicted. At the same time, 5 families (6 loci) were removed by filtering steps. A comparison with the current annotation shows that ∼69.0% of those t(s) see . For 12  general ,44.hsa-mir-499b (MI0017396). The homology stage of miRNAture suggests that it belongs to the mir-499 family, however, miRBase does not include it in this family. We argue that hsa-mir-499b is correctly annotated by miRNAture.Twelve miRNA families show overlaps with other miRNAs that are annotated as human-specific, see miRNAture predicted candidates that do not overlap with any currently annotated genomic element on the same strand. The miRNA families mir-544 (50), mir-548 (42), mir-1302 (27), mir-1289 (21), mir-649 (19), mir-290 (17), and mir-297 (11) account for nearly half of them. To further investigate those candidate loci, we intersected available annotation specifically at their ‘antisense’ strand and found 105 overlaps. Interestingly, more than half  a et al.  revealedmir-544 loci overlap with the DNA transposon MER (medium reiterated frequency repeat). It is interesting to note that the annotated mir-544 loci are located in the DLK1-DIO3 imprinted region [mir-548 loci overlap with Tc1/Mariner. An alignment of the mir-548 predictions is shown in mir-548 family has received detailed attention in the past [mir-548 loci have been reported to match Made1 elements in both reading directions [Many of the additional loci overlap specific repeat families, see d region . Similarthe past ,48, highrections . The fammir-764, mir-140 and mir-1306.Furthermore, 12 loci from 11 families passed the validation stage, but the predicted position in the human genome is not overlapping with annotation see . A bettemiRNAture, the strand may be mis-annotated, miRNAture based on the prediction of unusual positions of the mature sequences in relation to the secondary structure for the other strand. Examples are overlap of the mature sequence and the hairpin loop, or a multi-loop structure, see mir-101, mir-103 (2 loci), mir-122, mir-1245, mir-290, mir-451, mir-4536, mir-515, and mir-548. Difficulties with the mir-451 family are not unexpected due to its atypical biogenesis and a dominant mature product deriving from the loop region [For six loci that pass all evaluation steps of p region .miRNAture, 90 annotated miRNA loci from 13 families were not recovered. The majority of which were not reported due to the large number of detected homolog loci since miRNAture limits candidate lists to the 100 loci with the best bit-scores. This cut-off is intended to exclude candidates associated with highly repetitive sequences for which different, synteny-aware methods have to be used, see, for example, reference [potential candidates for later inspection by the user. 68 missing candidates can be explained in this manner. For example, mir-548 represents a highly repetitive miRNA (with 74 annotated loci in miRBase). miRNAture detected in total 6626 candidates by homology searches, highlighting the need for stringent cut-offs. From them, the best 100 bit-score candidates were subject to mature annotation and compared to the annotation, 63 were classified as potential and another 2 were predicted on the opposite strand. Among the remaining 22 of the 90 missing loci belonging to non-repetitive families are three predictions on the opposite strand  and miRNAture rejected a total of 19, including five mir-548 paralogs, four mir-378 sequences, and three mir-506 loci, while the remaining 7 undetected loci were not recognized by the corresponding HMM or CM miRNA family models.At the homology stage of eference . Candidamissing, see mir-550 family (MIPF0000334), for example, has five loci annotated in human. These were rejected by miRNAture because the mouse and chimpanzee loci retained in the input did no pass the secondary structure filters. Similarly, all three mir-1184 loci predicted at the homology stage were rejected because of the atypical secondary structure of the only remaining input sequences (from chimpanzee), see In the final result, that is, after the validation stage, 161 loci from 66 families are classified as missing candidates are the 28 families that matched the current annotation only partially, accounting for 116 loci. These include the ten loci assigned to opposite strand and the 66 highly repetitive loci tagged as potential that have been discussed above. Of the remaining 40 loci, 19 were not detected by homology and 21 were rejected by miRNAture at the evaluation stages. Here, the CM constructed from miRBase data after removing the human sequences did not match the annotated human loci. For example, hsa-let-7g was rejected at the validation stage because the common let-7 CM identified a sequence that was shifted relative to the paralog-specific results of oan-let-7g-5p, human mature sequences overlap the hairpin region, explaining the rejection.Another source of miRNAture pipeline is based on the observation that efficient homology search requires the interplay of fast and ideally loss-less identification of candidate loci in the genome of interest, and subsequent filtering to remove the false-positives. Since it is not difficult to increase the sensitivity of initial search , better and in particular more complete results can be achieved by developing more efficient filters. This is not a new principle, of course. HMMs improve over single-sequence blast queries by including patterns of sequence conservation, and covariance models provide another jump in accuracy by incorporating the conservation on secondary structure level. The trade-off, however, is the need for more and more information on the query side. While blast requires only a single sequence, nhmmer requires a multiple sequence alignment to derive the HMM model, and the CMs used by cmsearch need a consensus structure in addition to the sequence alignment. CMs thus are helpful only if the RNA family has an evolutionary well-conserved secondary structure.The miRNAture increases the achievable sensitivity by further restricting the scope of queries—its filters are highly specific for canonical microRNAs, i.e., those that share all the typical features of miRNA precursors, in particular a secondary structure that resembles a nearly symmetric stem-loop and a sequence conservation pattern governed by the location of the mature products on both sides of the stem region. Therefore, miRNAture tends to reject members of atypical families such as those associated with repetitive elements. The main use of miRNAture is to reliably process the typical cases and to limit the need for extensive manual analysis to miRNAs and miRNA-like ncRNAs with atypical features.miRNAture to canonical microRNAs. While general approaches can be extended to other classes of RNAs, such as box C/D snoRNAs or box H/ACA snoRNAs, class-specific filters need to be developed and tested. This requires extensive domain knowledge and thus makes it difficult to extend the strategy to poorly understood ncRNAs.The integration of the mature sequences and the evaluation of folding energies reaches beyond the information captured by HMMs and even CMs. This yields more stringent filters that make it feasible to increase the sensitivity of the initial homology search. The cost incurred for this advantage are the restriction of miRNAture pipeline is designed specifically to facilitate homology search for canonical miRNAs. The most obvious use case is the annotation of all conserved miRNA families in one or more new genomes. Complementarily, studies into the evolution of specific miRNA families require that (i) distant homologs can be detected reliably and (ii) no spurious apparent homologs are included. Only then is it possible to pinpoint the evolutionary origin of a miRNA family [miRNAture usefully assists both tasks, a number of issues remain that will require manual intervention and post-processing. Most importantly, the method relies on correct initial models for each microRNA. We recommend to use models that are specific to individual miRBase families, or—in the case of families with divergent paralogs—even paralog specific input alignments. While it is possible to use Rfam family models, these turned out to be too promiscuous in many cases, resulting in relatively large fractions of rejected candidates.The A family ,17,40. AmiRBase or Rfam by design contain entries that satisfy certain levels of evidence. These evidence criteria, however, imply massive ascertainment biases between organisms as a consequence to the large differences in the available empirical evidence. On the other hand, the definition of miRNAs as a class is fuzzy to certain extent as well, implying that not all database entries share all the features that are typical animal miRNAs. In miRNAture, very stringent quality criteria are implemented. While the evaluation against the human annotation shows that clear false positive calls are rare and largely confined to repeat-associated families, miRNAture fails on miRBase families with atypical features. The miRNAture pipeline also reports candidates of the homology stage that are later rejected by the automatic curation procedure to enable expert inspection. Such datasets are required to gather enough knowledge about miRNAs with atypical features.The study presented here also highlights the difficulty of benchmarking homology search tools for ncRNAs. The main reason is the lack of a gold standard of sufficient quality and coherence. Databases such as miRNAture and similar tools against simulated data with a guaranteed ground truth. The difficulty is that checking the sensitivity and specificity of the filters requires a way of simulating the evolution of artificial miRNAs that is independent of filter rules employed by miRNAture. It would be easy of course, to use miRNAture, that is, the MIRfix-based evaluation to model the selection pressures on miRNAs, but then our filters would be perfect by construction, and no information on the biological correctness of the filters could be gained. On the other hand, it is very simple to construct negative examples, since 10–20% of randomly placed point mutations is known to almost certainly destroy the secondary structure [miRNAture with the known, well-curated miRNA annotation of the human genome. Again a fully quantitative evaluation is difficult, because of the gray-zone between bona fide canonical miRNAs and other miRNA-like genes.In principle, it would be desirable to benchmark tructure . We havemiRNAture may serve as a blueprint for a new generation of homology search tools that rely on class-specific post filters. Here, we have manually constructed the homology and secondary structure filters, making use of explicit knowledge on structure and biogenesis of miRNAs. It seems tempting to use machine learning classifiers for miRNA gene detection  as an aSequence homology, Alignment scores, Annotation/Structure, and Consensus secondary structure. miRNAture workflow.The sequence/structure filters can be grouped by type of evaluation: Sequence homology searches are evaluated in terms of E-values, coverage and length of resulting high scoring pairs (HSPs) as suggested in [Pairwise comparisons with user-defined query sequences in ested in .nhmmer models as suggested in the HMMer userguide (http://eddylab.org/software/hmmer/Userguide.pdf (accessed on 18 December 2020)). Direct comparisons to miRNA CMs make use of the parameters calculated by cmsearch: E-value, bitscore and coverage with respect to the length of the CM. An uniform bitscore cutoff of N denotes the genome size. If a gathering cutoff ).HMM comparisons are evaluated w.r.t. the default inclusion thresholds of the f MIRfix   distance  between Ensembl FTP site (Release 100): gorilla (Gorilla gorilla: gorGor4), chimpanzee (Pan troglodytes: Pan_tro_3.0), sumatran orangutan (Pongo abelii: PPYG2), human (H. sapiens: GRCh38.p13), mouse (Mus musculus: GRCm38.p6), and Pacific transparent sea squirt (Ciona savignyi: CSAV 2.0).The following genome assemblies were retrieved from the Let-7 loci from G. gorilla, H. sapiens, P. abelii, P. troglodytes, M. musculus and C. savignyi were retrieved from miRBase v.22 in FASTA and GFF3 format. In addition, the let-7 loci reported in [let-7 loci from [miRBase were used as reference for evaluation.orted in  were retoci from  and fromlet-7 were retrieved from Rfam v.14.4 (RF00027), Hertel et al. [miRBase v.21, MIPF0000002). An additional CM was constructed using the bilaterian sequences from miRBase v.22, excluding both, paralogous sequences with 100% identity and sequences from the target species. All models were used as input for Infernal and for and Other_CM homology modes in miRNAture. All let-7 retrieved sequences from miRBase were used as queries using BLAST mode in miRNAture with strategies 1,2,3, 5 and 6.CMs for l et al.  (17 model et al.  curated miRNAture are stored in GFF3 format. Comparisons on genomic loci level were performed using bedtools [Match: overlaps on the same strand; Miss: locus in references without overlap in miRNAture output; Additional: candidates detected by miRNAture without overlap in reference.Both reference loci and the final results of bedtools  and clasmiRBase v.22 and corrected with MIRfix [Infernal [miRNA precursor and mature sequences were retrieved from h MIRfix  to creatmiRNAture was conducted to retrieve the final numbers from GFF3 files generated by miRNAture were compared to current annotation using bedtools v2.27.1 [Covariance models where built for 350 miRNA families, excluding all sequences corresponding to the human genome from the dataset described in  v2.27.1 . Specifi


Personal genomics and comparative genomics are becoming more important in clinical practice and genome research. Both fields require sequence alignment to discover sequence conservation and variation. Though many methods have been developed, some are designed for small genome comparison while some are not efficient for large genome comparison. Moreover, most existing genome comparison tools have not been evaluated the correctness of sequence alignments systematically. A wrong sequence alignment would produce false sequence variants.In this study, we present GSAlign that handles large genome sequence alignment efficiently and identifies sequence variants from the alignment result. GSAlign is an efficient sequence alignment tool for intra-species genomes. It identifies sequence variations from the sequence alignments. We estimate performance by measuring the correctness of predicted sequence variations. The experiment results demonstrated that GSAlign is not only faster than most existing state-of-the-art methods, but also identifies sequence variants with high accuracy.As more genome sequences become available, the demand for genome comparison is increasing. Therefore an efficient and robust algorithm is most desirable. We believe GSAlign can be a useful tool. It exhibits the abilities of ultra-fast alignment as well as high accuracy and sensitivity for detecting sequence variations. With the development of sequencing technology, the cost of whole genome sequencing is dropping rapidly. Sequencing the first human genome cost $2.7 billion in 2001; however, several commercial parties have claimed that the $1000 barrier for sequencing an entire human genome is broken  be the suffix array of T, such that SA[i] indicates the starting position of the i-th lexicographically smallest suffix. The BWT of T is a permutation of T such that BWT[i] = T[SA[i] − 1] . Given a pattern S, suppose SA[i] and SA[j] are the smallest and largest suffices of T where P is their common prefix, the range  indicates the occurrences of S. Thus, given an SA range  of pattern P, we can apply the backward search algorithm to find the SA range  of zP for any character z. If we build the BWT with the reverse of T, the backward search algorithm can be used to test whether a pattern P is an exact substring of T in O(|P|) time by iteratively matching each character in P. One of the BWT index algorithms was implemented in BWT-SW  be the i1-th nucleobase of P, and P be the sequence fragment between P[i1] and P[i2]. GSAlign finds LMEMs by searching against the BWT array with Q. Since each LMEM is a common substring that begins at a specific position of Q, it is represented as a simple pair  in this study and denoted by a 4-tuple , meaning P = Q and P[i2 + 1] ≠ Q[j2 + 1]. If the common substring appears multiple times , it would be transformed into multiple simple pairs. For example, if the substring Q is identical to P and P, it would be represented as two simple pairs  and . Note that an LMEM is transformed into simple pairs only if its size is not smaller than a user-defined threshold k and its occurrences are less than f. We investigate the effect of threshold k and f in the Given two genome sequences Q. It begins with Q[j1] (j1 = 0 at the first iteration) and stops at Q[j2] if it meets a mismatch at Q[j2 + 1], i.e., the SA range of Q = 0. The next iteration of BWT search will start from Q[j2 + 1] until it meets another mismatch. When GSAlign is running with sensitive mode, the next iteration of BWT search starts from Q[j1 + 5] instead of Q[j2 + 1]. In doing so, GSAlign is less likely to miss true LMEMs due to false overlaps between P and Q. The search procedure terminates until it reaches the end of genome Q.The BWT search iteratively matches every nucleotide of the query genome Q, GSAlign divides it into N blocks of equal size when it is running with N threads and each thread identifies LMEMs for a sequence block independently. The multithreading can be also applied in the following alignment step. We will demonstrate that such parallel processing greatly speedup the alignment process.Please note that the LMEM identification can be processed simultaneously if GSAlign runs with multiple threads. For each query sequence in P and Q and clusters those into disjoint groups. The clustering algorithm is described below.After collecting all simple pairs, GSAlign sorts all simple pairs according to their position differences between genomes Sk is a simple pair , we define PosDiffk = ik,1− jk,1. If two simple pairs have similar PosDiff, they are co-linear. We sort all simple pairs according to their PosDiff to group all co-linear simple pairs. The clustering starts with the first simple pair S1 and we check if the next simple pair (S2) is within a threshold MaxDiff . The size of MaxDiff determines the maximum indel size allowed between two simple pairs. If |PosDiff1 − PosDiff2| ≤ MaxDiff, we then check the PosDiff of S2 and S3 until we find two simple pairs Sk and Sk + 1 whose |PosDiffk − PosDiffk + 1| > MaxDiff. In such cases, the clustering breaks at Sk + 1 and simple pairs S1, S2, …, Sk are clustered in the same group. We investigate the performance of GSAlign with different values of MaxDiff and summarize the analysis in Suppose S1, S2, …, Sk by their positions at sequence Q . Since simple pairs are re-sorted by their positions at sequence Q, some of them may be not co-linear with their adjacent simple pairs and they are considered as outliers. We remove those outliers from the simple pair group. A simple pair Sm is considered as an outlier if |PosDiffm − PosDiffm − 1| > 5 and |PosDiffm − PosDiffm + 1| > 5 where Sm-1, Sm and Sm + 1 are adjacent. In such cases, we will perform a dynamic programming to handle the gap between Sm-1 and Sm + 1.We then re-sort PosDiff compared to the closest unique simple pair. Then we check every two adjacent simple pairs sa =  and sb = , we define gap = jb,1 − ja,2. If gap is more than 300 bp and the sequence fragments in the gap are dissimilar, we consider Sb as a break point of a similar region. To determine whether the sequence fragment in a gap are similar, we use k-mers to estimate their similarity. If the number of common k-mers is less than gap / 3, they are considered dissimilar. In such cases, we consider Sb as a break point of a similar region, and Sb will initiate another similar region. We investigate different gap size thresholds in the For those simple pairs of same positions at sequence Q , we keep the one with the minimal difference of 1. Simple pairs S1, S2, …, S8 are clustered in the same group since any two adjacent simple pairs in the group have similar PosDiff. For example, |PosDiff1 − PosDiff2| = 10, and |PosDiff2 − PosDiff3| = 0. By contrast, |PosDiff8 − PosDiff9| = 60, we break the clustering at S9. We then re-sort S1, S2, …, S8 by their positions at sequence Q as shown in Fig. 6 and S7 are not unique since the two simple pairs have the same position at Q. We compare S6 and S7 and keep S6 because it has the minimal difference of PosDiff with its neighboring unique simple pairs.We use an example to illustrate the process of simple pair clustering and outlier removing. Suppose GSAlign identifies nine simple pairs as shown in Fig. 1 and S8 since they are not co-linear with their adjacent simple pairs. S1 is considered an outlier because |PosDiff1 − PosDiff3| > 5 and |PosDiff1 − PosDiff6| > 5. After S1 is removed, the gap between S3 and S6 would probably form an un-gapped alignment since they have the same PosDiff. S8 is also an outlier because |PosDiff8 − PosDiff5| > MaxDiff. Finally, we confirm there is no any large gap between any two adjacent simple pairs in the group. Thus, the group of S3, S6, S2, S4, and S5 forms a similar region, and upon which we can generate a local alignment.We remove Ssa =  and sb = , we say sa and sb overlap if ia,1 ≤ ib,1 ≤ ia,2 or ja,1 ≤ jb,1 ≤ ja,2. In such cases, the overlapping fragment is chopped off from the smaller simple pair. For example, BWT index. Figure P and Q. In this example, “ACGT” is a tandem repeat where P has seven copies and Q has nine copies. GSAlign identifies two simple pairs in this region: A  and B . A and B overlap between P. In such cases, we remove the overlap from the preceding simple pair . After removing the overlap, A becomes  and we create a gap of Q. After removing overlaps, we check if there is a gap between any two adjacent simple pairs in each similar region. We fill gaps by inserting normal pairs. A normal pair is also denoted as a 4-tuple  in which P ≠ Q and the size of P or Q can be 0 if one of them is an deletion. Suppose we are given two adjacent simple pairs  and . If iq + 12 − iq2 > 1 or jq + 12 − jq2 > 1, then we insert a normal pair  to fill the gap, where ir – iq2 = iq + 12 – ir + 1 = 1 if iq + 12 − iq2 > 1; otherwise ir = ir + 1 = − 1 meaning the corresponding fragment size is 0. Likewise, jr – jq2 = jq + 12 – jr + 1 = 1 if jq + 12 − jq2 > 1, otherwise let jr = jr + 1 = − 1.Given two adjacent simple pairs in the same cluster, At this point, GSAlign has identified similar regions that consist of simple pairs and normal pairs. In this step, GSAlign only focuses on normal pairs. If the sequence fragments in a normal pair have equal size, it is very likely the sequence fragments only contain substitutions and the un-gapped alignment is already the best alignment; if the sequence fragments contain indels, gapped alignment is required. Therefore, we classify normal pairs into the following types:Type I if the fragment pair has equal size and the number of mismatches in a linear scan is less than a threshold;1) A normal pair is Type II if one of the fragment is a null string and the other contains at least one nucleobase;2) A normal pair is Type III;3) The remaining normal pairs are Type III require gapped alignment. GSAlign applies the KSW2 algorithm [Thus, only lgorithm  to perfoType III normal pairs require gapped alignment. This divide-and-conquer strategy not only reduces the number of fragment pairs requiring gapped alignment, but also shortens gap alignment sizes. Furthermore, GSAlign can produce the alignments of normal pairs simultaneously with multi-threads. Though MUMmer4 supports multi-threads to align query sequences in parallel, the concurrency is restricted to the number of sequences in the query.In general, GSAlign, MUMmer4, and Minimap2 follow the conventional seed-chain-align procedure to align genome sequences. However, the implementation details are very different from each other. MUMmer4 combines the ideas of suffix arrays, the longest increasing subsequence (LIS) and Smith-Waterman alignment. Minimap2 uses minimizers (k-mers) as seeds and identifies co-linear seeds as chains. It applies a heuristic algorithm to cluster seeds into chains and it uses dynamic programming to closes between adjacent seeds. GSAlign integrates the ideas of BWT arrays, PosDiff-based clustering and dynamic programming algorithm. GSAlign divides the query sequence into multiple blocks and identifies LMEMs on each block simultaneously using multiple threads. More importantly, GSAlign classifies normal pairs into three types and only GSAlign takes two genome sequences: one is the reference genome for creating the BWT index, and the other is the query genome for searching against the BWT array. If the reference genome has been indexed beforehand, GSAlign can read the index directly. After comparing the genome sequences, GSAlign outputs all local alignments in MAF format or BLAST-like format, a VCF file, and a dot-plot representation  for each query sequence.The correctness of sequence alignment is an important issue and variant detection is one of the major applications for genome sequence alignment. Therefore, we estimate the correctness of sequence alignments by measuring the variant detection accuracy. Though most of genome alignment tools do not output variants, we can identify variants by linearly scanning the sequence alignments. This measurement is sensitive to misalignments; thus we consider it is a fair measurement to estimate the performance of sequence alignment.n produces n mismatches. Thus, the ASI of the four datasets are 97.93, 93.86, 89.90, and 99.84%, respectively.We randomly generate sequence variations with the frequency of 20,000 substitutions (SNVs), 350 small indels (1~10 bp), 100 large indels (11~20 bp) for every 1 M base pairs. To increase the genetic distance, we generate different frequencies of SNVs. Benchmark datasets labelled with 1X contain around 20,000 SNVs for every 1 M base pairs, whereas datasets labelled with 3X (or 5X) contain 60,000  SNVs per million bases. We generate three synthetic datasets with different SNV frequencies using the human genome (GRCh38). The synthetic datasets are referred to as simHG-1X, simHG-3X, and simHG-5X, respectively. To evaluate the performance of genome sequence alignment on real genomes, we download the diploid sequence of NA12878 genome and its reference variants (the sources are shown in Supplementary data). We also estimate the Average Sequence Identity (ASI) based on the total number of mismatches due to the sequence variants over the genome size. For example, an SNV event produce one mismatch and an indel event of size The diploid sequence of NA12878 consists of 3,088,156 single nucleotide variants (SNVs) and 531,315 indels. The reference variants are generated from NGS data analysis. Please note that GSAlign is a genome alignment tool, rather than a variant caller such as Freebayes or GATK HaplotypeCaller (GATK-HC). GSAlign identifies variants from genome sequence alignment, while Freebayes and GATK-HC identify variants from NGS short read alignments. We use sequence variants to estimate the correctness of sequence alignment in this study. Table In this study, we compare the performance of GSAlign with several existing genome sequence aligners, including LAST (version 828), Minimap2 (2.17-r943-dirty), and MUMmer4 (version 4.0.0beta2). We exclude the others because they are either unavailable or developed for multiple sequence alignments, like Cactus , Mugsy , or MULTWe define true positives (TP) as those variants which are correctly identified from the sequence alignment; false positives (FP) as those variants which are incorrectly identified; and false negatives as those true variants which are not identified. A predicted SNV event is considered true if the genomic coordinate is exactly identical to the true event; a predicted indel event is considered true if the predicted coordinate is within 10 nucleobases of the corresponding true event. The precision and recall are defined as follows: precision = TP / (TP + FP) and recall = TP / (TP + FN).To estimate the performance for existing methods, we filter out sequence alignments whose sequence identity are lower than a threshold (for Mummer4 and LAST) or those whose quality score are 0 (for Minimap2). The argument setting used for each method is shown in the Table In terms of runtime, it can be observed that GSAlign spends the least amount of runtime on the three datasets. Minimap2 is the second fastest method. Though MUMmer4 is faster than LAST, it produces worse performance than LAST. We observe that LAST is not very efficient with multi-threading. Though it runs with eight threads, it only uses single thread most of time during the sequence comparison. Interestingly, GSAlign spends more time on less similar genome sequences (ex. simHG-5X) because there are more gapped alignments, whereas MUMmer4 and LAST spends more time on more similar genome sequences (ex. simHG-1X) because they handle more number of seeds. Minimap2 spends similar amount of time on the three synthetic datasets because Minimap2 produces similar number of seeds for those datasets. Note that it is possible to speed up the alignment procedure by optimizing the parameter settings for each method; however, it may complicate the comparison.The two sets of diploid sequence of NA12878 are aligned separately and the resulting VCF files are merged together for performance evaluation. Because many indel events of NA12878 locate in tandem repeat regions, we consider a predicted indel is a true positive case if it locates at either end of the repeat region. For example, the two following alignments produce identical alignment scores:AGCATGCATTG AGCATGCATTG.AGCAT----TG, and AG----CATTG.It can be observed that the two alignments produce different indel events.In such case, both indel events are considered true positives if one of them is a true indel.Table In terms of runtime, it is observed that GSAlign only spends 5 min to align the diploid sequences of NA12878 with HG38. Minimap2 is the second fastest method. It spends 65 min. LAST and MUMmer4 spend 1305 and 3898 min, respectively. In terms of memory consumption, it is observed that GSAlign consumes the least amount of memory among the selected methods. It requires 14 GB to perform the genome comparison, while MUMmer4 requires 57 GB.Though GSAlign is designed for comparing intra-species genomes, it can be used to identify conserved syntenic regions for inter-species genomes. Here we compare human genomes with whole chimpanzee genome and mouse chromosome 12. We compare human (GRCh38) and chimpanzee PanTro4) genomes using the four selected tools with 8 threads. Since the ground-truth alignment between GRCh38 and chimpanzee (PanTro4) genomes is unknown, we only show the total alignment length, the predicted SNV and Indel numbers of each alignment tool as well as their run time. We summarize their result in Table  genomes Mouse chromosomes share common ancestry with human chromosomes . Here weIn this study, we present GSAlign, a new alignment tool that handles genome sequence comparison. We evaluate the correctness of sequence alignment by measuring the accuracy of variant detection. GSAlign adopts the divide-and-conquer strategy to divide genome sequences into gap-free fragment pairs and gapped fragment pairs. GSAlign is a BWT-based genome sequence aligner. Therefore, it requires less amount of memory than hash table-based or tree-based aligners do. GSAlign also supports multi-thread computation, thus it is more efficient when comparing large genomes. We evaluate the performances of GSAlign with synthetic and real datasets. The experiment result shows that GSAlign is the fastest among the selected methods and it produces perfect or nearly perfect precisions and recalls on the identification of sequence variations for most of the datasets. We also found that the diploid genome sequence of NA12878 is not fully compatible with the reference variants derived from NGS data.As more genome sequences become available, the demand for genome comparison is increasing. Therefore, an efficient and robust algorithm is most desirable. We believe GSAlign can be a useful tool. It shows the abilities of ultra-fast alignment as well as high accuracy and sensitivity for detecting sequence variations.Project name: GSAlign.https://github.com/hsinnan75/GSAlignProject home page: Operating system: Linux.Programming language: C/C++.Other requirements: N/A.License: MIT License.Additional file 1 : Table S1. A summary of several existing genome sequence alignment tools. Table S2. The effect of minimal LMEM size k and their maximal frequency f for GSAlign on the Sim_Chr1 dataset. Table S3. The effect of MaxPosDiff for GSAlign on the Sim_Chr1 dataset. Table S4. The effect of gap size threshold on the Sim_Chr1 dataset. TableS5 lists the argument setting for each method tested in this study. Aligner and their arguments used on the benchmark datasets, where fa1 and fa2 are input genomes with FASTA format.


The vast number of protein structures currently available opens exciting opportunities for machine learning on proteins, aimed at predicting and understanding functional properties. In particular, in combination with homology modelling, it is now possible to not only use sequence features as input for machine learning, but also structure features. However, in order to do so, robust multiple structure alignments are imperative.Here we present Caretta, a multiple structure alignment suite meant for homologous but sequentially divergent protein families which consistently returns accurate alignments with a higher coverage than current state-of-the-art tools. Caretta is available as a GUI and command-line application and additionally outputs an aligned structure feature matrix for a given set of input structures, which can readily be used in downstream steps for supervised or unsupervised machine learning. We show Caretta’s performance on two benchmark datasets, and present an example application of Caretta in predicting the conformational state of cyclin-dependent kinases. While sequence alignment aims to use amino acid substitution patterns and physicochemical properties to make a residue-residue correspondence between sequences of related proteins, structure alignment instead usually focuses on making an optimal superposition of the 3D coordinates of backbone atoms to establish such a correspondence. In many cases these two approaches agree with each other, especially in cases where the proteins under consideration share a high sequence similarity. However, it has been repeatedly observed Until recently, structure alignment has had applications in understanding evolutionary conservation and divergence patterns between proteins across different species Though machine learning is not a new field, its popularity and applicability in bioinformatics has recently grown at a tremendous pace. In the protein and enzyme world, machine learning has successfully been applied to predict protein function, protein–protein interactions, drug-target binding, enzyme substrate specificity, thermostability, catalytic rates, binding affinity, and so on The typical input for a machine learning algorithm has a tabular format, with each row representing an input protein and each column representing a particular feature or attribute extracted across all the proteins considered. Naturally, the construction of such an input table is often performed by means of a multiple protein alignment. Each column then consists of a particular feature value measured across all the residues in a particular alignment position. This then allows the prediction algorithm to look for patterns in these columns which are correlated with the desired response. For example, in an alignment of ten proteins, if one position is a Trp in the five proteins with a high catalytic rate and a Gly in the five with a lower catalytic rate then this residue position may be implicated in the reduction of catalytic activity. The power of machine learning algorithms lies in finding much more complex and interconnected patterns such as this one. Regions in the alignment with many insertions and deletions, however, can be more difficult to handle, as functionally equivalent residues may be split across multiple columns. This makes it harder for a predictor to spot patterns in a single column or link them together. Often, columns with too many gaps without feature information have to be discarded completely from the analysis, with the risk of losing out on predictive and catalytically important residues simply due to an alignment not fit for the task at hand.Although there are a number of multiple structure alignment tools, different tools excel in different settings. Many existing multiple structure alignment algorithms, such as Matt Here we present Caretta, a multiple structure alignment tool that additionally outputs aligned structural feature matrices. Caretta uses a combination of dynamic time warping We demonstrate that Caretta covers more residues in its alignments than competing tools while still maintaining accuracy. Testing on the widely used Homstrad dataset 2vs. all pairwise alignment step followed by the construction of a guide tree for progressive alignment, to finally output a multiple alignment. Each intermediate pairwise alignment step uses the dynamic programming approach detailed in Section 2.1et al.DPAlign.The algorithm underlying Caretta is dynamic programming alignment with affine gap costs as described by Altschul 2.2i.e. an alignment, leading to a chicken and egg problem for pairwise alignment. Caretta solves this by making an initial superposition of two structures using the best out of two coarse alignments: the first based on secondary structure  kernel derived from the Euclidean distance between two (superposed) x and y:This score is summed across all paired residues to derive the score of an alignment between two proteins PairwiseAlignment in Caretta uses the scoring scheme When more than two structures are required to be aligned, pairwise alignments are made for all input structures. This step is essential for the guide tree construction described in the next section.2.3The idea behind a progressive alignment approach is to perform step-wise alignments of two stacks of previously aligned structures (or single structures) to result in a final stack of all aligned structures. The order of addition of structures is a crucial factor in the performance of this method. Aligning similar structures first, with a smoother progression towards distantly related structures, increases the chances of a good alignment. We construct a guide tree for determining the order of progression using maximum linkage neighbor joining MultipleAlignment. While progressive alignment typically consists of independent pairwise alignment steps, the algorithmic novelty of Caretta lies in the introduction of a feedback loop between the state of the multiple structure alignment and each pairwise alignment, explained in detail below. For this purpose, an additional consensus row, of length equal to the protein length, is maintained for each structure, initiated with a consensus weight parameter . This row is concatenated to the coordinates With the guide tree in place, the progressive alignment steps start, as illustrated in consensus row of each neighbor is multiplied by half the number of structures represented by the other neighbor. This ensures that when their difference is taken during the calculation of consensus row attached), the positions with equal consensus values in both receive high scores (as the difference is close to zero), increasing their chances of being aligned. An intermediate node is created with a length equal to the length of the resulting pairwise alignment, representing the aligned stack of structures from the two neighbors. The z coordinates of this intermediate node are calculated by averaging the coordinates of the two initial structures after superposition, across the alignment. At each position in the alignment, the secondary structure code in the intermediate node is taken as the code of the input which does not have a gap, or the code of the first input if both are aligned. The consensus term for each alignment position is the number of aligned residues at that position times the cw, i.e. well-aligned positions with fewer gaps have a higher consensus value. This way, Caretta tends to maintain fully aligned core regions by avoiding the insertion of gaps at these locations as progressive alignment proceeds as such gaps are unlikely to happen in conserved protein families.Before two neighbors in the guide tree are aligned, the 2.42.4.1Caretta takes as input a list of PDB files, along with optional chain identifiers and start and end residue indices. All PDB file parsing is done using ProDy Caretta was tested on two benchmark datasets, Homstrad 2.4.2•Gap-less positions – Positions in an alignment that do not contain any gaps.•Homstrad equivalence score – Percentage of gap-less positions which are present in the corresponding Homstrad reference alignment.•RMSD – The root mean square deviation between two superposed structures in a pairwise alignment is given by:•Structurally equivalent residues – Residues in the same alignment position of a pairwise alignment within 4 Å of each other after superposition.To measure the quality of multiple structure alignments we make use of various metrics. The last two are defined for pairwise alignments, and are calculated for every pair of structures in the multiple structure alignment after superposing all structures to one reference structure, the longest protein. These are then averaged over all pairs to give the final score for a multiple alignment.2.4.3To estimate Caretta running times, we randomly chose 25 protein structures from the Homstrad dataset with differing lengths as “seeds”. Each seed was used to form multiple groups of proteins to be aligned by Caretta. Forming each of these groups involved introducing noise to the seed coordinates to create a given number of members, from 13 to 93 in increments of 30. Caretta was then used to align these groups on a Linux workstation using 20 threads.2.5In addition to multiple structure alignment, Caretta was designed specifically in order to enable feature extraction for downstream machine learning.Structural features are extracted for each input protein, aligned according to Caretta’s multiple structure alignment. All atom-level features are converted into ProDy i and the C = O of residue DSSP features are calculated using ProDy Residue depths are extracted using BioPython 2.63FZP, chain A) using PyMOL Caretta’s alignment and feature extraction capabilities are further demonstrated on the task of predicting the functional state of cyclin-dependent kinases (CDKs). PDB IDs of these proteins, along with the corresponding active/inactive labels, were obtained from 2.7•Structure alignment – displays the superposed 3D structures of the input proteins;•Sequence alignment – displays the multiple sequence alignment, colored by hydrophobicity;•Feature alignment – displays aligned structural features. The feature name under consideration can be changed using a drop-down box.The Caretta GUI was built using Dash and Dash-Bio These three panels are interlinked via interactive capabilities. Clicking a protein or a residue position in any of the three panels highlights the corresponding protein or position in the other two. All three panels can also be exported to different file formats for downstream use.33.1We compare Caretta with two popular multiple structure alignment methods, Matt and mTM-align. Matt uses a fragment-based approach, which allows for local flexibility between fragment pairs from two input structures and then a dynamic programming algorithm to assemble these intermediate pairs These two MSA tools were tested along with Caretta on the popular Homstrad and SABmark-Sup datasets. Assessing the quality of multiple structure alignments is a difficult task and, depending on the metric used, different aspects of the alignment come under consideration. While RMSD (root mean square deviation) is often used, it has been observed that fewer aligned residues can easily lead to smaller RMSDs at the expense of a very gap-filled alignment The Homstrad dataset is unique in that it provides manually curated and annotated alignments, representing a ground truth. This dataset has examples from various homologous protein families, typical of the kinds of applications where machine learning would be applied. Since these proteins are homologous, a high alignment coverage is expected as many of the residues are functionally equivalent, with few insertions and deletions. In vs. average percentage of gap-less columns for Matt, mTM-align and Caretta on the SABmark-Sup dataset. While Matt and mTM-align have a gap-less percentage range typically within 20–60%, this increases to 40–80% for Caretta, often still within the same RMSD range. This indicates that Caretta also performs well at fragmented substructure alignment, though optimizing the gap penalties and consensus weight may improve results further for individual cases.The SABmark-Sup benchmark dataset consists of proteins from the same superfamily with distant homology n is the number of proteins and l is the length of the longest protein in the alignment. The time complexity of Caretta’s alignment algorithm is 3.2Apart from An application of Caretta for a classification task is presented using a dataset of cyclin-dependent kinases (CDKs). This family of enzymes is involved in cell cycle regulation and its members share a high degree of structural similarity. Classical kinase inhibitors bind to the ATP site of CDKs and compete for substrate binding Fortunately, a large number of CDK structures have been experimentally solved. We used Caretta to align 80 CDK structures and extracted bond angles, Gaussian and anisotropic network model residue fluctuations, residue depths and solvent accessibility features from these structures. The alignment had a mean pairwise RMSD of 3.08 Å and 128 gap-less positions. We trained a logistic regression model to predict active/inactive states of CDKs using features aligned according to these gap-less positions and attained a mean cross-validation accuracy of 98%, with only 60 structures used for training in each split. The performance is summarized in 3.3www.bioinformatics.nl/caretta for aligning selected structures, from either a Pfam domain or a custom folder, and exploring their structural features. A Caretta GUI application can be found at 4Multiple sequence alignment is an integral part of a broad range of bioinformatics research topics, including phylogenetics, functional domain identification, co-evolution analysis and machine learning to predict functional properties of proteins. Compared to protein sequences, protein structures echo an even deeper evolutionary history that in a more direct way relates to their function. Previously, this kind of analysis was hindered by the scarcity of protein structures available. However, the number of solved protein structures is increasing at a great pace, and structural modelling methods are also improving rapidly, in part due to the use of co-evolutionary information when reliable structural templates are not available. This means it is now possible to analyse patterns correlating with function in a protein family by aligning, comparing and applying machine learning on a large set of solved or modeled structures.We contribute to this field with Caretta, a multiple structure alignment suite which returns accurate alignments with an increased ratio of aligned positions to make the best use of structural features from functionally comparable residues. Dong et al. p small n” problem in machine learning, where the number of descriptors far exceeds the number of labeled data points from which to learn. Feature selection in such cases removes noisy and irrelevant features, and can be used to find residue positions correlated with the response variable. We demonstrated this in our application on predicting the conformational state of cyclin-dependent kinases, where we found a small set of predictive residues, some of which lie in previously studied motifs known to be involved in conformational change.In the Caretta GUI, we coupled structural alignment and feature extraction with a visual interface to pinpoint relevant proteins and residue positions for downstream prediction tasks. This kind of feature selection becomes necessary as proteins typically have many hundreds of residues, each of which is described by a number of structural features. This quickly leads to what is known as the “large More research into protein families using the approach we present for dealing with structural alignments and residue selection across a large set of structural features will lead to improvements and novel techniques for feature selection, dimensionality reduction, and learning that work well on such large, hierarchically structured data. Given the prominent role in present-day bioinformatics of both machine learning and homology modelling, this will lead to further breakthroughs in using protein structures to analyse protein function.The authors declare no conflicts of interest.Mehmet Akdel: Conceptualization, Methodology, Software, Validation, Formal analysis, Writing - original draft. Janani Durairaj: Conceptualization, Methodology, Software, Validation, Formal analysis, Writing - original draft. Dick de Ridder: Writing - review & editing, Supervision. Aalt D.J. van Dijk: Writing - review & editing, Supervision, Project administration.