The 3D structures of RNA play a critical role in understanding their functionalities. There exist several computational methods to study RNA 3D structures by identifying structural motifs and categorizing them into several motif families based on their structures. Although the number of such motif families is not limited, a few of them are well-studied. Out of these structural motif families, there exist several families that are visually similar or very close in structure, even with different base interactions. Alternatively, some motif families share a set of base interactions but maintain variation in their 3D formations. These similarities among different motif families, if known, can provide a better insight into the RNA 3D structural motifs as well as their characteristic functions in cell biology.In this work, we proposed a method, RNAMotifComp, that analyzes the instances of well-known structural motif families and establishes a relational graph among them. We also have designed a method to visualize the relational graph where the families are shown as nodes and their similarity information is represented as edges. We validated our discovered correlations of the motif families using RNAMotifContrast. Additionally, we used a basic Naïve Bayes classifier to show the importance of RNAMotifComp. The relational analysis explains the functional analogies of divergent motif families and illustrates the situations where the motifs of disparate families are predicted to be of the same family.https://github.com/ucfcbb/RNAMotifFamilySimilarity.Source code publicly available at  The availability of 3D structural data for different types of RNAs is increasing gradually. The understanding of these 3D structural data uncovers a new dimension to RNA-based research and therapeutics . Due to RNAMotifScan , RNAMotiIn this work, we have proposed a method, RNAMotifComp, that analyzes the similarities in 3D structures among the input instances of different structural motif families of RNAs and generates a similarity graph to represent the relation among the known families. This similarity is analyzed not only based on their 3D formations but also from their isosteric base-interaction point of view. RNAMotifComp takes two or more motif families as input and generates a similarity graph by analyzing the input motif instances to show if they have any structural resemblance. It also identifies the instances that are similar to the instances of another motif family and writes them in separate text files as output for further analysis. To find the similarities from the interaction point of view, we employed the alignment tool from RNAMotifScanX  that actTo show the similarities among different motif families using RNAMotifComp, we have utilized a dataset of known RNA structural motif families. The same motif instances were used to benchmark our analysis. To find the 3D structural resemblance from the base interaction point of view, we have employed the alignment tool from RNAMotifScanX. On the other hand, to explore the similarities from the coordinate point of view, we have utilized the RNA-align alignment tool. In both cases, the alignment results are used to calculate the Root-Mean-Square Deviation (RMSD) values between two motif instances. The closeness of any two motifs is determined by considering the alignment result . RNAMotifContrast is used as a clustering tool on an unannotated motif dataset to verify the results produced from RNAMotifComp. We have also applied a basic Naïve Bayes classifier to show the effect of structural similarity in motif family identification. A brief description of the dataset, procedures of RNAMotifComp, and the benchmarking steps are given in the following subsections.We have used a total of 360 instances of internal loop motifs from 11 well-known structural motif families for this work. The distribution of these motif instances is given in To generate the interaction-based alignment between two motifs using RNAMotifScanX, we needed to provide the base-pairing and base-stacking interactions along with their nucleotide sequences. To do so, we have downloaded all the PDB files corresponding to the motif locations that were used as input for our analysis. We then generated the annotations for all these PDBs using the DSSR  annotatiRNA-align aligns the motif instances based on the TM-score and it uses the 3D coordinates of the backbone atom while doing the alignments. Moreover, it requires the whole PDB files to work with and RNA-align aligns the first chains of each PDB file. To simplify the process and to generate the alignments for specific motif locations only, we have to extract those locations and generate partial PDB files for each motif instance. These partial PDB files were utilized by RNA-align to generate the coordinate-based structural alignments.N is the total number of atoms, and i.While doing the structural alignment of two motif instances, we get an aligned set of nucleotides as the alignment result. Alignment length is the total number of aligned nucleotides of that outcome excluding any insertion and deletions. In both types of alignment cases (interaction-based and coordinate-based), we have considered the coordinates of all heavy atoms from the backbone and the sugar ribose for the aligned nucleotides. We have calculated the centroid of these heavy atoms and then calculated the RMSD from the centroid to centroid using the following formula:where We have used two different alignment tools to show the relationships among the structural motif families from two different points of view. The alignment results are then utilized to calculate the RMSD between two motifs. A key point to note here is that the RMSD value is not length independent and the value can vary based on the alignment length. To handle this scenario and to make sure the two motifs with low RMSD are actually structurally similar, we considered the alignment length threshold for both motif families while measuring the similarity. We made sure the alignment length is equal to or higher than the threshold of both motif families so that their own structural features could be conserved. Another key concern of using two different alignment tools is that the tools are not performing the same type of alignment. RNAMotifScanX aligns the interactions and provides locally aligned regions as output. On the other hand, RNA-align aligns the structure using their 3D coordinates and their result is a global alignment. As the working criteria for the two alignment tools are quite different, it is highly likely to have a bit of divergence in the alignment result as well as in the RMSD value for a specific pair of motif instances. In both cases, RMSD is calculated based on the aligned nucleotides only. As RNAMotifScanX performs local alignment and RNA-align performs global alignment, the RMSD value found from the RNA-align alignment result would be comparatively higher than the RMSD value calculated from the RNAMotifScanX alignment. We have considered this imparity while setting the cutoff value for RNAMotifComp.In RNAMotifComp, two different motif families are defined as similar if a subset of the instances of one motif family is found structurally closer to a subset of another motif family. The closeness is measured from two different points of view: interaction-based and coordinate-based. The two participating subsets from two different motif families closely aligned in the interaction based alignment result with a lower average RMSD value based on their aligned region are identified as structurally similar from interaction point of view. Similarly, the two participating subsets will be similar from the coordinate point of view if they are better aligned to each other while using coordinate based alignment tool and also they have low average RMSD value. The process of calculating RMSD for both interaction-based and coordinate-based alignment is discussed in the previous subsection. At the beginning of our analysis, we generated all pair alignments between two well-known motif families. To focus on the interaction-based structurally similar instances, after generating all pair alignments between two motif families, we selected only the best-aligned pairs based on the following criteria. The first step is to select the alignments for which the alignment length is greater or equal to both family-specific alignment length thresholds. This family-specific alignment length threshold is calculated by taking the average of all pair alignments among the instances of the same motif family. The family-specific threshold implies that, if a motif instance is to be considered a member of a specific motif family, it needs to have an alignment length of equal or higher value compared to the threshold value of that particular family. The next step for an alignment pair to be considered as best alignment is having an RMSD value greater than or equal to a specific threshold. From our experimental data based on the RNAMotifScanX alignment, on average around 60% of all intra-family motif pairs have RMSD 1.0 Å or better. The percentage is slightly lower for Kink-turn, reverse Kink-turn, C-loop, E-loop, and Hook-turn motif families due to the high internal variation that can be confirmed by the subfamiliy distribution from RNAMotifContrast  paper. BNow, for these best-aligned pairs between two families, we identified the number of instances for each family that is taking part in the alignments as these are similar to instances from another motif family. We calculated the percentage of these participating instances for each motif family. We are saying two motif families are similar if the percentage of participating instances from each family is 20% or higher . In the similarity graph that we have prepared, the minimum of these two percentage values is added as the edge label along with the average RMSD and average alignment length of the selected alignment pairs. We have utilized RNAMotifContrast to group similar motif instances from an unannotated dataset. Here, by unannotated we are referring to the motif instances without their family information. To get the unannotated version, we merged all the instances of different motif families and provided them as a single family to RNAMotifContrast. RNAMotifContrast normally takes a single family as input and subdivides the motifs of that family into separate groups which are then defined as subfamilies. The motif instances of a subfamily have higher similarity than the instances of the other one. We have utilized this feature of clustering similar motif instances from RNAMotifContrast to group the similar motifs from our unannotated input instances. We have generated the clusters using both interaction-based and coordinate-based alignment. There are several parameters to control the output of RNAMotifContrast, and the “connectivity test threshold” is one of them. This parameter is used to control the merging criteria of Connected Motif Groups (CMG) in RNAMotifContrast. If the value is higher, it will be less likely to merge multiple CMGs into one and vice versa. As we wanted to decrease the rate of merging the subclusters, we changed the connectivity test threshold from 50% to 80% for both cases . The “RMSD threshold for merging” is another parameter that controls whether to merge two CMGs based on a cutoff value, the default is set to 1.0 Å in RNAMotifContrast. To match the cutoff with our experimental RMSD threshold (which is discussed in the “Compute similarities between two motif families” subsection), we kept the default value of this parameter for interaction-based benchmarking and used the cutoff of 1.5 Å for coordinate-based benchmarking. All other parameter values were kept the same as the default ones. The outcome of RNAMotifContrast is then annotated using the family information from our original input dataset.As a representative of various machine learning models, we have used a basic classifier to show the utilities of RNAMotifComp. Due to the independent nature of features and supposedly Gaussian distribution of continuous feature values, the Gaussian Naïve Bayes (GNB) Classifier is selected. A variance smoothing of 1e-9 is used to improve the accuracy of the classifier. As our main objective is to demonstrate the consequences of having similar motif instances across motif families using this GNB classifier, and since it is important to have a good amount of instances in each class, we handpicked some particular motif families to train our machine-learning model. Here, for two different types of alignments (interaction-based and coordinate-based), we have performed two different types of analysis, referred to as Interaction-based GNB (IGNB) analysis and Coordinate-based GNB (CGNB) analysis. A summary of the similar motif families identified by RNAMotifComp is shown in To perform the IGNB analysis, different interaction-based structural features such as interaction-based alignment score, alignment length, alignment RMSD, number of matching base pairs, and number of base stackings are generated using the tool RNAmotifScanX . First, For each binary classifier, the motifs belong to one of the two classes, Class 1 and Class 0, where Class 1 represents the motifs belonging to a certain family and Class 0 represents the motifs that do not belong to a certain family. Then using the random undersampling technique, an equal number of motifs were selected for both classes. For example, in the IGNB analysis, to assess the performance of the E-loop GNB binary classifier, the first 235 motifs belonging to the selected four families  are taken. From there, the 49 motifs belonging to the E-loop family are assigned to Class E-loop (Class 1) and the remaining motifs are assigned to Class non-E-loop (Class 0). Then a random undersampling is performed so that both Class E-loop and non-E-loop contain an equal number of motifs (49 motifs). A similar approach was followed for all the GNB binary classifiers to handle data bias. Then, to evaluate the performance of each binary classifier, repeated stratified 3-fold cross-validation is used to split the dataset into three non-overlapping folds, maintaining the same class ratio throughout the three folds as the ratio in the original dataset. At each run, one fold is used as the testing dataset while the other two folds are combined and used as the training dataset. After training the GNB model using the training dataset, the performance of the model for that run is tested using the non-overlapping testing dataset. For each model, there were a total of 15 runs (3 splits * 5 repeats), and performance is evaluated by considering the average of accuracy, sensitivity, and specificity of these 15 runs. Following this approach, for both IGNB and CGNB analysis, mean accuracy, specificity, and sensitivity have been calculated. Here, accuracy represents the percentage of correctly identified labels, sensitivity defines the percentage of correctly identified positive labels (Class 1), and specificity defines the percentage of correctly identified negative labels (Class 0).By analyzing the internal loop motifs from the input dataset, RNAMotifComp generates a graph that represents the similarity relationship among the motif families. The image generated from our visualization method depicts their relation with associated data. Then we verified our experimental result using RNAMotifContrast  which moThe generated similarity graph based on the interaction-based and the coordinate-based resemblance is shown in According to the results shown in To get a clear idea, we have generated the consensus structure of the interaction-based similar E-loop and Tandem-shear instances given in Kink-turn and E-loop are both well-known 2-loop motif families . AlthougTo see the relationship among the motif instances from different aspects, we have employed RNAMotifContrast  as a cluOne important point to be noted here is, although the RNAMotifContrast result supports our analyzed outcome, it does not fully match our findings. For example, both SR and L1C are assigned to the category of motif families that do not have interaction-based or coordinate-based structural similarity from our analysis which is summarized in Characteristics of different known motif families are generally defined based on the frequent occurrence of sequence patterns, 3D structural patterns, base-pairing, and base-stacking interactions. The existing base-interaction and shape-based 3D structural similarity between motif families can often cause indecision while identifying known motif family instances or discovering new motif families. One such example is explored here using the Naïve Bayes classifier as a machine learning model. These 3D structural similarities are increasing the complexity of the model in identifying motif families based on their 3D structural features, thus degrading the overall performance.In order to verify if a machine learning model faces difficulty while trying to identify motifs of structurally similar families, we used the Naïve Bayes Classifier . Here, wFor IGNB analysis, four binary classifier models using the GNB algorithm have been trained separately for the four selected families EL, TS, KT, and SR. Each binary classifier predicts if a given motif belongs to a certain motif family or not. For example, the binary classifier for EL is trained based on the data of the E-loop motif family (labeled as Class 1) and the other three motif families (labeled as Class 0). This binary classifier specifically predicts if a given motif family belongs to the E-loop family or not. Finally, to observe the overall performance, an MGNB classifier is used which predicts if a given motif instance belongs to EL, TS, KT, or SR. The performance of the IGNB classifier and the related MGNB classifier is listed in Similarly, for CGNB analysis, five separate binary GNB classifiers were built for the five motif families EL, TS, HT, CL, and SR. Then their accuracy along with the sensitivity and specificity are calculated. Finally, an MGNB classifier model is developed to predict if an input motif instance belongs to EL, TS, HT, CL, or SR. The performance of the coordinate-based binary GNB classifiers (CGNB) and the multiclass GNB classifier model (MGNB) are also depicted in From Besides structural variation among the motif instances of a known motif family, there are also structural similarities within motifs of different motif families. While trying to identify and predict motifs belonging to known motif families, these structural variations and similarities definitely create additional complexity. Due to lack of higher resolution data and annotation errors, annotating the structural motifs can be very confusing for computational algorithms, even for humans. For example, The similarity we discovered from interaction-based alignment could be attempted to be found using a subgraph matching algorithm. But that will require generating a graph representation of each motif instance incorporating all base-pairing and base-stacking interactions of a motif. The complexity will then be no less than the complexity of RNAMotifScanX alignment as it uses maximal clique finding to align one motif instance to another. Moreover, that graph matching algorithm will not work for finding similarities based on the 3D coordinates that we have achieved using the RNAalign tool. btad223_Supplementary_DataClick here for additional data file.


However, to fulfil this promise the software must remain usable, be able to accommodate a large amount of data and allow use of modern high performance computing infrastructure. In this study we present a reimplementation as well as an extension of a technique using indicator vectors to compute and visualize similarities between sets of nucleotide sequences. We have a flexible and easy to use python program relying on standard and open-source libraries. Our tool allows analysis of very large complement of sequences using code parallelization, as well as by providing routines to split a computational task in smaller and manageable subtasks whose results are then merged. This implementation also facilitates adding new sequences into an indicator vector-based representation without re-computing the whole set. The efficient synthesis of data into knowledge is no trivial matter given the size and rapid growth of biological sequence databases. Based on previous results regarding the properties of indicator vectors, the open-source approach proposed here efficiently and flexibly supports comparative analysis of genetic diversity at a large scale. Our software is freely available at:  The study of biological evolution through comparing linear sequences from different species, now known as molecular phylogeny, was first proposed by Crick in a 1957 lecture . The pas“Biologists should realize that before long we shall have a subject which might be called ’protein taxonomy’ -the study of the amino acid sequences of the proteins of an organism and the comparison of them between species. It can be argued that these sequences are the most delicate expression possible of the phenotype of an organism and that vast amounts of evolutionary information may be hidden away within them.”Linear sequence analysis of proteins from different species combined with paleontology to estimate chronological time of their divergence gave rise to the concept of a ‘molecular clock’ , 4, whicWoese and colleagues pioneered the use of small subunit ribosomal RNAs to construct phylogenies that encompass all of cellular life , 11. SSUMitochondrial genomes are small in size, lack recombination, and are technically straightforward in terms of molecular biology because their copy number is approximately a thousand times greater than nuclear genes. Mitochondrial analysis both confirmed and refined other methods for species and population identification, especially among animals , 14 but The Barcode Of Life Database (BOLD) is an open access compilation including mtCOI barcode data from approximately five million individuals that collectively cover a great deal of the extant animal kingdom . The BOLA method of choice to extract and recapitulate the evolutionary information contained in a set of homologous sequences is to construct a phylogenetic tree based on the of likelihood of a substitution model. These methods have benefitted from numerous advances in the past decades, but the explosion of the number of possible tree topologies as the number of leaves increase—(2n-5)!!, unrooted topologies for n leaves–still makes the reconstruction of phylogenetic trees with tens of thousands of sequences using likelihood methods daunting . Aside https://phe.rockefeller.edu/barcode/klee.php), which runs on an older version of a proprietary commercial software, Matlab2009a, a not only unmaintained version, but whose documentation is no longer available on the developer’s website. We used the opportunity of a re-implementation to introduce significant extensions to the original methods, in particular to accommodate a variable number of sequences per species, and provide a more flexible interface to the computational steps involved in going from a multiple sequence alignment to a structure matrix. Refactoring the process into separate steps allowed an efficient and scalable parallelization of the pipeline, as well as adding flexibility. In the previous Matlab implementation updating a structure matrix with one or several new sequences required recomputing it entirely. Separation of different steps in the implementation grants additional flexibility, both computationally and conceptually. Real-life datasets from BOLD (The Barcode Of Life Database) are used to illustrate the new implementation and in places contrast it with the published Matlab-based method.Sirovich and colleagues developed an innovative approach to allow multiple DNA sequences both within and among many animal species to be compared such that the overall structure of animal life within and among species can be visualized , 24. TheAfter a series of pre-processing steps, which are detailed in the result section of this manuscript as well as in , the muli is represented as matrix si of size m×s⋅4, where m is the number of sequences in the sequence set and s is the sequence size, in nucleotides.A set of sequence i is computed as the eigenvector with the largest positive eigenvalue of matrix:N is the number of sequence sets, as defined in equation 15 of ).To compute the structure matrix from a multiple sequence alignment, the DNA sequences are first pre-processed and transformed to vectors of four numbers. The pre-processing consists in a tentative imputation of the ambiguous N characters: any N character is replaced by the modal nucleotide at this position among its sequence set , unless that modal value is the gap character (“-”), or there are multiple non-gap character modal values, in which case the value stays N at this position. This pre-processing step is unchanged from the original method of . Then thi is represented as matrix si of size m×s⋅4, where m is the number of sequences in the sequence set and s is the sequence size, in nucleotides.The numerical representations of each sequence are then grouped by set into matrices where each row is the numerical vector of a single sequence. Thus, sequence set This series of pre-processing steps are represented in i as described in the pre-processing steps above and the top part of This step corresponds to the computation of a matrix representation of the diversity of a DNA sequence across several individuals or sets of individuals  that will serve as a reference against which individual sequence will later be contrasted (see step 2).After sequences have been transformed into numerical vectors and grouped into matrices si matrices, normalized by the number of sequences in each set mi:And then summing all SSee the O(Ns2) and a memory complexity of Reference matrix computation has a time complexity of When executed as an independent script, it takes as input a multiple sequence alignment in the fasta format, and as output the reference matrix in a simple binary format, chosen for its read/write performance.The script is parallelized using MPI and thus it can be deployed on multiple CPU architectures. Briefly, the different sequences sets are split between the different processes, each computing a local reference matrix, which are then combined before being written to a file.Furthermore, we provide a utility script to merge pre-existing reference matrix together (provided they were computed on an independent set of sequences and that no grouping/species are shared). This allows one to update a previously obtained matrix with new sequences.Additionally, it also permits the computation of a reference matrix to be subdivided in any number of subtasks, by simply splitting the input alignment, and subsequently merging the resulting matrices. This makes the computation of very large datasets tractable and easy to deploy on one or multiple HPC architecture.This step corresponds to the computation of indicator vectors for each sequence or sets of sequences provided.As per , the indSee the O(Ns2). When executed as an independent script, it takes as input a multiple sequence alignment in the fasta format, the reference matrix (as obtained from the previous step), and it outputs indicator vectors in a csv format file.Indicator vectors computation has a time and a memory complexity of The computation of each individual indicator vector being independent from the rest, the parallelization of this script using MPI is fairly straightforward.As with step1, it is possible to further split the computations in many subtasks by splitting the input alignment before invoking pyKleeBarcode. The subsequent merging of the resulting file is done by simple concatenation.In this step a structure matrix, containing the correlation between the indicator vectors of different sets of sequences, is computed.In pyKleeBarcode it may be computed either in one go as the inner product of in two fashions. Either in one go, following the formulation of , with a The first option is faster—albeit both options have the same time complexity–but more memory-intensive because the whole structure matrix needs to be held in memory at once, while the second one is slower but requires less memory.The switch between these two options is, by default, set at 5,000 indicator vectors, corresponding to about 800Mb of memory, and can be modified to suit the available resources.O(Ns) and a memory complexity of O(N2s) when the number of indicator vectors is small, and O(Ns) otherwise.Structure matrix computation has a time complexity of This step takes as input the set of indicator vectors, in csv format, for which to compute pair-wise correlations; and it outputs the structure matrix (in a binary format of its lower triangular portion).While the previous steps could be parallelized and split in subtasks in a straightforward manner, the structure matrix computation matrix computation is a more complicated affair, because it must look at all pairs of indicator vectors.Nevertheless, we have devised an algorithm, and provide the corresponding script, to update an existing structure matrix with new indicator vectors .In fact, as structure matrices grow quadratically in size with the number of sequence sets they represent , the structure matrix file format has been explicitly devised with the goal of allowing an update without having to read, or even re-write the whole file; the new information is merely appended to it.Overall, throughout our various benchmarks and experiments, Step2 has consistently been the step that took most of the computational time . When using a single MPI process, pykleebarcode has a performance of the same order of magnitude as the original Matlab code see ; the usaTo assess the effect of the impact of an incomplete reference matrix on the indicator vectors and structure matrix we designed three experiments as described in the methods section. The “primates-only” experiment serves to assess a case of extrapolation: computing indicator vectors on sequences which are outgroups to the ones included in the reference matrix. In contrast, the “no-Laurasiatheria” explores a case of interpolation, albeit one an extreme one where an entire super-order is missing. Finally, the “missingXX%” experiment addresses mixed cases of extrapolation and interpolation but where no large taxonomic group is missing.−12), the actual difference in median is only about 0.0001. The results of the “no-Laurasiatheria” experiment, presented in PyKleeBarcode builds on the previous works on indicator vectors, and allows several new possibilities, such as an extended handling of ambiguous nucleotide character, but more importantly the ability to have differences in the number of sequences included per group, as well as the possibility of efficient integration of new results onto a pre-existing structure matrix—something hitherto not possible. As our experiments have shown, this has only minimal impact on the computed indicator vectors and similarities between sequences, except in the direst of scenarios, such as in the primates-only experiment, or when the percentage of new sequences is above 50%. Consequently, our method is appropriate to integrate a growing set of sequences, continuously integrating new specimens to see where they fit among the existing structure, which could only need to be entirely recomputed for major version releases for example.Considering PyKleeBarcode, and the indicator vectors approach in general, in the context of other methods used to extract insight from homologous sequences, it first presumes that a multiple sequence alignment has already been obtained, using MAFFT , 33 for As we noted above our approach can be contrasted with phylogenetic tree reconstruction methods: pyKleeBarcode does not create an explicit evolutionary scenario, but only a distance matrix. Consequently, pyKleeBarcode is robust to cases where sequences do not follow a strictly tree-like evolutionary scenario and is not hindered by the unfavorable combinatorics of the number of possible tree topologies. In contrast, likelihood-based tree reconstruction methods must constantly engage in compromises between result approximation and runtime to completion.Furthermore, the distance matrix produced by pyKleeBarcode may be used as the basis to construct a tree, using a distance-based method such as Neighbor-Joining  or UPGMAThe indicator vector approach is most closely related with pairwise-distance methods and shares some of their limitations such as a sensitivity to genetic saturation. The main specificity of our approach relies on the elegant approach of  to conseSome properties of indicator vectors are still unclear. For example, the contrastive nature of the approach should make them robust against heterogeneities in the mutation patterns of sequences, but this remains to be tested. Similarly, it would be of interest to investigate potential biases introduced by sampling differences between studied clades.With pyKleeBarcode we propose a flexible and up-to-date interface to compute indicator vectors and structure matrix from a multiple sequence alignment in a manner, adapted to the size of modern datasets and large computer infrastructures.We anticipate that pyKleeBarcode will help undertake deeper analyses of biological sequence databases, including BOLD, and allow new insights into large scale features of extant life.S1 FileSupporting information for “PyKleeBarcode: Enabling representation of the whole animal kingdom in information space “Appendix A: Benchmarking of pykleeBarcode against the previous Matlab implementation, and Appendix B: Investigating the saturation of the COI5-P among mammals evolution.(PDF)Click here for additional data file.S1 FigEvolution of execution time (A) and peak RAM usage (B) for the computation of a structure matrix with the number of DNA sequences for different implementations.(TIF)Click here for additional data file.S2 FigThe red line represents the average on a rolling window of 20 points.(TIF)Click here for additional data file.S3 FigThe red line corresponds to a rolling average of a 20 MYA window.(TIF)Click here for additional data file.


We present here a software framework for the alignment-based training, evaluation and application of machine learning models with user-defined parameters. Instead of focusing on the one-size-fits-all approach of pervasive in silico annotation pipelines, we offer a framework for the structured generation and evaluation of models based on arbitrary features and input data, focusing on stable and explainable results. Furthermore, we showcase the usage of our software package in a full-genome screen of Drosophila melanogaster and evaluate our results against the well-known but much less flexible program RNAz.The  Experimental and theoretical work in molecular biology typically presupposes the correct annotation of genomic data and therefore depends on the compilation and curation of accessible sequence databases. This demand for high-quality genomes led to the emergence of high-throughput sequencing (HTS) techniques and huge amounts of data since the 2000s ,2.Automated annotation pipelines increasingly supplement – and to a certain extent replace – time consuming, error-prone and often poorly reproducible manual curation techniques . Still, de novo identification of ncRNA elements. Both minimum free energy (MFE) structures and intrinsic properties of nucleotide sequences are used in tools such as RNAz  against a background of protein-coding sequences. Prime examples of such tools include and CNCI . Effectiand CNCI . In the and CNCI ,26.While the reality of pervasive transcription of genomic DNA is widely accepted, it has remained a matter of debate what fraction of the transcriptional output conveys recognizable biological function and whether there are high levels of ‘transcriptional noise’ . The latMachine learning methods are utilized in many automatic annotation pipelines in the field of comparative genomics ,30. A keSvhip , a novel software pipeline for the calibration of diverse machine learning models used in genome-wide non-coding and/or protein-coding RNA screens. The software is available as source code on Github as well as provided via the conda environment manager. The key feature of Svhip is the ability to train a multitude of different models automatically while giving full control over individual steps to the user. We offer the possibility to generate both models for the identification of structurally conserved non-coding RNA elements against an ambiguous background as well as for the differentiation between non-coding RNAs, protein-coding sequences and/or undefined others. Generated models can be used in the framework of Svhip itself or, in the case of two-class models, be exported to be used with the highly successful annotation tool RNAz  with 3,832 alignments each. This data set was later reduced in size to 3,060 alignments in each subset, as all alignments containing only two sequences have been removed. This was necessary, because SISSIz shows some substantial drawbacks in the simulation of such alignments, see To test the capabilities of our own genomic background simulation approach for future data set generation, we shuffled each non-coding RNA alignment on a column by column basis using the Drosophila melanogaster as reference provided by the UCSC genome browser (https://hgdownload.soe.ucsc.edu/goldenPath/dm6/multiz27way/maf/). The corresponding annotation in GTF format was obtained from FlyBase   ,38. To pDrosophila alignment windows for evaluation purposes were assigned using the annotation in GTF format as follows: For each window we calculated the overlap with annotated transcripts, either coding or non-coding, as a fraction of the total window length. If the overlap exceeded 0.5, we assigned the corresponding label to the window. The cutoff at 0.5 was chosen to leave room for disadvantageous window slices and to increase the likelihood of at least one window covering most genes with sufficient length for detection. The alignment windows that were not filtered are then considered as representative of their respective class and serve as the basis for the generated training set. We use five features to capture essential information from these alignments while keeping redundancy at a minimum.True labels of Svhip accepts multiple sequence alignments in Clustal or MAF format as input. In the first processing step, these are grouped by the label assigned to them during initialization, either ncRNA or other in the case of two-way classification or ncRNA, protein-coding and other for three-way classification. We then use the rnazSelectSeqs.pl script for the initial selection of sequences. All sequences with a pairwise identity beyond a certain threshold (defaul is 98%), are rejected. This is to ensure that redundancy in the data set is reduced to a minimum.The data processing pipeline is explicitly intended for the preparation of raw alignment data for model training. It is not to be used for prediction runs, as the filtering steps implemented here may disrupt signals. RNAfold. Subsequently, the average tree edit distance is calculated using RNAdistance for the window as an approximation of the remaining structure conservation after shuffling. Both of these tools are part of the ViennaRNA package  is a metric for the alignment-wide presence, size and frequency of sites that are conserved in their secondary structure. The SCI compares the MFE of individual sequences with the consensus sequence MFE. Significant deviations indicate structured elements that are not present in other sequences. Thus, a value greater or equal to 1 represents a very strong level of conservation, while a value close to 0 indicates none. The SCI for the alignment x ∈ A are the individual sequences, Ex is the MFE of x, and Econsensus is the MFE of A’s consensus sequence as predicted by RNAalifold.where z-score of MFE is used. It can be calculated using one of the following two procedures. In the first case, a given sequence is 100 times dinucleotide shuffled using the Altschul–Erickson algorithm sequences with an MFE that significantly deviates from that of sequences with equal nucleotide composition but no notable secondary structure. To detect these, the lgorithm  and the z-score of the MFE for the full alignment can thus be expressed as:The mean normalized x denotes the standard deviation, Ex the MFE and μx the mean for sequence x. Explicit shuffling, however, is a highly expensive operation in terms of computation time per input alignment as it effectively requires the sampling and estimation of the unimodal distribution for each sequence in the alignment. For this reason, we implemented support vector regression models that estimate the expected standard deviation and mean based on fast-to-calculate, sequence-intrinsic features. These are: all 16 dinucleotide frequencies, C to G+C nucleotide fraction, A to A+U nucleotide fraction and the normalized length of the sequence. Normalization was carried out by dividing the sequence length by the maximum considered length of 200, thereby yielding a fraction between 0 and 1. The models were trained on multiple large sets of synthetic sequences representing GC-content fractions in 0.1 intervals from 0.2 to 0.8 and sequence lengths from 50 to 200. The estimated values are within an acceptable deviation from those obtained with the exhaustive approach outlined above, see Figure where σH is a metric for the sequence variation not otherwise captured by the features outlined here. It is calculated for each alignment asThe Shannon entropy b at the ith of  n alignment columns.where The hexamer score  is the empirical probability to find hexamer Hi in a coding sequence and F′(Hi) is the empirical probability assuming a non-coding background  for each two sequences in an alignment is calculated asIn terms of alignment-based coding potential estimation, the tool  release . It reliAi and ith aligned codon in the first and the second sequence, respectively. N is the number of aligned codons. A rudimentary phylogenetic tree for the input alignment is estimated using a neighbor joining (NJ) approach based on pairwise sequence identity. The NJ algorithm is implemented using the corresponding function of the Phylo module of the Biopython library , random forests (RFs) and logistic regression (LR). In principle, every feature set can be used to train each type of model, even though there are some qualitative differences between them. SVMs have by far the highest computational complexity, whereas LR has the lowest . When we trained RF classifiers instead of SVMs using the same training data and using the same test data, we received marginally worse accuracy results (data not shown). However, we believe that the size of the data sets is too small for generalized conclusions here.By default, e lowest . FurtherSvhip offers automatic hyperparameter optimization for all models using either a grid search or a random walk based approach. Both options are fully customizable in their search depth and number of cross-validation steps, however, we found that more than five cross-validation iterations typically do not offer a significant improvement in accuracy given the features used (data not shown). Generated models are saved in a binary format and can be loaded using the predict option. We also note that, under certain circumstances, the expected increase in accuracy by hyperparameter optimization might be marginal when compared with the quality of the training data. Parameters associated with training, including hyperparameters and scaling parameters for normalization are saved together with the model and loaded as required for prediction purposes.Svhip classifies alignment data by either employing one of the built-in prediction models, or by resorting to a custom model previously generated using the features command as described above. For the classification of input data, the alignments have to be cut into overlapping windows to achieve accurate results, analogous to their preparation for feature calculation. An alignment length of 50–200 nucleotides with a sliding overlap of 40–80 nucleotides yields reliable results for the cases presented here. For the preparation of windows, the software rnazWindow.pl of the RNAz framework was applied. The predicted class labels are assigned to the data sets on a per-window basis. Furthermore, an overall class probability is reported which, in the case of SVM classifiers, is based on a regularized maximum likelihood score  identical sequences, respectively. The expectation here was a statistically significant reduction of the resulting capability to differentiate positive from negative class instances, presumably below both the optimized Svhip-generated classifier and RNAz.We note that far more care could have been taken to ensure the selection of high-quality and representative input data sets. However, an important point to demonstrate here is the underlying hypothesis that the data preprocessing pipeline as implemented in Svhip-trained classifier and showcase a basic experiment with a retrained classifier, we performed a genome-wide screen on D. melanogaster full genome Multiz alignments in MAF format and compared predicted loci with a contemporary annotation. Alignment data for D. melanogaster was prepared as outlined in section Test Data preparation. Two different Svhip-trained classifiers were utilized for this study: one already trained and tested in the previous section, and one with added data for protein-coding sequence differentiation.To acquire a reasonable estimate of the expected accuracy of a Drosphila annotation and then downloaded all corresponding sequences from the database. To further reduce the amount of data, we limited ourselves to the Ephydroidea (containing Drosophilids) and Oestroidea superfamilies. Supplementary Table S3 summarizes all protein-coding genes that were processed using the Svhip pipeline and produced feature vectors used in training.Training data for the protein classification was assembled as follows. Coding sequences were sampled from the InsectBase . To thismafmerge to merge short MAF blocks into larger consensus alignments. The tool, which is available as part of the Svhip software, performs the following steps: The reference genome is traversed in sense direction and for each individual alignment block, the following block is analyzed for consensus of species aligned, genomic distance and continuous reading direction between sequences aligned in both blocks. By default we use a threshold of 75% of aligned species having to be present as consensus in both blocks for merging. Furthermore, all aligned sequences have to be directly continued in the next block with respect to their genome coordinates, i.e. the start nucleotide index of one sequence has to be its end nucleotide index plus one in the other alignment block. The merging process of continuous blocks terminates automatically when a length threshold is surpassed. A default value of 1,000 was chosen based on the observation that the majority of blocks in the Drosophila alignments do not get joined past this point due to lack of a sufficient consenus of aligned species, see Supplementary Figure S3. Considering the compatibility of genomic coordinates of non-reference sequences implicitly adds information regarding synteny in the resulting alignments. There is no indication that this introduces biases that might adversely affect the accuracy of secondary structure prediction.We implemented and applied Svhip pipeline and derived models for a range of different test sets and scenarios. The primary goal here was to properly estimate the specificity and expected false positive rate on both alignments of known non-coding RNAs as well as from the context of a full genome screen on D. melanogaster. Specifically, we also wanted to estimate the prediction quality on non-coding RNA data against the RNAz framework, which shares several of the implemented machine learning features, but comes with a hard-coded decision model, not allowing for retraining of the underlying classifier. In particular, we demonstrate that our preprocessing workflow for the raw training data does in fact yield high-quality positive instances, while the generation of synthetic negative data successfully reduces false discoveries to a minimum. We also show the separability of coding and non-coding data in an alignment context using our features designated for estimating coding potential.We tested the Svhip offers different modes of action, denoted as data, data3, training, features and predict. The first two serve to generate training instances for the machine learning engine from multiple sequence alignment data, either assuming a two-way  or three-way classification , respectively. The data and data3 commands also evaluate the suitability of a given set of alignments for training the classifier by analyzing properties such as the average minimum free energy, levels of conservation and the overall quality of separation achievable with selected features. Every step of the pipeline is accompanied by graphical output to facilitate the assessment of the results and potential issues, Figure training command computes a classification model based on a data set generated with either of the previous commands. The remaining two modes, features and predict, serve to calculate the features required for prediction of alignment data, and to perform the actual classification using an existing model, respectively.RNAcode is implemented in the codon conservation score that sacrifices some accuracy for substantially improved speed, see Materials and Methods — Data processing and Supplementary Figure S5 for a comparison of run times.We generalized the hexamer score, commonly used to estimate coding potential of individual sequences, to be applicable to alignments. Furthermore, a reduced version of the scoring scheme applied in RNAcode software, we did a comparative analysis between both approaches on the Drosophila genome alignments . All alignment windows containing at least three sequences were screened using RNAcode and their respective codon conservation scores were calculated. The restriction of the sequence number was necessary as RNAcode does not natively support the calculation of scores for alignments of only two sequences. As RNAcode reports scores for multiple possible subalignments while the codon conservation score is alignment-wide, only the best RNAcode hit was taken into account for this comparison. The resulting Pearson and Spearman correlation coefficients of 0.49 and 0.5 indicated only a weak correlation of both scores. In part, this can be explained by the fact that the codon conservation score takes the full alignment length into account and not just the subalignment with the highest individual score. A more relevant question, however, is whether both scores are suited metrics to tell coding and non-coding sequence alignments apart. Using annotated alignment windows as a basis and separating them into coding alignments and other alignments, we performed a Wilcoxon rank sum test on both sets of data and metrics, respectively. Using this approach, the null hypotheses that both sets of measurements originate from the same underlying distribution can be rejected for both metrics with a very high confidence as indicated by p-values smaller than 10−15. Thus, while there are certain intrinsic differences in both approaches, they do achieve a strong differentiation between coding and non-coding sequence alignments.As the codon conservation score was derived as a simplified version of the algorithm employed by the RNAcode yields higher raw discrimination power in comparison to the hexamer score (data not shown) when screening explicitly for the highest scoring, connected area in an alignment. In this sense, a protein-coding training set that was pre-screened using the RNAcode approach would, in most cases, be of higher reliability than one relying exclusively on the alignment-wide hexamer score. The former is however associated with significantly higher computational cost in an already computationally expensive pipeline. We therefore leave the decision for either of the methods to the user.The approach utilized by By combining information of InsectBase and FlyBase, we generated protein-coding alignments and investigated the distribution of the alignment-wide hexamer score and codon conservation score in coding and non-coding context. The codon conservation score shows a very strong separability of classes just based on this feature alone, while the alignment-wide hexamer score exhibits a moderate performance, Figure RNAz and two Svhip-trained classifiers were compared on the original RNAz test set. The difference between the latter two lies in their optimization by filtering for secondary structure significance of the input. The structure filtered classifier refers to the one trained by applying the secondary structure filter method and serves to illustrate the relative effectiveness in increasing the specificity by comparison . All three classifiers show a high degree of differentiation between true positive and true negative instances, Table Svhip classifier with structure filter enabled reaches the level of differentiation achieved with RNAz and surpasses the non-optimized classifiers by a notable margin, see Table Accuracy, sensitivity and specificity of Svhip classifier shows specificity well within the lines of expectation. We note that the classifiers with no structure filter fall short in terms of specificity when compared to the other classifiers and that the higher sensitivity can thus be partially attributed to this. This as well is to be expected as we suggest a strong link between the selection of statistically significant training examples and overall performance. We note however that, in the context of genome wide screens, a low FPR is in many cases more valuable than a high sensitivity. This is the case for two reasons: Firstly, using a sliding window approach, the average screened genome will be separated into millions of overlapping windows, leading to an expected false positive classification of thousands of windows. Considering this, even a reduction of FPR by one percent can be considered beneficial, as the experimental validation of each individual discovered locus is substantially more costly than the genome screen itself. Secondly, as most genomic loci are larger than one window of around 120 nt, there is a reasonable chance for at least strongly conserved subsequences of the full locus to be classified correctly, such that many candidates can be identified even at lower sensitivity. For these reasons, we strongly suggest an optimization towards the lowest possible FPR feasible for a given use case. Aside from these considerations, we also note again that the analyzed classifiers are not optimized in terms of initial input data and serve more to showcase the overall capabilities of the pipeline. Alternative avenues of investigation here include for example the specific selection of only certain types of non-coding RNAs as input, or a restriction to certain species of origin. We further note that the stagnation of raw accuracy at around 0.96 seen here indicates that a certain threshold is probably reached in terms of classifying this particular data set and any further attempts at improvement risk overfitting.Based on the false positive rate on both control sets, the optimized D. melanogaster. A running time analysis for an increasing number of windows shows that the Svhip classifier out-performs RNAz, see Supplementary Figure S6. We expected that the overall classification accuracy of structurally conserved non-coding RNA elements would further increase when considering protein-coding sequences as a third class. This approach, however, cannot be compared to RNAz directly, which exclusively screens for conserved RNA structures in an otherwise ambiguous genomic background and does not take protein coding sequences as a separate class into account. This latter fact makes a second comparison necessary, with RNAz on the one side, the better-performing, Svhip-generated classifier on the other, and a third classifier that differentiates non-coding RNA, protein-coding and background genetic locations, which will be discussed in the following section.Given the above observations, we decided to use only the classifier trained with structure filter for a non-coding RNA screen on the genome of D. melanogaster as reference and the current annotation from FlyBase v2 as ground truth  performs highly similar to the RNAz classifier in recovering snoRNAs, miRNAs and tRNAs, Table Svhip pipeline is well capable of isolating high-quality training instances even from suboptimal input data. It has to be noted, however, that it is difficult to correctly determine the false positive rate of the classifiers, especially for the scenario of a full genome screen. This is primarily caused by the lack of gold standard data that would guarantee correctness of the used annotation, and the higher abundance of spurious alignments containing non-homologous or very dissimilar sequences thus introducing the possibility for reduced conservation signals despite there being a genuine conserved transcript encoded. The ambiguity and likely incompleteness of available annotations is our primary motivation for restricting the analysis to the well-studied cases of tRNA, snoRNA and miRNA genes. Although the two primary sub-types of snoRNAs are covered in the input equally well, C/D-box snoRNAs are recalled at a substantially lower rate than H/ACA-box snoRNAs, Table We performed genome-wide screens based on Multiz alignments with nd truth ,38. CompACentral , a comprMultiz whole genome alignments we noted that a primary issue in the screening for conservation signals is the quality and coverage of the input alignments provided. Even a hypothetical ideal classifier would fail to recover all known annotated genes if the total coverage of the provided MAF blocks for these genes is insufficient. To investigate the magnitude of this problem, we analyzed the number of annotated genes theoretically covered by sliced alignment windows and discovered that in most cases less than half are actually present in the final input with sufficient coverage, Table rnazWindow.pl, this becomes a serious issue on genome wide screens. In many cases these short blocks still contain valid information and can be combined into larger blocks using a quite simple approach as implement in mafmerge of the Svhip software, see Material and Methods for details. Independent of the classifier, a substantial increase in both gene coverage and total recall of ncRNA genes is observable for the merged alignments. This indicates that such preprocessing adds substantial value to the analysis.While analyzing the P > 0.5, we get 48,217 total hits with Svhip and 46,596 with RNAz, respectively. Of these, 29,015 (62%) are shared between the two, suggesting a relatively high consensus. In general, the confidence or P-value distribution of hits looks different for both classifiers, which is presumably an artifact of underlying model architectures, as LibSVM and scikit-learn diverged in development a substantial time ago, even though the latter’s SVM implementation is built on top of the former. Of the shared hits, 4,183 map onto existing annotations of either coding or non-coding genes, or 6,941 in the case of Svhip and 6,717 for RNAz. Considering the very similar proportions of total hits and fractions that map onto existing annotations, we come to the conclusion that the false positive rate of the Svhip classifier is likely at least as good as that of RNAz for this confidence level. Together with the observations from the previous section we conclude that our classifier is about as efficient for the purpose of Drosophila non-coding RNA genome annotation as the RNAz framework.Analyzing all hits with a minimal cutoff of Svhip, it showed subpar results in this particular use case when compared with the usage of two individual classifiers (data not shown).A new classifier for the differentiation of protein-coding sequences and genomic background was trained based on the data obtained from InsectBase. The entire pool of training data from the ncRNA classification experiment, i.e. both the exemplary Rfam non-coding RNA alignments and the automatically generated negative set, was reused as a negative set. The goal of this experiment was the identification of coding regions in a full genome screen under the exclusion of everything else. While generation of a three-way classifier differentiating between ncRNA, coding sequences and genomic background is in principal possible with Svhip protein classifier . Note that one gene can consist of multiple coding sequences and that such a region was counted as recovered if (≥2) consecutive hits occurred on the same strand within that gene’s boundaries. If we only count regions as recovered when at least 50% of their length is covered by hits, we still obtain a recall of 66.1%, corresponding to 11,787 recovered regions. Obviously, these numbers are heavily dependent on the quality and quantity of the available training data and thus the resulting classifier.Of the 17,835 protein-coding sequences covered by the preprocessed alignments, 13,129 were recovered by the Since the annotation of protein-coding genes is more mature as compared to that of non-coding RNA, the estimate of the expected false positive rate (FPR) can be considered to be much more reliable in this case. Of 477,593 alignment windows not annotated as protein-coding, 54,309 (11.3%) were classified as protein-coding loci. It is of course likely that the Flybase annotation does not include all coding sequences that exist in reality, in particular taking into account that also features currently annotated as non-coding can harbor some coding potential or code for sORFS/smORFS . TherefoDrosophila genome annotated as protein-coding that is not split over two or more consecutive alignment windows. This means that, in the case of protein-coding screens, isolated windows reported as hits are most likely false positives and can be ignored in almost all cases. Applying this correction as a post-processing step reduces the false positive rate to 34,156 (i.e. 7.2%) incorrectly classified windows.Usually, protein-coding genes are substantially longer than the non-coding RNA genes discovered with this approach. In fact, there is not a single region in the CPAT  secondary structures, Supplementary Figure S8. In this interpretation, the potential for conservation of secondary structure is highest in genuine non-coding RNAs and lowest in the genomic background, while coding sequences exhibit an intermediate conservation level. Thus, their removal by a preceding classification step based on coding potential could substantially reduce the margin of errors in a subsequent, conservation-based non-coding RNA screening. In general, the non-coding RNA classifier outlined in "Analysis of two-way classification performance on Drosophila genome screen" can be assumed to have a worst-case FPR ofor, in the case of merged alignments,where FP  refers to non-coding RNA classification hits on windows not overlapping with a non-coding RNA annotation and TN (true negatives) refers to hits where coding sequence or genomic background windows are correctly identified as not being non-coding RNA. We explicitly counted here all potentially incorrect hits—and not only the high-confidence hits—as we assume that the probability of overlapping with genuine but not annotated genes is higher. If the goal is to estimate an upper bound on the FPR given the classifier at hand, then all hits irrespective of assigned probability should be counted.There are two obvious options to attempt an exclusion of coding sequences from the false positive set during classification. One is to first make a full genome screen using the protein model from "Analysis of protein classification performance on the Drosophila genome screen", and subsequently classify the remaining alignment windows. In principal this introduces the possibility of misclassifying non-coding RNAs as coding, which then subsequently disappear from the pool of true positive non-coding RNA hits. During our analysis, however, this effect appeared to have no relevant impact on non-coding RNA classification. Unfortunately, this approach resulted only in a marginal reduction of FPR, which fell to 1.3% due to 206 alignment windows being removed from the false positive set.Svhip using the same parameters, yields a drastically improved FPR on the annotated Drosophila genome:The second option involves retraining the classifier after also including the protein-coding training set as part of the negative set for non-coding RNA classification, thus attempting to generate a classifier that more directly excludes coding sequences instead of only being based on the original negative set of synthetic alignments. Marking all training examples in the protein classifier training set as background (relative to the class of non-coding RNA) and retraining the non-coding RNA classifier with z-score of MFE, and entropy, as it was noted that the more coding-sequence specific features do not substantially add to the non-coding RNA classification performance. These features were thus excluded from the training set for this experiment.It should be noted that this classifier, as outlined in the result section "Analysis of two-way classification performance on alignments of known ncRNAs", only uses the features of SCI, We conclude that, depending on the use case, it may be worthwhile to manually add known negative examples to the training set instead of relying exclusively on the artificial generation of negative training instances.Svhip pipeline allows for the generation of a three-way classification model. We compared the efficiency with the chaining of multiple classifiers for different purposes as described in the previous section. For this experiment, the same training data was used as for training of individual classifiers. We note, however, that the three-way classifier performs notably worse using the same training data in a direct comparison. For instance, the number of correctly recalled protein-coding regions drops from 13,129 to 9,798 using this approach and the FPR becomes 5%. Due to these notable drawbacks, we limit the more in-depth analysis to the previously presented approach. On the other hand, preliminary attempts on yet incomplete training data yielded promising results for some more specific applications. We are therefore convinced that this approach may be a viable tool to consider for certain use cases. A more detailed analysis, however, is beyond the scope of this initial report.The Svhip software was used in different classification scenarios to illustrate the applicability with different research goals in mind. It can serve as both a tool for a quick screen of potential coding regions as well as for an in-depth analysis of non-coding RNAs across a variety of species. Although it was not the main motivation for implementing Svhip, we found that its runtime even for a rather small set of 2,000 alignment windows is already about 20% shorter compared to RNAz. A trend that is expected to get even more pronounced for genome-wide screens. We hypothesize that this speedup is a result of the efficient implementation of the SVM approach in scikit-learn in comparison to the old libSVM version used when RNAz was trained in 2010. It can be demonstrated that Svhip is also applicable in a more specialized scenario such as the search for high-confidence hits for coding potential in genes already assumed to encode non-coding RNAs with different biological functions. To our knowledge, no other tool currently unites these functionalities and such an investigation would typically require to make a screen with one specialized tool and another, already established annotation in mind. This use case has substantial downstream potential, as there is evidence that non-coding RNAs containing protein-coding subsequences are yet not close to being fully investigated (The stigated ,56.Svhip generates accurate classifiers with high recall and moderately low false positive rates even from subpar training data. The set is based on the first 50 Rfam alignments manually selected for known and significant secondary structure, Supplementary Table S1. Depending on the application much more care could be taken in order to assemble an appropriate training set from available, e.g. domain of life or clade specific, data sources. We used the Rfam database as reference mainly because RNAz was trained on Rfam alignments and for the non-coding RNA application we intended to reproduce its implementation as proof of concept. The situation is similar for the training data of the protein-coding classifier and even more obvious for dual functional RNAs, i.e. those that act as non-coding as well as protein-coding RNA. With Svhip we provide a standardized framework enabling users to exploit current data and the most recent available knowledge in order to approximate a ground truth or gold standard as good as possible. The tool is primarily intended for genome-wide screens instead of the classification of single alignments of full-length sequences, which makes the classification task already more difficult at the outset. In cross-validation scenarios using subdivisions of our training data, we obtain recalls and false positive rates that are markedly superior to the data obtained from a genome screen, as the results sections Analysis of two-way classification performance on alignments of known ncRNAs and Analysis of two-way classification performance on Drosophila genome screen indicate. One problem here is the slicing of genomic regions into overlapping windows, a process that inherently destroys certain conservation and structure signals at the window boundaries. An important, yet insufficiently addressed problem is the classification of long non-coding RNAs in genome-wide screens. Long non-coding RNAs are in many ways more difficult to classify by window-based approaches because focusing on local features like secondary structure conservation becomes less viable for longer candidate sequences. Approaching this problem will be a goal of future investigation. Potential solutions include better methods to join related, but not directly adjacent hits within the same gene and improving training set composition for this specialized problem.Our results clearly show that The software could be improved further by incorporating specifically designed clustering approaches for given groups of hits. As noted above, in many cases it may be insufficient to simply group adjacent hits together and declare them to be part of the same gene or region. This simplified approach is problematic in particular if either a very long coding sequence or a long non-coding RNA is to be analyzed. In the former case, the large number of successive alignment windows involved naturally increases the chance for false-negative hits, and any false negative would lead to an interruption of the chain and thus to a misclassification as two protein-coding regions instead of one. In the latter case, structure conservation or the overall presence of stable secondary structure is usually not evenly distributed, again leading to accidental subdivision of loci. Both of these problems have been observed in our case studies. The situation is complicated further by the presence of introns and alternative splicing events. These shortcomings emphasize the necessity to devise better methods for identifying clusters of hits, which will be the subject of future work.Svhip serves as a framework to train, test and apply machine learning classifiers in full genome screens for the annotation of protein-coding or non-coding RNA genes. We demonstrated the usage of the pipeline in a number of different scenarios and obtained comparable classification results to the RNAz framework when investigating non-coding RNA classification. The primary advantage of Svhip is the ability to in- or exclude different data sources on the fly, and to fine-tune parameters for different use cases. Further work is required to evaluate the applicability in different scenarios. One prime target is an in-depth investigation of plant genomes, many of which are currently lacking accurate and complete annotations. A particular challenge there is the high presence of gene duplication and transfer events, increasing the difficulty of the data preprocessing, and a highly volatile alignment quality and structure. We envision that the adaptability of the pipeline will serve to further improve the annotation of many different species.The software lqad072_Supplemental_FileClick here for additional data file.


Although precision and recall for single nucleotide polymorphisms (SNPs) now exceeds 99.5%, INDEL recall remains below 80% for standard R9.4.1 flow cells. We show that read phasing and realignment can recover a significant portion of false negative INDELs. In particular, we extend Needleman-Wunsch affine gap alignment by introducing new gap penalties for more accurately aligning repeated  As long read technologies have matured and basecalling accuracy has increased to over mentclass2pt{minimordingly . Nanoporordingly –6.The two current leading nanopore variant callers are Clair3 (developed by the HKUCS Bioinformatics Algorithm Lab) and PEPPER-Margin-DeepVariant  , 8. Both“an entirely different network architecture and learning tasks”. Clair3 then split the model into a pileup caller to filter out the noise and a higher-dimensional full-alignment caller to make the more difficult decisions  is the negative log probability that base x[i] was observed as base x[j], either through a mutation or sequencing error:Figure mentclass2pt{minimi occurring: Confusion matrices for insertions (eletions .n-polymer lengths l (up to 100). For each n, or repeat unit size 1–6, a penalty matrix was calculated using the following equation, where i is the expected repeat length and j is the measured repeat length.n-polymers where few examples were observed, the following two properties were enforced for each possible combination of N:Shorter INDELs are more likely: n-polymers are more likely to contain an INDEL of a given size: Longer First, confusion matrix n-polymers during alignment. For each possible n-polymer repeat unit length from l, the length or number of consecutive repeat units, and idx, the 0-based index of the current repeat unit  containing the accumulated penalty value, in addition to the predecessor matrix and consecutive movements (run) within that matrix for backtracking purposes.Before aligning read i, j in S, the reference annotations l and idx, are first retrieved for each n for R[j]. All dependencies (marked with S) in Fig. S). In other words, when looking at S, for each n, we do:S and L such as these are only allowed conditionally based on reference annotations (described in the following section). Note that if matrices Figure  penalty .D, I,  and M, the results for matrices S and L are stored several cells ahead of the current cell, and cell dependencies are only allowed conditionally based on the reference annotations. This ensures that matrices S and L only allow INDELs which change the copy number of tandem repeats and homopolymers. Here are the three conditions Unlike matrices M, and relies on predecessor and run length information computed during the forward pass. The selected optimal alignment path is computed by Algorithm 1 and reported in the output BAM file in the form of a CIGAR string. CIGAR strings are composed of the symbols M, I, and D. Reference Matches, Insertions, and Deletions, correspond to diagonal, vertical, and horizontal movements in the alignment matrix, respectively. An example alignment is denoted by Traceback occurs entirely within matrix r will be aligned to a small subsection of the reference m is the length of the reference sequence corresponding to r. Barring any large INDELs, O(|R||r|) to As mentioned previously, the primary goal of our read realignment algorithm is to more accurately model the mutations and sequencing errors in fine-grained alignment. Therefore, we can skip read mapping and trust the read start position reported by the previous aligner in the input SAM/BAM file’s POSition field. Read Moreover, we can use the SAM/BAM file’s existing CIGAR string to simplify our alignment problem even further. Our optimal alignment will likely follow a path close to that of the original alignment. Figure Kasahara , insteadA to a B saves significant space because nanopore sequencing read lengths |r| can be up to several million bases [Firstly, all M CIGAR operations are converted to ID, an insertion followed by a deletion. This change is shown in Fig. on bases , while rB with indices i, j, its position in matrix A can be computed using the following formula:Offset arrays INSs and DELs can be precomputed using the CIGAR Fig. . Given aR| is the length of the reference R, and n-polymer considered, and n-polymer length. Since our n-polymer score matrix N is of size , O(|R|). Furthermore, these annotations must only be computed once, and cost can be amortized over all the reads that are aligned to the reference. We found the time required for reference annotations to be insignificant compared to alignment. These annotations require The worst-case time complexity for computing the reference annotations is O(|R| |r|) time for each read r. The only additional overhead nPoRe incurs over Needleman-Wunsch with affine gaps is computing five Dynamic Programming (DP) matrices instead of three, as well as computing the new cell dependencies. All new penalties are conditional O(1) lookups. As discussed earlier, follow-banding further reduces both the time and space complexity of alignment from O(|R| |r|) to O(b|r|), where b is the band width. In total, the cost of aligning all reads is m is the number of reads, or equivalently O(db|R|), where d is the average depth of coverage.Once the reference annotations and score matrices have been computed, the nPoRe algorithm requires https://github.com/TimD1/nPoRe.All our code is open source and readily available at: We used the GrCh38 reference from the Genome-In-A-Bottle (GIAB) consortium .ONT Open Datasets’ May 2021 re-basecalling of HG002 PromethION R9.4.1 data using Guppy 5.0.6. Specifically, we used flow cell PAG07162, prepared using the Short Read Eliminator (SRE) protocol [We obtained our FASTQ files from protocol . Depth on-polymers provided previously. Regions were extended by a single base on each side (slop=1) to include variants occurring at the edges of n-polymer regions. These BEDS were then merged and complemented as necessary to create stratification BEDS for all n-polymer regions and non n-polymer regions.Stratification BED regions were calculated for The full training and evaluation pipelines for all three Clair3 configurations tested are shown in Table In order to add haplotype phasing information to Clair3, a single iteration of the ordinary pileup-based pipeline was first run. Proposed variants were then phased using whatshap phase, and reads were tagged by haplotype using whatshap haplotag. We then sorted reads by haplotype into three separate BAM files. When generating the input pileup tensor for training clair3-hap and clair3-npore-hap, for each position a pileup tensor was generated for both unphased reads, reads from the first haplotype, and reads from the second haplotype. These three pileup tensors were then concatenated to create a new input tensor for Clair3.n-polymer INDELs than SNPs, due to the reduced INDEL penalty. We found that clair3-npore-hap variant calling performance suffers if we train Clair3 with the original ground-truth VCF, since the realigned reads tend to report variants using an INDEL-heavy representation. To mitigate this, we altered the ground-truth VCF so that it reports variants using the same representation our aligner tends towards. An example of this “standardized” VCF is shown in Fig. Figure To achieve this, we copied our reference FASTA to create two haplotype FASTAs, and applied the phased ground-truth variants to each haplotype FASTA, storing the new CIGAR. Using a mapping position of 0, the generated haplotype references, and associated CIGARs, we considered these ground-truth haplotype references to be reads and aligned them to the original reference using nPoRe. Any substitutions, insertions, or deletions in the resulting alignment were then parsed into a new standardized ground-truth VCF file. This process ensures that the new “standardized” truth VCF contains the same exact ground-truth sequence as the original VCF when applied to the reference FASTA, but reports variants in a manner consistent with nPoRe.


Finding alignments between millions of reads and genome sequences is crucial in computational biology. Since the standard alignment algorithm has a large computational cost, heuristics have been developed to speed up this task. Though orders of magnitude faster, these methods lack theoretical guarantees and often have low sensitivity especially when reads have many insertions, deletions, and mismatches relative to the genome. Here we develop a theoretically principled and efficient algorithm that has high sensitivity across a wide range of insertion, deletion, and mutation rates. We frame sequence alignment as an inference problem in a probabilistic model. Given a reference database of reads and a query read, we find the match that maximizes a log-likelihood ratio of a reference read and query read being generated jointly from a probabilistic model versus independent models. The brute force solution to this problem computes joint and independent probabilities between each query and reference pair, and its complexity grows linearly with database size. We introduce a bucketing strategy where reads with higher log-likelihood ratio are mapped to the same bucket with high probability. Experimental results show that our method is more accurate than the state-of-the-art approaches in aligning long-reads from Pacific Bioscience sequencers to genome sequences. Many methods have been developed for solving the sequence alignment problem over the past decades15. Currently, the most popular alignment algorithms use the seed-chain alignment procedure. In this procedure a set of subsequences are extracted from the reference genome and indexed. Likewise a set of subsequences are extracted from a batch of query reads. Then each query read is only compared to sections of the reference genome that have subsequences in common. The assumption is that each read can only map to sections of the genome which share exact subsequences with the read. This assumption, however, is violated when error rate is high and thus the most popular alignment algorithms fail to identify a large portion of true alignments. One way to identify more true alignments in the high error rate regime is to relax the requirement that read and genome share exactly matching subsequences to read and genome share similar subsequences . However, methods for speeding up alignments of reads with high error rates using this relaxation are not available.Aligning millions of DNA sequences is significant for identifying functional and evolutionary relationships between organisms, assembling genomes, and analyzing single-nucleotide polymorphisms9. It has been shown that optimal inference is equivalent to the dynamic programming solution to the sequence alignment problem. In order to overcome the low true positive rate of existing sequence alignment methods when the error rate is high, we first model sequence alignment as an inference problem in a latent variable generative model . For each X, we compute Y that are in the same buckets as X. This strategy significantly reduces the number of sequence pairs for which A naive implementation of this natural algorithm suffers from the fact that 3. Algorithms based on HMM like probabilistic models are either slow or require heuristics3 with no theoretical bound on performance. The area of asymetric bucketing is relatively underexplored4. In order to detect homolog sequences, Mak et al.4 introduced a method that would put pairs of sequences in the same bucket if they shared the same subsequence plus or minus a few deletions and insertions. However, Mak et al. does not consider that computational complexity may increase due introducing this procedure and do not provide a methodology that adjusts the subsequence length and number of indels considered to various indel probabilities.In this paper, we introduce Distribution Sensitive Bucketing (DSB), a novel technique for efficient and accurate alignment of large read datasets against genomes. To do this, we first formulated the sequence alignment problem as an inference in two types of latent variable models, Hidden Markov Models (HMMs) and pair Hidden Markov Models (pair-HMMS). For both probabilistic models, we developed an efficient inference algorithm and derived its complexity. We further designed a family of asymmetric bucketing functions that minimize the complexity. Several alignment software packages have been developed using HMM like probabilistic modelsDSB is an algorithm that given indel probabilities will generate a bucketing data structure for sequence alignment that aims to keep sensitivity high while keeping computational complexity of sequence alignment low. Like Mak et al., DSB will compare reads that share similar subsequences, not just identical ones. Our results show that the DSB algorithm is at least as sensitive as competing methods in aligning reads at low error rates, and outperforms all methods when the error rates are high. When aligning reads to distant homologs, the DSB algorithm is 10 percent more sensitivity than the closest competitor.For the sake of presentation clarity, we provide bodies of algorithms in E. coli reads against reference genomes of distance homologs. In each scenario both Sensitivity and False Positives are reported. Sensitivity is measured as the proportion of aligned regions (read to genome) in the reference that are reported by the corresponding method. A sensitivity of 1 means that every region in the reference is recovered while 0 means no regions in the genome are recovered.We benchmarked DSB-SA against popular alignment methods in several scenarios. The scenarios include aligning simulated reads against a reference genome, aligning experimental reads against corresponding genomes, and aligning Distribution Sensitive Bucketing (DSB) consists of the following steps (see details in the Method Section). First, a directed graph is constructed, where each sink node represents a bucket. Then, starting from the source node, query and reference reads are mapped to one or several buckets through the decision graph . Finally, all the buckets are explored, and pairs of query and reference reads in each bucket are reported as matches. In the following, DSB-HMM refers to DSB algorithm in case of HMM model  and they do not generalize to alignment of reads to distant homologs. Furthermore, most of the methods that are used in practice use heuristics to speed up alignment.decision graphs . Thus the hashes for pair-HMMs are phs Fig.  rather tThe sequence alignment problem has been modeled as a statistical inference problem, and dynamic programming solutions have been proposed to achieve optimal inference. Since these statistical models can be parameterized by different insertion, deletion, and mutation rates, the dynamic programming solutions are capable of discovering alignments between distant homologs. Still dynamic programming is very slow and runtime grows linearly with the size of reference. Inspired by locality sensitive hashing, in this paper we introduced the distribution sensitive bucketing paradigm for efficient subquadratic inference in latent variable models. In this paradigm, given a joint and an independent model for true and random pair of sequences, the sequences are mapped to buckets in a way that pairs of sequences from the joint model are more likely to fall into the same bucket than random ones. By focusing on pairs that fall into the same bucket, Distribution Sensitive Bucketing (DSB) can not only find read/genome pairs over a wide range of insertion, deletion, and mutation rates, but also avoids many unnecessary computations.Brute force techniques (e.g. LALIGN) are incapable of handling large scale alignment tasks. On the other hand, faster methods (e.g. Minimap2 and DALIGNER) can not handle larger mismatch, insertion and deletion rates. DSB is capable of recovering a significant portion of alignments with a reasonable computation time. We do not advise using DSB in cases where error rates are small (e.g. Illumina reads). In these scenarios, our benchmark shows that Minimap2 outperforms other methods while maintaining high accuracy. In case of erroneous reads or distant homologs, DSB algorithm achieves high sensitivity, while being orders of magnitude faster than the brute force alternatives.16. However, our results show that the main disadvantage of MinHash based methods is in their low sensitivity and high false positive rate. Switching to densified MinHash can make this problem more severe, since it has been reported that densified MinHash slightly improves the speed of MinHash while sacrificing sensitivity17.An alternative strategy to the MinHash approach used in MHAP is densified MinHashCurrently, DSB is based on a simplistic assumption that insertions and deletions occur independently. The current approach paves the path toward more sophisticated models, such as affine gap penalties and non-uniform distribution of nucleotides.P. In the context of sequence alignment, one can treat Y as the query sequence and X as a sequence in the database. The inference problem for latent variable models is defined as follows: Given a latent variable model 18, and then finding P(Y|X) for all N is large. Also note that the forward algorithm is not applicable to the string alignment problem with insertions and deletions.Consider an HMM X, Y) with higher Here we introduce decision trees with bucket data structures, and show how they can be used for mapping database sequences G and a set of distinguished leaf nodes of G called “buckets”. Decision tree G is a directed tree with a set of nodes V. Each node v). For each w of v corresponding to a and b to the end of v to w is indexed by . Furthermore the set of D buckets D is referred to as the number of buckets. Later we will discuss how to set D and The data structure consists of a decision tree G and a set of buckets Given the decision tree v in position X starting at j, i.e. Y is mapped similarly. For example if j, J bands are needed in order to guarantee a true positive rate nearly one.A sequence j-th band. That is:G  pairs jointly generated under the HMMs that are captured in the same bucket. Using J bands we havej. In order to have a nearly one true positive rate, i.e. M is the number of sequences in N is the number of sequences in j, and X and Y in band j fall into, respectively.In the H, e.g. m”, “i”, and “d” represent match/mismatch, insertion, and deletion respectively. Given H, the generative model generates pre-sequences H is “i” (“d”), the corresponding entry X and Y by removing all “−” from Algorithm 1 is limited to standard HMMs, and does not generalize to other latent variable models. Unfortunately, standard HMMs are not sufficient for modelling the important features of the sequence alignment problem, as they can not model insertions and deletions. Here, we define a natural probabilistic model (very similar to pair-HMMs) that we refer to as the sequence alignment model, that takes into account insertions, deletions, and matches/mismatches between pairs of sequences. Our probabilistic model first generates a latent variable entclass1pt{minimaentclass1pt{minimaFor simplicity we have assumed that insertion, deletion, and match/mismatch events are happening independent of each other. In practice, these events are usually not independent, and a HMM is used for modeling intervals of insertions and deletions, that is equivalent to using an affine gap penalty. While in this paper we will focus on the independent insertion and deletion model, the algorithms presented can generalize to arbitrary priors.So far, we have shown that the sequence alignment model can be stated as a special case of latent variable models. We will show how the algorithms we have developed for HMMs (without insertions/deletions) can be adapted to the case of sequence alignment models.Here we develop analogous methods to solve the inference problem in case of the sequence alignment model. We present a method to align sequences via the bucketing strategy (Algorithm Supplementary Information.