Protein functional similarity based on gene ontology (GO) annotations serves as a powerful tool when comparing proteins on a functional level in applications such as protein-protein interaction prediction, gene prioritization, and disease gene discovery. Functional similarity (FS) is usually quantified by combining the GO hierarchy with an annotation corpus that links genes and gene products to GO terms. One large group of algorithms involves calculation of GO term semantic similarity (SS) between all the terms annotating the two proteins, followed by a second step, described as “mixing strategy”, which involves combining the SS values to yield the final FS value. Due to the variability of protein annotation caused e.g. by annotation bias, this value cannot be reliably compared on an absolute scale. We therefore introduce a similarity z-score that takes into account the FS background distribution of each protein. For a selection of popular SS measures and mixing strategies we demonstrate moderate accuracy improvement when using z-scores in a benchmark that aims to separate orthologous cases from random gene pairs and discuss in this context the impact of annotation corpus choice. The approach has been implemented in Frela, a fast high-throughput public web server for protein FS calculation and interpretation. Proteins can further be computationally compared regarding their molecular function and biological role in the cell, provided there is a set of standardized annotations that can be used to calculate measures of semantic similarity2. The Gene Ontology (GO) Consortium has been making major contributions in this area by establishing a controlled vocabulary of defined terms for annotating gene and gene product properties across species3. More specifically, GO provides three independent ontologies to describe biological molecules: Molecular Function (MF), Biological Process (BP), and Cellular Component (CC). Starting as a collaborative annotation project between fruit fly, yeast, and mouse model organism databases, GO now contains a growing collection of ontology terms as well as a large amount of annotations from all kingdoms of life. GO terms are organized in a directed acyclic graph (DAG) with the so-called root term being the most common term with only child terms. More specific terms are further away from the root and any term (other than the root) may have multiple child and parent terms.Gene products can be compared in many different ways, researchers for example have been performing comparisons between proteins based on the amino acid similarity4, defining a function that maps a pair of GO terms to a numeric value expressing the semantic similarity (SS) of this pair. Now, more than a decade later, roughly 100 different methods have been suggested to compute SS for pairs or sets of GO terms5. Several reviews are available9, which attempt to classify the different methods into groups with related SS measures. Generally, these measures make use of the structure of the DAG and/or combine this with information from a GO annotation corpus, which provides the mapping between GO terms and gene products. Following the classification suggested by Guzzi et al.7, measures may be distinguished by their use of ancestor terms, whether they utilize DAG structural properties such as term depths or path lengths, or if they consider the term information content, which relates to negative logarithm of the probability to observe a GO term or a more specific term in the annotation corpus given the DAG. The SS measures have also been classified as pairwise or group-wise, according to how the multiple GO terms annotated to each protein are taken into account. In pairwise measures two proteins are compared semantically by either computing all possible pairwise semantic similarities between the GO terms annotated to each of the two proteins and determining a suitable combined score (often referred to as mixing strategy), for example by taking the maximum of all SS values. Alternatively, group-wise measures perform a direct comparison of sets of GO terms. Ultimately, some measures apply vector space models, where the GO terms annotated to a protein are encoded as a vector, and two vectors are then used to compute the SS of the respective proteins.Soon after the introduction of GO, researchers started to apply semantic similarity concepts from natural language ontologies to GO-annotated gene products10, and conversely, prediction of false positives in PPI networks11, prediction of pathways from PPI genome-wide data12, imputation of missing values in expression data13, construction of SS networks to discover disease genes14, protein function prediction15, just to name some recent applications, see Pesquita et al. for a more complete overview9. Performance of SS measures was also benchmarked on PPI data and on their relationship to sequence similarity and gene expression data, summarized in the CESSM tool for assessment of SS measures16.Protein semantic similarity has been applied to answer various biological questions, such as protein-protein interaction (PPI) prediction17 and promising gene products (e.g. disease associated genes) or specific gene families having a higher number of annotations. These biases have been analysed over time18 and lead to correlations between the number of GO terms a protein is annotated with, which in turn affects applications that involve SS measures19. Protein annotations with GO terms include evidence codes, which inform about the source of the annotations and which are frequently interpreted as a quality index. In particular, use of automated annotations , which form the majority of all available annotations, has been investigated in most SS assessments. In general, these assessments report that including annotations with IEA evidence codes improves performance most of the time7, and it is encouraging to see electronic annotations improve over time20. Finally, certain SS measures are affected by shallow annotations, which are very generic/unspecific annotations close to the root, in the sense that two proteins furnished with only shallow annotations receive high SS scores21.Over time, several groups identified and discussed drawbacks of individual SS measures and the annotation process as a whole. Evidently, protein annotation is biased and is influenced by different research interests, with model organisms of human disease for example being better annotatedet al.7 and Gan et al.6, with subsequent work published as web services or extensions to existing web resources25 or as downloadable source code29. Notably, the GOssTo framework allows calculation of several popular SS measures both as a standalone Java program or through a web service30, whereas the Semantic Measures Library offers about 50 SS measures in a command line Java toolkit. Ultimately, Mazandu et al. provide the most complete set of SS measures both for online calculation31 and as a high-performance Python command line tool5.Considerable effort has been put into developing software packages and web servers to compute SS, see reviews by Guzzi http://frela.eurac.edu, which allows high-throughput calculations on an average of several ten thousand protein pairs per minute. The server facilitates biological interpretation of computed FS scores. The software, including the source code for the web server, is available for download from our web server.Here, we present an orthology-based evaluation of different functional similarity (FS) measures, resulting from the combination of six popular SS measures with five mixing strategies. We tackle annotation bias by introducing protein-specific z-scores and discuss their performance when compared to conventional FS scores. This work also reinvestigates the influence of electronic annotations and annotation corpus choice on the performance of the FS measures with respect to all three available gene ontologies. We furthermore combine FS scores from the different ontologies and compare their performance. The methods can be run on a new web server, simRes), Lin (simLin), Schlicker (simRel), information coefficient (simIC), Jiang and Conrath (simJC), and graph information content (simGIC). We combined these SS measures with each of the following five mixing strategies: average (fsAvg), maximum (fsMax), maximum of best matches (fsBMM), best matches averaged (fsBMA), and mean of best matches (fsABM), which gives a total of 30 FS measures that were investigated in our study.In this work, we have compared the performance of selected functional similarity measures in discriminating pairs of orthologous genes from random pairs and utilized protein-based z-scores as a means to improve the discriminatory power of these FS measures. To summarize, we have chosen the following six pairwise SS measures: Resnik (simGIC) discussed here make use of the most informative common ancestor (MICA), that is, the common ancestor of the two compared GO terms that has maximal information content. The earliest SS measure is simRes, which simply selects the information content of the MICA. This measure is not normalized and neglects any information about the contributing GO terms. Since simRes is not bounded from above, any normalization procedure  depends on the annotation corpus itself, and therefore hampers score comparison between corpora. To limit these drawbacks, simLin normalizes simRes by the sum of the contributing GO terms’ information contents, such that MICAs that are close to their GO terms receive a higher score than those that are higher up in the GO graph. In simRel, the level of annotation detail is addressed by weighting simLin with the counter-probability of the MICA. By this, shallow annotations receive less relevance than MICAs further away from the root. A criticism of this approach was that annotations close to the root or to the leaves receive too little weights, which was addressed by an alternative weighting scheme in the simIC measure. Initially designed as a metric33, then transformed into a SS measure34, simJC shows little difference to simLin35. Ultimately, simGIC takes into account all ancestor terms and, most notably, does not make use of MICAs.All SS measures  has a strong influence on scores. Figure simLin/fsAvg measure. The scores are computed both when taking into account annotations with evidence code IEA (IEA(+)) and also when excluding IEA annotations (IEA(−)), see Table simLin/fsAvg score. Each distribution exhibits a sharp peak in the lower score ranges that is absent in the case of scores calculated from non-electronic annotations. Since the scores are averages of proteins with randomly chosen partners, the peaks correspond to background similarity due to the electronic annotation process. This can be rather extreme, as for example is the case in the BP ontology  distribution is elongated by almost 30% of the length of the IEA(−) distribution. This tail accounts to proteins annotated with only IEA evidence codes.Several studies have examined annotation bias in GO. Early on, Wang ogy Fig. , where iogy Fig.  furthermet al.43 suggested using as a similarity threshold the 95% quantile derived from scores calculated from random protein pairs. Alternatively, we propose to improve the similarity scores of two proteins by taking into account their respective score background distribution and calculate a similarity z-score (see Methods) that is less affected by annotation biases of specific proteins. This requires calculation of mean and standard deviation for each protein P by evaluating FS scores from protein pairs , where proteins Q are randomly sampled. The so derived mean score for protein P represents a baseline score that varies from protein to protein  score of 0.22 in BP ontology. A similar score is computed for the unrelated pair of a human G-protein coupled receptor and a mouse histone protein (UPANr Q96LA9 and Q8CGP5). However, the latter protein pair has an average simLin/fsAvg score of 0.21 for 1000 randomly chosen mouse proteins, and in this context the particular score of 0.22 should no longer be considered high enough to call the pair similar. By calculating a z-score for these specific pairs of proteins, we find for the orthologous alcohol dehydrogenases z = 2.97, and for the unrelated pair of proteins we compute z = 0.41, now giving a clear distinction on protein functional similarity.As a first example, we consider the orthologous genes encoding human and mouse alcohol dehydrogenase 1 (UniProt accession numbers (UPANr) P07327 and P00329, respectively), which have a PARK2 disease gene, which encodes an E3 ubiquitin protein ligase. Mutations on this gene have been shown to be causative for various forms of Parkinson’s Disease. As a disease gene it is heavily annotated with 91 GO terms in BP ontology, and can therefore be considered an annotation-biased gene that tends to receive on average higher FS scores. In fact, the mean simGIC/fsBMA IEA(+) functional similarity score between PARK2 and 1000 randomly selected mouse proteins equals to 0.186, which corresponds to the 99.7 percentile of all human protein scores. In this context, comparison of PARK2 protein (UPANr O60260) with two other mouse proteins, FBXL2 (Q8BH16) and GATAD1 (Q920S3), results in a similar, but very high simGIC/fsBMA IEA(+) score of 0.53. However, z = 5.58 whereas z = 2.68 provides help to discriminate a protein involved in an E3 ubiquitin-ligase complex (FBXL2) from a zinc finger protein (GATAD1). (All examples can be reproduced on our Frela server).As another example we examine the et al. used orthology relationships to quantify the ability of a SS measure to distinguish orthologues determined by phylogeny from an equally sized set of randomly paired proteins to demonstrate the superiority of their newly developed SS measure over others27. We have extended this idea and made use of high quality orthology relationships to define optimal thresholds for separating pairs of orthologues from random protein pairs for both raw FS scores and z-scores. This allowed us to directly compare FS measures under various conditions, to determine the optimal measure and additionally to investigate possible improvements achieved by applying a z-score calculated on top of the conventional FS scores. Since each measure is evaluated on the same set of cases and controls, all measures considered were equally exposed to any existing annotations bias, shallow annotations, or any other flaws in the GO ontology. It is furthermore known that GO annotations are incomplete and erroneous45, and it is not guaranteed that no true positive was selected as a control by random sampling or conversely, that orthologues are not recognized due to very low function similarity described by GO itself. These are challenges to the orthology-based evaluation framework we have applied. However, all measures are tested on the same set of cases and controls, and therefore within this testing framework these effects cancel out when comparing measure performance on a relative basis9. Therefore, this allows ranking the tested measures; yet, we discourage to draw conclusions on an absolute scale such as transferring the error rates to other applications.In a recent study, Wu x-axis) and respective z-score (y-axis). In general, application of z-scores mildly improved accuracy; FS scores based on fsMax are the only exception. The overall lowest error rates were observed for the closely related and well annotated human/mouse orthologues: We found an error rate of approximately 2%  corpus  data: simIC/fsMax, simRel/fsMax, and simJC/fsMax  data fsMax behaves similar to the generally well-performing fsBMA mixing strategy due to small number of annotations per protein9. We have already mentioned the very low error rates for human/mouse orthologues, such that by transitivity the mouse/fly orthologues also yielded similar results as the human/fly comparison. We also found the same measures ranking top. For both IEA(+) and IEA(−), the error rate in mouse/fly orthologues was higher by one to two percent compared to the human/fly orthologue dataset  dataset when combined with simIC, simRel, and simLin SS measures  dataset  dataset distinguished themselves with the lowest error rates for any set of orthologues we have investigated. When restricting annotations to the IEA(−) dataset, various other measures ranked top, and simGIC/fsBMA consistently ranked in the top 20%  and simRes/fsBMA performed best in a sequence similarity-based benchmark using MF ontology. Since the CC ontology describes cellular locations, it is not surprising to find rather high error rates when using this ontology for protein functional similarity  annotation corpus, as this mixing strategy generally overestimates protein functional similarity. On the other hand, when looking for very good single matches of GO terms, fsMax will be a good choice, especially in an annotation corpus without electronic annotations, and has been recommended for protein-protein interaction analysis32. Broader partial matches are best detected by fsBMA. Overall, we find simGIC/fsBMA as a recommendable all-round FS measure.In all three ontologies we have observed that 47. In particular, Rogers and Ben-Hur47 identified a 16% increase in average accuracy as classifier overestimate when predicting protein function using BP ontology and electronic annotations. In BP, when using the IEA(+) dataset we find an average decrease of 3.9% in error rate for both raw and z-score based measures. One reason for this improvement is explained by orthologues that are annotated with only IEA evidence codes such as the pair given by UPANrs , which is annotated electronically only with GO term GO:0006508 (proteolysis). In our human/mouse orthology dataset, we found 221 orthologous gene pairs where both genes are annotated with IEA evidence codes only, implicating that GO terms have been transferred during an automated annotation process. Moreover, these 221 gene pairs make already 39% of all pairs that are only present in the human/mouse orthologue IEA(+) set, whereas the remaining 61% are pairs where only one protein is annotated exclusively with IEA evidence codes. The improvement with IEA can also be explained by the effect of IEA annotation being partially based on orthology, resulting in a similar GO annotation between orthologues and therefore the similarity measures achieve better discrimination from the background of unrelated proteins. Overall, when including automated annotations, accuracy was higher, but this is also due to minimally and solely electronically annotated proteins, which should be kept in mind when making a decision whether to involve automated annotations. Our web service therefore provides means to investigate FS score composition, so that such cases can be detected easily.Our results confirm previous findings that inclusion of electronic annotations improves measure accuracy46. It is therefore of interest if protein functional similarity should be computed with data retrieved only from the organism(s) under investigation rather than taking all available data from all annotated organisms. To examine this question within our orthology framework, we restricted the annotation corpus to include only data from the organisms that form the orthologue pairs (AOO) and compared it to our default corpora that do not impose any restrictions on organisms (ALL). For example, for human/mouse orthologues, SS measures were computed from GO graphs containing information content related to only human and mouse protein annotations (AOO), and were compared to SS measures computed from GO annotations of proteins from all organisms (ALL).The annotation corpus forms the basis for calculation of FS scores and therefore has an influence on the score magnitude, which needs to be kept in mind during score interpretationsimJC/fsAvg being the only exception where the null hypothesis that the two datasets’ mean ranks are equal was not rejected at the significance level of α = 0.01. On the other hand, the magnitude of the difference is rather small: overall, error rate differences are mostly below 1% of the respective case/control dataset size. The measures with the lowest error rates in BP, simJC and simGIC in combination with fsBMA  error rate drops by 1.48% from 716.73  to 665.51  false assignments in a dataset of 1726 mouse/fly orthologues and 1726 controls sampled 225 times . On the one hand, this is quite remarkable, as this observation holds for 29 SS/MS combinations, with BMA Fig. , showed (−) annotations, it is much more convenient to compute scores based on a single database derived from all available annotation corpora instead of using corpora tailored to organisms.To summarize, when discriminating orthologues from unrelated proteins it seems to be beneficial to rely on a larger set of annotation corpora instead of restricting to the annotations in the respective organisms. Even though gain of accuracy is generally small and is only evident when restricting to IEAet al. suggested combining FS scores from BP and MF ontologies into a single FS score by computing the root mean square of the two ontologies48, which to our knowledge has never undergone any performance investigations. The orthology testing framework described here, provided an appropriate environment for investigating the performance of FS measures based on BP+MF combined ontologies, as well as the effect of combing all three ontologies (BP+MF+CC) into a single FS score. Our analysis of BP+MF, and BP+MF+CC combined ontology FS scores reflects an increasingly specific filtering strategy starting with biological processes: an annotation to BP is made when a gene product activity is part of, or regulates, or is upstream of but still necessary for, a biological program49. This broad context makes BP annotations especially valuable for protein functional similarity calculations. Requiring additional similarity by MF, that is, comparable biochemical or signalling activity, will generally narrow down the list of matching candidate proteins and increase the overall sensitivity of the query. Demanding protein colocation by utilizing annotations from CC ontology presents the ultimate refinement in protein functional similarity that can be retrieved from GO. For completeness, the Frela server allows score calculation based on any combination of ontologies.A decade ago, Schlicker When looking at the results we need to keep in mind that combined ontologies require every protein be annotated to each single ontology, which results in a smaller number of orthologous gene pairs used in the assessment  dataset, with fsAvg and simLin/fsMax being the only exceptions . One of these orthologues is given by the human/fly pair Q6ZYL4/B7Z018 , but due to only a single annotation in MF for each of the two proteins, which evaluates to a raw MF score of 0.0, and therefore heavily penalizes calculation of the combined score, this orthologous pair became a false negative in the BP+MF combined ontology.In Fig. ent Fig. . NotablyThe results indicate that there are considerable differences between established FS scores, where some measures perform better in the identification of functionally related proteins. In addition, the results suggest that combining FS scores from the three ontologies  tends to improve the performance of the various FS measures. Future work should address several remaining issues, in particular problematic cases should be investigated, for example where functional similarity scores are unexpectedly low, in order to identify the limitations in the current functional similarity approaches and devise new solutions. Many approaches have been proposed for measuring functional relationships, but in the current work we have only assessed a few of the most established. It would be desirable to extend the current analysis to other approaches, as for example group-wise SS measures. Finally, the current work focused on orthologous genes as a reference set of functionally related proteins, future work should focus on investigating FS-based approaches for concrete applications like candidate gene prioritization.http://frela.eurac.edu, which supports calculation of different FS measures by combining any of the six SS measures with five different mixing strategies discussed in this study. Frela can be invoked in two major modes: the interactive mode receives input directly from the user through web entry forms, while the batch mode processes uploaded files. It is possible to perform calculations on either the IEA(+) or the IEA(−) dataset for any of the BP, MF, and CC ontologies or combinations thereof  to red (high similarity). Based on the chosen FS measure, various types of maxima are denoted by line styles of the affected cells, which greatly simplify understanding score calculation and further score interpretation.STX1B (UPANr P61266), which has been associated with Parkinson’s Disease50. When evaluating simGIC/fsBMA scores for the BP IEA(+) ontology, its fly orthologue Syx1A (UPANr Q7KVY7) ranks on position 50 of the hit list sorted by z-score (and position 107 when sorted by raw score). One reason for this rather low rank is that the fly protein is annotated with GO term GO:0002121, “inter-male aggressive behavior”, a term which is obviously missing in the annotation of the human protein . We have retrieved GO graphs and annotations as of September 2015 as MySQL dumps from the GO website and extracted data for human and the model organisms Drosophila melanogaster (fly) and Mus musculus (mouse), which form the basis for all of our comparisons. For each of the BP, MF, and CC ontologies we compute the term probabilities P(t) = N(t)/N(root), where N(t) denotes the number of proteins annotated with a term t or any of its descendants, and root is the ontology’s unique term without any ancestors. The GO term information content is then given by I(t) = −log(P(t)).An annotation corpus s and t be two GO terms we want to compare semantically and let S denote the set of all common ancestors of s and t. The measure defined by Resnik51 is given by52 takes into account the information content of the most informative common ancestor relative to the information content of the two terms,In this study we concentrate on six frequently used pairwise SS measures. Let therefore 39,53 by introducing the information coefficient measure asSchlicker weights Lin’s measure by common ancestor term probability33 has been shown to be equivalent with Lin’s measure35 but is included in our work for historical purposes. It is defined byJiang and Conrath’s measure9, which by design is a group-wise SS measure, as a pairwise measure for consistency as suggested by Li et al.53:Finally, we use the graph information content measureP is annotated with m GO terms t1, t2,.., tm and protein R is annotated with n GO terms r1, r2,.., rn, then the matrix M is given by all possible pairwise SS values sij = sim with sim being one of the SS measures introduced above, i = 1, 2,.., m and j = 1, 2,.., n. Functional similarity is computed from the SS entries of M according to a specific mixing strategy (MS), and in this work we investigate five different mixing strategies. A frequently used MS uses the maximum value of the matrix, fsAvg takes the average over all entries, Let us assume protein 48. We focus on pooling scores from BP and MF ontologies and from BP, MF, and CC ontologies. A functional similarity F is computed by combining a SS measure with any mixing strategy defined above over any of the different ontologies: biological process (FBP), molecular function (FMF), and cellular component (FCC). We compute the combined measures as the functions We additionally study the effect of combining multiple gene ontologies into a single score, as suggested previouslyP and R the similarity z-score asP and R, respectively, and F is a functional similarity measure between P and R based on a combination of any supported SS and MS. From the definitions above it follows that the mean values and standard deviations depend on the ontology, the organism, and the functional similarity, but for reasons of simplicity these parameters are omitted in the formula.Given a pair of target organisms, for each annotated protein in the respective corpora we compute a score background distribution by random sampling without replacement of 1,000 annotated proteins from the corresponding second organism. From this distribution we compute the mean and standard deviation. For example, if we choose human and mouse, in our dataset we have 21,212 human proteins furnished with BP annotations (excluding IEA) and therefore yield 21,212 mean values and standard deviations, one pair for each annotated protein. Conversely, we find 9,714 manually annotated proteins for mouse  is provided by a modified z-score54, which utilizes medians instead of means and may be considered for distributions deviating heavily from Gaussians. We have reviewed its applicability, and for most of our functional similarity measures we find only small differences between means and medians in representative sets of randomly paired proteins. We therefore remain with the more traditional definition of z-scores described above. Further discussion concerning modified z-scores is provided in Section “Modified z-Score” in the Supplementary Material.An alternative approach to score the protein pair (scores or raw scores when a distinction to z-scores is needed). First, since there is much discussion about the impact of automated annotations, we utilize datasets that include data inferred from electronic annotations (IEA(+)) and such that do not (IEA(−)). The latter is implemented by omitting annotations to GO evidence code IEA. Simultaneously, we examine the effects of incorporating annotation data from organisms other than those being compared. Whenever we compare proteins from our set of target organisms , we distinguish between scores calculated with all available annotations in GO (ALL corpus) and scores that are based only on annotations made to the organisms belonging to the proteins under investigation (AOO corpus).We investigate the influence of various compositions of annotation corpora on FS scores , and true and false negatives (TN and FN) according to standard nomenclature56. We then determine the optimal threshold h*, for which the error rate is minimized. The error rate (or fraction of incorrect) is the number of incorrectly assigned pairs divided by the number of all pairs, (FP + FN)/(TP + FP + TN + FN). Since this error rate calculation involves randomly drawn controls, the outcome will differ from one random control set to another. We therefore repeat optimal threshold determination and its associated error rate computation on 225 sets of control proteins and compute 99% confidence intervals (CI) of the mean for the so-obtained 225 error rates of both raw scores and z-scores. In this manuscript, we therefore always report mean error rates of 225 individual error rates, called “error rate” for brevity. Accordingly, reported thresholds, especially those shown in the Frela web interface, refer to an average over 225 optimal thresholds h*. We refer to Supplementary Data File We use orthology relationships as a means to evaluate a measure’s ability to discriminate between functionally related and unrelated pairs of proteins: orthologous genes should have similar functions and therefore we expect them to have higher scores than randomly selected pairs of genes. In particular, we use Ensembl 82 Biomart to retrieve one to one orthology relationships between human/mouse, human/fly, and mouse/fly organism pairs of protein coding genesWe specifically test if there are SS/MS measures that significantly perform better (or worse) when using the AOO annotation corpus instead of the ALL corpus. For an arbitrary fixed ontology and a SS/MS measure, we perform a Wilcoxon signed-rank test that compares the individual error rates based on z-scores of each of the 225 case/random control datasets between ALL and AOO corpora (with IEA evidence code annotations), using cases from all three orthology relationships we investigate, human/mouse, human/fly, and mouse/fly. In order to detect whether the ALL or the AOO corpus results in better error rates, we perform one-sided tests. A test is considered significant at the level of α = 0.01 after Bonferroni adjustment for multiple hypothesis testing of 30 measures (six SS measures times five mixing strategies).57 for functional similarity analysis and is implemented in the Python programming language. We make use of a client/server architecture, where the computation server is decoupled from the web server. This specifically allows employing a locally installed server that is queried by users from different hosts with computationally inexpensive client software. On a 2.3 GHz AMD Opteron processor with 32GB of RAM, Frela computes BP protein functional similarity scores of 10,000 random protein pairs from the human organism in about 10 seconds and scales linearly with the number of protein pairs compared. Time doubles when utilizing the simGIC SS measure due to more complex graph algorithms used in score calculation (for a more complete set of timings, see Supplementary Table http://frela.eurac.edu.Our software extends the Dintor frameworkSupplementary MaterialData File S1Data File S2


Protein cavities play a key role in biomolecular recognition and function, particularly in protein-ligand interactions, as usual in drug discovery and design. Grid-based cavity detection methods aim at finding cavities as aggregates of grid nodes outside the molecule, under the condition that such cavities are bracketed by nodes on the molecule surface along a set of directions . Therefore, these methods are sensitive to scanning directions, a problem that we call cavity ground-and-walls ambiguity, i.e., they depend on the position and orientation of the protein in the discretized domain. Also, it is hard to distinguish grid nodes belonging to protein cavities amongst all those outside the protein, a problem that we call cavity ceiling ambiguity.We solve those two ambiguity problems using two implicit isosurfaces of the protein, the protein surface itself  that excludes all its interior nodes from any cavity, and the outer isosurface that excludes most of its exterior nodes from any cavity. Summing up, the cavities are formed from nodes located between these two isosurfaces. It is worth noting that these two surfaces do not need to be evaluated , triangulated, and rendered on the screen to find the cavities in between; their defining analytic functions are enough to determine which grid nodes are in the empty space between them.This article introduces a novel geometric algorithm to detect cavities on the protein surface that takes advantage of the real analytic functions describing two Gaussian surfaces of a given protein. Macromolecules  are the building blocks of living beings. In particular, proteins are relevant for the cell chemistry inasmuch they perform a variety of different functions, such as catalysts, transporters, sensors, and regulators of cellular processes. Such functions depend on the interactions that establish with other entities in the cell, namely long entities like nucleic acids  and with small entities like nucleotides, peptides, catalytic substrates, and man-made chemicals. Thus, such interactions have some flavors, namely: protein-ligand, protein-protein, protein-DNA, and so forth. It is clear that these interactions involve both shape complementarity and physicochemical complementarity between a protein and any other fitting entity.putative binding sites of a given protein for ligands.Nevertheless, this article does not focus on physicochemical complementarity. Instead, the focus is on detecting cavities on the protein surface where ligands  may bind. The detection of protein cavities is instrumental as a first step to establish the shape complementarity between a protein and a ligand. As noted by Kawabata and Go , identifThe algorithms to identify binding sites on a molecular surface are divided into four categories: geometry-based, energy-based, evolution-based, and hybrid approaches . In thisGrid-based methods are characterized by mapping a protein onto an axis-aligned 3D grid, using then a particular geometric criterion to detect cavities on the protein surface. Well-known geometric criteria are those based on distance grid-and-k et al. . In bothwww.ebi.ac.uk/pdbsum/) [As mentioned above, this paper addresses a grid-and-surface method, here called GaussianFinder. This method combines two Gaussian surfaces of a given protein, called inner and outer surfaces, as a way of finding cavities as clusters of voxels located between those two surfaces. As shown further ahead, this solves the ambiguity problems of grid-based methods mentioned above, i.e., the problems faced in the delineation of the limits of protein cavities, without using any visibility criterion of the grid-and-surface methods above to find cavities on the molecular surface. GaussianFinder aims at finding protein cavities accurately relative to ground-truth binding sites certified by known databases, as the one known as PDBsum (pdbsum/) .Before proceeding any further, let us also mention the methods 3V and KVFinder due to Voss and Gerstein  and OlivIn regards to KVFinder, one obtains the cavity voxels by the difference of two but different voxelized volumes. KVFinder uses a solvent sphere of radius 1.4 Å and a default probe sphere radius of 4.0 Å. However, this method only operates on grid points outside of the molecule atoms. In the first step, KVfinder collects all outside grid points such that the solvent sphere centered at each outside grid point fits in the empty outside space without overlapping the molecule. The second step is identical to the first one, with the difference that one uses the default probe sphere instead of the solvent sphere. The cavity voxels are those that belong to the first voxelized volume, but not to the second one. Therefore, cavity voxels correspond to the empty outside space where the solvent sphere gets in, but the default probe sphere does not.GaussianFinder builds upon the concept of Gaussian surface, which is defined as the level set n of Gaussian kernel functions fi, one function per atom i, and where xi and ri stand for the center location and van der Waals radius of the i-th atom, respectively, while β represents the Gaussian kernel decay value. Therefore, the Gaussian surface depends on two parameters, c and β [where  c and β , 25.Fin(x)=c and Fout(x)=c of each protein  was set to  for both inner and outer surfaces because it is the one that more closely approximates the solvent-excuded surface (SES) [where ce (SES) , 26–28.http://www.rcsb.org) in an array on CPU side.Read atomic centers of a protein from the PDB file  and q=. These coordinates are then updated such that p=p−2R and q=q+2R, where R is the maximum atomic radius among the atoms belonging to the molecule, as needed to guarantee that the molecule lies in the box D.Determine the bounding box Fin, Fout, intermediate voxels between the inner and outer surfaces, and cavity voxels. These 3D arrays of voxels are size-congruent and depend on the voxel length Δ=1.0.Copy the array of atomic centers into GPU memory, and allocate GPU memory for the following 3D arrays, as needed for: voxels of bounding box, The diagram of the GaussianFinder method is shown in Fig. D, i.e., a grid of nodes.Voxelize the bounding box Fin(x) at every grid node.Calculate Fout(x) at every grid node.Calculate Fin(x).Calculate voxel flags for Fout(x).Calculate voxel flags for Identify intermediate voxels (or grid nodes) between the inner and outer surfaces.Identify cavity voxels among the intermediate voxels.After completing the pre-processing stage, GaussianFinder identifies the cavities of an input protein through the following seven steps on GPU: x, y, and z) allocated in memory is transferred to GPU memory using the CUDA  functioitecture , as needD consists in partitioning D into a grid of equally-sized voxels  of length Δ=1.0 Å. Considering that the voxels are all axis-aligned, it thus suffices using only the 0-th corner  of each voxel to represent it, because the remaining seven corners of a voxel are 0-th corners of its adjacent voxels. Therefore, it suffices to allocate a 3-dimensional array of such 0-th corners representing the voxels on GPU side; this array is named V. The location of each 0-th corner is also calculated on the GPU side.This is the first CUDA kernel. The voxelization of the bounding box N threads , one per 0-th corner. Each thread calculates the value of Fin (see Eq. (This kernel launches (see Eq. ) at eachFout .Fin. Each bit is associated with each voxel corner so that we have 28=256 possible configurations for each voxel. If Fin<c at a voxel corner, its bit takes the value 1; otherwise, it takes on the value 0. Therefore, the flag 111111112=25510 indicates that the corresponding voxel is outside the inner surface because the value of Fin decreases with the distance to the protein. The flags are stored in a 3D array, called FLAGIN, which is of the same size as FIN.To determine the intermediate voxels between the inner and outer surfaces in Step 6, we need to find the voxels outside of the inner surface. For that purpose, we determine the 8-bit flag for each voxel of the scalar field Fout instead of Fin. But, now we are interested in voxels whose flag is 000000002=010, that is, voxels inside of the outer surface. The flags are stored in a 3D array, called FLAGOUT, which is of the same size as FOUT.This kernel is the same as the previous kernel, with the difference that now the computation of voxel flags is for i,j,k) between the inner and outer surfaces is easily identified through the condition FLAGIN=255 and FLAGOUT=0, here called the intermediate condition.Based on the results of 4-th and 5-th kernels, an intermediate voxel =c. This triangulation is carried out entirely on GPU side using the variant of the marching cubes algorithm introduced by Dias and Gomes [The graphics visualization of each protein requires the triangulation of the Gaussian molecular surface defined by nd Gomes –34.Figure The experimental testing results were obtained using a methodology built upon the following aspects: (i) hardware/software setup; (ii) a ground-truth dataset of protein cavities; (iii) set of benchmarking protein cavity detection methods; (iv) performance quality; (v) GPU time performance; and (vi) GPU memory space consumption.Testing was accomplished using a desktop computer running the Linux Fedora 25 operating system and equipped with an Intel-Core I7 6800K 3.4GHz GHz Processor, 32GB RAM, one Nvidia Tesla K40, and one Nvidia Quadro M6000. Most computations to detect cavities of proteins and other molecules took place on the Nvidia Tesla K40. Also, all the computations needed to triangulate surfaces of molecules and their cavities were performed on the same Nvidia Tesla K40. The Nvidia Quadro M6000 was only used for graphics output and visualization.Furthermore, GaussianFinder was written in C/C++ together with CUDA 9.0 to run on GPU. As noted above, we used the DBSCAN clustering algorithm to form clusters of cavity voxels featuring protein cavities. This clustering step runs on CPU side. Triangulating and rendering surfaces of proteins and binding sites on GPU were performed using a variant of the GPU-based implementation of the marching cubes algorithm by Dias and Gomes –34.www.ebi.ac.uk/pdbsum/) as the ground-truth dataset of protein cavities because it provides us with already known binding sites for a set of proteins [www.rcsb.org). By inspection of the LigASite dataset in the PDBsum, we counted 8150 cavities on apo proteins, and 17850 cavities on holo proteins.We used PDBsum .SURFNET. It includes the sphere-based method proposed by Laskowski [askowski .PASS. It includes the sphere-based method proposed by Brady et al. [y et al. .Fpocket. It includes a triangulation-based method based on a Voronoi tessellation and alpha spheres on the top of a convex hull algorithm .GHECOM. It includes the sphere-based method proposed by Kawabata [Kawabata .ConCavity. It includes the grid-based method proposed by Laskowski [askowski .3V. This grid-and-sphere method was proposed by Voss and Gerstein [Gerstein .KVFinder. This grid-and-sphere method was introduced by Oliveira et al. [a et al. .For benchmarking sake with GaussianFinder, we used the following protein cavity detection methods: These methods and the GaussianFinder were run on the same desktop computer to guarantee a fair comparison between them. Note that the first six methods listed above are also part of Metapocket .Let us now to analyze the performance quality of each benchmark cavity detection algorithms relative to the PDBsum ground-truth dataset of apo and holo proteins. For that purpose, we first counted 8150 cavities on the 816 apo proteins, and 17850 cavities on the 1788 holo proteins of the ground-truth dataset.CP. These positive cavities include the true positive (TP) and false positive (FP) cavities  and false negative (FN) cavities . Recall that the number of apo protein ground-truth cavities is C=8150, while C=17850 is the number of ground-truth cavities for holo proteins. From Tables Sv>0.9), but GaussianFinder ranks behind PASS, GHECOM, and Fpocket regarding specificity because the value of TN is not much greater than the value of FP. However, these four methods possess an accuracy about 90% (Sc≈0.9). Among these methods, GaussianFinder ranks first because its rate of detected ground-truth cavities (rd) stands out above the other methods . Moreover, we used actual locations of binding sites of proteins (via PDBsum) as the ground-truth for the cavities detected by those benchmarking methods, including GaussianFinder.We have introduced a novel grid-and-surface based algorithm, called GaussianFinder, for identifying cavities on protein surfaces without using a visibility criterion. The leading idea of the method is to determine the grid nodes between two Gaussian isosurfaces of each molecule, which are then aggregated into clusters of nodes featuring cavities. This avoids possible geometric ambiguities (concerning the limits of cavities) inherent to the use of grid-based methods to detect cavities of the protein surface. GaussianFinder is considerably fast, with the cavity detection stage finishing in a matter of a few seconds on a GPU-based workstation equipped with a Nvidia Tesla K40 and a Nvidia Quadro M6000. Shortly, we intend to parallelize other cavity detection algorithms existing in the literature for a more comprehensive comparison between algorithms in terms of time performance.Project name: GaussianFinder;Project home page:https://sourceforge.net/projects/gaussianfinder;Operating system(s): Linux Fedora 25;Programming language: C/C++;Other requirements: CUDA 9.0;Any restrictions to use by non-academics: The source code is freely available under the GPLv3 License.


Most cellular functions involve proteins’ features based on their physical interactions with other partner proteins. Sketching a map of protein–protein interactions (PPIs) is therefore an important inception step towards understanding the basics of cell functions. Several experimental techniques operating in vivo or in vitro have made significant contributions to screening a large number of protein interaction partners, especially high-throughput experimental methods. However, computational approaches for PPI predication supported by rapid accumulation of data generated from experimental techniques, 3D structure definitions, and genome sequencing have boosted the map sketching of PPIs. In this review, we shed light on in silico PPI prediction methods that integrate evidence from multiple sources, including evolutionary relationship, function annotation, sequence/structure features, network topology and text mining. These methods are developed for integration of multi-dimensional evidence, for designing the strategies to predict novel interactions, and for making the results consistent with the increase of prediction coverage and accuracy. Homo sapiens [Drosophila melanogaster [Saccharomyces cerevisiae [Caenorhabditis elegans [Proteins perform their complicated functions by physically interacting with other proteins. Sketching a map of protein–protein interactions (PPI) is a significant topic of system biology and an important step towards understanding protein functions and cellular behaviors . Differe sapiens , Drosophnogaster , Saccharrevisiae , and Cae elegans . However elegans ,10. More elegans . As far Bioinformatics techniques of PPI prediction strengthen and flourish the study of protein interactions . BioinfoIn this review, the workflows for prediction pair-wise PPIs by combined evidence from studies building PPI networks on the genome scale level are presented and discussed. The presented workflows mainly consist of three basic steps: (1) Defining gold standard datasets/training datasets of interacting and non-interacting protein pairs; (2) Characterizing the interactions by annotating gold standard datasets with diverse and carefully chosen evidence; this is an encoding process to turn protein interaction features into machine-readable rules; (3) Determining the probability of particular interactions by individual evidence, and thus combining the probabilities (or encoded vector) of all evidences to uncover the novel subset of the interactome.Units of gold standard datasets are usually constructed for training or testing of PPI prediction. Datasets for training and testing units are generally independent. The quality and reliability of gold standard datasets for training affect the performance of different machine learning methods .The gold standard positive (GSP) datasets are basically PPIs with high experimental confidence or reference evidence. Some of the datasets are available in public databases, such as: the Biological General Repository for Interaction Datasets (BioGRID) , the IntGold standard negative (GSN) datasets generally cannot be obtained by direct experimental measures. There is a Negatome database (2.0)  which prThe characterization of existing interactions is usually processed to explore the crucial role of protein interactions. Interactions can convert proteins/polypeptides into transient or permanent complexes and the binding is determined by different elements such as cell physiology , biochemistry environment  and shape of the binding surface , which are further involved in the fields of functional genomes, dynamics, kinetics, mechanics etc. ,44. ExpeN-dimensional vector: N, number of genomes; Value = 1 or 0, presence or absence of protein/domain in an organism) and evaluate protein pairs by measuring distance.Evolutionary Relationship: Methods based on evolutionary information use genomic context of organisms to infer functional associations between proteins, including gene neighborhood , gene fuOrtholog: If a pair of proteins has high similarity to the sequences of another pair of genes or proteins with known interaction in other species (orthologous proteins), they are supposed to have similar functions which infers the relationship of interactions. This approach usually uses sequence alignment algorithms to define the similarity of full sequences or residues, which is regarded as an index to predict interactions between proteins ,50,51.Gene Function Annotations: This method is based on the hypothesis that two proteins functioning in the same biological process should be more likely to interact with each other than those two proteins not sharing the same biological process. Information of biological function is accessible from some hierarchically structured annotation systems, such as GO, KEGG, EC and MapMan  , which pCoexpression: It is generally acknowledged that a pair of interacting proteins has relative gene expression, although the gene coexpression methods are an indirect way to infer the protein interaction (some results indicated that there is no straight correlation between gene expression profiles and PPI associations under some conditions ). HoweveN = 3, total number = 8000 (203)) [Sequence-Based Code Signatures: Some studies implement the natural language processing (NLP) technique to encode sequences for perdition of PPIs. The language of protein sequences is translated into sequence-based signatures and mapped into high-dimensional vectors by using the occurrence frequencies of each kind of building block . Differe0 (203)) ,61. ORF 0 (203)) .3)-dimensional vectors to represent a given protein, and each element of this vector is the frequency of an AA triad; (4) The PPI pair is represented by concatenating the individual two vectors of corresponding proteins. It is noticed that, if we do not process the AA cluster step (in step 1), protein pairs will be required to get encoded as a 16,000-dimensional vector , which is too large for most classifiers. The rule of seven classes for 20 AAs is effective and convenient to operate, and is developed as a classical method that has been widely applied in interaction prediction and interaction site prediction based on sequences [The Conjoint Triad Method   is one oequences .Sequence-Based Structure Signatures: Structure and chemical properties of a protein sequence can be translated into structure signatures to represent characteristics of a residue interface. These signatures include: 1) Physicochemical properties of amino acids, such as hydrophobicity, hydrophilicity, polarizability, solvent-accessible surface area (SASA), relative surface accessibilities (RSA) of residues, side chain net charge index (NCI), charge, isoelectric point, etc.; (2) Signatures of protein structure, such as 3D structure indexes in PDB, protein fold alpha helices, beta sheets and coils, posttranslational modifications (PTMs), and domains  Physicoc. These sDomain methods aim to establish protein relationships by domain−domain interactions (DDIs), which are applied widely in sequence-based PPI prediction ,46,47,48Network Topology: Network topological parameters are generally calculated from positive datasets. They characterize the topological properties of currently available protein interaction networks to evaluate target protein pairs. Graph-theoretic invariants include weighted domination number, average eccentricity number, the eccentricity, circumference, weighted peripheral number, clustering coefficient of a protein pair, etc. .Text Mining: Protein–protein interactions can also be predicted using text mining (TM). TM technology could explore protein interactions from full-length papers through titles, abstracts, paragraphs, diagram texts and find co-occurrence of statistical significance between text corpuses . Some meStudies in this category make use of a classification algorithm to integrate interaction-related features. With these available physical and non-physical features, classifiers are trained to distinguish between positive and negative examples. It is a challenge to integrate evidence variants in confidence and coverage to increase PPI prediction coverage and accuracy. The common process of PPI prediction by evidence-combining methods includes several steps.Step 1: Choose appropriate evidence. Evidence must be carefully chosen with content specialized for each different network. Moreover, the following issue must be taken into consideration: Is this evidence a discovery of a global PPI in an unexploited species, or is it a meticulous digging of interaction sites in model species? It should be noted that there is a widespread misconception that “more evidence yields better results”. In a prediction process, blindly incorporating multiple sources of evidence could influence the results and yield other biases .Step 2: Encode protein pairs with evidence. The common encoding process transforms individual or homogeneous evidence into a feature vector representing each pair of proteins. The goal is to convert them to solve the problem of binary classification. These features may represent a particular source of information such as correlations of gene expression, phylogenetic profiles, sequence-based signatures, GO functional annotation and chemical properties. There are many modes to encode evidence sources into a featured vector, to choose statistical standard and data dimensions, and to check the normalization affect or the reliability of different computational predictions ,45.Step 3: Different strategies are adopted to merge classifiers into integrative datasets. Some studies use uniform evidence with a similarly encoded rule in one step. Some studies first train datasets with multiple independent evidence and then cross-validate and integrate multiple independent sets of training results to reduce potential bias. Others use single training or integrating probability score to uncover a novel subset of the whole interactome. Many classifiers have been introduced to predict PPIs including, Artificial Neural Network (ANN), Decision Tree (DT), K-Nearest Neighbor (KNN), Logistic Regression (LR), Naive Bayes (NB), Random Forest (RF), Support Vector Machine (SVM), etc. .However, studies of PPIs are diverse in target species, data sources, demand of accuracy and coverage, which are various in details and processes. In this paper, we are focused on introducing several independent strategies for integrative analysis. Some related studies are listed in Some studies encode evidence sources with uniform rule and use high-dimensional concatenated vectors to present information of uniformed evidence.Case 1: The Multi-Scale Continuous and Discontinuous (MCD) feature method    uses AC Proteins associate with each other through specific binding sites. These protein–protein interaction sites (PPISs) are believed to be good contributors to the recognition of binding residues under specific chemical and physical statuses. Since PPISs mark the central position of interactions and are less efficiently captured by experimental methods, computational approaches have been developed to model the discrimination between interacting and non-interacting sites for prediction of PPIS. Many studies proposed PPI site prediction methods by training with structure-based and sequence-based evidence. Computational approaches for PPI prediction using structural information have gained more attention due to the rapid growth of structural information (in PDB). In this review, the following individual studies are taken as examples.Case 1: A prediction server of PPIS named PSIVER  predictsCase 2: In a study by Dhole et al. (2014), L1-regularized logistic regression (L1-logreg) was developed as a classifier by training evidence based on PSSM, averaged cumulative hydropathy (ACH) and predicted relative solvent accessibility (RSA), which includes evolutionary conservation and chemical/functional information of amino acids .Case 3: The SSWRF method  is introGenerally, cross-validation is employed to evaluate the prediction of performance of the proposed method. Some studies evaluate the performance of prediction by cross-validating datasets from different sources . Some studies randomly divide testing datasets into several equally sized subsets, and each subset is used as a test set ,65,76.The following assessments are taken into account to perform evaluation: Precision, Recall (Sensitivity), Specificity, Overall Prediction Accuracy, Matthews’s Correlation Coefficient (MCC), F-measure, Receiver Operating Characteristic (ROC) and Area Under the ROC Curve (AUC). These assessments compute the accuracy and deviation to evaluate the feasibility and robustness of a PPI prediction method. Some are defined as follows:TP (true positive) is the number of the predicted PPIs found in the GSP; FP  is the number of the predicted PPIs not found in GSP; FN  is the number of PPIs in the GSP that failed to be predicted by the method false positive; TN (true negative) is the number of true non-interacting pairs predicted correctly. MCC, F-measure, ROC and AUC are important assessments. MCC is a measure of the quality of binary classification, which is a correlation coefficient between the observed and predicted results . F-measure is the harmonic mean of Recall and Precision which combines Recall and Precision with balanced weights. In addition, ROC curve and AUC value illustrate performance of a binary classifier system by graphical plot. ROC curve is generated by plotting the TP rate against the , along with the systemic understanding of the essential mechanism of PPIs. We hope that the present work will inspire PPI predictors toward further evaluation and improvements.


Genome-wide association studies for the detection of disease-associated genes or cancer drivers can only identify common variations or driver genes in a cohort of patients. Thus, they cannot discover unique disease-associated mutations or cancer driver genes on a personal basis. Moreover, even when there are such common variations, their significance is unknown. Here, we extend the machine learning based approach ENTPRISE developed for predicting the disease association of missense mutations to frameshift and nonsense mutations. The new approach, ENTPRISE-X, is shown to outperform the state-of-the-art methods VEST-indel and DDIG-in for predicting the disease association of germline frameshift mutations in terms of balanced measure Matthew’s correlation coefficient, MCC, with a MCC of 0.586 for ENTPRISE-X, versus 0.412 by VEST-indel and 0.321 by DDIG-in, respectively. Large scale testing on the ExAC dataset shows ENTPRISE-X has a much lower fraction of 16% of variations classified as disease causing, as compared to VEST-indel’s 26% and DDIG-in’s 65% of predictions as being disease-associated. A web server for ENTPRISE-X is freely available for academic users at  Identifying the causative genetic variations of a disease or cancer is a key step towards diagnosis and cure. A typical patient has thousands of genetic variations involving thousands of genes. It is thus a nontrivial task to pinpoint the single variation or group of variations responsible for the given disease. Traditionally, genome wide association studies (GWAS) are employed for detecting common variations among a cohort of same disease patients , 2. Due Although ENTPRISE has outstanding performance for both Mendelian and cancer driver mutations and requires only single patient exome information, like many other methods, it can only handle missense amino acid substitutions. In practice, based on the v76 version of the COSMIC database , while 6http://cssb2.biology.gatech.edu/entprise-x.In this work, we extend the ENTPRISE approach  that sucftp://ftp.ncbi.nlm.nih.gov/pub/clinvar/tab_delimited/. Two neutral data sets are used. One is EPS6500 from http://evs.gs.washington.edu/EVS/ [Pathogenic data (ClinVar) were downloaded from the NCBI site edu/EVS/ , and theedu/EVS/ . The mutTo compare ENTPRISE with VEST-indel and DDIG-in, we use the same frameshift testing set as employed in the VEST-indel . It conshttp://cancergenome.nih.gov/). The goal of ExAC is to provide a global “reference set” for filtering out harmless genetic variants observed in patients with some disease. However, since the data only excluded childhood diseases, it could still contain variations causing disease in adults. The ExAC variations excluding the Cancer Genome Atlas (TCGA) data are considered to be mostly, but not completely, neutral [For large scale testing of our method in comparison to existing algorithms, we downloaded the ExAC dataset  that exc neutral . A methoA brief summary of training and testing datasets on frameshift and nonsense used for ENTPRISE-X mutations are given in Owing to the success of the features in the ENTPRISE method  for miss(a)The reference amino acid type of the mutated protein position. This requires 20 variables to represent the 20 types of amino acids. For a given residue type, the specific variable is set to the value of 1 and the rest of the variables are set to 0. This feature reflects the fact that mutations of certain types of amino acids are more likely to cause disease than others . The wil(b)α atom is within 12 Å from the mutated position’s Cα atom assuming that the structure of the protein is in the same native conformation. The consideration of this feature is that if a mutation is at or close to a protein-protein interface or protein-ligand binding pocket, the mutation is more likely to be disease associated [VMT [N(a) is the number of residues of type a contacting the mutated position, and ∑N(b) sums over all 20 types of residues.20 variables for the amino acid composition of the residues in contact with the mutated protein position. This reflects the protein’s local structural environment at the mutated position and is defined as the amino acid composition of all residues whose Csociated . Since cted [VMT  as in ENted [VMT . The com(c)N(x) being replaced by the number of residues of type x in the whole domain.20 variables for the composition of the domain that contains the mutated protein position. The reason for considering this feature is that certain domains/proteins are more likely to be disease associated than others . Protein(d)fl is the normalized amino acid frequency profile generated by sequence search method PSI-BLAST [Sequence entropy of the mutated protein position. This characterizes the evolutionary conservation at the mutated position. Our assumption is that evolutionarily conserved residues are either functionally and/or structurally important. The sequence entropy is given bySI-BLAST . <> stanThe above 61 features are the same as in the ENTPRISE method and describe the features of the mutated position. We next introduce five additional new features that describe partial or whole protein functionality. The newly introduced features specific for frameshift & nonsense mutations are:(e)Nr − k)/Nr where Nr is the total number of residues of the protein, k is the sequential position of the mutation from the N-terminus. This feature describes how much of the protein is affected by the mutation that disrupts the structure. Our assumption is that the greater the fraction of the protein that is affected, the more likely it is that the mutation is disease associated.Fraction of the affected protein structure defined as ((f)Essentiality of mutated protein. If the protein is homologous (defined as having protein sequence identity > 35%) to an essential protein in the database of essential genes , a featu(g)Pathogenicity of the affected part of the protein. This feature is derived from the ENTPRISE calculation on the whole protein sequence. It reflects the average effects of the disrupted structure if all the positions have a missense mutation in turn, one position a time. The assumption is that if the probability of a missense mutation being disease associated in the missing dysfunctional part of a protein is larger, a frameshift or nonsense mutation that disrupts this region will be more likely to be disease associated. For any position in the reference protein sequence, there are 19 possible missense mutations. The pathogenicity of a position is defined as the ratio of the number of mutations having an ENTPRISE score ≥ 0.45 divided by 19. (Note that an ENTPRISE score ≥ 0.45 is considered to be pathogenic). The pathogenicity of the affected part of the protein is the average pathogenicity of all positions after the nonsense/frameshift position in the sequence.(h)Disease involvement of the protein. This feature provides the information that a protein is associated with disease when its function is altered. It is obvious a consideration for a mutation on the protein to be disease associated or not when it is mutated. If the protein is related to any disease as defined by the GeneCards database , this fe(i)Number of protein-protein interactions inferred from the HIPPIE protein-protein interaction database . This feTm is a decision tree, ε is the shrinkage factor or learning rate, and Ntree is the number of trees. Two objective function values are adopted: 1 for disease-associated mutations, and 0 for neutral mutations. The learning rate ε is set to 0.005 and the number of trees, Ntree, is set to 2000. These values are purely empirical and were not optimized. As mentioned above, we treat frameshift and nonsense mutations exactly the same. Since in the training sets, the total number of disease-associated mutations  is roughly twice the total number of neutral mutations , to train the model on balanced datasets, we shall train two models: one with a randomly selected half of the disease-associated mutations and all neutral ones, the other with the remaining half of the disease-associated mutations and all neutral ones. This way, both models are trained on balanced datasets. In prediction/testing, a prediction will be given by the average score from the two models. In case of a jackknife or cross-validation test, the same training/testing protocol was applied to the training/testing subsets.We shall employ the machine learning boosted tree regression method that has been employed in many other applications , 40. It Overall, we have tested the 66 features described above. However, not all types of features are useful. We shall employ a feature reduction procedure to eliminate features that have negative impact on performance. To this end, we cluster the whole training proteins into 10 clusters with a 35% protein sequence identity cutoff and employ a leave one cluster out cross validation (LOOCV) test . Then, for each cluster, we use nine clusters for training and the one that is left out for testing. To eliminate features, for each type of feature, we remove it from training and perform a LOOCV test; if the feature’s removal results in a better Matthew’s Correlation Coefficient (MCC) with a default cutoff of 0.5, then the feature is discarded. As a result of this procedure, the 20 features of domain composition (type (c) above) were eliminated, leaving us with total of 46 final features including all five newly introduced features.http://www.cravat.us/CRAVAT) and DDIG-in (http://sparks-lab.org/ddig/) are obtained from their respective webservers. We next tested ENTPRISE-X using the 46 final features for frameshift mutations on the VEST-indel test set  and compbaseline method, with the results also given in Since we have used the feature representing whether the given gene is disease-associated or not as learned from literature (similar to the “PubMed” feature in VEST-indel ), it is To see the importance of new features, we performed five separate tests by removing each of the five new features, one at a time:ENTPRISE-X-noratio—removing the fraction of affected protein part;ENTPRISE-X-noessential—removing the essentiality of protein;ENTPRISE-X-nopathogen—removing the pathogenicity score predicted by ENTPRISE;ENTPRISE-X-nodisease—removing the disease involvement of the protein;ENTPRISE-X-nointeract—removing number of protein-protein interactions.In practice, due to the fact that neutral variations always dominate over true disease-associated variations, even a small false positive rate will result in large number of false positive predictions. We employ the “false discovery rate” (FDR) defined as /. This rate depends not only on the false positive rate, but also on the relative abundance of true positives and true negatives in the evaluated data set. http://sift.jcvi.org/www/SIFT_chr_coords_indels_submit.html. Clearly, due to its larger false positive rate, DDIG-in has a much larger false discovery rate. SIFT-indel is better than DDIG-in and worse than ENTPRISE-X in terms of its MCC, F-score and false positive rate.In order to benchmark ENTPRISE-X for nonsense as well as frameshift mutations, we performed a ten-fold cross-validation of ENTPRISE-X using the complete training dataset. Proteins in the whole training set are clustered into clusters with a 35% sequence identity cutoff. We then randomly partitioned these clusters into ten sets. To test frameshift and nonsense mutations in each set, we use the mutations in the remaining nine sets for training. The test is done in three scenarios: (a) evaluating the test set on frameshift and nonsense mutations together; (b) evaluating the test set only on frameshift mutations, and (c) evaluating the test set only on nonsense mutations. The results are compiled in We next performed large scale testing of ENTPRISE-X, VEST-indel and DDIG-in on the ExAC set. The results are compiled in In order to estimate the false positive rate at an arbitrary prediction score, a smooth fitting function of the false positive rate distribution is needed. This is equivalent to finding the p-value  given an arbitrary prediction score. To derive a p-value measuring the statistical significance of ENTPRISE-X predictions, we utilize a total of 1,025 neutral frameshift mutations from the VEST-indel test set. The distribution of ENTPRISE-X scores is shown in pNtotal, where p is the p-value at the cutoff, Ntotal is the total number of genes evaluated as an approximation of the total number of neutral input genes. Then the false discovery rate can be estimated by pNtotal / Npred, where Npred is number of predicted genes to be disease-associated. The average false discovery rate over the 13 individuals is around 25% with the default cutoff 0.5. The dependence of the average false discovery rate for 13 human exomes on the cutoff is shown in Npred drops faster than the p-value does. Obviously, the FDRs of DDIG-in is too high to be practically useful as its FDR ranges from 0.55 to 0.88.Next, we apply the p-value that is equivalent to theoretical false positive rate at a given cutoff score to real exome data to estimate the false discovery rate of ENTPRISE-X prediction along with DDIG-in method. We obtained data from 13 individuals (private communication) and annotated their frameshift and nonsense variations using ENTPRISE-X. In this work, we have extended our machine learning approach ENTPRISE for predicting disease association of missense mutations to address frameshift and nonsense mutations in ENTPRISE-X. We show that ENTPRISE-X’s performance is superior to the state-of-the-art VEST-indel and DDIG-in methods. As with ENTPRISE, ENTPRISE-X has a much lower false positive rate than its peers while maintaining comparable sensitivity. In practice, for an entire exome, a large false positive rate will result in a method not useful because of the large number of mutations incorrectly identified as being disease-associated. The better performance of ENTPRISE-X demonstrates that selection of proper features and their integration yields better results. For example, the protein structure based local features help ENTPRISE-X to increase its sensitivity while maintaining specificity. As was shown for ENTPRISE , use of S1 File(ZIP)Click here for additional data file.