Genome-scale datasets have been used extensively in model organisms to screen for specific candidates or to predict functions for uncharacterized genes. However, despite the availability of extensive knowledge in model organisms, the planning of genome-scale experiments in poorly studied species is still based on the intuition of experts or heuristic trials. We propose that computational and systematic approaches can be applied to drive the experiment planning process in poorly studied species based on available data and knowledge in closely related model organisms. In this paper, we suggest a computational strategy for recommending genome-scale experiments based on their capability to interrogate diverse biological processes to enable protein function assignment. To this end, we use the data-rich functional genomics compendium of the model organism to quantify the accuracy of each dataset in predicting each specific biological process and the overlap in such coverage between different datasets. Our approach uses an optimized combination of these quantifications to recommend an ordered list of experiments for accurately annotating most proteins in the poorly studied related organisms to most biological processes, as well as a set of experiments that target each specific biological process. The effectiveness of this experiment- planning system is demonstrated for two related yeast species: the model organism Saccharomyces cerevisiae and the comparatively poorly studied Saccharomyces bayanus. Our system recommended a set of S. bayanus experiments based on an S. cerevisiae microarray data compendium. In silico evaluations estimate that less than 10% of the experiments could achieve similar functional coverage to the whole microarray compendium. This estimation was confirmed by performing the recommended experiments in S. bayanus, therefore significantly reducing the labor devoted to characterize the poorly studied genome. This experiment-planning framework could readily be adapted to the design of other types of large-scale experiments as well as other groups of organisms.
To understand the functions of gene products and the interplay between them, significant effort has been spent on performing and analyzing genome-wide expression profiling experiments. Compared to traditional experiments that study protein functions on the single-gene scale, modern high-throughput techniques efficiently characterize expression of the whole genome. One of the most popular techniques is the gene expression microarray, with thousands of expression profiles available for the commonly-studied species. For example, in the Gene Expression Omnibus repository, over 150 datasets comprised of 2400 conditions were available for Saccharomyces cerevisiae as of 2007 [1], with data continuing to appear at an enormous rate. These large scale data have been used to accurately predict gene functions [2]–[4], protein-protein physical interactions [5] and functional relationships for yeast [6] and other model organisms [7], [8], as well as human [9]. On the other hand, new genomes are being sequenced at an exponentially growing rate [10], with more than 2,200 genome sequencing projects completed or ongoing to date. These sequencing efforts accelerate our understanding on diverse species, but identifying the gene sequence is not sufficient to define the biological role of its product, and functional annotation of these genomes lags far behind sequencing. Many of these newly sequenced species are amenable to further experimental study in the lab. The lack of such functional annotation is partly due to the fact that experiments in poorly-studied species are still mainly based on expertise experience or heuristic trials, rather than using a systematic approach based on comparative functional genomics. Although the heuristic approach is useful in directing specific experiments, it is often far from optimal for a systematic functional annotation of all proteins (or at least the majority) in a newly-sequenced genome. Furthermore, experiments that target a specific biological process may also provide accurate functional signal for additional pathways. For example, hyperosmotic shock datasets not only elucidate stress responses, these experiments provide information on regulation of DNA replication initiation because of the cell cycle arrest that occurs under this condition. This functional coverage information is often implicit. We propose here that systematic analysis and quantification of this information in a well-studied species could be the foundation of a systematic experimental design scheme in related poorly-studied species. In recent years, computationally directed experiments have been applied to different fields. The most prominent application domain is the prediction of protein function with follow-up in vivo tests. For example, the prediction results of an ensemble of three algorithms have been used to direct experiments to find genes required for mitochondrial biogenesis [2]. Experiments that detect physical or genetic interactions have also been directed through computationally integrating quantitative genetic interactions and TAP-MS data [11]. The recent development of the Robot Scientist “Adam” marks the state-of-art pipeline of computationally directed studies, which generate hypotheses and experimentally test them [12]. However, these computational efforts have not been extended to direct experiments in a poorly studied species for functional annotation based on existing knowledge in a well characterized species. In this paper, we developed a systematic approach to recommend experiments for functional annotation of S. bayanus (a poorly-studied yeast species) based on the wealth of available gene expression data in the model organism S. cerevisiae (baker' s yeast). The system identifies experiments that are informative of genes participating in each function and then uses an optimized combination of the predictive power of each experimental treatment with the coverage overlap between treatments to rank a list of experiments that are able to predict the maximum spectrum of biological processes using the minimum number of arrays. Based on functional analysis we estimated that experimentalists can achieve similar functional coverage in the same or related species with less than 10% of the arrays. We further carried out these recommended experiments in S. bayanus and the resulting arrays achieved similar functional coverage to all the existing S. cerevisiae arrays with a 10 fold reduction in labor. Our approach is readily adaptable to other sequencing-based measurements of expression and to measurements of protein and metabolite levels, and is potentially applicable to other large-scale experiment types.
Genes play individualized roles in the cell and one gene product can be involved in several different biological processes. Hence for a given experimental treatment or genetic perturbation, we would expect that genes of some functional groups respond more strongly than others. Thus, different datasets are more or less informative of particular processes, including processes that are not necessarily the direct target of the experiment' s design. This information, i. e. , the informativeness of a dataset when used to predict certain biological process, could be used to select experimental treatments to target certain biological process. However, this information is often implicit and must be quantified statistically. The Brem et al. , 2005 dataset, for example, represents the progeny from an outcross between two strains, executed with the goal of using the resulting expression profiles as phenotypic traits in genetic mapping. It performs well in predicting a wide range of biological processes, including terms not directly related to genetic crossing such as electron transport and sulfur metabolic process (Figure 2A). Our method can rank the candidate experiments according to their informativeness, or how much information each experiment provides on telling whether a gene is related to a certain biological process. We propose to access such informativeness by assessing the predictive performance of a machine learner that uses the data in the experiment under consideration to predict proteins involved in that process. The intuition is that the machine learner will achieve higher accuracy if the training data provides more information about the specific process (at least in terms of functional annotation), and thus this experiment is likely to be highly effective in interrogating this functional group in the evolutionarily related, less well-studied organism. To do this, we use Support Vector Machine (SVM), a state-of-the-art machine learning algorithm [4], [13], though the method can be used with any machine learning approach. We used bootstrap cross-validation [14] to characterize the performance of our S. cerevisiae microarray data collection in predicting Gene Ontology (GO) biological process (BP) terms that are annotated with 10 to 500 genes in S. cerevisiae (Figure 2B). We employed several different measurements, including AUC (area under the receiver operating characteristic curve), which characterizes the overall ability of a dataset resulting from an experimental treatment to predict function for proteins in a certain biological process; and precision (accuracy) at 1 percent, 10 percent, 50 percent and 80 percent recall, which focuses more on discovering new genes. It is important to note that these measures assess the ability of experimental treatments to interrogate participants in specific biological processes. Although these measures are influenced by the quality of data as well, we found that they are sensitive to different experimental treatments, because these measures are highly correlated across GO terms in the same treatment between S. cerevisiae and S. bayanus (see below). If a different property of experimental design is desired, such as ability to assess regulatory interactions or binding partners, this particular characteristic can be optimized through the same evaluation methodology. With this information, experimental treatments can be ranked by how effective they are in predicting a given biological process. We make this method available to the scientific community through our interactive website, where the users can search for the most relevant experiment (s) for the biological process of interest. An important phenomenon we observed through the function-dataset informativeness analysis is that some biological processes are well represented in many datasets, while others are only reflected in a small fraction of datasets (Figure 2B). For example, signals for the group of ribosome- and translation-related biological processes are present in the majority of the S. cerevisiae expression datasets. Metabolism, mitosis, carbohydrate, and amino acid-related terms are also well represented by many datasets. However, most of the biological processes, for example, transcription, cell cycle, stress and transport-related terms, are only detectable in particular datasets. Datasets that have the best overall performance in our analysis showed this same range of variability in the terms that they could cover (Figure 2A). For example, transcription from RNA polymerase I promoter (GO: 0006360) and electron transport (GO: 0006118) are reflected in most of the top 10 datasets. On the other hand, response to toxin (GO: 0009636) has strong signal only in the Chitikila02 dataset [15] and the Brem02 set1 data [16], and function of proteins in peroxisome organization and biogenesis (GO: 0007031) is well represented only in the Boer05 dataset, which profiles the expression pattern of a leu3 mutant strain [17]. The phenomenon of different functional sensitivity of different expression datasets is consistent with previous studies using different machine leaning methods to estimate informativeness [1], [18]. Therefore, both the accuracy and the redundancy between datasets should be considered for planning experiments. To quantify the overlap in information between datasets, we calculated pair-wise conditional mutual information (CMI). This CMI analysis is highly informative of functional redundancies and therefore is critical to our experiment planning system as shown below. Intuitively, CMI quantifies the overlapping information between datasets in predicting functions. The CMI analysis effectively identified datasets that result from similar experimental treatments (Figure 3). For example, the Brem et al. 2002, Brem et al. , 2005 and Yvert et al. , 2003 datasets have very high mutual information, and in fact are overlapping subsets of the same type of experiment [16], [19], [20]. Less obvious is identification of datasets that are different in their treatments but essentially targeting the same biological processes, for example, the Tai et al. 2005 and Boer et al. , 2005 datasets have very high mutual information, although the former is a nutrient limitation treatment, while the latter studies the expression in leucine auxotroph mutants. This overlap is likely because both experiments were performed in chemostat culture which may induce similar responses in yeasts. [17], [21]. Less obvious relationships identified by the CMI analysis are those among datasets that do not directly share the same or similar experiment treatments but still contain high mutual information. For example, a set of cell cycle-related experiments are clustered together by their high mutual information, including Spellman98 cyclin [22] and the two technical replicates from Iyer, et al. 2001 [23] (Figure 3). These experiments altered key cell cycle regulators including the cyclins and the transcriptional regulator MBF/SBF. Although these experiments do not analyze a time course of synchronized cells, they measure the transcriptional response to these key regulators and so are very informative about gene expression regulation in the cell cycle. Similarly, stress-response experiments Rutherford01 [24], Fernandes04 [25] and Gasch00 HOresponse [26] are clustered together, despite the fact that they represent diverse experimental treatments, including iron homeostasis, hydrostatic pressure and hydrogen peroxide. The CMI quantification allows us to statistically identify redundant datasets and avoid such redundancies in experimental recommendations for the less-studied species. Large-scale microarray experiments designed for characterization of the whole genome tend to be of very high accuracy but often include a large number of arrays. For example, among the top performing datasets in predicting gene functions are the Brem et al. , 2005 experiments designed to detect segregation of expression patterns in an outcross [19] and the Hughes et al. , 2000 experiments designed for genome-wide mutation analysis [27]. They include 130 and 300 arrays respectively. Although both datasets are among the top recommended experiments, intensive labor is required to repeat these experiments in a new species. We attempted to minimize the number of arrays in these large-scale microarray experiments while retaining their function prediction capability. Through randomized selection of subsets of datasets, we could estimate the accuracy versus number of arrays included in the subsets. Surprisingly, a rather small fraction of the arrays (25–40) can achieve very similar performance in overall function prediction to the entire datasets (Figure 4A and 4B). Additional arrays only add to marginal improvement in performance. Therefore only a small proportion of the arrays of these very large scale experiments are required for our experiment planning system. Of course, this does not mean that these additional data are not biologically relevant, in fact, for genetic linkage experiments the entire dataset is informative. Rather, a subset of these experiments of optimized size can be used for this specific goal of functional annotation; if a different biological question is important, the size of the appropriate subset or entire dataset can be estimated specifically for that question (e. g. regulatory relationship prediction). Biological processes differ in their sensitivity to the number of arrays required for reasonable assessment of each process (Figure 4C). For example, we could accurately predict the biological process ‘ribosome biogenesis and assembly’ (GO: 0007046) with only about 15 arrays from the Brem et al. , 2005 [19] dataset. Additional arrays add no improvement in predicting this term. Predictions for many of the other biological processes have different sensitivity to the number of arrays. For example, ‘cellular lipid catabolic process’ (GO: 00044242) and ‘histidine metabolic process’ (GO: 0006547) could be captured with relatively small number of arrays. On the contrary, ‘co-factor biosynthetic process’ (GO: 0051188) and ‘G1 phase of mitotic cell cycle’ (GO: 0000080) require a large number of arrays to be well characterized. There are also terms like ‘cell growth’ (GO: 0016049), which cannot be captured even using the maximum number of array we tested. For this case, increasing the number of arrays is meaningless. The requirement for the number of arrays on a per-biological process basis was calculated and provided though our online searchable system. For the general recommendation process, we define the minimum number of arrays based on the average AUC across all GO functional SLIM terms [28] and therefore guarantee the overall performance. Our experiment planning system flexibly leverages both the accuracy of each experimental treatment in capturing different functions and the overlap in information between them. We determined the overall accuracy of a dataset by its average AUC across GO functional SLIM terms (as listed in Figure 2B), which are terms curated by biologists to represent functions specific enough for experimental characterization, but which do not have any parent terms satisfying this criterion [28]. The redundancy between datasets was quantified by pair-wise conditional mutual information as described above. A trade-off factor (α) between accuracy and redundancy, where a higher value means more weight on accuracy and vice versa, allows flexibility in the experiment design process. In our study, α was optimized through cross-validation; in the web-interface, the users can optimize this factor according to their specific preferences. Recommendation of datasets for functional annotation requires leverage between data precision and redundancy. We applied bootstrap cross-validation [14] to evaluate the ability of the selected set of data in predicting different functions (Figure 5A). We found that we could optimize the function prediction capability of the top 10 datasets by a trade-off factor α = 0. 9 (Figure 5B). Function prediction by SVM maps the original data into feature space, which relies on the accuracy of the datasets and penalized by the redundancy in information between them. Thus an adjustable trade-off factor is necessary and provides flexibility for the experiment recommendation process. We estimated how effective our approach is in reducing the number of experiments to characterize the overall functionality of the S. bayanus proteins based on the S. cerevisiae gene expression compendium. We integrated the information from our analysis of the minimum number of arrays in the very-large-scale microarray experiments, and the information from our accuracy and redundancy analysis. This gives us an ordered list of experiments in S. cerevisiae, including the number of microarrays that need to be completed in the very large experiments. We experimentally generated an S. bayanus expression data compendium based on the experiments proposed by our system (GEO accession GSE16544). The list of highly informative experiments (250 arrays) included cell cycle progression, meiosis, diauxic shift, nutrient limitation, stress conditions, and outcross progeny. The S. bayanus data we generated are highly informative for diverse biological processes (Figure 6). As no S. bayanus functional annotation exists, to assess the coverage of S. bayanus experimental data, we use the gene ontology annotations from S. cerevisiae orthologs of the S. bayanus genes. This is a conservative measurement because not all orthologs are conserved in function, but as most genes are likely to be conserved at least on the level of functional annotations, this measure should provide a reasonable lower bound on performance. We used bootstrapping and a linear SVM classifier to estimate the accuracy of the expression data in functional annotation of GO functional SLIM terms [28]. Interestingly, the ‘informativeness’ is highly similar between matched experiments between S. bayanus and S. cerevisiae, further supporting the validity of our approach (Figure 6). We find that our dataset of 250 S. bayanus arrays predicts gene function with an average AUC of 0. 74, which is very close to the AUC (0. 75) of predictions made with a set of 2547 S. cerevisiae arrays (Figure 7A). This is very similar to the theoretical analysis, where we estimated that less than 10% of the arrays of the total 2569 arrays available in the S. cerevisiae repository as of 2007 would achieve similar performance. Such performance requires selection of specific experimental treatments – computational simulation shows that selection of random subsets of experiments from the repository substantially decreases overall accuracy (Figure 7). This indicates that the experiment planning scheme can significantly reduce the human and technical resources necessary to characterize a newly sequenced species by providing effective guidance for the most informative sets of experiments for functional annotation based on related model organisms or other well-studied species. To further validate our approach, we also compared the performance of individually matched dataset pairs between S. bayanus and S. cerevisiae for all GO terms with more than 30 genes annotated to each. The correlation and the similar range of AUC between the two sets (Figure 6) further supports the recommendation of experiments based on closely-related model organisms. Signals for most of the biological processes are very well represented in our S. bayanus expression compendium, to an extent comparable to the theoretical maximum in S. cerevisiae (Figure 7A). However, a small set of biological processes, including aging, ion homeostasis, hyperosmotic shock, auxotroph starvation, and alternative carbon sources were not well-captured by the experiments, with an AUC less than 0. 65. Thus, we used the system to suggest a second round of experiments targeting these particular processes again based on the accuracy and redundancy analysis. The second round experiments included 54 arrays covering 11 biological treatments carried out in S. bayanus (Text S1). On average we gained a 0. 006 (2. 5% over random) improvement in AUC over all GO biological process terms with 10 to 300 genes annotated to each. This minor improvement indicates the saturation of the ability to predict functions based on expression data and transfer of annotation through homology. However, we observed an average of 0. 012 (10% over random) improvement in AUC in the targeted GO categories, which were poorly predicted in the first round. Five out of seven of the targeted GO categories achieved significantly improved AUC (Figure 7B). Therefore the second round recommends experiments that provide relatively orthogonal information to the first round, indicating the ability of our experimental planning system to extract the information contained in the existing data and to direct further specific experiments. The top improved terms during the second round (Text S1) are well-explained by the additional datasets. For example, we observed a 52% improvement (0. 071) in AUC for “double-strand break repair via nonhomologous end joining” (GO: 0006303) and a 56% improvement (0. 111) for “DNA catabolic process, endonucleolytic), most likely due to the addition of MMS and zeocin DNA damage datasets. Starvation experiments might explain the 177% improvement in “nitrogen utilization” (GO: 0019740) and “histidine biosynthetic process” (GO: 0000105). Experiments of alternating carbon sources, particularly glycerol, lead to a 35% improvement in our prediction power on “hexose biosynthetic process. ” These observations suggest that our evaluation scheme is well in accordance to the established knowledge in this field.
In this paper, we propose that existing genome-scale data in model organisms could facilitate the planning of experiment treatment in poorly-studied species. We demonstrate the feasibility of this approach by planning microarray experiments for a relatively poorly studied yeast species S. bayanus based on an available gene expression data repository for model organism S. cerevisiae. In this framework, we recommend an ordered list of experiments targeting the overall functional annotation of S. bayanus proteins as well as experiments targeting specific biological processes. We also detected the minimum number of arrays to achieve satisfactory performance for some very large-scale microarray experiments. Our framework results in a substantial reduction in the resources we need to characterize the functions of a poorly-studied genome. Our work represents the first attempt for large-scale experiment design of a relatively poorly studied organism based on available data in a related organism, rather than existing functional genomics data in that species, which in the case of S. bayanus does not exist. This method is complementary to designing experiments based on expert knowledge or intuition, which is irreplaceable when targeting in-depth aspects of biology but is likely not to be optimal for generating a large compendium for functional annotation. Of course, after such initial functional annotation, carefully designed experiments will be necessary to ascertain specific relationships within functions and to further explore the functional space. This task should be facilitated greatly by the availability of the initial functional annotations generated based on the experiment design system and the resulting data compendium. The experiment design system is adaptable, and can be extended to other related species groups. The current analysis is restricted to S. cerevisiae and S. bayanus, which are separated from each other by 20 million years. There are thousands of genome projects finished or ongoing, and several model organisms with large amount of genome-scale data. Analysis of these genome-scale data could be used to design experiments for the poorly-studied related species. The 20 millions years distance between S. cerevisiae and S. bayanus is comparable to the sequence divergence between human and mouse [29]. On the other hand, comparative genomics often focuses on less diverged groups, for example, the Drosophila species subgroup or the other sensu stricto yeast species. The extendibility to further related groups, however, remains to be validated by future investigation in the intelligent experiment design field. Nevertheless, currently, GO annotations are often transferred between species of vast distance based on sequence alone. Experiment recommendations such as those described here provide a complementary approach to the current annotation scheme. Indeed, expression patterns of the majority of genes are conserved across species over vast distances (e. g. from human to mouse, and from Candida to S. cerevisiae) [30], [31], suggesting the likely value of applying such experimental design methods across further distances. Applying the experiment design system could not only facilitate the annotation of these genomes but also provide invaluable resources for cross-species expression comparison. In addition, our current study focuses on the prediction of biological processes. The same approach could be extended to molecular function, cellular component and pathway predictions, as well as predictions of particular types of relationships among proteins (e. g. physical, regulatory interactions, etc). The basic framework could also be extended to other data types, more complex data, and higher organisms. Our current work focuses on microarray data because it is currently the most abundant functional genomics data source for most organisms. This methodology can be readily applied to sequencing-based measurements of expression and to measurements of protein and metabolite level. We expect that as more data of these types become available, applications of this and similar methods will become more common. An extension of this methodology could be developed for other types of large-scale experimental methods, including yeast two hybrid, affinity precipitation, chromatin IP datasets, etc. These data, like microarrays, are often readily available in diverse well-characterized model organisms. Furthermore, data from several model organisms could be integrated together so that more confident experiment planning system could be established. Novel methodology that integrates both sequence data and information from these types of large-scale datasets will ultimately allow us to more accurately and quickly understand the differences and similarities of functions between species.
We collected an extensive compendium of S. cerevisiae microarray datasets from diverse sources [32]–[36]. This compendium includes 125 datasets with 2569 arrays. A complete list of publications for these datasets is available on the website supporting this publication http: //exprecommender. princeton. edu. To allow reasonable comparison between datasets, we carried out the following normalization steps. For each raw dataset, genes that are represented in less than half of the arrays were removed, and missing values were inserted using KNNimpute [37] with K = 10, Euclidean distance. Technical replicates are averaged, resulting in datasets with each gene followed by a vector representing its expression values in a series of arrays. Based on the analysis of experiment performance, we observed that some of the biological processes have strong signal in a wide range of experiments, but others are only sensitive to one or a very limited set of experiments. Furthermore, two very accurate datasets may interrogate a highly overlapping set of processes, thus providing largely redundant information in terms of functional annotation. Therefore, when designing a set of microarray experiments for global function profiling, we should not only consider the accuracy of each experiment in predicting function, but also weigh the overlap in information between datasets. We found using a trade-off factor between the two could allow flexibility in experimental design for different applications (see Results). In the following section, we will describe our measurements of accuracy and redundancy and the combination of the two. To leverage the trade-off between accuracy and mutual information, we introduced a trade-off factor α, which linearly combines the two factors: Where P (X) is the overall precision of the dataset X in consideration, k is the number of experiment selected before X. This approach allows iterative selection of datasets. Therefore it is suitable for selecting datasets in a species with several experiments available already, which is more likely to be true in the real-world situations. We provide this feature on our website, allowing the user to select the experiments already performed and the desired tradeoff value, so that our system can recommend additional experiments. We identified GO functional SLIM biological processes that are weakly represented in the first round datasets (below 0. 65 in AUC and with a minimum of 30 genes annotated to it in S. cerevisiae). This list included 7 biological processes in total (Figure 6). Experimental treatments that best cover each of these biological processes were ranked by accuracy and carried out in S. bayanus in the second round. Based on the recommendations, we carried out the experiments in S. bayanus. The resulting array data are accessible from the GEO database with accession ID GSE16544. For both first round and second round datasets, we borrowed annotations from S. cerevisiae through orthology and applied bootstrap cross-validation to estimate the error rates as describe in S. cerevisiae accuracy estimation. We developed a website to facilitate exploration of the functional analysis of the microarray data and our recommendation of datasets for yeast species related to S. cerevisiae. This website supports searchable recommendations for datasets targeting specific biological processes and ones targeting the entire functionality of the genome given existing datasets in poorly-studied species. In addition, the number of arrays (on a per biological process basis) required for the large-scale datasets is also searchable by the users. This website is publicly available at http: //exprecommender. princeton. edu.
Microarray expression experiments allow fast functional profiling of an organism' s entire genome and significant efforts are devoted to analyzing the resulting data. Available genome sequences are also increasing quickly. However, it is unexplored how to use available functional genomics data to direct large-scale experiments in newly sequenced but poorly studied species. In this paper, we propose a strategy to systematically plan experimental treatments in the poorly studied species based on their model organism relatives. We consider both the accuracy of the datasets in capturing different biological processes and the redundancy between datasets. Quantifying the above information allows us to recommend a list of experimental treatments. We demonstrate the efficacy of this approach by designing, performing and evaluating S. bayanus microarray experiments using an available S. cerevisiae data repository. We show that this systematic planning process could reduce the labor in doing microarray experiments by 10 fold and achieve similar functional coverage.
genetics and genomics/genomics
genetics and genomics/gene expression
genetics and genomics/functional genomics
Systematic Planning of Genome-Scale Experiments in Poorly Studied Species