Once the manifold has been constructed the raw microarray data is projected onto it and clustering and classification can take place. In contrast to earlier fusion based methods, the prior knowledge from the KEGG databases is not used in, and does not bias the classification process—it merely acts as an aid to find the best space in which to search the data. In our experiments we have found that using our new manifold method gives better classification results than using either PCA or conventional Isomap.Microarray databases are a large source of genetic data, which, upon proper analysis, could enhance our understanding of biology and medicine. Many microarray experiments have been designed to investigate the genetic mechanisms of cancer, and analytical approaches have been applied in order to classify different types of cancer or distinguish between cancerous and non-cancerous tissue. However, microarrays are high-dimensional datasets with high levels of noise and this causes problems when using machine learning methods. A popular approach to this problem is to search for a set of features that will simplify the structure and to some degree remove the noise from the data. The most widely used approach to feature extraction is principal component analysis (PCA) which assumes a multivariate Gaussian model of the data. More recently, non-linear methods have been investigated. Among these, manifold learning algorithms, for example Isomap, aim to project the data from a higher dimensional space onto a lower dimension one. We have proposed  In machine learning as the dimensionality of the data rises, the amount of data required to provide a reliable analysis grows exponentially. Richard E. Bellman referred to this phenomenon as the “curse of dimensionality” when considering problems in dynamic optimisation In the last ten years, machine learning techniques have been investigated in microarray data analysis. Several approaches have been tried in order to: (i) distinguish between cancerous and non-cancerous samples; (ii) classify different types of cancer and (iii) to identify subtypes of cancer that may progress aggressively. All these investigations are seeking to generate biologically meaningful interpretations of complex datasets that are sufficiently interesting to drive follow-up experimentation.Many methods have been implemented for extracting only the important information from the microarrays thus reducing their size. The simplest is feature selection, in which the number of gene probes in an experiment is reduced by selecting only the most significant according to some criterion such as high levels of activity. A number of investigations of this kind have been used to examine breast cancer Feature extraction methods have also been widely explored. The most widely used method is principal component analysis (PCA) and many variations of it have been applied as a way of reducing the dimensionality of the data in microarrays k-nearest points are found by constructing a minimum spanning tree using an ε-radius hypersphere. Isomap has been tried on microarray data with some very good results An approach to dimensionality reduction that can take into account potential non-linearity is based on the assumption that the data (genes of interest) lie on an embedded non-linear manifold which has lower dimension than the raw data space and lies within it. Algorithms based on manifold learning work well when the high dimensionality of the data sets is artificially high; although each point is defined by thousands of variables, it can be accurately characterised by just a few. Samples are drawn from a low-dimensional manifold that is embedded in a high-dimensional space KEGG pathways database. KEGG (Kyoto Encyclopedia of Genes and Genomes) We have been investigating a novel way of constructing the manifold which makes use of prior knowledge. Prior knowledge has previously been used in microarray studies Our method of building the manifold is as follows. In common with all previous methods we first build an affinity matrix from a set of microarrays. A gene-by-gene affinity matrix is a square matrix whose dimension is the same as the number of gene probes in the microarray data. The matrix is symmetric and each entry is a similarity measure (for example covariance) of the expression levels of the two genes that index it. We then fuse information from the KEGG pathways increasing the values in the affinity matrix for gene pairs with a strong relationship in KEGG. Next we apply a conventional manifold learning method to the fused affinity matrix to find the manifold. Having found the manifold of the gene probes we then project the raw data onto it so we can carry out classification experiments. This means that the KEGG pathway data is only involved in building the manifold. In contrast to previous data fusion approaches a priori manifold learning against the original Isomap algorithm and PCA. We used the Dunn Index which is a metric for evaluating the density and the structure of the clusters in the embedding. We also employed the k-Nearest Neighbours (k-NN), Support Vector Machines (SVMs) and Linear Discriminant Analysis (LDA) classifiers with 10-fold cross validation to test the accuracy of the model. Nine different types of cancer were used to evaluate the methods and we used a smaller dataset to visualise the results. The datasets are described in To verify the effectiveness of our method we tested The first metric we used to evaluate the density of the clusters is the Dunn Index. The Dunn Index is a way to measure the difference of the objects in a cluster with the mean of the same cluster. The higher the index value the better the state of the clusters. For our experiments the Dunn Index can indicate how well the resulting embedding separates the samples according to their label, since it uses the labels of each sample as the cluster indicators. In practice manifold learning does not create any clusters but if the embedding is done in a successful way many points will end up being next to each other, since the embedding is just a mapping from the original dataset to a different space. We ran this experiment for different dimensional embeddings (2 to 50 components) as the components we will end up using in the embedding is heavily dependent on the complexity of the data. We applied it on both sample-by-sample affinity matrices, shown in a priori manifold learning creates denser clusters in all cases except colon, uterine and lung cancer. From the graph induced from the colon dataset for both sample-by-sample and gene-by-gene experiments and the uterine dataset in the gene-by-gene experiments we can see that a priori manifold learning outperforms PCA and Isomap for embeddings with lower dimensions. Our goal is to create an embedding with as few components possible to represent the original high-dimensional data. For the lung dataset in the sample-by-sample experiments we need more samples to create a more accurate embedding.The results for the Dunn Index in sample-by-sample experiments in k-NN and LDA classifiers with ten fold cross validation to measure the accuracy of our method. In order to get the values we used the trapezoidal rule which approximates the definite integral of the plots. Results are shown in k-NN. The corresponding results for LDA is shown in a priori manifold learning outperforms the rest of the methods. It should be noted that the variance is small enough so we can compare the individual accuracies of the experiments safely. The variances for the k-NN classifier for the gene-by-gene experiments are shown in k-NN gene-by-gene experiments the graphs are shown in a priori manifold learning outperforms PCA and Isomap for 6 out of 9 datasets. These are the same datasets for both sample-by-sample and gene-by-gene experiments. For the datasets that a priori manifold learning does not perform as good as the other two methods the problem might lie to the lack of a sufficient number of pathways in the KEGG database.To evaluate the accuracy of the embeddings we used the a priori manifold learning since it needs the genes for constructing the affinity matrix therefore a priori manifold learning only operates on a gene-by-gene affinity matrix. For the GEMLeR dataset, the sample-by-sample affinity matrix has dimensions 1545 by 1545. This is the number of microarrays in the dataset. The gene-by-gene affinity matrix is 10935 by 10935 which is the number of gene probes in each microarray.The sample-by-sample affinity matrix cannot be computed directly using a priori manifold learning.In addition we created the Receiver Operating Characteristic (ROC) curves to illustrate the ratio of true positives and false positive results. We have used three different classification methods for illustrating the effectiveness of k-NN classifier the results we got for the ROC curves agree with the 10-fold cross validation results. A priori manifold learning performs better in all the gene-by-gene experiments as shown in For the a priori manifold learning performs better in 7 out of 9 datasets for the gene-by-gene experiments  dataset for leukaemia to demonstrate how the different cells were clustered. We have chosen the ALL dataset as it is simple enough to visualise and has been used before in k-nearest points and creates the neighbouring graph by utilising KEGG pathway information. The KEGG data is a form of prior knowledge which is better curated and more reliable than the microarray data. Once the manifold has been constructed the raw microarray data is projected onto it and clustering and classification can take place. We called this method a priori manifold learning and we compared it to the original Isomap and the PCA algorithms, since PCA is the most commonly used method for dimensionality reduction. By incorporating prior knowledge we argue that we are able to have less variable and more biologically significant clusters. Information taken from KEGG pathways is a way of decreasing the noise in the microarray experiments. We produced results using ten different datasets of cancer data, where we tried to distinguish among different types of cancers. Nine out of ten datasets are considered to be high dimensional.Conventional manifold learning algorithms, such as Isomap, aim to project the microarray data to a lower dimensional space in which functionally different clusters are better separated. The lower dimensional space is a manifold (hypersurface) contained in the original data space and found from the local distribution of the data. A large representative dataset is used to compute the manifold. Our method provides a way of improving the way Isomap finds the A priori manifold learning produces more compact, well - separated clusters when compared with PCA and the original Isomap. In some cases a priori manifold learning performs better only for embeddings with a smaller number of components which is still useful since we are more interested in embeddings with a lower number of dimensions. There were also cases were the samples and the KEGG signatures were not enough for a priori manifold learning to perform better than PCA and Isomap.The results were similar across the different datasets. In the first set of results, we showed, using the Dunn Index, that our algorithm is able to create denser clusters with objects that lie closer to the mean of the cluster with a small variance. Incorporating prior knowledge using KEGG pathways is not only limited to cancer data but it can be applied to a number of diseases that have KEGG signatures. This, along with the fact that the method does not require any other information, makes it easy to adapt to any kind of biological problem. Other studies A priori manifold learning can only be computed using the gene-by-gene affinity matrix. This is because the prior knowledge extracted from the KEGG data base is in the form of similarities between gene pairs. Our results show that both PCA and Isomap perform better using the sample-by-sample affinity matrix. A priori manifold learning on average performs better in all cases when using the LDA and SVM classifiers. It does not do so well in classification experiments where PCA and Isomap are computed using the sample-by-sample affinity matrix using the k-NN classifier. In this case there is no significant difference between the three formulations. A possible reason for this is that both LDA and SVM classifiers create a model of the underlying classes, but k-NN is a parametric method which depends on the local distribution of the data, and consequently may be more susceptible to noise.When performing cross validation experiments both PCA and Isomap features can be computed using either the gene-by-gene affinity matrix or the sample-by-sample affinity matrix. The latter is a square matrix with dimension equal to the number of microarrays used in the experiment. Each entry represents the similarity (or distance) between the corresponding pair of microarrays. It is considerably smaller than the gene-gene matrix and consequently more robust to noise. a priori manifold learning produces better formed clusters than either PCA or Isomap, and also performs better in classification experiments using either SVM or LDA methods. One of the drawbacks of the method is that it has only been formulated using the gene-by-gene affinity matrix, and this makes it more susceptible to noise than methods that can be computed directly on a sample-by-sample affinity matrix. Consequently a current direction of further work is to investigate methods whereby prior knowledge can be used in a sample-by-sample formulation. We are also investigating ways in which we can make the prior knowledge more specific to the particular type of cancer under investigation. By doing so we hope to make inroads into the harder problem of recognising subtypes of a cancer that will progress aggressively.Overall we see that sklearnIn this paper we present a method which incorporates manifold learning along with a novel approach for estimating the neighbourhood graph. The cluster validation and accuracy measures, along with the original Isomap algorithm and PCA were implemented using the Kernel PCA. The Isomap algorithm works as follows:k nearest points (k - Isomap) or the closest points based on distance (ϵ-Isomap)Determine the neighbours: For all points in a fixed radius, find the k nearest points away with the edge length set to their Euclidean distance.Construct the neighbourhood graph: Points are connected to each other if they are Dijkstra or Floyd-Warshall) to construct the matrix of pairwise geodesic distances between different points.Find the shortest path between all the nodes on the graph using a graph algorithm There is large number of samples for different tumour typesAdditional information is available for combined genotype-phenotype studiesAcute lymphoblastic leukaemia (ALL) is a form of leukaemia characterised by excess lymphoblasts. There are two main types of acute leukaemia: T-cell ALL and B-cell ALL. T-Cell acute leukaemia is aggressive and progresses quickly but is more common in older children and teenagers. B-Cell ALL leukaemia Information on the contents of the datasets is shown in a priori manifold learning and Isomap are non-linear methods and they need more time to fit the data.Our algorithm takes approximately 45 minutes for each embedding which is the same as the original Isomap algorithm. PCA is however a lot faster since is only takes ten minutes to fit the data and create an embedding. This is because PCA is linear while Figure S1Accuracy with variance for all nine datasets for gene-by-gene affinity matrices k-Nearest Neighbours. Accuracy with variance calculated for a priori manifold learning (blue) compared with PCA (Green) and Isomap (Red) computed using the gene-by-gene affinity matrix and the k-NN classifier.(TIFF)Click here for additional data file.Figure S2Accuracy with variance for all nine datasets for sample-by-sample affinity matrices using k-Nearest Neighbours. Accuracy with variance calculated for a priori manifold learning (blue) compared with PCA (Green) and Isomap (Red) computed using the sample-by-sample affinity matrix and the k-NN classifier.(TIFF)Click here for additional data file.Figure S3Accuracy with variance for all nine datasets for gene-by-gene affinity matrices using Linear Discriminant Analysis. Accuracy with variance calculated for a priori manifold learning (blue) compared with PCA (Green) and Isomap (Red) computed using the gene-by-gene affinity matrix and the LDA classifier.(TIFF)Click here for additional data file.Figure S4Accuracy with variance for all nine datasets for sample-by-sample affinity matrices using Linear Discriminant Analysis. Accuracy with variance calculated for a priori manifold learning (blue) compared with PCA (Green) and Isomap (Red) computed using the sample-by-sample affinity matrix and the LDA classifier.(TIFF)Click here for additional data file.Figure S5Endometrium Cancer. How the (TIFF)Click here for additional data file.Material S1The Value. We show how the (PDF)Click here for additional data file.Material S2Accuracy Variance. We present the error bars with one standard deviation of uncertainty for the 10-fold cross validation with a k-NN classifier in (PDF)Click here for additional data file.


It is widely accepted that orthologous genes between species are conserved at the sequence level and perform similar functions in different organisms. However, the level of conservation of gene expression patterns of the orthologous genes in different species has been unclear. To address the issue, we compared gene expression of orthologous genes based on 2,557 human and 1,267 mouse samples with high quality gene expression data, selected from experiments stored in the public microarray repository ArrayExpress.In a principal component analysis (PCA) of combined data from human and mouse samples merged on orthologous probesets, samples largely form distinctive clusters based on their tissue sources when projected onto the top principal components. The most prominent groups are the nervous system, muscle/heart tissues, liver and cell lines. Despite the great differences in sample characteristics and experiment conditions, the overall patterns of these prominent clusters are strikingly similar for human and mouse. We further analyzed data for each tissue separately and found that the most variable genes in each tissue are highly enriched with human-mouse tissue-specific orthologs and the least variable genes in each tissue are enriched with human-mouse housekeeping orthologs.The results indicate that the global patterns of tissue-specific expression of orthologous genes are conserved in human and mouse. The expression of groups of orthologous genes co-varies in the two species, both for the most variable genes and the most ubiquitously expressed genes. Over the past two decades, both tissue specificity and the conservation of expression between orthologous genes have been much discussed but comparative analysis at the transcriptome level has produced ambiguous results. While studies suggested that orthologous genes do not share similar expression patterns -5, otherThe challenge of comparing expression patterns of orthologous genes in different species is mainly due to different affinities of probes on different chips, leading to difficulties in comparing data from different platforms. Different approaches have been tried to compare gene expression patterns in different organisms (reviewed in ). Some sOur study used data generated on species-specific microarray platforms. Only human data from the Affymetrix HG-U133A array and mouse data from the Affymetrix MG_U74Av2 array were considered to exclude between-array variability within each species. These two whole genome arrays were selected because they have been used for the highest number of human and mouse samples in ArrayExpress. Raw data consisting of 5,372 and 1,323 high quality human and mouse CEL files were selected from ArrayExpress. Each CEL file corresponds to the hybridization of one biological sample. Since the data matrices are extremely large and the information content is very rich, we first normalized and filtered for human-mouse orthologous probesets, then used principal component analysis (PCA) to reduce the data dimensions. PCA has been often used to study high-dimensional data generated by genome-wide gene expression studies -25. In aIt is intriguing whether these major gene expression patterns are conserved across evolutionarily diverse species such as human and mouse. We answer this question positively and report a similar PCA analysis of the 1,323 mouse hybridizations. Similar to what was observed in the previous study of human data , the mouAn integrated mouse gene expression dataset based on Affymetrix platform MG_U74Av2 was created as described in Materials and methods. It can be downloaded from the ArrayExpress website , accessiFurther analysis demonstrated that samples of a particular tissue type are always represented by multiple experiments , as described in Materials and methods. The data can be downloaded from our web site  in the fIn the combined analysis, we observe the same cluster pattern as in the mouse-only analysis. The four predominant groups are a central cluster of mostly cell line samples, and three tissue-specific clusters: muscle/heart, nervous system, and liver samples Figure . Human sSamples such as mammary gland and hematopoietic system were removed from the analysis presented in Figure It is interesting to observe that all mouse clusters are closer to the center than their human counterparts Figure  and 5. THow the data were normalized before they were merged into a combined matrix has profound impact on the PCA landscape. In all PCA results we presented so far, the data were normalized by probeset across all samples to minimize the platform differences among samples; thus, the data are more comparable cross-species. If we normalized the human and mouse data matrices by sample, in the combined matrix, the platform difference is the largest variance captured in the top principal component  was hierarchically clustered . For liver, muscle/heart, nervous system, cell lines, adipocyte tissues, immune system, skin and gastrointestinal organs, human and mouse data clustered side by side on both X and Y axis Figure .Cross-platform comparison of gene expression data is always a challenge. Even for the same tissue type, human and mouse samples differ in many ways; thus, it is difficult to take a pair of orthologous genes between the two species and compare their expression levels directly. A condition that induces or suppresses the expression of a gene in one species may not be applicable to another species. To minimize sample and platform variations, we used a measurement called correlation of correlation coefficient or corCor . It compThe distribution of corCor for all 6,180 probesets shows that orthologous genes have high corCor compared to a negative control Figure : in the In contrast to what we observed in Figure Since corCor is not suitable to identify correlating human and mouse genes at the tissue level, an alternative approach was attempted to identify orthologous genes that are expressed in a correlated fashion in the two species. The expression variance of every gene was calculated one tissue and one species at a time. For each tissue type, the genes are sorted based on their variance. When comparing the sorted gene lists for a human tissue and its corresponding mouse tissue, we observed that, on average, 42% of the most variable 600 genes in one species have ortholog counterparts in the most variable 600 genes in the other species Figure . For theet al. [et al. at the highest threshold (top 1/6). We identified 13 overlap genes between the two lists. Our study used 6,108 orthologs, whereas Chan's study used 3,074, with an overlap of 1,344 genes. Of the 273 genes we identified, 51 are in the 1,344-gene set, and of the 110 genes Chan et al. identified, 79 are in the same 1,344-gene set. A simple hypergeometric probability test shows that the chance of having 13 overlaps between 51 and 79 genes randomly taken from a common pool of 1,344 genes is low (P = 2.9 × 10-6), suggesting the overlap of the results from the two studies is significant. The same comparison was also done in heart/muscle and liver; similar overlaps with more significant P-values were observed between the two methods, showing significant overlap between gene sets identified by the two studies  term over-representation in the gene list using ONTO-EXPRESS . ONTO-EXWith large amounts of gene expression data obtained from public repositories, we investigated the transcriptomes of human and mouse across a large variety of experimental conditions. Where single experiments benefit from reducing experimental variability to discover gene-specific expression regulation, by instead selecting as wide a variety of experimental and sample conditions as possible, we can gain insights into regulation at a higher level of complexity. When analyzing samples from a large variety of tissues, such large-scale studies revealed that the patterns of global gene expression are strong enough to segregate samples based on key biological properties, despite vast variations in experiment conditions, genetic background, age, sex and other sample characteristics. The results confirmed the common belief that samples of similar tissue types share similarities at the transcriptome level. At the same time, the patterns of this segregation, as detected by PCA, are similar between mouse and human and indicate that, on a global level, the signals driving tissue specificity are similar between the species. It supports previous findings -9 that aOur results provide strong evidence that, on a global level, gene expression patterns of human-mouse orthologs are conserved. The cross-species conservation of expression profiles of tissue-specific genes and housekeeping genes is the foundation for the similar landscapes of sample clustering between human and mouse in large-scale transcriptome comparison. A recent publication  documentsimpleaffy package and four quality control measurements were produced: average background (AvgBg), scale factors (sfs), percent present (PP) and RNA degradation slope (RNAdeg). Arrays were selected for inclusion in this study based on these quantities using the following ranges: AvgBg, 20 to 150; PP, 25 to 65; RNAdeg, <1.7; sfs, 0.1 to 2.5 .simpleaffy assessments, the CEL files selected were further evaluated by probe level model (PLM) using the Bioconductor's affyPLM package. Two quality assessments were derived from the PLM fitting output: normalized unscaled standard error (nuse) and relative log expression (rle). The cutoffs were set as: nuse, 0.97 to 1.05; rle, -0.15 to 0.15. Arrays not passing these criteria were discarded from further analysis.In addition to the RMA package [The resulting 1,323 CEL files were pre-processed using Bioconductor's justRMA function. In the resulting matrices, each column contains data for one sample and each row data for one probeset. The two matrices were then reduced to a subset of probesets representing orthologous genes between mouse and human. The pairing of these orthologous probesets was done based on gene orthologs obtained from Ensembl Compara [The high quality CEL files of 5,372 human samples tested on the HG-U133A microarray were selected and prepared as previously described . The higAfter we discarded rows with non-orthologous probesets from the human and mouse matrices, the remaining data on each matrix were normalized either by probeset or by sample. To normalize by probeset, we first centered data row by row on median zero by subtracting the row median from each value in the row. Then the centered values were divided by median absolute deviation to scale the data. To normalize by sample, we used the same procedure but centered and scaled the data by columns instead of by rows; column median was used to center the data and column median absolute deviation was used to scale the data. After normalization either by probeset or by sample, the two data matrices of centered and scaled values were merged into one matrix by concatenating the sample columns of orthologous probesets. In the merged matrix, the rows are probesets and the columns are human and mouse samples.prcomp with default settings provided in the R statistic package was used to perform PCA on different data matrices throughout this study. The results were visualized by scatter plots.PCA is a technique that transforms a dataset onto a linear space spanned by a number of orthogonal components, ordered by decreasing variance of the data when projected on it. The technique facilitates dimensionality reduction and noise filtering by the projection of data onto a number of the principal components, maximizing the variance retained. The function heatmap.2.The combined data matrix of 2,557 human samples and 1,267 mouse samples created as described above was used for hierarchical clustering. The matrix contains gene expression values centered and scaled by probeset. Each sample in the matrix is assigned to one of 13 general tissue categories that are well represented in both species so the total number of annotation types is 26 (tissue combining species). We extracted 26 submatrices containing data from samples of 26 different annotation types; Pearson correlation coefficients were calculated for 26 × 26 permutations of the submatrices; for each pair of submatrices, a mean correlation coefficient was taken and placed in a 26 × 26 matrix. Hierarchical clustering of the samples in the matrix was performed by R function MergeMaid was used for this analysis [For a gene A on the human array composed of n genes, we computed its pair wise Spearman correlation coefficient with every gene on the same chip, giving a vector v(A) of length n - 1. Given gene A' is the ortholog of gene A on the mouse array, we similarly computed its pair wise correlation coefficient with every mouse gene as v(A') of length n - 1. The correlation coefficient between v(A) and v(A'), corCor, provides an indication of whether A and A' are correlated in mouse and human on the transcriptome level, regardless of the vast sample variations. The higher the absolute corCor value, the stronger correlation of the orthologous genes is; negative corCor indicates negative correlation. The R package analysis .corCor: correlation of correlation coefficient; GO: Gene Ontology; PCA: principal component analysis; PLM: probe level model.The authors declare that they have no competing interests.XZ designed and carried out all analyses and wrote the manuscript. JR participated in the design and interpretation of the study and contributed to manuscript writing. HP participated in the design and coordination of the study. AB conceived the study and participated in its design and helped to draft the manuscript. All authors read and approved the final manuscript.AB is a senior team leader and senior scientist at EMBL-EBI and serves on the board of FGED  Society.PCA plot of the integrated mouse gene expression data matrix. The two axes are components 2 and 3; each dot represents a sample, colored by experiment accession number. While experiments with more than 15 samples are labeled as individual experiments, experiments with smaller numbers of samples are grouped into one category, 'small exp' (light brown). Tissue clusters observed in Figure 1 are circled. No apparent clustering of samples based on experiments is observed.Click here for fileExperiments and samples used for the mouse PCA.Click here for fileDistribution of gene expression variances for the top 50 principal components. The histograms were plotted for PCA results of the combined human mouse data matrix normalized by (a) probeset or (b) sample.Click here for filePCA plot of a combined human and mouse gene expression data matrix . The samples are labeled by (a) species and (b) tissue type. Four major sample clusters are indicated: muscle/heart samples (red), nervous system samples (blue), liver samples (purple) and cell line samples (green). For these clusters, human and mouse samples exhibit subclustering in proximity to each other.Click here for filePCA plots of a combined human and mouse gene expression data matrix with all samples. The samples are labeled by (a) species and (b) tissue type. Unlike previous PCA plots, samples such as mammary gland and hematopoietic system whose presentation is mostly one-sided in one species were removed from the analysis; this PCA included all high quality data from both human and mouse. The clustering of samples from nervous system (green), muscle/heart (lilac), cell lines (brown), and liver (pink) is still evident among the overwhelmingly dominant hematopoietic samples (blue) and mammary gland samples (turquoise). The corresponding human and mouse sample clusters resemble each other. Samples of unknown tissue type annotation are colored white and labeled as '0'.Click here for filePCA plots of a combined human and mouse gene expression data matrix normalized by sample. The samples are labeled by (a) species and (b) tissue type. Mouse samples (black) and human samples (red) are well separated on the axis of component 1. Tissue clusters in the two species are projected to the second principal component in a similar order: nervous system (blue), muscle/heart (red), liver (purple) and cell lines (green).Click here for fileHierarchical clustering heatmap of Pearson correlation coefficients between different types of tissues in human and mouse. Tissues in which human and mouse data clustered together are outlined by boxes.Click here for fileDistribution of corCor between human and mouse ortholog genes in specific tissues. The X-axis is the corCor value between human and mouse gene expression levels in (a) nervous system and (b) cell line samples. The Y-axis is the number of orthologs. In these analyses, corCor distribution is not very different from a randomized negative control .Click here for filePercentage of common genes in the top 10% most variable genes between different tissues of the same species, as well as between different tissues of human and mouse. The numbers in bold are those represented in the top 10% group in Figure 5.Click here for fileFunctional analysis of orthologous genes shared between mouse and human in the top 10% most variable genes and the top 10% least variable genes. (a-c) The top 10% most variable genes and (d) the top 10% least variable genes:  nervous system; (b) muscle/heart; (c) liver. In (a-c), GO over-representation was sorted by corrected P-value and then by level of GO term enrichment; only the top ten categories are displayed. Genes with tissue-specific functions are colored in orange. The over-represented GO terms in (d) were sorted by count of genes in each category; the top categories are mostly housekeeping molecular functions.Click here for file


Comparison and classification of metagenome samples is one of the major tasks in the study of microbial communities of natural environments or niches on human bodies. Bioinformatics methods play important roles on this task, including 16S rRNA gene analysis and some alignment-based or alignment-free methods on metagenomic data. Alignment-free methods have the advantage of not depending on known genome annotations and therefore have high potential in studying complicated microbiomes. However, the existing alignment-free methods are all based on unsupervised learning strategy . These types of methods are powerful in revealing major similarities and grouping relations between microbiome samples, but cannot be applied for discriminating predefined classes of interest which might not be the dominating assortment in the data. Supervised classification is needed in the latter scenario, with the goal of classifying samples into predefined classes and finding the features that can discriminate the classes. The effectiveness of supervised classification with alignment-based features on metagenomic data have been shown in some recent studies. The application of alignment-free supervised classification methods on metagenome data has not been well explored yet.We developed a method for this task using k-tuple frequencies as features counted directly from metagenome short reads and the R-SVM (Recursive SVM) for feature selection and classification. We tested our method on a simulation dataset, a real dataset composed of several known genomes, and a real metagenome NGS short reads dataset. Experiments on simulated data showed that the method can classify the classes almost perfectly and can recover major sequence signatures that distinguish the two classes. On the real human gut metagenome data, the method can discriminate samples of inflammatory bowel disease (IBD) patients from control samples with high accuracy, which cannot be separated when comparing the samples with unsupervised clustering approaches.The proposed alignment-free supervised classification method can perform well in discriminating of metagenomic samples of predefined classes and in selecting characteristic sequence features for the discrimination. This study shows as an example on the feasibility of using metagenome sequence features of microbiomes on human bodies to study specific human health conditions using supervised machine learning methods. Microbes play important roles in human health and habitat environment. There are a large amount of microbes accreting with human, forming many microbiomes on human bodies such as on human skins -3, mouthThe 16S rRNA sequencing approach is a conventional way of identifying microbes and many researchers have used this method in comparing microbome samples ,12. The The basic approach for analyzing metagenome data is alignment-based methods. NGS reads are first mapped to known datasets of microbial genomes and genes , and a catalogue of taxonomy or genes for each metagenomic sample is obtained. Comparison of the metagenome samples can be done based on the catalogues or abundances of known microbial genes. Both supervised and unsupervised machine learning methods can be applied on such data to classify samples into informative classes or to find intrinsic clusters in the samples. The limitation of these types of methods is also obvious. Firstly, reference databases of microbial genomes and genes are still far from complete. Known microbial genome sequences only occupy a small part of the whole microbial world. Secondly, the alignment of the huge amount of short reads to multiple references often means a high complexity of computing. Alignment-free methods are therefore a promising alternative approach for analyzing massive metagenome data.Alignment-free methods based on sequence signatures are powerful for analyzing genomic data without using reference genomes. The basic strategy is to profile the composition of short oligonucleotides  as signatures for distinguishing sequences from different organisms -30. SuchSupervised classification has been successfully applied on 16S rRNA data and on alignment-based features of metagenomic data in recent years -38. HoweThe basic idea of alignment-free methods is to use the occurrence or frequency of k-tuples  as sequence signatures of the studied genomes or metagenomes, and use such signatures as features for the clustering or classification. Existing works using unsupervised clustering approaches have shown that such sequence signatures contain information that can separate metagenomes of different characteristics into meaningful groups, and can cluster genomes according to their phylogenetic relations ,41,42. Fkwords are taken as the candidate features and their occurrences are scanned and counted in every short read. Summing up all counts gives the total count vector of dimension4k, with each component as the count of the corresponding k-tuple word. We standardized the vectors in two steps: The frequencies of all k-tuples in each sample were normalized by the total count of k-tuples to remove the effect of different sequencing depths between samples; then for each feature, the component values across all samples were standardized to mean = 0, variance = 1.The k-tuple frequency is the relative occurrence frequency of each word of k nucleotides in a metagenome sample. This can be counted directly from the NGS short reads of the metagenome data, without the need of full or partial sequence assembly. For a fixed length k of k-tuples, all 4Recursive SVM (R-SVM)  is a modn is the number of samples in the training set, x is the feature vector of a test sample, xi is the vector of training sample i and yi ϵ { − 1, 1} is the corresponding class label. The parameters αi’s and b are trained from the training dataset by maximizing the separation margin and minimizing the prediction error on training data. And the weight vector of the features is k is small, the number of all features can be less than 1000. We then start with all features and use the biggest number in the list that is smaller than the number of features in the next selection.where ribed in  in the cTo assess the classification performance at each feature selection level and to choose the level of best performances, we use the accuracy of leave-one-out cross validation (LOOCV) to evaluate the result of classification and feature selection. In each round of LOOCV, one sample is left out and R-SVM is trained on the remaining samples. At each feature selection level, an SVM model is trained and is used to predict the class label of the left-out sample. After all the samples are left-out once, the rate of correct prediction is calculated at each level of feature selection. It should be noticed that in this procedure, feature selection must also be included in the validation step. That is, the test sample should be left out before any feature selection, otherwise the assessment of the performance can be biased and over-optimistic , which cFor a particular classification question defined on a phenotype property of the host of the microbiome, such as a specific human disease, it is not sure whether there is an association between the classification and metagenome features. As the number of samples is small but the number of features is huge, chance alone may result in some apparent accuracy in the classification on some datasets. Therefore, it is necessary to check the statistical significance of the classification performance by comparing the accuracy obtained on the real data with the null distribution of accuracies on similar data when there is no association between the features and classification labels. We implemented a set of permutation experiments to answer the question in this study. In each experiment, 1000 times of permutations were applied on class labels of the samples, with labels randomly assigned to the samples while keeping the total number of each label consistent with the true labels. This is a way to generate multiple datasets with real data that have no association with the classification labels. The same R-SVM LOOCV procedure was applied on each permuted dataset, and the best LOOCV result on each permuted data was recorded. With 1000 permutations, we obtain an estimated null distribution of the classification performance when there is no true classification signal in the data. Comparing the performance we obtained on the true data to this null distribution, we obtain an estimation of the statistical significance of the achieved performance with the permutation p-value.We first created a series of simulated genomic data to study the performance of the method. The simulation data are summarized in Table N genomes with N of them belonging to class 1 and the other N belonging to class 2. In the two classes, genome lengths and background models are the same, but sequence seeds are inserted with different strategies .In this step, there are two key elements: the background model and seed insertion strategy. We generated 2N = 25 with a uniform model in which PA = PC = PG = PT and genome length n = 10,000 for all simulated genomes. When inserting seeds of fixed length l, we randomly choose an insertion position pins(1 ≤ pins ≤ n − l + 1), and replace the l letters from this position with the seed we want to insert. Each seed are inserted randomly for several times according to the seed density parameter. The density of one seed is defined as l is the length of the seed, #seed is how many times this seed is inserted into the genome and n is the length of the genome. Table l = 4, 5, 6, 7 are all experimented . All seed sequences are randomly generated, and the seeds we used for inserting are the same in each row of Table We fix β = 50, coverage C = 10, and no sequencing error.In the second step, we use Metasim  to randoAs the differences between the two classes of simulated data are known, we use these data to investigate how well the proposed method can discriminate the two classes with sequence features, and how well it can reveal the true differences  between the two classes.http://www.ncbi.nlm.nih.gov/sra?term=SRP001298; Data for Moraceae: http://www.ncbi.nlm.nih.gov/sra?term=SRP001299).To test how the proposed method performances on real data, we first applied it on a real tree genome dataset that has been used in , as a siWe applied our method on the metagenome data of faecal samples of 124 European individuals from  to classk varying between 3 and 8. Table We did a total of 14 × 4 simulation experiments as described in Method and Data. In each experiment we used k-tuple features with length For each of the 14×4 experiments, we can obtain a result similar to the example of Table Table Table In Sim-1, we changed different seeds and use k-tuples with different lengths as features to study the relation of the result and the underlying information in the sequences. Again, taking the result in Table When we use other k-tuples lengths, we may still get 100% accuracy, but not with the least number of features. For example, when we use 5-tuples, the 100% accuracy appears at 20 features level. When we use 7-tuples, the 100% accuracy appears at levels higher than 20 features. From this example and from the summary in Table What features can be revealed by the recursive feature selection and classification method? As an example, Figure The top 20 features with length 7 can also give 100% accuracy. All these 20 selected features contain the seeds or their reverse compliment. When the selection level is too low (too few selected features), even though they can contain the seeds, the feature is one letter longer than the seed so the discrimination information is weakened.From these simulation experiments, we can see that the feature set selected at the best result or nearly best result can recover the true characteristic differences of the sequence groups.We tested our method on the NGS short read data to classify 10 tree genomes into 2 families. Figure We permuted 1000 times on these 10 tree genome data with 6-tuple features, and observed 3 out of the 1000 permutations can result in a 100% accuracy by the same R-SVM experiments. This gives the permutation p-value of 0.003. The reason why 100% accuracy can be achieved on 3 of the 1000 permutated data is that the sample size is too small (only 10 samples) and therefore even some random assignment of class labels may coincide with the true class labels with some probability.k increases from 4 to 8. When k = 7, at the selection level of 200 features, we get the best LOOCV result . This shows that although the IBD samples and control samples cannot be separated by unsupervised clustering with sequence signatures, they can be discriminated by supervised learning methods. This illustrates the power of using alignment-free supervised classification to reveal underlying sequence signatures that distinguish groups of metagenome samples.We applied the proposed method on the real metagenome dataset to classify IBD and non-IBD samples. IBD (Inflammatory Bowel Disease) is an inflammatory disease occurring in the colon or small intestine. It can affect people at any age group and is not easy to treat . People We did permutation test to study the significance of this classification result. One thousand permutated datasets were generated and the same R-SVM method was applied on all of them with different k-tuple lengths as on the real data to choose a best LOOCV result for each permutation dataset. None of the 1000 permutations can give us the accuracy equal to or higher than 88%, which gives the p-value less than 0.001. Figure After the LOOCV experiment, we re-run the R-SVM method on all 50 training samples with k-tuple length of 7 to obtain a classifier to be applied for predicting other samples. We applied the classifier on the remaining 74 samples  as the independent test data and found 78% of them are correctly predicted.We did 4 extra experiments by randomly choosing a different group of 25 control samples in the training data each time. Table In this paper, we developed an alignment-free supervised classification approach to classify metagenome samples into predefined classes with sequence signatures from NGS data. We conducted a series of simulations to study the performance of the method and effects of parameters. Simulation results show that the method is powerful in classifying the classes, and can successfully reveal underlying features that distinguish the two classes. Then we applied the method to a real genome dataset and a real metagenome dataset to test its ability to handle real NGS sequences. It can give 100% LOOCV accuracy on the genome dataset and ~88% LOOCV accuracy on the metagenome dataset. These results are proved to be significant by permutation test.The rapidly developing next-generation sequencing technologies provided great opportunity for studying the microbial communities of different environments or human niches. The lacking of sufficient reference genomes and also the computational burden of assembling microbial genomes from metagenome data hinders the wider application of such technology. Experiments in this study showed that these problems can be solved by adopting the alignment-free strategy together with machine learning methods. Sequence signatures can be analyzed directly on the short reads, and the biological functions can be studied by downstream analyses of sequence signatures that are selected as discriminative for the classification.The method R-SVM adopted in this work is a representative for supervised classification and feature selection methods. There are many other optional methods that can be applied, such as the Naive Bayes method, k-nearest neighbor method, decision tree and random forest, etc.. These method need to be combined with some feature selection methods for the application. R-SVM represents a category of methods that integrate the feature selection and classification steps in one “wrapper” method. Brown et al discussed that “wrapper methods have higher learning capacity so are more likely to overfit” . Extra cA current limitation of the alignment-free methods is that after a set of sequence signatures are found to be discriminative for the classification, the biological indication of these k-tuple features is still not clear. This is a limitation for both supervised and unsupervised methods. The biological meaning of selected k-tuple sequence signatures is still an open question for future study. Any breakthrough on this direction may open a new gate for understanding functions of microbiomes and their interaction with the host systems they parasitize or accrete with.PCA: Principal component analysis; NGS: Next-generation sequencing; R-SVM: Recursive support vector machines; LOOCV: Leave-one-out cross validation; IBD: Inflammatory bowel disease; RDP: Ribosomal database project; OTU: Operational taxonomic unit; NCBI: National center for biotechnology informationThe authors declare that they have no competing interests.XZ initiated and designed the study. HC did all the experiments. HC and XZ analyzed the results and wrote the manuscript. Both authors read and approved the final manuscript.


Keloids are protrusive claw-like scars that have a propensity to recur even after surgery, and its molecular etiology remains elusive. The goal of reverse engineering is to infer gene networks from observational data, thus providing insight into the inner workings of a cell. However, most attempts at modeling biological networks have been done using simulated data. This study aims to highlight some of the issues involved in working with experimental data, and at the same time gain some insights into the transcriptional regulatory mechanism present in keloid fibroblasts.Microarray data from our previous study was combined with microarray data obtained from the literature as well as new microarray data generated by our group. For the physical approach, we used the fREDUCE algorithm for correlating expression values to binding motifs. For the influence approach, we compared the Bayesian algorithm BANJO with the information theoretic method ARACNE in terms of performance in recovering known influence networks obtained from the KEGG database. In addition, we also compared the performance of different normalization methods as well as different types of gene networks.Using the physical approach, we found consensus sequences that were active in the keloid condition, as well as some sequences that were responsive to steroids, a commonly used treatment for keloids. From the influence approach, we found that BANJO was better at recovering the gene networks compared to ARACNE and that transcriptional networks were better suited for network recovery compared to cytokine-receptor interaction networks and intracellular signaling networks. We also found that the NFKB transcriptional network that was inferred from normal fibroblast data was more accurate compared to that inferred from keloid data, suggesting a more robust network in the keloid condition.Consensus sequences that were found from this study are possible transcription factor binding sites and could be explored for developing future keloid treatments or for improving the efficacy of current steroid treatments. We also found that the combination of the Bayesian algorithm, RMA normalization and transcriptional networks gave the best reconstruction results and this could serve as a guide for future influence approaches dealing with experimental data. Keloids are large protruding claw-like scars that extend well beyond the confines of the original wound and do not subside with time . They unThe goal of reverse engineering methods is to infer gene networks from observational data, thus providing insight into the inner workings of a cell ,9. ThereIn addition to these two modeling approaches, reverse engineering methods also differ in terms of the mathematical formalisms used and can be static or dynamic, continuous or discrete, linear or nonlinear and deterministic or stochastic . For theMicroarray data from previous studies will be used to learn the networks. However, learning the structure of a gene network using the influence approach is difficult as the number of possibilities scale exponentially with the number of variables. Therefore, modeling and testing such large structures would require large amounts of data for accuracy. Due to our limited data, we have decided to focus on small networks of genes that have been found to be differentially expressed from our previous work. Furthermore, to increase the number of samples, we will also use data from Smith et al , which iIn total, we have four different treatment conditions  and two different cell derivations  from multiple patients. Although some of our datasets consist of time-series data, the gap between each time point is very large (in the order of days) and may lead to inaccurate results if used to infer time-series regulatory networks. Therefore, we have limited our study to steady state conditions with the assumption that each time point is statistically independent from others. This is a possibly valid assumption as the sampling time is very long. Furthermore, the genes were not directly perturbed by knockdown or overexpression in our experiments and it is very likely that the different conditions used will result in multiple unknown perturbations. As such, inference algorithms such as dynamic Bayesian networks (which require numerous closely spaced time points) and differential equation approaches (which require either time series data or knowledge of perturbations) cannot be applied in our case.To date, most attempts at modeling biological networks have been done using simulated data. We hope that this work would highlight some of the issues involved in working with experimental data. Furthermore, we also hope that insights gained from this endeavor would provide some clues about the different transcriptional regulatory mechanisms present in keloid and normal fibroblasts.Keloid and normal fibroblasts were selected from a specimen bank of fibroblast strains derived from excised keloid specimens. All patients had received no previous treatment for the keloids before surgical excision. A full history was taken and an examination performed, complete with color slide photographic documentation before taking informed consent prior to excision. Approval by the NUS Institutional Review Board, NUS-IRB was sought before excision of human tissue and collection of cells. Remnant dermis from keloid or normal skin was minced and incubated in a solution of collagenase type I (0.5 mg/ml) and trypsin (0.2 mg/ml) for 6 h at 37°C. The cells were pelleted and grown in tissue culture flasks. The cell strains were maintained and stored in liquid nitrogen until use.4 cells/ml in 10% FCS until confluency and subsequently starved in a serum-free medium for 48 hrs. After 48 hrs, the serum free medium was replaced and fibroblasts were harvested after another 24 hrs (day 1), 72 hrs (day 3) and 120 hrs (day 5). Cells were grown and processed in five batches. Each batch consisted of one keloid and one normal sample harvested at the three different time points. KF1, NF1, KF2, NF2, KF4, NF4 and KF5, NF5 were samples from different patients while KF3 and NF3 were samples from the same patient. In another experiment, one keloid fibroblast sample was grown, treated with hepatoma derived growth factor (HDGF) and harvested for RNA after 6 hours, day 1 and day 2.Five different keloid fibroblast samples and five different normal fibroblast samples that were previously maintained and stored at -150°C were thawed and used for the experiments. Fibroblasts were seeded in 15 cm dishes at a density of 1 × 10RNA was extracted using the RNeasy-kit  according to the manufacturer's protocol. Purified RNA was quantified by UV absorbance at 260 and 280 nm on a ND1000 spectrophotometer . Labeled complementary RNA (cRNA) was produced from total RNA using the GeneChip One-Cycle or Two-Cycle Eukaryotic Target Labeling and Control Reagents  according to the manufacturer's protocol.Fragmented cRNA was then hybridized to preequilibrated Affymetrix GeneChip U133A or the newer Genechip U133 2.0 Plus arrays at 45 °C for 15 hours. The cocktails were removed after hybridization and the chips were washed and stained using Affymetrix wash buffers and stain cocktails in an automated fluidic station. The chips were then scanned in a Hewlett-Packard ChipScanner  to detect hybridization signals.In addition to microarray data generated by our lab, raw microarray data in the form of .CELS files from Smith et al's experiments were also downloaded from the GEO database . FollowiHuman genomic sequences 1000 base pairs upstream from the transcriptional start site if known, or from the initiation codon, were extracted from the Ensembl database . As fREDeg is the mean value of gene g under our condition of interest, μcg os the mean value of gene g under control conditions, Vareg is the variance of gene g under our condition of interest, Varcg is the variance of gene g under control conditions, and ne and nc are the number of samples under our condition of interest and under control conditions respectively. This statisitic is similar to the z-statistic used by the fREDUCE creators [where g is the index over genes, μcreators . We thencreators  for possKEGG pathways that were found to be enriched when comparing keloid to normal fibroblasts from a previous study were used for the influence approach (unpublished data). These were the antigen presentation and processing pathway, cytokine-cytokine receptor interaction and toll-like receptor signaling pathway. Genes that were used as nodes for modeling were chosen on the basis that there is only one gene representing that particular node, all other genes will be assumed to be hidden nodes. The following 5 pathways were eventually selected for the influence approach Figure . PathwayExpression values of selected genes from all the different data sets available were used for the influence approach. To enable comparison between the different data sets, gene expression for all the relevant nodes were normalized using the average of GAPDH and B-actin expression. GAPDH and B-actin were first plotted to determine their correlation and outliers were removed from the dataset. Three keloid experiments from the serum starvation U133A dataset did not meet this criteria and was removed giving a total of 28 keloid experiments and 24 normal experiments. We ran ARACNE and BANJO on the keloid and normal inputs separately, and also on the MAS 5 and RMA normalized expression values separately. All parameters were left at their default values. For ARACNE, kernel width and number of bins were automatically detected by the software while DPI tolerance to remove false positives was set at 0.15. For BANJO, the Proposer/Searcher strategies were chosen as random local move and simulated annealing, respectively, and the amount of time BANJO uses to explore the Bayesian Network space was set to one minute. All the other parameters such as reannealingTemperature, coolingFactor, and so on, were left with their default values. Parameter values were selected as best values (in terms of network inference accuracy) as shown by Bansal et al . In ordeIn order to assess the inference performances we computed the Positive Predicted Value (PPV) and the Sensitivity scores as described by Bansal et al . The folTP = Number of True Positives = number of edges in the real network that are correctly inferred; FP = Number of False Positives = number of inferred edges that are not in the real network; FN = Number of False Negatives = number of edges in the real network that are not inferred.The following were then computed:In order to compute the random PPV we considered the expected value of a hypergeometrically distributed random variable whose distribution function and expected value are, respectively:where N = number of possible edges in the network, M = number of true edges and n = number of predicted edges. Then,All statistical tests are done using the one tailed paired t-test.Binding motifs found using the gene expression values from set A  are shown in Table No binding motifs were found for unfiltered RMA normalized set B , but binding motifs were found for the other conditions Table . Of partBinding motifs were found for set C (keloid treated with steroid versus serum induced keloid fibroblasts) and D  when fREDUCE was run using parameters length 6 with 0 IUPAC substitutions. Other parameters did not produce any results. Furthermore, results were only obtained when MAS 5 normalization was used. The effect of hydrocortisone appears to be realized through the binding motifs GGAGGG and GCCCCC and this was consistent for both keloid Table . Of partfREDUCE found few binding motifs for set E  and no binding motifs for set F (keloid treated with HDGF versus untreated keloid fibroblasts). Binding motifs for set E were found only when the MAS 5 unfiltered condition and the RMA filtered condition were used Table . FurtherOn average, BANJO was significantly more sensitive compared to ARACNE in recovering influence networks Figure . HoweverTranscriptional networks . However, a cellular signaling network has been successfully reconstructed previously using a Bayesian approach . It shouin silico study done by Bansal et al. [Between the two influence methods tested, BANJO produced significantly better results compared to ARACNE. The superior performance of BANJO for small data sets with 'global' perturbations can also be seen in the results of the l et al. . The lowl et al. . Having The fact that performance for the transcriptional network involving NFKB  using influence methods was better for normal fibroblasts compared to keloid fibroblasts suggests that the influence between NFKB and its targets was weaker in keloid fibroblasts, or that there were more links in the keloid network that were not captured by our simplified diagram. This would imply that targeting of NFKB alone may not be sufficient in reducing the expression of its targets in keloids. However, more work needs to be done to verify this hypothesis.In this study, we have attempted to reverse engineer gene networks from all microarray expression data that we have collected. Using the physical approach of correlating expression values to binding motifs, we found some consensus sequences that were active in the keloid condition, as well as some sequences that were responsive to steroid treatment which is one of the commonly used treatments for keloids. These consensus sequences are possible transcription factor binding sites and could be explored for developing future keloid treatments or to improve the efficacy of current steroid treatments. Using influence approaches on experimental data, we found that the combination of the Bayesian algorithm, RMA normalization and transcriptional networks gave the best reconstruction results and this could serve as a guide for future influence approaches dealing with experimental data. Furthermore, our results show that the NFKB transcriptional network inferred from normal fibroblast data was more accurate than that inferred from keloid data, suggesting a more robust network in the keloid condition.The ability to infer molecular interactions in cellular systems is one of the most exciting promises of systems biology. As the most widely available high throughput technology, gene expression microarrays provide a good test set for the application of inference algorithms that infer dynamic models from static, genome-scale data. However, the critical assumption underlying this methodology is that mRNA measurements are predictive of molecular activity. This assumption has been thrown into question as new studies reveal the substantial role of alternate regulatory mechanisms, such as translation, post-translational modifications, genetic and epigenetic factors, as well as the increasingly appreciated regulatory role of non-coding RNAs. Furthermore, data from the microarray platform is typically noisy, and is also hidden in multiple probes that can be combined in multiple ways to produce different expression values. Yet in spite of all these difficulties, the topic of reverse engineering gene networks is surely worth pursuing, as it provides us with a means of understanding biology not only in terms of the genes themselves, but also through their interactions.The authors declare that they have no competing interests.BNSO conducted the experiments, carried out computational analysis and drafted the manuscript. PTT made contributions in the design of the experiments, interpretation of biological data and in the drafting of the manuscript. All authors have read and approved the final manuscript.


Genomic datasets generated by new technologies are increasingly prevalent in disparate areas of biological research. While many studies have sought to characterize relationships among genomic features, commensurate efforts to characterize relationships among biological samples have been less common. Consequently, the full extent of sample variation in genomic studies is often under-appreciated, complicating downstream analytical tasks such as gene co-expression network analysis.cor, can distinguish biologically meaningful relationships among subgroups of samples. Specifically, we find that cor reveals the profound effect of Huntington’s disease on samples from the caudate nucleus relative to other brain regions. Furthermore, we find that this effect is concentrated in specific modules of genes that are naturally co-expressed in human caudate nucleus, highlighting a new strategy for exploring the effects of disease on sets of genes.Here we demonstrate the use of network methods for characterizing sample relationships in microarray data generated from human brain tissue. We describe an approach for identifying outlying samples that does not depend on the choice or use of clustering algorithms. We introduce a battery of measures for quantifying the consistency and integrity of sample relationships, which can be compared across disparate studies, technology platforms, and biological systems. Among these measures, we provide evidence that the correlation between the connectivity and the clustering coefficient (two important network concepts) is a sensitive indicator of homogeneity among biological samples. We also show that this measure, which we refer to as These results underscore the importance of systematically exploring sample relationships in large genomic datasets before seeking to analyze genomic feature activity. We introduce a standardized platform for this purpose using freely available R software that has been designed to enable iterative and interactive exploration of sample networks. Genomic studies capture an enormous amount of information about the molecular organization of biological systems. Understanding this organization poses a challenge for biologists. In most genomic studies, the number of features  far exceeds the number of biological samples under investigation. Consequently, while network methods are often used to illuminate patterns among pairwise relationships of genomic features, the rich information contained in the connectivity patterns among samples remains comparatively untapped. However, patterns of co-variation in genomic feature activity ultimately reflect heterogeneity among biological samples. It is therefore critical to understand the extent of sample heterogeneity before analyzing genomic feature activity, and whenever possible to relate sample heterogeneity to known sample traits, which may include both biological and technical sources of variation. In practice, biologists’ efforts to explore sample relationships in genomic studies are an integral component of data pre-processing, yet they are often performed in a perfunctory fashion using platform-specific and qualitative criteria.k in k-means clustering) . The mean adjacency is also known as the density of the network. In sample networks, we often refer to the density as the mean intersample adjacency (ISA).where Sample network interpretation of the density: Using our proposed measure of sample adjacency (signed weighted network with β = 2), we find thatif all sample correlations are > 0.6. Thus, the mean adjacency is roughly equal to the mean correlation in sample networks.Z.K and Z.C exhibit a progressively linear, inverse relationship. A similar relationship has been observed in unweighted (binary) networks, where the relationship between the (unstandardized) connectivity and (unstandardized) clustering coefficient of network nodes, i.e. the C(k) curve, has previously been reported to follow a scaling law  are corrected, w C ≅ kα,31), wit, witZ.K w  curve as a scatter plot between Z.K and Z.C where Z.K and Z.C denote the standardized sample connectivity and the standardized sample clustering coefficient, respectively. We also introduce a new network concept, cor, which we define as the Spearman correlation between Z.K and Z.C. We note that other measures of correlation could also be used (e.g. Pearson correlation). Since the Spearman correlation is invariant with respect to a monotonically increasing transformation (e.g. standardization), we find thatWe define the standardized k denotes the unscaled connectivity. As described in Results, we find that cor is inversely related to the density (i.e. mean adjacency) in simulated networks. However, because cor is invariant if one scales all off-diagonal adjacencies by a constant, it is more accurate to consider cor as an indicator of network heterogeneity. The network concept Heterogeneity is defined as:where aij = CF(i) CF(j)) ) exhibited no relationship to a simulated sample trait (referred to as “disease status”), and ii) cor(KC) distinguished “control” subjects from those with “moderate” or “severe” disease status. The module was simulated to contain two unrelated sub-modules of 200 and 300 genes, respectively. The first sub-module contained a signal for the simulated sample trait, while the second sub-module contained noise genes with no relation to disease status. The first sub-module was simulated in two steps. First, we used a seed ME as input for the simulateModule function from the R package WGCNA [To further illustrate the utility of ge WGCNA . This fudecentralization and homogeneity, as well as summaries of node-based measures such as the mean correlation, mean connectivity, mean clustering coefficient, mean intersample adjacency (or density), and mean maximum adjacency ratio (MAR). When applied to sample networks, these concepts provide a battery of measures for comparing the consistency of sample behavior within and across datasets. These network concepts are calculated automatically by SampleNetwork and are discussed further in Additional file In addition to characterizing sample networks via the connectivity and the clustering coefficient, it is also possible to characterize sample networks using additional network concepts. Such concepts include t-test to determine whether the mean expression levels of the ME differed between groups (distinguished by HD diagnosis). An advantage of this approach is that the extent of modular DE can be summarized by a single P-value. Future efforts may seek to incorporate higher-order representative features  to explore additional relationships between gene co-expression modules and disease status [t-test on log2-transformed expression values. The resulting P-values were corrected for multiple comparisons by controlling for the false-discovery rate [Q-values), along with mean expression levels for CTRL and HD, are reported for all genes in the salmon module in Additional file To determine whether specific CN gene co-expression modules were associated with DE in HD, for each CN module we calculated the ME , which is a vector that summarizes the characteristic expression pattern of a module . We thene status . Differeery rate . The reshttp://www.ingenuity.com/) was used to determine whether gene co-expression modules identified in [P < 0.001) were used to search for enrichment. Network construction was restricted to experimentally verified, direct physical interactions. IPA reported false-discovery rate (FDR)-corrected P-values for the 500 most enriched functionally annotated categories of genes in each module. Results for the salmon module are reported in Additional file Ingenuity Pathways Analysis ; BA9, Brodmann’s area 9 ; CB, cerebellum; CN, caudate nucleus; cor, the Spearman correlation between the standardized connectivity and the standardized clustering coefficient; CTRL, control; DE, differential expression; Dx, diagnosis; HD, Huntington’s disease; IPA, Ingenuity Pathways Analysis; ISA, intersample adjacency; p.DE, P-value for the significance of differential expression; p.Diff.cor, P-value for the significance of differences between cor; WGCNA, weighted gene co-expression network analysis; Z.K, the standardized connectivity; Z.C, the standardized clustering coefficient.The authors declare that they have no competing interests.MCO and SH designed the SampleNetwork and ModuleSampleNetwork R functions and drafted the manuscript. MCO created the SampleNetwork and ModuleSampleNetwork R functions and the R tutorial; MCO performed the data analysis. PL performed a simulation study  and created the web site. All authors read and approved the final manuscript.Supplementary information. PDF file containing Supplementary Methods, Supplementary References, and Supplementary Figures (1–8).Click here for fileSample information. XLS table that summarizes sample information, including Gene Expression Omnibus (GEO: http://www.ncbi.nlm.nih.gov/geo/) sample ID, sample labels, diagnosis, severity grade, age, sex, individual ID, hybridization date, hybridization batch assignment, and country of processing, as described in ref. [ in ref. .Click here for fileSampleNetwork R tutorial. DOC file containing annotated R code and detailed instructions for executing the SampleNetwork and ModuleSampleNetwork R functions. The datasets that are referenced in the tutorial and analyzed in this study can be downloaded from: http://www.genetics.ucla.edu/labs/horvath/CoexpressionNetwork/SampleNetwork.Click here for fileSummary of differential expression, functional enrichment, and module membership for genes in the salmon module. XLS table that summarizes the extent of differential expression, functional enrichment, and membership strength for the salmon module. Differential expression analyses include CTRL vs. HD human caudate nucleus samples [P-values are taken from ref. [ samples , CTRL vs samples , and wil samples . Functio samples . Module rom ref. .Click here for fileIngenuity Pathways Analysis of salmon module genes. XLS table that reports false-discovery rate (FDR)-corrected P-values for the 500 most enriched functionally annotated categories of genes in the salmon module from Ingenuity Pathways Analysis .Click here for file


P procedure and the bootstrap step-down minP procedure perform the best among all tested procedures, when sample size is as small as 3 in each group and either familywise error rate or false discovery rate control is desired. When sample size increases to 12 and false discovery rate control is desired, the permutation maxT procedure and the permutation minP procedure perform best. Our results provide guidance for high-throughput data analysis when sample size is small.Resampling-based multiple testing procedures are widely used in genomic studies to identify differentially expressed genes and to conduct genome-wide association studies. However, the power and stability properties of these popular resampling-based multiple testing procedures have not been extensively evaluated. Our study focuses on investigating the power and stability of seven resampling-based multiple testing procedures frequently used in high-throughput data analysis for small sample size data through simulations and gene oncology examples. The bootstrap single-step min With rapidly developing biotechnology, microarrays and next generation sequencing have been widely used in biomedical and biological fields for identifying differentially expressed genes, detecting transcription factor binding sites, and mapping complex traits using single nucleotide polymorphisms (SNPs) –7. The mP  values or test statistics, resulting in lower type II errors. The commonly used resampling techniques include permutation tests and bootstrap methods. Resampling-based multiple testing procedures are widely used in high-throughput data analysis , especially when the sample size is small or the distribution of test statistic is nonnormally distributed or is unknown. Resampling-based multiple testing procedures can account for dependent structures among P  values even when sample size is small. Permutation tests are nonparametric statistical significance tests, where the test statistics' distribution under the null hypothesis is constructed by calculating all possible values or a concrete number of test statistics  from permuted observations under the null hypothesis. The theory of the permutation test is based on the work by Fisher and Pitman in the 1930s. Permutation tests are distribution-free, which can provide exact o(n−1/2) to o(n−1), makes the bootstrap method one order of magnitude more accurate than the delta method. The P  values computed by the bootstrap method are less exact than P  values obtained from the permutation method, and additionally, P  values estimated by the bootstrap method are asymptotically convergent to the true P  values [The bootstrap method, first introduced by Efron  and furt  values . Different resampling methods can draw different conclusions, however, when applied to the same data set. An investigation of multiple testing error rate control, power, and stability of those resampling methods under different situations is necessary to provide guidance for data analysis, so that optimal methods in different scenarios could be used to maximize power and minimize multiple testing error rates.In this paper, we focus on investigating the power and stability properties of several commonly used resampling-based multiple testing procedures: (1) the permutation tests ; (2) thet-test statistics.Compute the test statistics for the observed data set, such as two sample Permute the original data in a way that matches the null hypothesis to get permuted resamples and construct the reference distribution using the test statistics calculated from permuted resamples.α test based on the upper α percentile of the reference distribution, or obtain the raw P value by computing the proportion of permutation test statistics that are as extreme as or more extreme than the observed test statistic.Calculate the critical value of a level To carry out a permutation test based on a test statistic that measures the size of an effect of interest, we proceed as follows.P  values to control the multiple testing error rates. One is single-step minP procedure and the other is single-step maxT procedure.Westfall and Young  proposedP adjusted P  values are defined as [T adjusted P  values are defined in terms of test statistics Ti, namely, [HM is the complete null hypothesis. Pl is the raw P value for the lth hypothesis, and Tl is the observed test statistic for the lth hypothesis.The single-step minfined as (1)p~i=P namely, (2)p~iT=t-tests. In SAM, genes are assigned with scores relative to change in gene expression and its standard deviation of repeated measurements. Scatter plots of the observed relative differences and the expected relative differences estimated through permutation identifies statistically significant genes based on a fixed threshold.The Significance Analysis of Microarrays (SAM) procedure proposed by Tusher et al.  identifiti for each gene i  .Compute a test statistic ti)( such that t(1) ≤ t(2) ≤ ⋯≤tg), ≤ tb(2), ≤ ⋯≤tg),b( versus the expected Form a quantile-quantile (Q-Q) plot (SAM plot) of the observed i = i1 such that i1 are called significant positives. Similarly, starting at the origin and moving down to the left, find the first i = i2 such that i2 are called significant negatives. Define the upper cut point Cutup(Δ) = min{ti)( : i ≤ i1} = ti1)( and the lower cut point Cutlow(Δ) = max{ti)( : i ≥ i2} = ti2) is estimated by computing the number of genes with ti,b above Cutup(Δ) or below Cutlow(Δ) for each of the B permutation and averaging the numbers over B permutations.For a given threshold, the expected number of false rejections E(V)/r)) under the complete null hypothesis, at an acceptable nominal level.A threshold Δ is chosen to control the Fdr   is set to be 2000 with the fraction of true null hypotheses (m0/m) at 50%. In the two-group comparison, the standardized logarithms of gene expression levels are generated from multivariate normal distribution. One group has 50% of genes with means at μ and the remaining with means at 0. All genes in the other group have means at 0. The mean expression level μ on log2 scale is set to be from 1 to 6 with step 0.50 for the first simulation study. The variances of the standardized logarithm of gene expression levels are equal to 1 in both groups. Thus, the mean differences of μ in gene expression between the two groups are the Cohen's d effect sizes. The pairwise correlation coefficients of test statistics are set to be 0 in our simulation study. The test statistics used are equal variance t-test throughout the simulation study. The FWER/FDR level is set at 5% (α = 0.05).In our first simulation study, each set includes 100 independently generated groups of two samples with equal sample size of 3 or 12 in each group. 100 repetitions are chosen because computationally 100 is more efficient than 1000 or even higher repetitions. 1000 repetitions are also tried and similar results are obtained. Thus, 100 repetitions are chosen for computational efficiency. The total number of genes (m) is set to be 1000, with the fraction of differentially expressed genes ((m − m0)/m) equal to 10%, 25%, 50%, 75%, and 90% to cover all possible scenarios. In the two-group comparison, the gene expression level on log2 scale is generated randomly from a multivariate normal distribution with μ = 0 and σ = 1. The correlations between genes are randomly fluctuated between 0 and 1 to mimic the correlations in real microarray data. The mean differences are set between 1 and 2, with the step equaling the inverse of the number of differentially expressed gene (m − m0). The variances are set to 1. Equal variance t-tests are used for this simulation study, and the FWER/FDR level is set at 5% (α = 0.05).We conducted another simulation study to examine the effect of fraction of true null hypotheses on power and stability. In our second simulation study, each data set includes 100 independently generated samples of two groups with equal sample size of 3. The total number of genes , single-step minP (ss.minP), step-down maxT (sd.maxT), and step-down minP (sd.minP).The mt.maxder Laan  was execP  values and the adjusted P  values of comparisons between the chemotherapy favorable group (3 subjects) and chemotherapy unfavorable group (3 subjects) were calculated using the resampling-based multiple testing functions in the multitest package and the siggenes package in R. Ovarian cancer is a common cause of cancer deaths in women . MicroarSimulation studies were conducted to compare the power and stability across all tested multiple testing procedures for normally distributed data with either independent or randomly correlated test statistics. The sample size is 3 or 12 in each group for independent test statistics and 3 in each group for randomly correlated test statistics.P procedures outperformed all other tested procedures when sample size is 3 in each group  had no power to detect any significant difference between groups, and their FWER estimates were close to 0. The power of the bootstrap maxT procedures (ss.maxT and sd.maxT) were between the permutation procedures and the bootstrap minP procedures. The estimated variances of true discoveries and total number of discoveries were around 0 for all tested resampling-based multiple testing procedures. The estimated FWER, power, and stability were constant across effect sizes.For independent test statistics with FWER controlled at 5%, the two bootstrap minch group . Both thP procedures remained to have the largest power among all tested procedures when FDR was controlled at 5% and sample size was 3 in each group  showed higher power than all other tested procedures across various proportions of nontrue null hypotheses, when test statistics are dependent and FWER is controlled. The two bootstrap minP procedures had desired FWER control as well, when the proportions of nontrue null hypotheses were greater than 50% (T procedures (ss.maxT and sd.maxT) had lower power than the two bootstrap minP procedures. They showed desired FWER control, however, when the proportions of nontrue null hypotheses were over 25%. The permutation single-step maxT and minP procedures had no power to detect any significant difference between groups. All resampling-based procedures had estimated variances of true discoveries and total discoveries around 0 across various proportions of nontrue null hypotheses, when sample size was as small as 3 in each group.The two bootstrap minthan 50% . The twoP, sd.minP, ss.maxT, and sd.maxT) and the two permutation methods (mt.maxT and mt.minP) showed similar results, when FDR were controlled, as that when FWER were controlled  and moderate sample size data (12 in each group) when FWER control is desired. The bootstrap single-step and step-down minP procedures are the best when FDR control is desired for data with small sample size (3 in each group). The permutation maxT and minP procedures perform the best for data with moderate sample size when FDR control is desired. The SAM procedure overestimates FDR, although it has higher power than the permutation and bootstrap maxT and minP procedures.This paper investigated the power and stability properties of several popular resampling-based multiple testing procedures for both independent and dependent test statistics, when sample size is small or moderate, using available functions in The simulation results also showed that the permutation test procedures have no power to detect any significant differences between groups when sample size is as small as 3 in each group; the permutation test procedures perform well when sample size increases to 12 in each group; the SAM procedure has no power for detecting significant differences when the proportion of nontrue null hypotheses is less than 50% and sample size is 3; the bootstrap multiple testing procedures perform better than the permutation test procedures and the SAM procedure for small sample size data.T procedure and permutation single-step minP procedure when sample size is only 3 in each group. Thus, the smallest raw P  value from the permutation procedures will be 0.05. After adjusting the raw P  values to control FWER or FDR, all adjusted P  values will be larger than 0.05, thus no hypotheses will be rejected. As such, the estimated FWER, FDR, and power will all be zero.The zero power of the permutation test procedures is due to its limited number of permutated test statistics for data set with small sample sizes. For example, the complete number of enumeration is only 20 for both permutation single-step maxOur current investigation only focuses on normally distributed data. Further investigation is needed to extend the distribution in the simulations from multivariate normal to other distributions, such as lognormal and binomial distributions. To examine the power and stability properties of those resampling-based multiple testing procedures, under nonnormal distributions, will be a focus for our future research.