In transcriptome‐wide association studies (TWAS), gene expression values are predicted using genotype data and tested for association with a phenotype. The power of this approach to detect associations relies, at least in part, on the accuracy of the prediction. Here we compare the prediction accuracy of six different methods—LASSO, Ridge regression, Elastic net, Best Linear Unbiased Predictor, Bayesian Sparse Linear Mixed Model, and Random Forests—by performing cross‐validation using data from the Geuvadis Project. We also examine prediction accuracy (a) at different sample sizes, (b) when ancestry of the prediction model training and testing populations is different, and (c) when the tissue used to train the model is different from the tissue to be predicted. We find that, for most genes, the expression cannot be accurately predicted, but in general sparse statistical models tend to outperform polygenic models at prediction. Average prediction accuracy is reduced when the model training set size is reduced or when predicting across ancestries and is marginally reduced when predicting across tissues. We conclude that using sparse statistical models and the development of large reference panels across multiple ethnicities and tissues will lead to better prediction of gene expression, and thus may improve TWAS power. This methodology uses reference panels with measured gene expression and single‐nucleotide polymorphism (SNP) genotypes to build a predictive model representing the genetically regulated component of gene expression. This model is then applied to GWAS genotype data to impute gene expression values, which can be tested for association with the phenotype of interest. The methodology is implemented in multiple software packages  ancestry, sample size, and tissue matching. Ancestry is known to be important in the prediction of complex traits. Polygenic risk scores estimated in a population of one ancestry typically perform poorly when applied to populations of alternative ancestries  at predicting gene expression levels, using data from the Geuvadis project in which genome‐wide genotype and lymphoblastoid cell line (LCL) gene expression data were measured in 462 individuals of European or African origin. We also compare the performance of these statistical modeling approaches with pre‐existing prediction models from the PrediXcan software package. Finally, we examine the impact of sample size, ancestry and tissue matching on the accuracy of gene expression prediction.22.12.1.1R2 < 0.8 in the individuals for whom genotypes were imputed. We also removed SNPs with missing data in any samples. Gene expression data were used as is, with no further processing or quality control after download from the Geuvadis website.The genotype data for 465 individuals  and PEER‐factor normalized gene expression counts of 23,722 genes from LCLs from 462 individuals measured by RNA‐seq were downloaded from the Geuvadis website  from WTCCC1 , and genotypes at all SNPs within 1 megabase of the gene start or end site were used as predictors (x), with no additional covariates included in the regression.In all methods applied here, gene expression was used as the phenotype  and LASSO . In this study, the elastic net was used twice—once with α set to .5, and once in which α was determined by cross‐validation. The λ parameter was determined by cross‐validation in both instances. Here we implemented an elastic net using the R package glmnet.By including both penalties, and using the parameter 2.2.4µ represents the mean of the trait of interest, β represents a vector of fixed (sparse) effects, u is a vector of random effects, ε is a vector of errors, π represents the proportion of variants which have an additional effect (over the polygenic effect provided by the LMM component), τ−1 represents the residual variance, aσ is the magnitude of the non‐zero fixed effects, δ0 is a point mass at zero, bσ is the magnitude of the random effects, K is a variance‐covariance matrix of genotypes, and nI is an identity matrix.The BSLMM is a spike‐and‐slab prior model that combines a standard linear mixed model (LMM) with a Bayesian variable selection regression approach (BVSR). The LMM aspect of the model assumes that all covariates have a small effect on the trait of interest (polygenic), while the BVSR portion also allows some variants to have a larger effect on the trait. The model can be considered asρ), which is the proportion of variance explained by the sparse and random effects together, and PGE (h), which is the proportion of variance explained by only the sparse effects. ρ and h, in addition to π, are the model hyperparameters estimated through Markov chain Monte Carlo (MCMC). Here, BSLMM was implemented through GEMMA  operating in individual i and εi is the residual term. These are assumed to follow a normal multivariate distribution:G represents a genetic relationship matrix (GRM) defined using genotypes of all SNPs within 1 Mb of the gene start or end sites, p represents the number of SNPs used to generate the GRM, and I is an identity matrix. This model tends to produce a more polygenic solution where many SNPs have small effects on the phenotype of interest. Further details on the determination of marker effects and predictions can be found in the Supporting Information note of de Los Campos, Vazquez, Fernando, Klimentidis, and Sorensen,  and the outer loop . The 373 samples were split into 10 groups of equal size. For a single fold of the outer loop, nine groups were combined into a single data set to be used for training a gene expression prediction model (training set), and the remaining one group was used to evaluate the performance of the model (test set).α = .5) and LASSO, this inner loop was used to tune the λ parameter. For elastic net , the inner loop was also used to tune the α parameter. In each case, the parameter value was chosen as that which maximized the correlation coefficient (R) between the predicted and the observed expression values, within this inner loop. BSLMM and BLUP did not have any parameters that could be tuned via cross‐validation, so this inner loop was not performed for these methods.The inner loop 10‐fold cross‐validation was then performed on the training set. This inner loop was used to tune any model parameters. For ridge regression, elastic net (R) between predicted expression and measured expression was then calculated in the test set. This gave a single R‐value for onefold of the outer loop.Once appropriate parameter values had been chosen using the inner loop, a gene expression prediction model was trained on the full training set using the appropriate parameter values. The resulting model was then applied to the test set, and expression was predicted for these samples. The correlation coefficient  for each gene with available gene expression data.2.3.2R estimates obtained using this nested cross‐validation with the reduced training set were then compared with R estimates from our original cross‐validation.To examine the effect of sample size on gene expression prediction accuracy, we repeated the 10‐fold nested cross‐validation procedure described above, using the 373 EUR Geuvadis samples. However, in each fold of the outer loop of cross‐validation, we used 1 of the 10 groups as the training set, and the remaining nine groups combined as the test set. The rest of the procedure was unchanged. 2.3.3R) between gene expression predicted by the EUR‐trained models and measured gene expression. The analysis was then repeated using 90% of EUR samples to train models, and 37 YRI samples to test models, giving training and testing set sample sizes equivalent to those used for onefold of the 10‐fold nested cross‐validation on EUR samples.To evaluate the predictive ability of models trained using samples of one ancestry to predict the expression of samples of an alternative ancestry, we used the 89 YRI samples. Initially, gene expression prediction models were trained using the whole set of 373 EUR samples, with model parameters tuned via 10‐fold cross‐validation within the EUR samples. These prediction models were then applied to the 89 YRI samples, and predictive performance was evaluated as the correlation  in the 10‐fold nested cross‐validation using the 373 EUR Geuvadis samples were used as the input for the “gene2func” option. The MHC region was not excluded.Gene set enrichment was conducted using FUMA  analysis implemented in GCTA . Each set of prediction models was applied to genotype data for the 373 EUR samples in Geuvadis, and for each of these prediction models, the correlation coefficient (R) between measured Geuvadis expression and expression predicted using the GTEx‐informed model was calculated.Gene expression prediction models for 48 different GTEx tissues from the PrediXcan software package were downloaded from 33.1α parameter set to .5), elastic net , LASSO, BSLMM, BLUP, and Random Forests. To compare the approaches, LCL expression levels of 23,722 genes from the Geuvadis project were used. For each of the seven approaches, 10‐fold nested cross‐validation was performed on data from the 373 Geuvadis samples of EUR origin. In each of the 10 folds, 90% of samples (the training set) were used to train a model, which was applied to the remaining 10% (the test set). Correlation (R) between the predicted and measured expression of each gene in the test set was calculated in each fold. For each gene, we defined prediction accuracy as the mean of the 10R estimates.We first aimed to compare the performance of seven different statistical approaches for the prediction of gene expression from SNP genotypes. The chosen modeling approaches were: ridge regression, elastic net (with the R = 0.0743) across these 22,218 genes . In total, we estimated that 208,702 of the 225,170 BSLMM models fitted during the 10‐fold cross‐validation showed poor MCMC mixing (indicative of poor convergence) for at least one of the hyperparameters, raising concerns about its use for prediction of gene expression.In total, the expression of 22,218 genes could be predicted by all seven methods. Overall, the BSLMM showed the best average prediction performance  and R = .885 with LASSO, but only achieved R = .425 with ridge regression and R = .332 with BLUP. In total, for 108 genes we observed a difference in R of at least 0.3 between any two methods. In 94 of these cases, a method with assumptions of sparsity showed the greater R, while a more polygenic method showed the lower R. For these genes, there was a clear benefit in using the more sparse method.We observed that r Figure , indicatR estimates for the 22,218 genes were heavily skewed toward zero for all seven methods  showed R ≥ 0.5 from any of the seven methods, and are subsequently defined as “well‐predicted” genes. When considering only these genes, the modeling methods with assumptions of sparsity outperformed the more polygenic methods even more strongly than observed previously  and  showed strong concordance with estimates of prediction accuracy obtained using elastic net . On average, heritability estimates were slightly larger than prediction accuracy estimates, indicating that there was some potential room for improvement to prediction accuracy.The upper bound of prediction accuracy achievable here is given by the heritability of gene expression attributable to local SNPs. To identify this upper bound for each gene, we next estimated the heritability of each transcript that was attributable to the SNPs within 1 Mb of the gene using GCTA. Overall, estimates of local heritability were highly correlated  GWAS data from WTCCC1.z scores were highly correlated , 22,490 genes had predictions from both the analyses using the larger and the reduced training sets. Of these, 14,019 (62.3%) showed a smaller R in the analysis using the reduced training set , and interestingly, many genes that achieved large R estimates in the 90% training set analysis also achieved large R estimates at the reduced sample size. For example, RPS26 showed R = 0.913 in the larger training set analysis and R = 0.888 in the reduced training set analysis. Similarly, AC008957.1 showed R = 0.915 in the analysis using the larger training set, and R = 0.889 in the reduced training set analysis. Thus, it seems that for genes where the relationship between expression and local SNPs is strong enough, even small sample sizes are sufficient for constructing models that predict expression well.Overall, the average prediction accuracy was reduced when using the smaller training set. For elastic net . This relationship did not plateau at the largest available sample size, indicating that a further increase in sample size may boost the average prediction accuracy observed here. Despite this, a plateau in prediction accuracy was observed for some genes, with no further increase beyond a given sample size , indicating that the upper limit of prediction accuracy imposed by the local heritability of expression at these genes had been reached at the sample sizes used here.3.4α = .5) and applied to genotype data for 89 YRI Geuvadis samples, and the correlation (R) between predicted expression and measured expression was calculated. These R measures were then compared with the R achieved by elastic net (with α = .5) from 10‐fold nested cross‐validation within the EUR samples.We next sought to investigate how well prediction models trained using samples of one ancestry were able to predict the gene expression of samples of different ancestry. To do this, prediction models were trained on the 373 EUR Geuvadis samples using elastic net  showed larger R from 10‐fold nested cross‐validation within EUR samples. Across the 22,493 genes, the average prediction accuracy obtained from 10‐fold cross‐validation on EUR samples (mean R = 0.0625) was greater than the average accuracy from the application of EUR models to YRI samples (mean R = 0.0332). However, there was no consistent pattern, with some genes showing greater R from the application of EUR models to YRI samples. We repeated this analysis with the other six statistical methods and consistently observed poorer average prediction when predicting into YRI samples  and applying them to the EUR samples, and calculating R between predicted and measured expression in the EUR samples. We also performed 10‐fold nested cross‐validation using only the 89 YRI samples and then compared the R estimates from this YRI‐only 10‐fold nested cross‐validation with R estimates from the application of YRI‐trained models to EUR samples. Across the 22,334 genes, the average prediction accuracy from 10‐fold cross‐validation on YRI samples (mean R = 0.0317) was greater than the average accuracy from the application of YRI models to EUR samples , reinforcing our earlier findings. As above, to account for sample size differences, we repeated the analysis by training models on 90% of YRI Geuvadis and applying them to nine EUR samples, matching the sample sizes to onefold of the 10‐fold cross‐validation on the 89 YRI samples. While R estimates showed much variation, likely reflecting the smaller sample sizes used for both prediction and testing, we again observed that average prediction accuracy was poorer when predicting across populations .We next performed the reverse of this analysis by training models on the 89 YRI samples using elastic net (with α = .5). R values achieved in this mixed sample were highly correlated (r = .780) with those achieved from 10‐fold nested cross‐validation using only the 373 EUR samples . Across the 22,498 genes, the average prediction accuracy estimate obtained from this mixed sample 10‐fold cross‐validation (mean R = 0.0609) was similar to the average prediction accuracy estimate obtained from 10‐fold cross‐validation on EUR samples (mean R = 0.0625), demonstrating that even when the population contains samples from different ancestries, the prediction accuracy mimics that achieved when using a single ancestry, as long as the composition of the training population matches that of the testing population.Finally, we combined the EUR and YRI samples into a single group of 462 samples. We then extracted 373 samples, keeping the proportion of EUR and YRI samples the same as in the larger group of 462 samples. Using this mixture of EUR and YRI samples, we performed 10‐fold nested cross‐validation on this group using an elastic net  were applied to Geuvadis EUR samples. Correlation between expression predicted with GTEx‐trained models and measured Geuvadis expression was calculated.α = .5; Figure RPS26, which achieved R = 0.905 from the application of the GTEx model to Geuvadis, and R = 0.913 from 10‐fold nested cross‐validation within Geuvadis. Overall, R estimates from 10‐fold nested cross‐validation and from GTEx‐trained models were highly correlated (r = .86), although those from GTEx‐trained models tended to be slightly smaller.Overall, prediction accuracy estimates from using the PrediXcan GTEx‐trained models were very similar to those achieved from 10‐fold nested cross‐validation on Geuvadis data using elastic net (3.6R = 0.188) with the GTEx LCL models (Table S2).We next sought to examine how models trained using GTEx expression data from a range of tissues performed at the prediction of Geuvadis LCL gene expression. To do this, 47 sets of gene expression prediction models taken from the PrediXcan software package, each trained using GTEx expression data from a tissue other than LCLs (in which Geuvadis expression was measured), were applied to Geuvadis data and the correlation between predicted and measured expression was calculated. For each of the 47 GTEx non‐LCL tissues, the average level of prediction accuracy achieved (across gene‐specific prediction models) was lower than that achieved . This resulted in the non‐LCL prediction models achieving slightly poorer prediction accuracy than the LCL prediction models on average.RPS26. Prediction models for this gene were available for all 48 GTEx tissues, and prediction models for all tissues achieved R ≥ 0.8 when applied to the Geuvadis data . In contrast, we found that 10.3% of prediction accuracy estimates achieved by non‐LCL prediction models were at least 0.2 less than that achieved by the LCL model for the same gene. One example of this was NDUFAF1, for which the GTEx LCL model achieved a prediction accuracy estimate of 0.686, whereas the GTEx transverse colon model achieved an estimate of 0.128. In instances such as this, the LCL models clearly outperformed the non‐LCL models.In total, 53.3% of prediction accuracy estimates achieved by non‐LCL prediction models were within 0.05 of the prediction accuracy estimate achieved by the LCL model for the same gene, indicating similar performance and potentially shared regulation across different tissues. One gene for which non‐LCL prediction models achieved similar prediction accuracy to the LCL model was 4In recent years, TWAS has become a popular post‐GWAS approach for integrating genotype and expression data to identify genes potentially underlying phenotypes of interest. The power of TWAS to detect true gene expression‐phenotype associations relies partially on accurate imputation of gene expression values. Accuracy of imputation is dependent on a number of factors, including (but not limited to) the statistical method used to construct the prediction model, sample size, ancestry matching, and tissue matching. Here, we examined how each of these factors affects the accuracy of gene expression prediction, and thus how they may impact power in TWAS.By using cross‐validation to train and test prediction models, we found that statistical models that assumed sparsity predicted gene expression levels slightly better than those that assumed polygenicity, corroborating similar findings from previous studies  observation from our study is that the average prediction accuracy across the ~22,000 genes examined was typically very low, with many genes showing a cross‐validation We examined the effect of the sample size of the model training data set on expression prediction accuracy, finding that a reduced sample size led to poorer prediction. This has been observed in the context of prediction of other complex traits .Next, we examined the issue of prediction across tissues. By applying GTEx‐trained models for non‐LCL tissues to Geuvadis data, we found that average prediction accuracy was reduced when the tissues of the model training data set and the model testing data set were different. We found that for many genes, models trained in non‐LCL tissues were able to predict LCL gene expression with similar accuracy to models trained using LCL expression data. Given the evidence of strong sharing of eQTLs across many tissues  generally achieve the best gene expression prediction accuracy, although the actual prediction accuracy is low for the majority of genes. We have also demonstrated that increasing sample size and careful matching of ancestry and tissue between model training and testing populations improves prediction accuracy. Further increases in sample size and development of population‐specific reference panels across multiple tissues may help to further improve gene expression prediction accuracy and thus improve the power of future TWAS.The authors declare that there are no conflict of interests.Supporting informationClick here for additional data file.Supporting informationClick here for additional data file.Supporting informationClick here for additional data file.Supporting informationClick here for additional data file.Supporting informationClick here for additional data file.Supporting informationClick here for additional data file.Supporting informationClick here for additional data file.Supporting informationClick here for additional data file.Supporting informationClick here for additional data file.Supporting informationClick here for additional data file.Supporting informationClick here for additional data file.


To better understand the molecular basis of respiratory diseases of viral origin, high-throughput gene-expression data are frequently taken by means of DNA microarray or RNA-seq technology. Such data can also be useful to classify infected individuals by molecular signatures in the form of machine-learning models with genes as predictor variables. Early diagnosis of patients by molecular signatures could also contribute to better treatments. An approach that has rarely been considered for machine-learning models in the context of transcriptomics is data augmentation. For other data types it has been shown that augmentation can improve classification accuracy and prevent overfitting. Here, we compare three strategies for data augmentation of DNA microarray and RNA-seq data from two selected studies on respiratory diseases of viral origin. The first study involves samples of patients with either viral or bacterial origin of the respiratory disease, the second study involves patients with either SARS-CoV-2 or another respiratory virus as disease origin. Specifically, we reanalyze these public datasets to study whether patient classification by transcriptomic signatures can be improved when adding artificial data for training of the machine-learning models. Our comparison reveals that augmentation of transcriptomic data can improve the classification accuracy and that fewer genes are necessary as explanatory variables in the final models. We also report genes from our signatures that overlap with signatures presented in the original publications of our example data. Due to strict selection criteria, the molecular role of these genes in the context of respiratory infectious diseases is underlined. Viral diseases of the respiratory system are frequently studied on the molecular level of the transcriptome ,2,3. ThiIn this work, we reanalyze two transcriptomic datasets from study contexts of respiratory, viral-caused diseases to demonstrate how data augmentation can help to improve the performance of ANNs based on high-dimensional gene-expression data. Gene-expression data for the two selected studies are freely available from the gene-expression omnibus (GEO) archive . One of d of the data is much larger than the sample size n, i.e., To obtain robust ML models, a high quantity of annotated training data is usually required . Howeverformance , i.e., tformance , and canformance . Data auformance . Recent formance . Studiesformance ,23. For In the next section, we detail the results of the comparison of the three data augmentation techniques in the two respiratory datasets. We also compare the sets of genes in the molecular signatures from our ANN models with the signatures from the original publications. The results section is followed by a discussion about the computational and biological findings. The datasets used and the methods for data augmentation and classification are described in the last chapter.Using the two gene-expression datasets on respiratory infections, we investigated the applicability of data augmentation methods to both continuous microarray data and discrete RNA-seq count data. More specifically, we compared three different augmentation approaches: (a) a simple approach that is based on mixing observations, hereafter referred to as the mixed weighted observations (MWO) approach, as described in We first describe the incremental signature selection for both datasets via ML without data augmentation. By increasing the number of genes, their influence on the performance of the classifier was examined. As mentioned in the introduction ANNs were used as classification model, more specifically we fitted multilayer perceptrons (MLP) with two hidden layers with 150 and 50 neurons each and rectified linear unit (ReLU) activation. The number of input neurons corresponded to the number of genes that were used as input signature. The output layer was one single neuron with linear activation. The ANN architecture is described in more detail in the methods D was split into A 10-fold cross-validation was used to estimate the performance of each classification model, i.e., the original dataset p-values in ascending order, i.e., the gene with the lowest p-value being the first important input feature, and so on. This approach assumes that the genes with the most evidence for differential expression are also important for the discrimination of phenotypes. As a result, including an increasing number of the top ranked genes, i.e., the first 50, 100, 150, …, 750 most significant genes, as input features for training and testing the classification ANN, led to different classification performances. To identify the minimum number of genes as input features required for optimal ANN classification performance, an incremental feature selection was carried out for each cross-validation training subset Performances of the ANN models for both datasets when using data augmentation are provided in the following subsections.Here, we describe the performance of ANN models trained with genes selected in the microarray study that aimed to distinguish between viral and bacterial origin of ARIs. Since the above detailed differences between augmented and non-augmented scenarios describe tendencies, significant difference can also be detected using the duality between statistical test and confidence interval, as we have shown 95%—confidence intervals (CI) in all figures and tables. According to the duality between statistical test and CI, a significant difference can be inferred with a significance level of 5% when the 95%—CI for two scenarios do not overlap. Using this principle, one can see in the figures and tables clearly that data augmentation yields significantly better results in some of our simulated scenarios. Regarding the different influence of the number of genes being included in the model, from As the previously described results of the 2-fold augmentation already led to promising results, we tested further augmentation factors, namely 1-fold, 3-fold and 5-fold augmentation. Regarding the WGAN-GP approach, it could be shown that the accuracy and the Brier score improved with increasing augmentation; an improvement in the Brier score could be observed for the MWO approach. In contrast, when increasing the number of GAN-generated samples for training, the trained classification network performed worse on the test data. The detailed results are part of the Besides studying the use of data augmentation for assessing the value of a molecular signature, we were also interested how well each augmentation method maintained the correlation structure among genes. When performing data augmentation, it is important that the distribution of the artificial data closely meets that of the original data. That applies, for example, to the correlation structure between genes, which is represented, for example, by the covariance matrix of the expression matrices. Since a covariance matrix between thousands of genes cannot be estimated unbiased, we randomly selected small subsets of genes for which we studied the covariance matrices of original and artificial data. Although Although the training data in this example had a rather large sample size of 171, many microarray studies are much smaller. Therefore, we randomly selected a subset of 30 samples as training data, and compared again classifier performance with and without data augmentation. In this analysis, from a total of 190 samples, the 160 remaining samples were used as test samples. The whole procedure was repeated ten times to obtain confidence intervals for accuracy and the other performance measures. We compared the classification results for 2-fold and 5-fold augmentation with the classification results obtained without augmentation. Both 2-fold and 5-fold augmentation, i.e., having 30 original and 60 (150) artificial samples as training data, led to a significant improvement of the classification performance in terms of the Brier score, whereas considerable improvements of the accuracy could be observed only when including 50–200 genes into the classifier. The results for the different augmentation methods and different numbers of genes for the 2-fold and 5-fold augmentation of randomly chosen subsets are shown in the Here, we describe the performance of classification models trained with genes selected in the RNA-seq study that aimed to distinguish between SARS-CoV-2 and other viral origins of respiratory infections. Again, the duality-principle between statistical test and CIs can be used to show that augmentation yields a significantly better performance of a classifier than training only with original data or between a particular number of top genes included into a classifier. One can for example infer a significant difference between the Brier score in As for the microarray study, also 1-fold, 3-fold and 5-fold augmentation scenarios were tested to investigate the quality of the augmented samples. With further augmentation, the results improved further for the MWO and the WGAN approach, whereas the results for the ANNs trained with GAN augmented training data were not consistently better and showed an increasing variety in the classification of the test data. Detailed results can be found in the We also studied in this RNA-seq example whether data augmentation would also be helpful with a smaller set of training data. From the 258 samples, we selected randomly a subset of 30 samples as training data, and compared again classifier performance with and without data augmentation. In this analysis, all remaining 228 samples were taken as test data. The whole procedure was repeated ten times to obtain confidence intervals for accuracy and the other performance measures. To augment the training data, we considered 2-fold augmentation and 5-fold augmentation of the small training subset. The results of the analysis support the previously shown results that augmentation improves classification performance, since here too the accuracy increased, and the Brier score decreased with increasing augmentation; for the WGAN-GP and the MWO approach more than for the GAN approach. Detailed results are part of the p-value was <0.05 and the absolute log fold change was >2. Using the number of cross-validation folds is a sharper criterion than statistical testing on the whole dataset, because multiple reduced datasets  are used and genes are validated in the left-out dataset.To select genes as molecular signatures from the microarray and RNA-seq studies, we counted in how many folds of the cross-validation a gene was selected. In each fold, the selection criterion for a gene was that the FDR-adjusted In the microarray ARI study, 44 genes were selected in each of the 10 cross-validation runs. In the RNA-seq SARS-CoV-2 study, 88 genes were selected, accordingly. Of these 88 genes, 41 coded for small nuclear RNAs.We compared the selected genes with those presented as signatures in the original publications of the two datasets ,16). It . It 16])Most of these genes encode for molecules that are potentially involved in the pathogenesis of viral respiratory disease, such as antiviral and interferon-associated proteins of the innate immune response , transcription factors regulating cell differentiation (BATF3), chemokines , cytokines (IFNL1), molecules of membrane receptor-ligand systems , and molecules involved in calcium pathways  ,27,28,29Microarray example, biological processes: antigen processing and presentation of endogenous peptide antigen (GO:0002483); antigen processing and presentation of endogenous peptide antigen via MHC class I (GO:0019885); antigen processing and presentation (GO:0019882); antigen processing and presentation of peptide antigen via MHC class I (GO:0002474); regulation of dendritic cell differentiation (GO:2001198); dendritic cell differentiation (GO:0097028); macrophage cytokine production (GO:0010934); antigen processing and presentation of endogenous antigen (GO:0019883); phagocytosis, engulfment (GO:0006911); antigen processing and presentation of peptide antigen (GO:0048002)Microarray example, cellular component: phagocytic vesicle (GO:0045335)Microarray example, molecular function: MHC protein binding (GO:0042287); MHC class I protein binding (GO:0042288); MHC protein complex binding (GO:0023023)RNA-seq example, biological processes: macrophage activation (GO:0042116); T-cell differentiation involved in immune response (GO:0002292); leukocyte activation involved in immune response (GO:0002366); regulation of macrophage activation (GO:0043030); CD4-positive, alpha-beta T-cell differentiation involved in immune response (GO:0002294); cell activation involved in immune response (GO:0002263); alpha-beta T-cell activation involved in immune response (GO:0002287); alpha-beta T-cell differentiation involved in immune response (GO:0002293); regulation of interleukin-1 production (GO:0032652); regulation of interleukin-1 beta production (GO:0032651); myeloid leukocyte activation (GO:0002274); regulation of cytokine production(GO:0001817); regulation of interleukin-17 production (GO:0032660); immune response (GO:0006955); T-cell activation involved in immune response (GO:0002286); response to molecule of bacterial origin (GO:0002237); cytokine production (GO:0001816); response to lipopolysaccharide (GO:0032496); cellular response to oxidative stress (GO:0034599); NIK/NF-kappaB signaling (GO:0038061); regulation of microglial cell activation (GO:1903978); T-helper cell differentiation (GO:0042093); microglial cell activation (GO:0001774); leukocyte activation involved in inflammatory response (GO:0002269); interleukin-1 production (GO:0032612); reactive oxygen species metabolic process (GO:0072593); leukocyte activation (GO:0045321); interleukin-1 beta production (GO:0032611); plasma membrane bounded cell projection assembly (GO:0120031); positive regulation of cytokine production (GO:0001819); interleukin-17 production (GO:0032620); mitochondrial translation (GO:0032543); cytokine-mediated signaling pathway (GO:0019221); CD4-positive, alpha-beta T-cell differentiation (GO:0043367); lymphocyte activation involved in immune response (GO:0002285); cell projection assembly (GO:0030031)RNA-seq example, molecular function: antigen binding (GO:0003823); virus receptor activity (GO:0001618); immune receptor activity (GO:0140375); cytokine receptor binding (GO:0005126); double-stranded RNA binding (GO:0003725); cytokine receptor activity (GO:0004896); chemokine binding (GO:0019956)Besides looking at individual genes, we also performed gene ontology (GO) enrichment analysis , again bEarly diagnosis of patients with respiratory diseases caused by viral infections is important to provide a good treatment. Since changes in the transcriptome are among the early responses upon infection, classifiers based on high-throughput gene-expression data would provide a reasonable means for diagnosis. Early diagnosis is important to allow rapid therapeutical intervention, preventing progression of tissue damage and respiratory distress, resulting in a better prognosis. Additionally, early diagnosis is essential for the reduction of viral shedding and the initiation of early disease control strategies in the environment of affected individuals, which limits further virus spread.Several studies have been performed to study the differentiability of patients with respiratory diseases of viral origin, usually using standard ML models fitted to transcriptomics data. We selected two of such studies to demonstrate that ANNs in combination with a step of data augmentation, can provide improved classification accuracy. Although data augmentation has a long tradition in classifying samples based on other data types , it has rarely been considered for transcriptomic data. We selected two different kinds of transcriptomic data, i.e., one study based on DNA microarray data and one based on RNA-seq data. Although high-dimensional gene-expression data are distributed on a continuous scale as fluorescence values from DNA microarrays , discretWe mainly focused on accuracy and Brier score when presenting the results. A common metric for evaluating medical diagnoses is the F1-score, which is the harmonic mean of sensitivity  and positive predictive value (precision) . LookingFurthermore, our simulations have shown that the amount of augmentation has an impact on the classifier performance. In principle, the larger the amount of augmentation, the larger was the observed performance. In some cases, already with a 1-fold augmentation, a significant improvement was observed. However, a significant improvement might not necessarily mean a clinical relevant improvement. With a 5-fold augmentation in the RNA-seq example, one could obtain a significant improvement of the accuracy by 20%. This would mean a clearly larger number of correct classifications, and would be important if such a classification had an impact on therapy decisions. Therefore, it is not reasonable here to make a general statement regarding the minimum required amount of augmentation for classifiers with high-dimensional gene-expression data. Instead, we recommend checking the performance individually when other scientist use augmentation for such data.In this manuscript, we have evaluated the use of data augmentation in the context of ANNs. Moreover, we did some additional evaluations of transcriptomic data augmentation with other classifier methods, specifically SVMs, LDA and RF, but could not see the same amount of improvement. Regarding LDA, it might be that additional observations generated by the studied approaches do not much change the estimates for group means and covariance matrices. For SVMs, additional observations generated by the studied approaches may not change much the shape of the separating hyperplane. It may be that other augmentation methods could work with SVMs, LDA and RF. Therefore, more research will be necessary to understand how augmentation of transcriptomics data can be performed in combination with different classifier models.Independent from the question, whether augmented data can improve the classifier performance, we have compared covariance matrices of randomly selected gene sets to see, whether artificial data would also be useful for computer simulations, where researchers want to have data that reflects the correlation between genes. We found that the difference between the covariance matrix of original and artificial data was smallest for the MWO approach but larger for the GAN and WGAN-GP approaches. Although we observed these significant differences, the general shape of the covariance structure seems to be maintained as can be seen in Our classification results are not comparable with that of the ML models presented in the original publications of the two example datasets. First, we did use different groupings of samples, and second, we used other classifier models. In general, it is difficult to reimplement exactly ML models from other publications. However, our basic goal was to show that data augmentation in combination with ANNs can improve classification performance in contrast to ANNs fitted on the original data only, and that is what we have demonstrated here.Knowledge about changes in the transcriptome during a disease can significantly contribute to a better understanding of the molecular processes of the pathogenesis and therapy response. Thus, it can be helpful to know which genes are significantly up- or down-regulated, and which GO terms are involved. We found that some of the significantly enriched GO terms have been mentioned in the context of other infectious diseases. In the microarray example, we selected the biological process GO:0002483 which was also found enriched in a study on juvenile dermatomyositis , which iTypically, differentially expressed genes are detected by statistical tests to judge significance and by the log fold change to quantify the strength of the effect. We have argued in the introduction that there is more evidence for the molecular importance of a set of differentially expressed genes, if these genes show additionally a high performance in a ML model. To estimate this performance more robust, we used the concept of data augmentation. In the two datasets, we have shown that the selected signatures have also a higher diagnostic power when using data augmentation for fitting the ANN models. Following our above argumentation, we think that there is good evidence regarding the molecular role of the selected set of differentially expressed genes and GO terms.We developed a standardized pipeline with varying data augmentation methods followed by training a classifier using an increasing number of preselected features and testing the classifier for unseen test data. The pipeline was performed for two real world datasets to assess the applicability of those augmentation methods and their influence on the robustness of a classification model. For the latter, we chose an artificial neural network with two hidden layers. The number of input features was successively increased, with the genes being sorted based on how strongly they are differentially expressed, i.e., incremental feature selection . Two maihttps://www.ncbi.nlm.gov/geo, accessed on 15 April 2020) under the accession number GSE63990, containing microarray data of patients with respiratory infection. In the original study [Data were taken from the GEO archive  under the accession number GSE163151 and contained RNA-sequencing data from nasopharyngeal swabs of humans with either SARS-CoV-2 infection, other viral ARI or non-viral ARI. Originally, the data also included whole blood samples and were used to investigate the differential host responses to SARS-CoV-2 infection [Samples were taken from GEO archive  K times: for each K = 10 subsets preserving the percentage of samples per class  by a sii has the outcome 1, denoted by Let The proportion of correctly classified samples, i.e., In contrast, the Brier score measures the mean squared difference between the probability of belonging to class 1 and the actual outcome of the sample :(2)BrierThus, the Brier score provides a more detailed measure for the performance of a classifier than the accuracy.For classification, a deep neural network with two hidden layers with 150 and 50 neurons each was used. The number of input neurons corresponded to the number of input features (genes). In the hidden layers, rectified linear unit activation (ReLU) function was used to transform the input values, i.e., the output of the respective previous layer. ReLU function is a common type of non-linear activation used in hidden layers that maps negative values to zero and for positive values the function returns the input value . The outp-values were obtained for gene ranking and the resulting lists of ranked genes were used to extract the top n genes.The number of input features that were passed to the classification ANN was successively increased, with those features being sorted accordingly to their importance, i.e., the most differentially expressed genes. Therefore, a feature selection was carried out for each cross-validation fold and accordingly for each training subset, and the classification network was trained and tested with different numbers of those preselected genes. For both microarray data and RNA-seq data, the selection of genes followed a test for differential expression which was conducted using the R packages ‘Limma’  and ‘DESp × n matrix, with p being the number of features and n being the number of samples is multiplied by a (n × 1) vector of randomly chosen small weights. Repeating this calculation p matrix, with Z-transforming that matrix, we obtain a synthetic dataset consisting of the new generated arrays that lie within the range of the original data.The approach to augment data by weighted mixed observations is mainly used for image data augmentation and is described by Shorten et al.  as mixinA generative adversarial network (GAN) is a type of generative model based on deep learning  which waz to degrade the classification performance of the discriminator, i.e., to maximize the cost of the discriminator [c epochs to minimize its loss before the generator is trained for one epoch. Through this competition, the model theoretically learns a generator that creates realistic data, i.e., that samples from the complex, high-dimensional training distribution [A conventional GAN consists of two deep neural networks which are embedded in a competitive process. The discriminator works as a classifier and discriminates between real and fake samples. The generator generates synthetic samples from a latent vector iminator . As suggiminator , the disribution . Viñas eribution  point toribution .z, consisting of 100 randomly selected values between 0 and 1 from the normal distribution, processes it through the hidden layers and maps it onto a high-dimensional vector that describes an artificial sample. Thus, the generator has as many output neurons as the number of genes that make up an original sample, each with hyperbolic tangent (Tanh) activation. The discriminator takes such a vector as input vector, i.e., its number of input features corresponds to the number of output neurons of the generator, processes it through the hidden layers and maps it to a single output neuron with linear activation. Binary crossentropy with logits loss was used as the criterion for backward propagation for both discriminator and generator. Adam optimization algorithm [In our model, the GAN consisted of a generator, which in turn was a MLP with three hidden layers with 250, 500 and 1000 neurons each with ReLU activation, and a discriminator, which was also a MLP with three hidden layers with 1000, 500 and 250 neurons each with leaky ReLU activation. The generator takes a latent vector lgorithm  with a l PyTorch .As one problem with GANs is their training instability , much ofIn our study, we employed a Wasserstein GAN with gradient penalty (WGAN-GP). For this we used the same architecture for the generator ANN and the discriminator ANN as with the GAN, but changed the loss function by adding a gradient penalty. For implementing the gradient penalty term, we modified code from Persson , who wro


The following paper explores the popularity and the performance of all the GSA methodologies and software published during the 20 years since its inception. "Popularity" is estimated according to each paper's citation counts, while "performance" is based on a comprehensive evaluation of the validation strategies used by papers in the field, as well as the consolidated results from the existing benchmark studies.Gene Set Analysis (GSA) is arguably the method of choice for the functional interpretation of versus performance, results show discrepancies between the most popular and the best performing GSA methods.Regarding popularity, data is collected into an online open database ("GSARefDB") which allows browsing bibliographic and method-descriptive information from 503 GSA paper references; regarding performance, we introduce a repository of jupyter workflows and shiny apps for automated benchmarking of GSA methods (“GSA-BenchmarKING”). After comparing popularity The above-mentioned results call our attention towards the nature of the tool selection procedures followed by researchers and raise doubts regarding the quality of the functional interpretation of biological datasets in current biomedical studies. Suggestions for the future of the functional interpretation field are made, including strategies for education and discussion of GSA tools, better validation and benchmarking practices, reproducibility, and functional re-analysis of previously reported data.The online version contains supplementary material available at 10.1186/s12859-021-04124-5. Bioinformatics method and software selection is an important problem in biomedical research, due to the possible consequences of choosing the wrong methods among the existing myriad of computational methods and software available. Errors in software selection may include the use of outdated or suboptimal methods (or reference databases) or misunderstanding the parameters and assumptions behind the chosen methods. Such errors may affect the conclusions of the entire research project and nullify the efforts made on the rest of the experimental and computational pipeline .popularity (defined as the perceived frequency of use of a tool among members of the community) and the performance . The study is focused on the field of “gene set analysis” (GSA), where the popularity and performance of bioinformatics software show discrepancies, and therefore the question appears whether biomedical sciences have been using the best available GSA methods or not.The following paper discusses two main factors that motivate researchers to make method or software choices, that is, the omics data, and, for the purposes of this paper, we define it as the comparison of a query gene set  to a reference database, using a particular statistical method, in order to interpret it as a rank of significant pathways, functionally related gene sets, or ontology terms. Such definition includes the categories that have been traditionally called 'gene set analysis', 'pathway analysis', 'ontology analysis', and 'enrichment analysis'. All GSA methods have a common goal, which is the interpretation of biomolecular data in terms of annotated gene sets, while they differ depending on characteristics of the computational approach . The mos Fig. . , Functios Fig. . , and Pats Fig. . , which hs Fig. . .Fig. 1StThe first part of the analysis is a study on GSA method and software popularity based on a comprehensive database of 503 papers for all methods, tools, platforms, reviews, and benchmarks of the GSA field, collectively cited 134,222 times between 1999 and 2019, including their popularity indicators and other relevant characteristics. The second part is a study on performance based on the validation procedures reported in the papers introducing new methods together with all the existing independent benchmark studies in the above-mentioned database. Instead of recommending one single GSA method, we focus on discussing better benchmarking practices and generating benchmarking tools that follow such practices. Together, both parts of the study allow us to compare the popularity and performance of GSA tools but also to explore possible explanations for the popularity phenomenon and the problems that limit the execution and adoption of independent performance studies. In the end, some practices are suggested to guarantee that bioinformatics software selection is guided by the most appropriate metrics.omics tools  measure of a GSA method’s popularity. 350 references to GSA papers of methods, software or platforms have been collected  report that they specifically outperform GSEA in at least one of many possible ways; such methods are highlighted on Additional file omics data analysis process may undermine the efforts on the rest of the pipeline.It is hard to estimate how many conclusions from how many articles should be rectified if we rigorously apply our current knowledge on GSA and use the best performing methods in every paper that ever used GSA. However, taking into account that our database , where prioritization is a concept related to the rank of the target pathways for a given method. Zyla et al. have recently extended Tarca's work and recommended using a set of five different metrics: Sensitivity, False positive rate (FPR), Prioritization, Computational time, and Reproducibility. In their approach, reproducibility assigned a high score to methods that showed similar results for different datasets coming from the same condition/disease and the same technical platform, but different authors/labs [A second issue is the selection of the benchmark metrics that quantitatively determine who is the best performer. Well-known metrics such as sensitivity, specificity, precision, or the area under the ROC curve, have been traditionally used. Tarca et alors/labs . The choors/labs , and theA third issue is that validation procedures should not allow the authors to subjectively choose the methods that the authors want to compare to their new method, as it would be possible for the authors to choose those methods that they can outperform. One alternative to that is using well-established, independent, and comprehensive benchmark studies as a reference; then, when a new method appears, validation should be done by comparison to the top methods from such independent benchmarks. This practice is not common; as an exception, the authors of LEGO  explicitAs a final thought, ensembles of methods have been suggested to give better results than any of the single methods . That imhttps://gsa-central.github.io/benchmarKING.html.The previous analysis makes us think that future GSA benchmarks should include the most recent developments on benchmark theory, as well as be performed on more GSA methods, in order to extract more meaningful conclusions. In order to do this, we have also created a benchmarking platform, “GSA-BenchmarKING”, which is a repository of apps/workflows/pipelines that follow the above-mentioned good benchmarking practices and allow benchmarking of GSA software in an easy and automated way. Currently, GSA-BenchmarKING contains Jupyter notebooks with full workflows for benchmarking GSA methods, and shiny apps that allow benchmarking with the click of a few buttons. The initial benchmarking tools added to the repository were created by our students as an example, following the previous guidelines, but they are open software and can be permanently improved. The initial tools are focused on two types of methods: Single-sample GSA  and genomic-region GSA . All apps allow the user to define different gold standard datasets (or use ours), select the GSA methods to compare, and select the comparison metrics to plot. In addition, it is possible to keep adding more GSA methods to the app with the help of the community, as each app includes instructions for programmers interested in adding new methods. Given the large amount of GSA methods, more benchmarking tools are needed and welcomed. GSA-BenchmarKING can be accessed at: The GSARefDB has been used to make some initial data exploration on the relationship between popularity and performance of GSA tools. Besides the observations highlighted in this article, such a database can also be the source for more in-depth research. For example, some limitations of our approach include (i) that some papers describe more than one method or platform, and (ii) that some methods are only cited for comparison purposes. We observe that such problems are the exception and not the rule, but future studies might want to take them into account and GSARefDB can still be used as a data source for it.In general terms, it is evident that performance studies are still few, small, inconsistent, and dependent on the quality of the benchmarks; however, they tend to recommend tools different to the popular and friendly ones. Among the possible reasons for the discrepancies between popularity and performance, it has been suggested that software selection may not be entirely related to performance but to factors such as a preference for user-friendly platforms or user-friendly concepts or plots . Also, due to the fact that the objective evaluation of the performance of the different GSA methods is a complicated and time-consuming issue, or to the fact that new methods need more time to be accepted. In network analysis, popularity can be explained by the “rich-gets-richer” (popular-gets-more-popular) effect. Using concepts of the “consumer behavior” field, software selection can be studied as the choice of a popular brand, that is, variables such as: Confidence in experience (the respect to the researcher/institution associated to the software), social acceptance and personal image (following the software that everyone else is using), or consumer loyalty  . In a reHowever, in the specific case of GSA, there is a complementary explanatory hypothesis: Besides all the progress in GSA theory leading to better-performing methods, at the end of the pipeline, the users usually extract the evidence that they consider relevant from the resulting rank of gene sets, choosing the pathways or GO terms of their interest and ignoring the rest of the ranking, and, therefore, any differences between methods regarding lower p-values, prioritization order, and so on, become less important. It is said that in the step of gene set selection from the final gene set ranking, researchers bring "context" to the results but, this way, their subjectivity may be projected to the study. One way out of this is to stimulate the research on “context-based GSA”. For example, a recent effort called contextTRAP  combinesGiven that popular methods are not necessarily the best, bioinformatics software users should not only be guided by popularity but mainly by performance studies. However, performance studies, in time, must be guided by the general guidelines that we have discussed, that is, researchers should only follow the most convincing benchmark procedures. Such strict recommendations are problematic because performance studies are few, low-coverage, and have a variable quality; therefore, we need more open tools to dynamically review popularity and performance of bioinformatics software, such as those introduced here.omics data, as well as more teaching of new and sophisticated methods in bioinformatics courses, more guidelines for selection of a tool, and more popularization of the functional interpretation shortcomings. Also, requesting deeper discussions of the selection of GSA methods for all biomedical papers.Having more information and discussion regarding the nature and scope of the existing functional interpretation methods of The permanent evaluation of GSA methods, including better gold standards, more and more comprehensive comparison studies, and better benchmarking practices. The fact that two prestigious computational biology journals have recently created a special edition and a collection specialized on benchmarks are welcome steps in that direction , 38.Paying attention to reproducibility and offering open code in code-sharing platforms (such as Github), containers with the specific software and library versions used in their work (such as Docker), notebooks (such as RStudio and Jupyter) including scripts with a detailed explanation of their methods, and other strategies to allow reproducibility.omics datasets in a streamlined manner.The creation of a culture of functional re-analysis of existing data using new GSA methods, as well as the computational tools to functionally re-analyze existing More rigorous validation procedures for GSA tools. Also, bioinformaticians should get as much training on scientific validation methods and tools as they get on using and building bioinformatics tools.Based on the previous results and discussion, we argue that the functional interpretation field would benefit from:https://gsa-central.github.io/gsarefdb.html and https://gsa-central.github.io/benchmarKING.html.Proper tool selection is fundamental for generating high-quality results in all scientific fields. This paper suggests that tool performance and tool selection studies, via the popularity-performance evaluation based on an exhaustive reference database, is a methodology that should be followed up, to keep track of the evolution of the tool selection issues in a scientific field. We have also introduced examples of popularity and performance-measuring software that could help making such studies easier. The reader is here invited to keep following our work for the GSA field at: The concepts involved in this study have been defined in several different ways in the literature. For example, the field under study has been called "Pathway Analysis" , "EnrichGene Set Analysis (GSA): GSA methods have been defined as a group of "methods that aim to identify the pathways that are significantly impacted in a condition under study" [r study"  or as "tr study" . More spOver-Representation Analysis (ORA): A subset of GSA methods based on comparing a list of query genes  to a list of genes in a class or gene set, using a statistical test that detects over-representation. ORA "statistically evaluates the fraction of genes in a particular pathway found among the set of genes showing changes in expression" [ression" .Functional Class Scoring (FCS): A subset of GSA methods in which the values of a gene-level statistic for all genes in the experiment are aggregated into a gene-set-level statistics [atistics , and genatistics .Pathway-Topology-based (PT): A subset of GSA methods that weights enrichment scores according to the position of a gene in a pathway. Only applies to pathway data and not to other types of gene sets.Network Interaction (NI): A subset of GSA methods that not only includes the given gene sets but also the gene products that interact with the members of such gene sets when located on top of an interaction or functional annotation network.Popularity: The frequency of use of a method or tool among members of a community.Performance: The value of a quantitative property  that measures the agreement between the method's output and either empirical data, simulated data, or the output of another method.Benchmark study: A systematic comparison between computational methods, in which all of them are applied to a gold standard dataset and the success of their gene set predictions are summarized in terms of quantitative metrics .Simulation study: A systematic comparison of computational methods based on building artificial datasets that possess the properties that we specify for them.Gold standard: A "perfect gold standard" would be an error-free dataset that can be used as a synonym for truth ; however, in practice, we are limited to use "imperfect" or "alloyed gold standards", which are datasets confidently linked to the truth, but not necessarily datasets lacking error [ng error .Google and PubMed searches of keywords such as: "Pathway analysis", "Gene set analysis", and "Functional enrichment" (approx. 10% of the records).Cross-references from all the collected papers and reviews (approx. 50% of the records).Email alerts received from NCBI and selected journals (approx. 40% of the records).The Gene Set Analysis Reference Database (GSARefDB) was built from the following sources:https://scholar.google.com). Only methods that associate omics data to annotated gene sets were included  Generic methods/software/platforms, usually dealing with mRNA datasets; (ii) Reviews/benchmark studies; (iii) Genomic GSA, which includes GSA applied to enrichment of genomic regions ; and (iv) ncRNA GSA, which includes methods dealing with miRNA and lncRNA datasets. All information in the database was manually extracted from the papers. Numbers of citations were extracted from Google Scholar Examples of application of the method followed by highlighting the reasonableness of the results (without comparing to other methods),(ii)Arguments  stating that new assumptions are better than old ones, without any comparison to empirical or properly simulated data, or(iii)Comparisons of capabilities between old and new software .The scientific validation approaches followed by the top 153 GSA tool papers in our database were manually reviewed  was created to store and share different tools to measure GSA method performance. For a benchmarking software to be accepted into the repository, it should: (i) Be open software ; (ii) Have a clear reason for selecting the GSA methods included; for example, because all of them belong to the same type of methods; (iii) Include both a gold standard dataset and options to upload user-selected gold standard datasets; (iv) Include either a list of target pathways linked to the gold standard dataset or disease relevance scores per pathway for the diseases related to the gold standard; (v) Give the user the option of selecting different benchmarking metrics ; (vi) Possibility of selecting ensemble results; (vii) Possibility of adding new GSA methods to the code in the future.The “GSA-BenchmarKING” repository (see Fig. https://github.com/gsa-central/gsarefdb, https://github.com/mora-lab/ss-shiny, and https://github.com/mora-lab/gr-shiny.The “GSARefDB”, “ss-shiny”, and “gr-shiny” apps were built using the “shiny” package. “GSARefDB” and “ss-shiny” were built using R 3.6.2, while “gr-shiny” was built using R 4.0.0. Open code can be accessed at: Additional file 1. GSA Reference DB.Additional file 2. Detailed Performance Study.


Gene expression-based multiclass prediction, such as tumor subtyping, is a non-trivial bioinformatic problem. Most classifier methods operate by comparing expression levels relative to other samples. Methods that base predictions on the expression pattern within a sample have been proposed as an alternative. As these methods are invariant to the cohort composition and can be applied to a sample in isolation, they can collectively be termed single sample predictors (SSP). Such predictors could potentially be used for preprocessing-free classification of new samples and be built to function across different expression platforms where proper batch and dataset normalization is challenging. Here, we evaluate the behavior of several multiclass SSPs based on binary gene-pair rules  and compare them to centroids built with centered or raw expression values, with the criteria that an optimal predictor should have high accuracy, overcome differences in tumor purity, be robust across expression platforms and provide an informative prediction output score.We found that gene-pair-based SSPs showed excellent performance on many expression-based classification tasks. The three methods differed in prediction score output, handling of tied scores and behavior in low purity samples. The k-Top Scoring Pairs and Random Forest approach both achieved high classification accuracy while providing an informative prediction score. Although gene-pair-based SSPs have been touted as being cross-platform compatible (through training on mixed platform data), out-of-the-box compatibility with a new dataset remains a potential issue that warrants cohort-to-cohort verification.https://cran.r-project.org/package=multiclassPairs) (https://doi.org/10.1093/bioinformatics/btab088) is freely available and enables easy training, prediction, and visualization using the gene-pair rule-based Random Forest SSP method and provides additional multiclass functionalities to the switchBox k-Top-Scoring Pairs package.Our R package ‘multiclassPairs’ (Bioinformatics online. Before such a classifier can be used, a new dataset needs to be processed and normalized to ensure that it conforms to the expected data distribution. Rule-based classifiers have been proposed as an alternative approach where non-cohort-normalized expression data  is converted into binary gene-pair rules. The concept is that characteristic gene ratios can be used to identify different subtypes. A gene-pair decision rule can be described as ‘if Gene A > Gene B is TRUE this indicates Class X, else non-Class X’. Although information carried by continuous expression values is lost when the data is featurized, these binary rules are not dependent on between-sample normalization making the classifier applicable to samples in isolation i.e., a single sample predictor (SSP). The featurizing also enables data from different platforms and preprocessing pipelines to be used during training to find class specific gene ratios captured across methods in order to create a more broadly applicable classifier, as demonstrated in earlier works   of selected genes whose expression differ between subtypes are commonly used. A new sample is classified by determining which centroid it most closely resembles, as measured by correlation or other similarity measure. Centroid variations such as Prediction Analysis of Microarrays (PAM)  advance er works . Geman eir (TSP) . Here, we.g., fresh frozen or FFPE, as well as mRNA quantification methods, and sample purity can vary broadly both due to intratumor immune and stroma levels or adjacent normal tissue in the biopsy. These factors, taken together, are challenging to overcome with standard batch correction methods and can strongly impact prediction results of a conventional centroid-based classifier. This has hindered robust validation efforts and confident adoption of any one tumor classification scheme proposed for the disease, which motivated us to evaluate alternative classifier methods that could overcome these problems.We apply these methods to bladder cancer subtype prediction and compare them to a currently utilized single sample centroid approach (https://gdc.cancer.gov)  (GSE83586) (cer.gov) , obtaineA total of 1084 samples from 18 bladder cancer datasets was used to further examine cross-platform performance . This da2.3.1P-value < 0.05) and a ‘Subtype vs Rest’ AUC above 0.6, followed by ranking on mean fold change. Log2(TPM + 1) data was used for the TCGA dataset and log2 was used for the Lund2017 dataset. The mean raw expression of the selected genes for each subtype was used as centroids. Pearson correlation was used as distance metric (1-Pearson's r) to determine the nearest centroid for each sample. We evaluated centroids of increasing sizes by gradually including more genes per class based on the Subtype versus Rest gene ranking. The smallest centroid used the top 10 upregulated and 10 downregulated genes per class, while the largest used 125 up- and downregulated genes per class, resulting in centroids ranging from 67 to 774 genes. Training accuracy is reported by a direct reapplication of the centroids to the training data.Centroids built on log-transformed raw expression data have recently been utilized as SSPs for bladder cancer subtyping  (The code for Absolute Intrinsic Molecular Subtyping (AIMS) was obtained from Github (ainAIMS) . AIMS ushttps://www.bioconductor.org/) (The k-TSP classifier R package switchBox (SB) was installed from Bioconductor (or.org/) . ReporteRandom Forest (RF) uses an ensemble of decision trees built through randomized feature selection and subsampling (bagging) , which aWe compared how conventional centroids, single-sample centroids, Absolute Intrinsic Molecular Subtyping (AIMS), switchBox k-TSPs and rule-based Random Forest performed on the 5-molecular subtype prediction task in bladder cancer.3.1.1Conventional centroids were trained on the Lund2017 and TCGA datasets separately. The Lund2017 centroids had prediction accuracies ranging between 0.83 and 0.88 when applied to the TCGA data  , while 3.1.2The AIMS method was applied with default settings . The peak average cross-validation accuracy was reached at 22 rules per subtype for both the Lund2017 (0.84) and TCGA (0.87) models . The Lun3.1.3We used switchBox (SB) to train one-versus-rest k-TSP classifiers for each subtype. The five classifiers were used for multi-class prediction by applying each classifier to a sample and calling the subtype assignment based on the highest individual classifier score. The optimal number of rules was determined independently for each classifier, meaning that a different number of rules may be selected for each class. With default settings, the Lund2017 k-TSP model had a training accuracy of 0.94 and a prediction accuracy of 0.80 on the TCGA dataset, while the TCGA model had a training accuracy of 0.92 and a prediction accuracy of 0.76 on the Lund2017 data . The modWe evaluated modifications to the gene and rule selection methods implemented in switchBox, including one-versus-one gene filtering, platform-wise gene filtering, platform-wise rule scoring and one-versus-one rule scoring. For bladder cancer subtype prediction, none of these modifications resulted in significant improvements . We exam3.1.4We evaluated seven models with different combinations of input genes (g) and rules (r) per class . The Lund2017 classifiers had OOB accuracies between 0.88 and 0.95 and prediction accuracies between 0.80 and 0.85 applied on TCGA data . The TCG3.1.5e.g., TUBB2B, CHGA, ENO2, SYP), and low expression of many epithelial genes. Notably, the OOB prediction scores of an RF model trained on the full combined dataset closely matched the OOB patterns of the individual dataset models, indicating that the score reflects true biological differences  using supplied centroid-derived reference labels. The k-TSP models had training accuracies between 0.81 and 0.88 and prediction accuracies between 0.78 and 0.85, while RF models had between 0.88 and 0.92 OOB accuracy and 0.86–0.89 prediction accuracy  and rectal adenocarcinoma (READ) merged into a COADREAD class) and performed the prediction on the remaining 5044 samples. The k-TSP methods achieved prediction accuracies between 0.89 and 0.92, while the RF approach (using 25 genes and rules per class and 200 from the ‘All’ model) had an OOB training accuracy of 0.97 and a prediction accuracy of 0.96 . We obsee.g., clinical tests or prospective molecular subtyping, where invariant predictions of individual samples are needed. Similar to relative classifiers, this is simplified if the platform remains identical. Given the large amount of publicly available transcriptomic research data across diverse platforms, a SSP with robust performance also across platforms would be highly useful. The rule-based classifiers that were trained on isolated bladder cancer datasets showed transferability between array and RNA-seq platforms, performed well when trained on two bladder cancer datasets and applied to 16 separate cohorts or vice versa, and performed well on most dataset in lung cancer. In the lung cancer evaluation, we saw platform issues with an older microarray dataset that was not represented in the training cohort. The rule-based classification approach boils down to the identification and utilization of subtype-informative gene-pair ratios that are captured across gene expression profiling methodologies. Gene-length normalized RNA-sequencing data  can be expected to better reflect the quantitative ranking of gene mRNAs compared with signal intensities from microarrays . If data from a new platform deviates greatly in dynamic range from that of the training cohort, or does not at least partly reflect true mRNA quantities, a number of rules may be affected to the detriment of the prediction results. Therefore, as with any research classifier, compatibility with new data should be evaluated and verified, particularly if the expression platform was not represented in the training data. Inclusion of mixed cohorts during training improved the cross-platform behavior of the selected rules, and this approach has been used in published rule-based classifiers such as the AIMS breast cancer subtype predictor. This could potentially be extended further using additional versions of the training data, preprocessed using different RNA-seq quantification or microarray summarizing methods to find stable rules. We found that the rule-based SSPs performed excellent when trained and tested on 50/50 data-splits of SCANB and Pan-Cancer TCGA data, indicating robust performance when applied to uniformly generated data. Prediction results in most tests were also stable even when relatively few genes were used, suggesting that these methods may be applicable to panel-based mRNA quantification methods like Nanostring nCounter. Many tumor classification systems include sample categories partly defined by their high degree of non-tumor cells. Such infiltrated categories can be relatively uninformative regarding tumor cell features. Intrinsic tumor cell properties certainly play a role in the composition of the tumor mass, e.g., by being immune-attracting or having a propensity for invasive growth, but low purity can also be caused by general inflammation or tissue sampling. Bladder cancer tissue specimens may come from papillary (exophytic), flat or inverted (endophytic) or invasive lesions, contributing to the variability of biopsy cellularity even if tumor-rich areas are microdissected for analysis. An analysis of purity across TCGA cohorts indicated that bladder cancer had among the lowest average purity scores and high variability between samples , meaning that they rely on relative expression differences between samples. Prediction results from such classifiers can be severely affected if the subtype distribution of a new cohort differs significantly from that of the training cohort. If the data from the new cohort has been generated with the exact same method it is possible to effectively normalize and integrate it for compliance with a prediction model. However, as research cohorts from different studies can be generated on diverse platforms and may differ in many other clinical, biological or methodological aspects, such normalization strategies are often challenging. A SSP is applied to a new sample in isolation without relying on cohort row-centering, effectively bypassing the issue of relative expression differences. This approach is advantageous in n purity , and expbtab763_supplementary_dataClick here for additional data file.


Glioblastoma (GBM) can be divided into subtypes according to their genomic features, including Proneural (PN), Neural (NE), Classical (CL) and Mesenchymal (ME). However, it is a difficult task to unify various genomic expression profiles which were standardized with various procedures from different studies and to manually classify a given GBM sample into a subtype.An algorithm was developed to unify the genomic profiles of GBM samples into a standardized normal distribution (SND), based on their internal expression ranks. Deep neural networks (DNN) and convolutional DNN (CDNN) models were trained on original and SND data. In addition, expanded SND data by combining various The Cancer Genome Atlas (TCGA) datasets were used to improve the robustness and generalization capacity of the CDNN models.The SND data kept unimodal distribution similar to their original data, and also kept the internal expression ranks of all genes for each sample. CDNN models trained on the SND data showed significantly higher accuracy compared to DNN and CDNN models trained on primary expression data. Interestingly, the CDNN models classified the NE subtype with the lowest accuracy in the GBM datasets, expanded datasets and in IDH wide type GBMs, consistent with the recent studies that NE subtype should be excluded. Furthermore, the CDNN models also recognized independent GBM datasets, even with small set of genomic expressions.The GBM expression profiles can be transformed into unified SND data, which can be used to train CDNN models with high accuracy and generalization capacity. These models suggested NE subtype may be not compatible with the 4 subtypes classification system.The online version contains supplementary material available at 10.1186/s12885-022-09191-2. Glioblastoma (GBM) is one of the most lethal tumors affecting human, which is the most common primary malignant tumor in brain . DespiteDeep learning has great potential to deal with complicated biological data, and has been used to recognize genetic, histopathological, and radiographic features of GBM or low-grade glioma –14. PartTotally 10 GBM datasets were used in the present study, among which 9 GBM datasets were used to train the neural networks ; Unified data transformed into “unified standardized normal distribution N” data ; a combined dataset including Unified and Validation data transformed in to SND-data (SND-Train2Sets data); a combined data including all of the 9 training datasets (SND-Train9Sets). All of the datasets can be found in online materials.Two kinds of NNs were developed: deep neural networks (DNNs) and convolutional deep neural networks (CDNNs). DNNs were composed by 4 layers, and CDNNs contained totally 5 layers: 2 convolutional layers, 1 subsampling layer, 1 density layer and 1 output layer. Large ranges of super-parameters were trained to get optimized DNNs with high accuracies, including number of iterations, epochs, dropout and momentum values, number of layers and number of nodes in each layer, activation functions in each layer, et al. For CDNNs, there are more super-parameters, including kernel size in the convolutional layers and stride values in the subsampling layer. For each kind of the above 4 training datasets, to train the DNNs, the dataset was shuffled and split into training (70%) and testing (30%) datasets. The trained DNNs was further validated in untrained datasets. For the SND-Train9Sets dataset, 10% of the randomly selected samples were first preserved as validation data , then the remaining 90% samples (Train9Sets-TrainTestData) were further split into training (70%) and test (30%) datasets to train the DNNs.Totally 6 kinds of NNs were trained in the present study Fig. : 1. a DNn. We first generated a value array containing n elements obeying the N distribution, which is denoted as N.Produce standardized normal distribution N. Let the number of genes for each sample is Rank the genes according to their internal expression levels. Order the gene expressions for each sample by their expression levels.Transform the rank values of each gene into corresponding values with the same rank value in N.The widely used q-PCR is actually a method based on ranks of gene expression levels which determines the relative expression level of genes normalized to internal reference genes such as β-Actin or GAPDH. Based on these assumptions, the gene expressions of each sample were unified into SND-data by the following procedures Fig. A:ProduceTo ensure that different samples have comparability from each other, the genes were re-ordered according to a same fixed gene order . In the present study, the RGL were defined by ordering the genes ascendingly according to their expression levels in the PN subtype in the unified dataset. If the gene in the RGL was not found in the dataset, then its value was set to a default value of 0. In addition, if the gene was not included in the RGL, then the value would be discarded.http://deeplearning4j.org).Our NNs implementation was based on the Deeplearning4J package, which is an open source, distributed deep-learning project in Java and Scala  = 91.23%. The classification accuracies for other subtypes were similarly calculated.Classification of GBM samples based on their genomic expressions had been performed by different groups primarily based on data cluster analysis. The same TCGA datasets were then classified by different researchers with certain inconsistency. Here, we investigated the common GBM samples  classified by Brennan, C. W. et al. and Ceccarelli et al. (cell2013 and cell2016 data), respectively . SimilarP-values are 2-tailed, and P < 0.05 was considered statistically significant. Statistical analysis was performed with SPSS v.13.0.0.Statistical analyses were performed using Student’s t-tests and one-way ANOVAs with least-squared-difference post-hoc tests, as appropriate. All The datasets used and DNN models investigated in the present work were elucidated in Fig. One of the major problems is that gene expressions in the above datasets were normalized with different approaches. Therefore, a key issue is to unify these datasets into a uniformed pattern. After data unification, gene expressions from different datasets should have the similar distribution. However, data unification is normally based on the distribution features of the whole dataset, making it difficult to transform and test single or a few samples. Therefore, we are trying to establish a simple unification method dependent on as few as additional data and investigate whether it can be used as input to classify the GBM samples by the DNN.To unify the gene expressions, we first observed the data distribution of the gene expression. The frequency chart analysis revealed that the expressions are unimodally distributed and similar to normal distribution . As a result, we got SND-DNNs with an accuracy of 97.10% at the testing dataset, slightly higher than that of DNN. Then we used the SND-DNN to classify the 6 original GBM datasets. The 6 GBM datasets were normalized into SND-data and then classified by the CDNN. As a result, the SND-CDNN classified these GBM datasets with accuracies between 33.5 ~ 57.87% (43.72% ± 9.25%), whose performance was improved when compared to the original-DNN  to train the dataset. Because we have addressed that SND-data had better performance, we first normalized the data as described above, after which perform CDNN training (SND-CDNN). In fact, CDNNs obtained by training the original-data but not SND-data  got an accuracy of more than 97% at the testing dataset, but these original-CDNN also have problems of generalization, which classified the other original-data with accuracies between 15.74 ~ 73.10%. Therefore, we next trained CDNNs by using the SND-data (SND-CDNN).p < 0.01, paired t-test; Supplementary Table First, the data were normalized into SND-data, after which the one-dimensional gene expression SND-data were arranged into two-dimensional data array 2D-array) with square of the gene numbers as the length of the 2D-array. The missing data in the 2D-array was set into default value of 0  to train the CDNN. Therefore, we next added the validation data to the training datasets (designated as “Train2Sets”) and trained a new CDNN with the same super-parameters (SND-CDNN-Train2Sets). As a result, we obtained SND-CDNN-Train2Sets with accuracies more than 99% at the testing dataset. Interestingly, these SND-CDNN-Train2Sets classified the 6 original GBM datasets at accuracies of 61.37 ~ 83.87% . First, importantly, these SND-CDNN-Train9Sets classified the 10% Train9Sets-validate data, which are not included in the training process either as training or as testing data, at accuracies between 83% ~ 87% Fig. A-B. Intep < 0.001, ANOVA, Fig. When we analyzed the detailed classification accuracies of the SND-CDNN-Train9Sets models to classify the 10% Train9Sets-validate data for each subtype, we noted that, interestingly, these models classified the NE subtype with an averaged accuracy of only 64.17% (55.00% ~ 70.00%), much lower than that of PN, CL and ME subtypes, which were classified at averaged accuracies of 91.23% (84.21% ~ 98.25%), 85.14% (82.43% ~ 86.49%) and 91.37% (89.16% ~ 92.77%), respectively  , which o010 , whiIn this study, we demonstrated the utility of deep learning to classify GBM subtypes. As a kind of malignant brain tumor without ideal treatment, more and more studies were performed to decipher the molecular and pathological characteristics of GBM. Molecular classification was considered to be more important than traditional pathological classifications in recent years. Based on genomic and clinical investigations, GBMs were classified into 4 subtypes with different features of genomic mutations, expression profiles and clinical characteristics, that is PN, NE, CL and ME subtypes. In addition, core gene signatures were identified to define each subtype. The initial classification of GBM subtypes were primarily based on genomic expression data clustering analysis. However, genomic based classification is mainly used for clustering analysis, and infeasible to classify single or a few samples.Deep learning is useful to classify dataset, even for complicated data, such as genome expression profile, which is infeasible for human to recognize. Another critical issue in genomic data analysis is the normalization procedure of expression values. The normalization approach used in different research groups would have inconsistences, and would result in differences in the expression profiles. For example, the same TCGA samples classified by two groups have a total consistency of about 80% . We also studied and compared the SVM model to classify the GBM data, by using a LIBSVM program . As a reAlthough exhibited better performance than the SVM models in larger datasets, the CDNN models still have some limitations. First, because the fundamental principles of the CDNN is not fully clarified, this model has little value to provide insights into the underlying biological processes, and therefore has poor interpretability when it comes to translational cancer genomics. Next, similarly, although the CDNN models had better performances than the DNN, it is difficult to demonstrate the exact meanings of the transformed two-dimensional data of the genomic profiles. Nevertheless, in practice, given that obtaining the genomic profiles of a given sample would need lower price and shorter time in the future, it would be acceptable to identify the subtype of a GBM sample based on its genomic profile. The present work provided potential ways to make this process more feasible. Specifically, the internal-rank based SND transformation provides a concise algorithm to unify the genomic data. In addition, because sample size is important for the training of deep learning models, accumulated data in the future would further improve the performance of the CDNN models.In conclusion, the present work established approaches to normalize and classify GBM samples based only on their internal ranks of the genome data. Several networks were trained on the internal rank data of the genomic profile to classify GBM subtypes with high performance. In addition, the CDNNs analysis suggested to exclude the NE subtype from the four GBM subtype classification system.Additional file 1: Supplementary Table S1. Super parameters for the DNN. Supplementary Table S2. Super parameters for the CDNN. Supplementary Table S3. Averaged Accuracies of SND-CDNN-Train9Sets to classify Train9Sets-validate data for each subtype. Supplementary Table S4. Averaged Accuracies of SND-CDNN-Train9Sets to classify the whole Train9Sets data for each subtype. Supplementary Table S5. Averaged Accuracies of SND-CDNN-Train9Sets to classify GBM datasets for each subtype. Supplementary Table S6. Accuracies of SND-CDNN-Train9Sets-IDH-WT to classify the corresponding 10% validation datasets for each subtype. Supplementary Table S7. Accuracies of SND-CDNN-Train9Sets to classify the GSE84010 dataset for each subtype. Supplementary Fig. S1. Deep neural network architecture of the DNN models. Supplementary Fig. S2. Averaged subtype classification accuracies of the SND-CDNN-Train9Sets models on the whole combined GBM dataset (Train9Sets data). * p < 0.01 compared to NE. Supplementary Fig. S3. Averaged classification accuracies of the SVM models on the 6 original TCGA datasets . Supplementary Fig. S4. Averaged subtype classification accuracies of the SVM-Train9Sets models (trained on 90% Train9Sets data) on the validation dataset (10% Train9Sets data). * p < 0.01 compared to NE.