Gene expression data from different cancer types offer the opportunity to identify cancer tissue-of-origin specific biomarkers and targets. In this study, we used pan-cancer gene expression data to train a deep learning neural network model to identify cancer tissue-of-origin specific gene expression signatures. We identified 976 genes that can reliably classify different cancer types with >97% accuracy.Cancer tissue-of-origin specific biomarkers are needed for effective diagnosis, monitoring, and treatment of cancers. In this study, we analyzed transcriptomics data from 37 cancer types provided by The Cancer Genome Atlas (TCGA) to identify cancer tissue-of-origin specific gene expression signatures. We developed a deep neural network model to classify cancers based on gene expression data. The model achieved a predictive accuracy of >97% across cancer types indicating the presence of distinct cancer tissue-of-origin specific gene expression signatures. We interpreted the model using Shapley additive explanations to identify specific gene signatures that significantly contributed to cancer-type classification. We evaluated the model and the validity of gene signatures using an independent test data set from the International Cancer Genome Consortium. In conclusion, we present a robust neural network model for accurate classification of cancers based on gene expression data and also provide a list of gene signatures that are valuable for developing biomarker panels for determining cancer tissue-of-origin. These gene signatures serve as valuable biomarkers for determining tissue-of-origin for cancers of unknown primary. The identities and phenotypes of different cell types are governed by underlying gene expression pattern. While all cells contain the same genetic information, only a subset of genes are expressed in a given cell type. Identification of unique gene expression signatures associated with different cancers are valuable as diagnostic biomarkers and therapeutic targets. For example, the expression of prostate-specific antigen (PSA) is elevated in prostate cancer patients . IdentifAdvent of computational method called deep learning has enabled researchers to utilize multilayer neural networks to analyze these large data sets. Deep learning is a cutting-edge artificial intelligence technique that provides a powerful framework for supervised learning . AccordiWe carried out a comprehensive pan-cancer gene expression analysis using deep learning to analyze transcriptome data from TCGA to identify cancer tissue-of-origin specific gene signatures. We developed a model that classified different cancer types based on transcriptomic data with >97% accuracy. Interpretation of this model revealed 976 genes that constitute cancer tissue-of-origin specific gene expression signatures including specific genes that can effectively discriminate among different cancer types. These gene signatures can be used to guide the development of more reliable gene panels for tissue-of-origin diagnosis and monitoring of cancers. In addition, this DNN model can be extended to identify tissue-of-origin for cancers of unknown primary.https://portal.gdc.cancer.gov, accessed on 20 August 2020) using the GDC data transfer tool. We analyzed only protein-coding genes. In order to reduce noise and to focus on genes that were reliably expressed, we excluded genes that did not exhibit expression levels ≥ 5 FPKM in at least 50% of samples in a given cancer type. These gene lists were then merged to create a non-redundant gene list for model construction. FPKM values were log10 transformed before using the data for model training and testing. Sample labels  were encoded using OneHotEncoder from the sklearn python package. Samples of each cancer type were randomly assigned to three data sets: (1) a training set (70%), (2) a validation set (10%), and (3) a testing set (20%).We obtained fragment per kilobase per million mapped fragments (FPKMs) values for 14,237 tumor samples representing 39 cancer types from the TCGA data portal (T-distributed stochastic neighbor embedding (t-SNE) is an unsupervised nonlinear dimensionality reduction technique . We empln is the total number cancer types, i)pq for developing, testing, and validating the model for the classification of cancer types based on gene expression data. The model architecture and training were implemented on an NVIDIA Tesla M60 GPU with 128 GB RAM using TensorFlow 2.0 and Python 3 . The DNNThe model’s performance was assessed using the most commonly used evaluative metric, i.e., accuracy. However, it tends to be biased towards over-represented classes for class imbalanced data. As such, cases existed in our data set; thus, we also used precision, recall, and F1-score values to assess model performance on the test data.To identify the most important model driving genes, we conducted model interpretation using Shapley additive explanations (SHAP) values . SHAP ishttps://dcc.icgc.org/releases/current/Projects, accessed on 26 August 2021). We first removed samples that were common between ICGC and TCGA data sets. We also discarded normal tissue samples. Additionally, we eliminated Ewing’s sarcoma (BOCA-FR) and pediatric brain tumor (PBCA-US) cohorts. As the model was not trained using Ewing’s sarcoma and PBCA-US data with transcript-level expression. In total, there were 2085 samples spanning across 18 cancer types in the resultant data set. ID mapping was performed using ensemble BioMart and HGNC Multi-symbol checker tools. Read count was set to zero for missing genes. Before using the data for model testing, the FPKM values were calculated and log10 transformed.To test the model on an independent data set apart from TCGA, we downloaded gene expression data pertaining to the latest release from the ICGC data portal (The backend was built using Amazon Web Services (AWS), while the user interface was designed using Angular framework, CSS, and HTM5. Simple Storage Service (S3) buckets were used to store input and output files. A docker image was created to analyse the input data. It was configured to support the scikit-learn and TensorFlow2 framework using python 3.7. Elastic Container Repository (ECR) was used to deploy the docker image. A lambda function was created to communicate between the S3 bucket and ECR docker image. The frontend and backend were linked using the REST API.We used transcriptome data from 14,237 tumor samples from 39 cancer types available on the TCGA data portal, including adenocarcinomas, squamous cell carcinomas, haematological malignancies, neuronal tumors, melanomas, germ cell tumors, soft tissue tumors, and other cancer types . The datRNA-seq is one of the most powerful techniques for gene expression profiling. However, accurate noise-free quantification remains a challenge . Deep leWhen log-transformed data was used for the t-SNE unsupervised clustering, it exhibited better separation between cancer types B. As expWe developed a pan-cancer classification model using deep neural networks (DNN) and RNA-seq data from TCGA. We opted for DNN based architecture as it is better suited to learning latent patterns in data where input variables are independent of one another. Our first DNN model was composed of 5 dense hidden layers with 50 nodes, and its accuracy was below 40%. The learning capacity of the model was improved by adjusting the number of hidden layers and nodes which resulted in improved model accuracy of over 95%. Then we performed a grid search to identify the best optimization algorithm and learning rate using TensorBoard HParams . Our finTo evaluate the performance of DNN model, we used it to classify a test data set containing gene expression data from 2848 samples corresponding to 37 different cancer types. The model accurately predicted cancer type for 2772 tumor samples and misclassified 76 samples . The ovex-axis and predicted cancer type is represented on the y-axis. Number of samples in the test data set that were accurately classified are represented on the diagonal.Actual cancer type is represented on the The model performed poorly when classifying CHOL samples, with a test accuracy of 71.42%. This may be due to the limited number of samples in the training data set and the anatomical location of this cancer type. Small sample sizes hinder model learning and result in the construction of a poor classifier. CHOL originates from the liver and performing a biopsy of these tumors without liver cell contamination is non-trivial. Hence, CHOL samples often exhibit gene expression profiles similar to those associated with LIHC. This was also evident from t-SNE unsupervised clustering results where these two cancer types formed proximal clusters B. Even tClassification ambiguities were also observed among different cancers arising from the same organ systems. For example, three acute lymphoid leukemia (ALL) samples were predicted as AML samples, and four lung adenocarcinoma (LUAD) samples as LUSC. Same was observed for kidney renal papillary cell carcinoma (KIRP), KIRC, KICH, CESC, UCEC, STAD, and ESCA samples . ClassifWe performed stratified five-fold cross-validation of our model by randomly dividing TCGA data into five data sets. Each time, we used one of those data sets as a test set and the remaining four as a training set for the model. Our model achieved consistent results in this cross-validation analysis, with a training accuracy of more than 99% . PrecisiWe next used SHAP values to interpret our model to identify gene signatures that drive classification . To idenTo evaluate the validity of gene signatures identified through our model interpretation, we built a new DNN model using expression profiles of only these 976 gene signatures. The model architecture remained the same except for the input layer, which was reduced from 13,250 to 976. The model exhibited consistent performance comparable to the original model and achieved 99% and 97% training and testing accuracy, respectively. Similarly, a weighted average of precision, recall, and f1-score values were consistent with the original model (97%). The model achieved consistent performance even during five-fold cross-validation . We perfTo validate the findings on an independent data set, we used ICGC data. The ICGC test data was composed of primary and metastatic tumor samples from 18 cancer types including AML, BRAIN, BRCA, CESC, CORE, HNSC, KIRC, KIRP, LIHC, LUAD, neuroblastoma (NHL), OV, pancreatic adenocarcinoma (PAAD), PRAD, SKCM, STAD, THCA and UECE. The DNN model correctly predicted cancer types for 1905 out of 2085 samples. The overall accuracy of the model was 91.37%. Interestingly, our model predicted the correct cancer type for 348 of 395 metastatic samples (88.10%) and classified the following eight cancers with 100% accuracy: AML, BRAIN, HNSC, KIRP, LUAD, PRAD, THCA and UCEC. The model accuracy was below 90% for just four cancers, which included BRCA (89.21%), CORE (80%), PAAD (73.84%), and CESC (33.33%). Most of the misclassified PAAD samples were derived from the PACA-CA cohort (74/102), of which 15 and 59 samples had 14 and 35 missing gene signatures, respectively.+/KRT20−, whereas prostate cancer cells are negative for both KRT7 and KRT20 [Keratin staining is widely used in cancer diagnosis. Most epithelial cells express keratins. The human genome encodes 54 different keratins, some of which exhibit a cell/tissue-specific expression pattern . Owing tnd KRT20 ,30. Manynd KRT20 , whereaswww.deepcap.org, accessed on 26 August 2021). It allows users to upload gene expression data and obtain cancer classification. AWS cloud services, such as simple storage system (S3), elastic container repository (ECR), and lambda, were used to implement a serverless architecture  cancer-type predictions and (2) t-SNE analysis. For a given sample, the DNN model assigns a probability value to each of the 37 cancer types. The cancer type of the input sample is the one with the highest probability value. Deepcap provides a bar chart to compare probability values . A dropdThe bar chart shows cancer-type probabilities predicted by the DNN model prediction for a given sample, while the scatter plot represents a 2D visualization of the user’s data with respect to TCGA data.One of the primary goals of cancer genomics and transcriptomics studies is identification of cancer biomarkers and therapeutic targets. Availability of cancer genome and transcriptome sequencing data from multiple cancer types has enabled pan-cancer analyses to identify cancer tissue-of-origin specific biomarkers and therapeutic targets. In this study, we utilized transcriptomic data from 14,237 samples from the TCGA database to carry out pan-cancer analysis and identified gene expression signatures associated with different cancer types. We used a deep learning approach for pan-cancer analysis and identified cancer-specific gene expression signatures. Our model was capable of predicting cancer types with greater than 97% accuracy based on gene expression data alone. Classification ambiguities were primarily observed when comparing different cancer subtypes within a given organ system such as tumors of the blood, kidney, lung, and gastrointestinal organs. This is likely attributable to the high degree of molecular similarity between different cancer types originating from the same tissues. In this study, we successfully implemented the SHAP method and identified a comprehensive list consisting of 976 pan-cancer gene expression signatures. These gene signatures were evaluated for their ability to accurately identify different cancer types. The model showed consistent performance, achieving over 97% accuracy on an independent test data set. Statistical measures that are not biased towards over-represented cancer types, including precision, recall, and f1-score values, were also above 97%, and the five-fold cross-validation of our model yielded consistent results. Furthermore, to assess the generalizability of the model, we used ICGC data as an independent test data set from TCGA. The model classified ICGC data with a 91.37% accuracy. The ICGC data set consisted of 395 metastatic tumor samples of which 348 were correctly classified. In other words, our model was capable of predicting tissue of origin for metastatic tumors with 88% accuracy. Additionally, we developed Deepcap, a cloud-based platform to make our deep learning model available to the scientific community. It predicts the cancer type of a given sample from the gene expression profile. It also performs unsupervised nonlinear dimensionality reduction to visualize input data in relation to TCGA data.Our analysis resulted in the identification of a number of cancer biomarkers across different cancer types. These included several known markers confirming the validity of our approach. In addition, we identified several novel biomarkers that exhibited a cancer tissue-of-origin specific expression pattern. Such signatures are useful for the development of gene panels that can be used to screen and diagnose cancers. The model and gene signatures presented in this study can also be used to identify the tissue of origin for cancers of unknown primary (CUP). CUP accounts for 2–5% of all cancer diagnoses and represents a major area of unmet clinical need. Our list of cancer tissue-of-origin specific gene signatures can therefore be used for a number of applications including elucidation of novel therapeutic targets.https://www.fda.gov/medical-devices/vitro-diagnostics/nucleic-acid-based-tests, accessed on 28 November 2020). There were only twelve genes that overlapped between our list and the FDA-approved marker list. All 12 genes were part of FDA-approved tests for breast cancer offered by Veridex, LLC (one gene), Agendia BV (three genes), and Nanostring Technologies (eight genes). However, only two of those 12 genes  were predicted as breast cancer signatures in our data set. The remaining 10 genes were part of the gene signature of other cancers. For example, MLPH and MIA genes were identified as signatures associated with UVM and SKCM. Both of these genes showed higher expression in UVM and SKCM compared to breast cancer, suggesting they are better markers for melanoma.We compared cancer tissue-of-origin specific gene expression signatures identified in our study with the list of United States Food and Drug Administration (FDA)-approved gene expression-based in vitro cancer diagnostic tests (There are some limitations associated with our pan-cancer analysis-based assessment of cancer tissue-of-origin specific gene expression signatures . For exaIn conclusion, we implemented deep neural networks to accurately classify pan-cancer gene expression data. We then implemented the SHAP, a model agnostic method, to identify cancer tissue-of-origin specific gene signatures. The gene signatures showed a cancer tissue-of-origin specific expression pattern. In addition, the model based on gene signatures showed consistent performance during five-fold cross-validation. The cancer tissue-of-origin specific gene expression signatures identified through this pan-cancer analysis may represent valuable biomarkers and therapeutic targets that can guide future patient diagnosis and care. Moreover, the performance of the gene signature model was promising on ICGC metastatic data. In the future, it can be subjected to transfer learning using a large metastatic tumor gene expression data set to predict tissue of origin of CUPs.


Pooling metabolomics data across studies is often desirable to increase the statistical power of the analysis. However, this can raise methodological challenges as several preanalytical and analytical factors could introduce differences in measured concentrations and variability between datasets. Specifically, different studies may use variable sample types  collected, treated, and stored according to different protocols, and assayed in different laboratories using different instruments. To address these issues, a new pipeline was developed to normalize and pool metabolomics data through a set of sequential steps: (i) exclusions of the least informative observations and metabolites and removal of outliers; imputation of missing data; (ii) identification of the main sources of variability through principal component partial R-square (PC-PR2) analysis; (iii) application of linear mixed models to remove unwanted variability, including samples’ originating study and batch, and preserve biological variations while accounting for potential differences in the residual variances across studies. This pipeline was applied to targeted metabolomics data acquired using Biocrates AbsoluteIDQ kits in eight case-control studies nested within the European Prospective Investigation into Cancer and Nutrition (EPIC) cohort. Comprehensive examination of metabolomics measurements indicated that the pipeline improved the comparability of data across the studies. Our pipeline can be adapted to normalize other molecular data, including biomarkers as well as proteomics data, and could be used for pooling molecular datasets, for example in international consortia, to limit biases introduced by inter-study variability. This versatility of the pipeline makes our work of potential interest to molecular epidemiologists. Metabolomics is a powerful tool for investigating candidate etiological pathways for chronic diseases ,2,3,4. Up metabolites (or features) measured in n samples has been generated, pre-processing usually involves (i) feature and sample filtering, where low-quality features and samples are excluded, (ii) data imputation, to take care of missing values, and (iii) data normalization, to correct for sources of unwanted variation in metabolomics data, such as batch effects and other factors related to the handling of samples [As with other-omics technologies, pre-processing metabolomics data is critical before relating them to phenotypes, such as cancer endpoints or lifestyle exposures ,12. Afte samples ,14,15,16 samples . CorrectData on common quality controls assayed in all studies and/or reference assay data from a subset of samples in each study can be used for normalization ,18. Howen = 3172 samples) [n = 1706) [n = 112), liver cancer  [n = 1213) [n = 6020) [n = 946 and n = 2295, respectively). As displayed in n × p) matrix with n = 16,060 samples and the p = 118 metabolites measured in all studies. Specifically, they included 13 metabolites (amino acids) measured by a quantitative LC-MS/MS method (liquid chromatography coupled with tandem mass spectrometry) and 105 metabolites  acquired by a semi-quantitative FIA-MS/MS method . Targeted metabolomics data acquired within the EPIC study and centralized at the International Agency for Research on Cancer (IARC) included 16,060 pre-diagnostic blood samples originating from eight case-control studies nested within EPIC (details in samples) , endomet = 1706) , gallblan = 596) , kidney  = 1213) , prostat = 6020) , and colFor the exclusion of metabolites and samples exceeding a given threshold of missingness, we applied the method described in As displayed in Based on PC-PR2 analysis, metabolite concentrations were normalized using the method described in Intraclass correlation coefficients (ICC) were computed for each metabolite to assess their technical reproducibility, using measurements from 2 × 219 = 438 duplicate samples, i.e., samples measured once in two different studies , suggesting that quantified metabolites might be less prone to unwanted variations due to analytical factors. Our normalization approach eliminated most of the inter-study and inter-batch variability and improved the technical reproducibility of a large proportion of semi-quantified and quantified metabolites, with most ICC values greater than 0.75. Normalization using the ComBat approach , which rAs samples within each individual EPIC study were all assayed in the same laboratory with the same LC-MS instruments, and mostly originated from the same blood matrix (except for GLBD that included serum and heparin plasma samples and BREA that included EDTA and citrate plasma samples), the variability due to these factors was encompassed into the inter-study variability and could not be assessed by the PC-PR2 analysis. In particular, although the large inter-study variability in the non-normalized data supported the presence of inter-laboratory and inter-instrument variability, as previously reported for the AbsoluteIDQ p180 kit , correctA critical step of normalization procedures that use linear mixed models, or more generally models with location/scale adjustments , is the In multicenter investigations like EPIC, study center is a sensitive variable as it expresses technical  variation, likely the result of specific procedures for blood collection, sample treatment, and storage, as well as biological variation reflecting specific lifestyle exposures, often characterized by geographical gradients. In addition, in a multicenter context, the relationship between two sets of variables could be evaluated at the overall level, at the center level, or at the individual level . In thisThe decision to implement data normalization largely depends on the ultimate objectives of the analysis. As the relationship between metabolites and cancer risk is generally quantified in conditional logistic regression models for matched case-control studies, metabolite measurements are compared within each matched case-control pair. If cases and controls are assayed within the same batch (as was the case in the EPIC metabolomics data), the effects of study and batch on the means of the measurements are not a concern and normalization is not required unless the variances of the measurements also vary across studies or batches. However, if the evaluation focuses on the investigation of lifestyle determinants of metabolomics data, as for example in mediation analysis, the matching is “broken” and control for inter-batch and inter-study variability is required .K-nearest neighbors approach has been shown to perform particularly well for the imputation of missing data [Although originally developed for the normalization of metabolomics data acquired in different studies, our pipeline could be used for data acquired in a single study, for example to correct for inter-batch variability while preserving biological variability and to correct for potential heteroscedastic structures of concentration levels across batches. Our pipeline could also be adapted to the normalization of biomarker data and other molecular data, possibly with some modifications. In particular, for the normalization of untargeted LC-MS metabolomics data, a step to exclude features based on comparison with blank samples should be added to the data cleaning , and a King data ,30 in thing data .With the increasing availability of metabolomics data in large scale epidemiological investigations, such as those participating in the COnsortium of METabolomics Studies (COMETS) , poolingEPIC is a large prospective study of over 500,000 men and women recruited in 1992–2000 in 23 centres in 10 European countries [All participants provided written informed consent to participate in the EPIC study. This study was approved by the ethics committee of the International Agency for Research on Cancer (IARC) and all centers.p metabolites acquired on n samples, our pipeline implemented four main steps, as summarized in https://code.iarc.fr/viallonv/pipeline_biocrates, accessed on 14 September 2021.Given a matrix of b% and Nb the total number of batches. Samples assayed in batches with less than 10 samples were also excluded to ensure enough information during batch-specific data imputation  criteria. First, the pipeline excluded metabolites and samples exceeding a certain threshold of missingness  in each study separately. Missing values were here defined as fully missing values, for which no information on the real value was available. In particular, they did not include out of measurable range values, which corresponded to values that were missing because they were below the batch-specific limit of detection (LOD), below the kit-specific lower limit of quantification (LLOQ), or above the kit-specific upper limit of quantification (ULOQ). An extra step was implemented to exclude outlying samples within each batch based on principal component analysis (PCA) , using aputation  and normputation .All missing values, including the out of measurable range values, were imputed in the cleaned dataset in each batch separately. Values below batch-specific LOD, below kit-specific LLOQ, or above kit-specific ULOQ were set to LOD/2, LLOQ/2, and ULOQ, respectively. Values below an unknown batch-specific LOD were set to LOD/2 after setting batch-specific LOD to study-specific medians of known LOD values. Fully missing values were set to the batch-specific median of non-missing values if less than 50% of the measurements in the batch were missing and to the study-specific median of the batch-specific medians otherwise. Measurements were log-transformed to reduce skewness.1,…, WQ, in multiple linear regression models, and the partial R2 for each covariate Wq was estimated for each component (Ck). For example, the partial-R2 for W1 conditional on the (Q-1) other covariates for component k wask|Wj,…, WQ) expressing the residual sum of squares in the linear regression model of component Ck on variables Wj,…, WQ, for j = 1, 2. For variables with a nested structure, for example study (S) and batch within study (B), the formula wasThe PC-PR2 technique was used to identify main sources of variation in the metabolomics data . The PC-2partial (W1) was obtained by the average of terms R2partial,k (W1) weighted by the eigenvalue of each component. This overall estimate provides a measure of the variability in the ensemble of metabolite concentrations that each explanatory variable contributes to explaining. The PC-PR2 technique is implemented in the pcpr2 R package available on GitHub.An overall Ry is the n-vector of the measurements for the metabolite/feature under consideration , the matrix X expresses variables corresponding to sources of variations that should be corrected for, and the optional matrix Z expresses variables corresponding to biological variations that should be preserved. Variables expressed in matrices X and Z typically include some of the variables W1, …, WQ of the PC-PR2 analysis with largest R2partial. The vector of parameters β associated to matrix X may include both fixed- and random-effects, while the vector θ associated to matrix Z contains fixed effects only. Parameter α is the intercept and vector ε~ Nn corresponds to the random error of the model. Residuals ε are independent of the random effects of the model. Random effects are Gaussian, centered, and were further assumed to have diagonal covariance matrix in our illustration.In order to correct for unwanted sources of variability while preserving biological variability, a random effects model was used for each metabolite separately , asy = α,β, θ, and the vector of residuals ε under model (1) are estimated by, say, a, b, c, and e. Normalized residual measurements are computed asParameters Z, while any association with variables in X is eliminated. As mentioned in the Discussion, variables describing biological variations of interest should be included in matrix Z if they are associated with variables included in matrix X ; otherwise, some of the variation they explain would be removed because of the adjustment for X. In our illustration, study center indicators, gender, body mass index, and alcohol intake were included in matrix Z, while batch and study indicators were included in matrix X.In this way, normalization preserves the association between the metabolite and variables in ε of residuals has the same variance: Σ = σ2nI for some σ2 > 0, where we denote by pI the identity matrix of size p for any positive integer p. However, in practice, pre-analytical and/or analytical factors may not only influence the means of the measurements via the term Xβ in model (1), but also their variance. For example, variances of components of ε may vary across studies. This was accounted for by working under heteroscedastic random effect models with a specified structure for the variance matrix Σ of the residuals, e.g., Σ was made of blocks of the form σs2nsI for observations corresponding to study s (with ns the number of observations in study s). Then, residuals e were replaced by the Pearson residuals in Equation (2), after rescaling them to ensure that their overall variance equals that of the standard residuals. Homoscedastic models were implemented with the lmer function of the lme4 R package, while heteroscedastic models were implemented with the lme function of the nlme R package, using the weights instruction to specify the within-group heteroscedasticity structure.In the simple homoscedastic random effect models, each component of the vector K = 2 principal components of the PCA; other choices for the value of parameter K were considered and led to similar or lower reproducibility  is a subject-specific random effect , ik~N is the residual random error for replicate k of subject i, and Cov = 0. Under this model, ICC = Var(Iγ)/Var(I +γikξ), so the ICC estimate was defined as the ratio of the estimated between-subject variance to the estimated total variance (between- and within-subject). Model (3) above can be estimated even if only a portion of the subjects have replicated samples. It was implemented using the lmer function of the lme4 R package, and 95% confidence intervals (CI) of the ICC values were derived using the parametric bootstrap implemented by the bootMer function of the lme4 R package.The EPIC data included duplicate samples corresponding to aliquots of a baseline blood sample from the same subject measured twice in different batches or in different studies. These duplicated samples were used to assess the technical reproducibility of metabolomics measurements, and in particular to compare technical reproducibility before and after normalization. In sensitivity analyses, ICC values were estimated using only duplicate samples originating from distinct blood matrices (serum and citrate plasma). For each metabolite, we estimated its ICC using a linear mixed effects model of the form mik = γi


In such a case, the BC and ROI sequences, their possible mutant variants, and unambiguous BC–ROI associations have to be identified, whereas all uncertain cases have to be excluded from the analysis. Besides the preparation of additional “mapping” samples for NGS, this also requires specific bioinformatics tools. Here, we present a pipeline for processing raw MPRA data obtained by NGS for reporter construct libraries with a priori unknown sequences of BCs and ROIs. The pipeline robustly identifies unambiguous  BCs and ROIs associated with them, calculates the normalized expression level for each BC and the averaged values for each ROI, and provides a graphical visualization of the processed data.Massively parallel reporter assays (MPRAs) enable high-throughput functional evaluation of numerous DNA regulatory elements and/or their mutant variants. The assays are based on the construction of reporter plasmid libraries containing two variable parts, a region of interest (ROI) and a barcode (BC), located outside and within the transcription unit, respectively. Importantly, each plasmid molecule in a such a highly diverse library is characterized by a unique BC–ROI association. The reporter constructs are delivered to target cells and expression of BCs at the transcript level is assayed by RT-PCR followed by next-generation sequencing (NGS). The obtained values are normalized to the abundance of BCs in the plasmid DNA sample. Altogether, this allows evaluating the regulatory potential of the associated ROI sequences. However, depending on the MPRA library construction design, the BC and ROI sequences as well as their associations can be  Although numerous regulatory elements have been identified in eukaryotic genomes , so far cis-paired ROI variants using next-generation sequencing (NGS) ], whereas ROIs are typically outside this unit . As a reng (NGS) . For thang (NGS) . More spIt should be noted that ROIs can be either (i) preselected native, mutant, and/or synthetic sequences  usually of the same length  or (ii) a priori known or not. Completely predetermined MPRA libraries are generated by using sequences synthesized on custom high-density DNA microarrays  robustly identifies unambiguous (hereafter genuine) BCs and their mutant variants as well as associated ROIs, (ii) calculates the normalized expression level for each genuine BC and the averaged values for each ROI, and (iii) provides a graphical visualization of the processed data. The functionality of the pipeline was demonstrated using a data set obtained for an MPRA library designed to study the effects of sequence variations located at a certain distance downstream of the transcription termination site (TTS) of the eGFP reporter on its expression at the transcription level.The MPRAdecoder pipeline described in this study was developed for the processing of NGS data generated for MPRA libraries with eGFP reporter, was generated earlier  are located in 3′ UTR and downstream of TTS, respectively  names and lengths of all expected parts in the mapping and normalization/expression reads for each MPRA library (including indexes), (ii) sequences of the predetermined parts , (iii) relative orientation of BC sequences in mapping and normalization/expression reads, (iv) a maximum allowed error rate and the Phred quality score threshold for different parts, (v) a minimum number of read counts required for a BC and a BC–ROI association, and (vi) settings for identification of genuine BCs and associated ROIs, are specified in the configuration file. A detailed description of this file is available on the GitHub page of this project. Second, a user has to manually input the following details in the command prompt: (vii) names of the appropriate fastq file(s) and their locations as well as a location for output files, (viii) a number of replicates of each sample for each MPRA library, (ix) names of indexes used for sample multiplexing and (x) information on whether the fastq file(s) should be demultiplexed by the pipeline.The pipeline is able to process either fastq files that are already demultiplexed, for example, by the Illumina software, or fastq files containing custom-designed index sequences at the beginning of the reads. In the latter case, detection of a predetermined index sequence in each read is performed using a pairwise sequence alignment tool from Biopython . For thaDetection of the mCP1, mCP2, mCP3, neCP1, neCP2 , and ref6 to calculate the reads per million (RPM) values. After that, unique BC–ROI associations and BCs are assessed for reproducibility and robustness. Although preliminary results can be obtained using single replicates of the mapping, normalization, and expression samples, at least two replicates of each sample are strongly recommended. Under such conditions, only the BC–ROI associations that are revealed with at least m raw read counts  in at least two out of any available number of replicates of the mapping data are retained for further analysis. Also, only the BCs with n raw read counts  in each replicate of the normalization data are kept. For the expression data, the threshold read count e is set by default to zero, as some BCs might be present with very low frequency or even completely absent in the reporter transcripts due to the properties of particular ROI sequences. The threshold values m, n, and e are arbitrarily set in the configuration file. Finally, a list of BCs that are common for all samples is generated considering that some BCs might have zero counts in some or all replicates of the expression data.At the next step, the number of supporting reads for each BC (with a random or reference sequence) is counted for each replicate of all samples. Then, these numbers are divided by the total number of effective reads  in a replicate and multiplied by 1 × 109 to 1 × 1012 unique BCs, some of which might be different from each other just at one position. However, in practice, the size of a typical MPRA plasmid library is significantly less (by orders of magnitude) than the theoretical values. Nevertheless, in MPRA data sets, BCs with similar sequences do appear, partly due to errors introduced during PCR amplification and NGS steps. Thus, there is a need to find similar BC sequences, group them, and identify the genuine BCs in each such group (referred to below as a cluster). Two BC sequences are considered to be similar if they differ at no more than s positions , where s is equal to the maximum allowed error rate for this part. By default, up to two mismatchesOligonucleotides with a totally randomized part  of 15–20 nts in length can ensure cloning of ∼1 × 10are allowed for BCs of the example data set, as suggested previously .s+ 1) rounded down to the nearest whole number. Next, BCs sharing each particular k-mer are directly compared by using the pairwise sequence alignment (see above), taking into account their normalized read counts . Then, similar BCs are grouped into clusters, and a number of quality control steps are applied to ensure the absence of overlap between the clusters (ambiguous cases are removed).Because identification of similar BCs by the means of alignment approaches is rather time-consuming, especially for thousands or even millions of sequences to compare , the MPRs + 1 position(s) .After that, for each cluster, it is verified whether the most abundant ROI associated with the most abundant BC is supported by the majority of normalized read counts obtained for all ROI sequences present in a cluster . As a deOnce genuine BCs are identified, their RPM values in the normalization and expression replicates are averaged. Next, for each genuine BC, the normalized expression value is calculated as a ratio between its expression and normalization RPM values. Then, if reference constructs were spiked in the plasmid library, the pipeline can further normalize data by dividing them by the value obtained for one of these references . After that, values obtained with different genuine BCs but for the same ROI sequence are averaged. The raw and normalized read counts per unique BC–ROI association for each replicate of the mapping samples and per unique BC for each replicate of the expression and normalization samples, the RPM values averaged over these replicates as well as the ultimate expression values obtained for genuine BCs after each step of the data normalization and averaging are saved as tab-delimited text files. Also, the important details of data processing are reported in additional files. Among them are the numbers of allowed mismatches in the expected parts of the reads; the list of input fastq files used for a run; and statistics on (i) total and effective read counts per fastq file, (ii) numbers of unique and genuine BCs, and (iii) numbers of genuine BCs per ROI.Finally, the pipeline generates a number of plots to help evaluate data quality and interpret the results . In partThe pipeline can process 1 million reads of a non-demultiplexed fastq file in ∼20 min using the hardware and software specified in Materials and Methods. For larger data sets, the processing time can be estimated by assuming a linear dependence on the read number.cis-regulatory DNA elements in a high-throughput manner  . In addition, such design allows excluding as much as possible mutant or just very similar BC sequences from the analysis. Namely, only such BCs (referred to as genuine)  are usednce ≥ 3) , but nevThe use of oligonucleotides with randomized stretches of nucleotides to clone the ROIs and BCs as well as regular primers to amplify the mapping, normalization, and expression samples means that the following considerations should be taken into account during the processing of raw MPRA data. First, although synthetic oligonucleotides are purified by polyacrylamide gel electrophoresis (PAGE) or high-performance liquid chromatography (HPLC), their actual length in the preparation may vary due to the presence of deletions (more often) and insertions (less often) . Second,m and n), is associated with the fact that, when performing the mapping procedure, it is more important to identify the fact of different BC–ROI association(s) although data from the normalization samples are eventually quantified. Moreover, both of these parameters, as well as the parameter e, which determines the minimum number of reads for each unique BC in replicates of the expression samples, largely depend on both the complexity of a particular MPRA library (the number of unique clones in it) and the sequencing depth of the samples. Fourth, the threshold level of 0.9 controlling the identification of genuine BCs can be increased if necessary. This parameter is also highly dependent on the expected number of unique BC–ROI associations in the samples and their sequencing depth.Because many of the pipeline settings are arbitrary (set in the configuration file), it is important to note the following. First, of course, it is possible to set the allowed error level for all expected parts of reads to 0%; however, in the case of the example data set, this leads to a decrease in the number of genuine BCs by more than two times compared with the default settings described above. Second, because it is well known that the quality of sequencing gradually decreases toward the end of the reads, it seems appropriate to map the mCP3 and neCP2 regions in the reads not completely, but only by their beginnings. In particular, the use of only 10 instead of 17 nts for mCP3 and 20 instead of 86 nts for neCP2 for the example data set ultimately makes it possible to detect more than ∼1.5 times more genuine BCs with the error level in all parts of the reads set to 0%, but this gives only negligible gain (<0.1%) with the default settings described above. Third, the difference in the number of minimum reads, in which unique BCs should be detected in replicates of the mapping and normalization samples (parameters Although it is strongly recommended to obtain at least two biological replicates of the mapping, normalization, and expression samples, we notice that the pipeline nevertheless can process single replicates of these samples as well. This option can be useful when performing pilot experiments for a quick and preliminary evaluation of the results. Also, it is possible to load raw data obtained for different MPRA libraries into the pipeline simultaneously.Finally, the results obtained for the example data set  indicatehttps://github.com/Code-master2020/MPRAdecoder. The example input data as well as expected outputs are included in the GitHub repository. Detailed information on program can be found in the GitHub repository.The MPRAcode code written in Python is publicly available at AL and AP conceived the study. AL, EO, and AI developed the pipeline. EO and AL performed experiments and applied the pipeline to the obtained data sets. AP supervised the project. AP, EO, and AL wrote the manuscript. All authors contributed to the article and approved the submitted version.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


The genomics data analysis has been widely used to study disease genes and drug targets. However, the existence of missing values in genomics datasets poses a significant problem, which severely hinders the use of genomics data. Current imputation methods based on a single learner often explores less known genomic data information for imputation and thus causes the imputation performance loss.In this study, multiple single imputation methods are combined into an imputation method by ensemble learning. In the ensemble method, the bootstrap sampling is applied for predictions of missing values by each component method, and these predictions are weighted and summed to produce the final prediction. The optimal weights are learned from known gene data in the sense of minimizing a cost function about the imputation error. And the expression of the optimal weights is derived in closed form. Additionally, the performance of the ensemble method is analytically investigated, in terms of the sum of squared regression errors. The proposed method is simulated on several typical genomic datasets and compared with the state-of-the-art imputation methods at different noise levels, sample sizes and data missing rates. Experimental results show that the proposed method achieves the improved imputation performance in terms of the imputation accuracy, robustness and generalization.The ensemble method possesses the superior imputation performance since it can make use of known data information more efficiently for missing data imputation by integrating diverse imputation methods and learning the integration weights in a data-driven way. With the coming of biotechnology era, a lot of gene expression data are generated by DNA microarray technology to measure the expression levels of genes . The anaGenetic data is marked as missing values when the detected signals are very weak or far apart from biological knowledge. That happens due to various factors in the microarray experiment, such as the contamination of microarray surfaces, inappropriate manual operations, insufficient resolution, and systematic errors during the laboratory process, etc. –12. MissSo far, many efforts have been made to develop effective imputation methods for missing values in genomics –16. The Unlike KNNimpute, a set of neighboring genes were selected by Pearson correlation for a target gene in local least square imputation (LLSimpute) , and theIn , missingk most significant eigenvalues. In comparison with KNNimpute, SVDimpute is relatively sensitive to the missing rate and noise in the data.The above imputation methods have one thing in common, namely, only local similarity structure in gene data set is explored for missing value imputation. On the contrary, some research efforts were made to develop global imputation methods. For example, in singular value decomposition based imputation method (SVDimpute) , the misk principal axis vectors was built to model the missing data, and the model parameters were estimated within the framework of Bayesian inference. It has been shown that BPCA outperforms the representative methods mentioned previously. However, the shortcoming of BPCA is that it is difficult to determine the number of principal axes. In [In Bayesian principle component analysis (BPCA) , a probaaxes. In , the misThe support vector regression for imputation (SVRimpute) was first developed in , where rThe imputation method based on multilayer perceptron networks  was proposed in . The metA category of hybrid imputation methods was developed by combining local and global learning methods. A typical method is named LinCmb , where tThere are some works focusing on incorporating the relationships between diverse omics data or biological knowledge for the imputation. In , for impThe biological knowledge, such as the functional similarities of genes, the regulatory mechanism, information from multiple external data sets, was applied to the missing data imputation in –40. TheyNotice that most of the existing imputation methods make use of only a certain characteristic of the genetic data to impute the missing values, resulting in the weak generalization or even the database-dependent performance. To solve the problem, a comprehensive method based on ensemble learning is proposed in this paper. First, a set of representative single imputation methods are built and individually applied for predicting the missing values with the use of the bootstrap sampling. Then, the predictions output by all the individual predictors are combined into the final prediction using weighted average. And the weights for the linear prediction model are learned by using this model to estimate known gene data and minimizing the imputation errors. The proposed method has two prominent advantages: (1) more information from known genomics data is allowed to be used for the performance improvement; (2) the good generalization can be achieved by a weight learning approach involved in the training procedure.A basic framework for the ensemble learning based imputation method is proposed, where bootstrap sampling is introduced to train a set of base predictors, and the base predictors are integrated by the weighted average. On the framework, a strong predictor can be derived by the combinations of weak base predictors.The learning scheme of the combination weights is provided for the ensemble imputation. In this scheme, a linear regression model is built for the combination weights, and the expression of the optimal weights is derived in closed form.A specific ensemble imputation method is carefully described, including the choice of base predictors and the generation of multiple implementations for each predictor. The proposed method is extensively tested on several typical genomic datasets, and compared with the state-of-the-art imputation methods. The experiments confirms that our method achieves the significant performance improvement.The remainder of this paper is structured as follows. First, the problem model for missing value imputation is given and some basic definitions and conventions are formulated. Next, the ensemble imputation method with the bootstrap sampling is presented. Here, the imputation procedure and the weight learning scheme are carefully described. Detailed derivation of the optimal weights is provided in the sequel. The theoretical performance is subsequently analyzed in terms of the imputation errors. In addition, the choice of base imputation methods and the generation of the base predictions are explained. After that, a series of tests are done to evaluate the presented method. Finally, we conclude the paper.The main contributions of this work are as follows: M genes in N experiments. The element of i, j) is designated by ith gene produced in the jth experiment.Throughout the article, we will use a matrix ith gene locate at the ith row of ith gene. The missing rate Due to various reasons, e.g., media or experimental conditions, the elements of i, can be written as Further, for the sake of explanation, a vector or matrix operator The basic idea of missing value imputation is to estimate the missing gene expression data by the use of the known gene expression data. Using the above notations, the process can be generally expressed asThe performance of an imputation method is usually assessed by the normalized root mean square error (NRMSE), which is the most widely used metric to evaluate the accuracy of a prediction approach. For the imputation problem, NRMSE is defined asAs a major learning paradigm, an ensemble method tries to construct a set of learners from training data and combine them to generate a desirable learner . The proK imputation methods are selected as the component predictors in the proposed ensemble method. According to the generalization error analysis for ensemble learning in [Step 1: A set of rning in , the usei by each component predictor multiple times, L samples lth sampled set of the known column position set Step 2: In order to predict kth imputation method, the imputation function i with the use of the data in sample Step 3: For the A few observations are in order about the proposed imputation method. First, in step 2, the sample datasets are generated by bootstrap sampling for each utilized base imputation method, by which the performance loss of them from sampling process can be reduced.Second, the predictions given by the individual predictors are combined into the final prediction in step 4. It has been theoretically shown  that theA more intuitive explanation of the theoretical results is that each individual predictor is only adapted to a data space with a certain characteristics and the combination of them is capable of expressing a data space with various characteristics and forming a better imputation method. The specific performance analysis for the ensemble method will be addressed later.Third, the optimal weight vector e method .L predictions obtained by a base imputation method are combined with the use of different weights, while the same weight is assigned to the predictions of a base learner in the existing stacked regression methods. From this perspective, the proposed imputation method utilizes a more general combination rule.In addition, Equation  indicateith element of In the expression , the weiThis is a convex optimization problem with linear constraints. Solving the problem yieldsIn , the vecith component of the vector The detailed derivation is presented as follows. To solve the optimization problem  with theAccording to , we can nstraint , it is etituting .Further, applying the inequality in  to 7), , 7), we equality  and due mentclasspt{minima,0)Last,  can be ok and l satisfy In the sequel, the theoretical performance for the ensemble imputation method is accessed by the sum of squared regression errors, denoted by rding to  and 4),,\documenFurther, let nt}Er in  is19\docIn addition, we may assess the effectiveness of the ensemble method by the estimation bias P nearest neighbor genes, i.e., the local gene information, for the imputation, however, they determine the candidate genes by different gene matching rules. The last one achieves the aim by the use of the support vectors corresponding to the Q largest singular values, which contains the global gene information. As a result, the key characteristic diversity of base predictors can be ensured to obtain a good ensemble. Detailed descriptions for the chosen base predictors are presented as follows.To design the ensemble method, four early and relatively primitive imputation methods are adopted as base predictors: KNN imputation , LLS impi should be filled for the neighboring gene searching. As in [We establish the imputation functions g. As in , the rowith row and the columns in i and each of other genes is computed as Then, taking the vector P nearest neighbor genes of gene i are determined as the candidate genes for the imputation, whose expression data composes a matrix of size Next, by Euclidean distance, the ith gene are estimated byith gene and the lth sampling. For KNNimpute, the jth element of the weight vector jth row of Last, the missing gene data for the Since KNNimpute estimates the missing data by exploiting the local structure information in the target dataset, the imputation performance largely depends on the local similarity of gene data. Moreover, it is clearly unable to make use of the global information contained in the data.We use LLSimpute to establish the imputation functions P nearest neighbor genes of gene i. For any two genes i and j, the Pearson correlation Specifically, in LLSimpute, Pearson correlation based gene matching rule is adopted to find out the The weight vector Clearly, LLSimpute is a local imputation method with the use of the correlation among gene data. It has the same shortcomings as KNNimpute, and is more sensitive to the number of reference genes and the missing rate.The imputation function ted into  and 24)\documentIt has been shown that ILLSimpute achieves the improved imputation quality by multiple iterations of imputation, but fails to capture some unique properties that non time series datasets have. That is, ILLSimpute presents the good performance only on some kind of datasets.SVDimpute is finally used to construct the imputation functions Then, the resulted real matrix Q largest eigenvalues are selected from i can be represented by . The data matrix is derived by the use of a subset of the cancer genomic atlas database on Glioma cancer study with all missing values removed , called T but the larger computational cost will be caused. The parameters for the utilized component imputation methods KNNimpute and SVDimpute take the optimal values as suggested in [Q of the selected eigengenes is P as that of KNNimpute for avoiding the optimal parameter searching as done in [The bootstrap sampling is performed ested in . That is done in , 22. TheFirst, we test the imputation performance by varying the missing rate from 1 to 20%. The results for all the tested methods are shown in Fig. Next, the imputation performance is evaluated for different sample size, where the incomplete data matrix is generated at a given missing rate after a specific number of samples are selected randomly. The effect of sample size varying from 20 to 45 is demonstrated in Fig. Last, considering the measurement of gene expression data itself has a lot of noises, the imputation performance shall be evaluated in the existence of noise. For this purpose, the generated incomplete data matrices undergo additive white Gaussian noise (AWGN) with standard deviation being within the range 0.1–0.9. The robustness of all the tested methods to noise is demonstrated in Fig. Simulations are also done on another subset of the cancer genomic atlas database , called In principle, on the second data matrix, the tested imputation methods perform worse than on the previous one in terms of the imputation accuracy. Moreover, among the individual imputation methods, the relative performance depends on the utilized datasets. For example, in Fig. Again, we can see that, in most cases, the proposed method still outperforms other methods significantly despite the change of the data matrix. The good generalization ability is obtained for our method takes use of more data information by integrating diverse imputation methods. Moreover, the weights for each base method are optimized in a data-driven way, by which the importance of the global information or the local information depending on the used dataset can be expressed. This ensures that the ensemble output is best in the sense of Statistics.Furthermore, the proposed ensemble method is compared with SKNNimpute , IKNNimpSimulations are performed on the data matrix called GDS38 for a study of cell-cycle-regulated genes in Saccharomyces cerevisiae . GDS38 cstrategy . MLPimpuThe obtained NRMSE for the tested imputation methods is plotted in Fig. The proposed ensemble method is also compared with several typical hybrid imputation methods including LinCmb , BPCA-iLThe results obtained on GDS38 dataset is shown in Fig. Clearly, comparing with the single imputation methods in the last test, all the hybrid methods can obtain much lower NRMSE at the same missing rate, which confirms that the combination of multiple imputation methods is an effective strategy for the imputation performance improvement. Among the existing hybrid methods, LinCmb receives the NRMSE smaller than that of BPCA-iLLS and RMI at each missing rate. Moreover, the performance of LinCmb degrades more smoothly when the missing rate varies from 5 to To investigate the effects of the number of base predictors, we vary the number of base predictors in the proposed ensemble method from 2 to 4, and conduct the experiments on GDS38 dataset. The obtained results are shown in Fig. As can be seen, among the variants of our full method, the one combining KNNimpute and SVDimpute, called KNN-SVD in Fig. Further, the performance of the variants of our ensemble method (full model) in the test is explained based on the theoretical results given previously. We compute the sample mean mentclass2pt{minimrding to  and 20)\documentNotice that SVD presents the sample mean and standard deviation in imputation errors close to that of LLS. The sample covariance of KNN and LLS in imputation errors is apparently larger than that of KNN and SVD, and the sample covariance of LLS and ILLS in imputation errors is also much larger than that of SVD and ILLS. According to the results and considering  and 20)20), we cTumor classification is an important application of gene expression data, which is of great significance to the diagnosis and treatment of cancer diseases. In the application, missing data in gene expression data is first imputed and then tumor classification is performed on the imputed data. Therefore, the performance of the imputation method is straightforwardly related to the classification accuracy. In the sequel, we evaluate the performance of the proposed method by conducting a tumor classification experiment.Tumor cell gene expression data set GDS1761 is used in the experiment, which was sampled from gene expression profile data of 64 cell lines of tumors . This dak-nearest neighbor (KNN) and support vector machine (SVM) [The missing data in the dataset are imputed by applying the ensemble method and other popular imputation methods respectively. After that, the dimensionality of each sample is reduced from 9706 to 16 by the use of principle component analysis (PCA) , 47. Thene (SVM) , 49.For the performance evaluation method, the leave-one-out cross validation is adopted by considering the small sample number of each tumor category. And the classification performance is assessed by the average accuracy, i.e., the ratio of the number of samples to be correctly classified in all the cross-validation tests to the total times of the cross-validation tests. The experimental results are summarized in Table Obviously, the classification performance is closely related with the utilized imputation method. A good imputation method helps improve the classification performance. For instance, our ensemble method brings about the significant performance advantage over other tested methods for the two typical classifiers. The behavior indicates that the imputation method with high precision definitely plays a positive role in the subsequent data analysis. However, surprisingly, the simple imputation method, MEANimpute, is more effective than KNNimpute, LLSimpute, SVDimpute and their improved versions in the task of gene data classification. This reveals that the data imputation is less reliable if only exploiting local information  or global information . In addition, SVM possesses better classification performance than KNN for the same imputation method.In this paper, an ensemble method has been proposed for missing value imputation by constructing a set of base imputation methods and combining them. Four commonly used imputation methods served as the base methods and were trained by applying the bootstrap sampling to reduce their dependence. The final predictions were produced by weighting and summing the predictions given by all base methods, where a learning scheme was developed to derive the optimal weights by minimizing the imputation errors with known gene data. Moreover, we theoretically evaluated the performance of the proposed method.The ensemble imputation method has been extensively tested on several typical genomic datasets, and compared with the state-of-the-art imputation methods including KNNimpute, IKNNimpute, SKNNimpute, LLSimpute, ILLSimpute, SLLSimpute, SVDimpute, SVRimpute, MLPimpute, LinCmb, BPCA-iLLS, and RMI. Experimental results confirmed the advantage of the proposed method over other tested methods consistently in all three different scenarios in terms of lower value of NRMSE. Of particular importance is that our method yields much better generalization and universality.


RNA sequencing has become the standard technique for high resolution genome-wide monitoring of gene expression. As such, it often comprises the first step towards understanding complex molecular mechanisms driving various phenotypes, spanning organ development to disease genesis, monitoring and progression. An advantage of RNA sequencing is its ability to capture complex transcriptomic events such as alternative splicing which results in alternate isoform abundance. At the same time, this advantage remains algorithmically and computationally challenging, especially with the emergence of even higher resolution technologies such as single-cell RNA sequencing. Although several algorithms have been proposed for the effective detection of differential isoform expression from RNA-Seq data, no widely accepted golden standards have been established. This fact is further compounded by the significant differences in the output of different algorithms when applied on the same data. In addition, many of the proposed algorithms remain scarce and poorly maintained. Driven by these challenges, we developed a novel integrative approach that effectively combines the most widely used algorithms for differential transcript and isoform analysis using state-of-the-art machine learning techniques. We demonstrate its usability by applying it on simulated data based on several organisms, and using several performance metrics; we conclude that our strategy outperforms the application of the individual algorithms. Finally, our approach is implemented as an R Shiny application, with the underlying data analysis pipelines also available as docker containers. A common application of next-generation sequencing (NGS) is RNA-sequencing (RNA-Seq), which is the genome-wide monitoring of gene expression across organisms, tissues, and lately, single cells . Due to Although RNA-Seq has been used for a variety of applications, its most common use remains the genome-wide gene expression monitoring across different biological conditions in the search for alternatively expressed genes, gene sets and encompassing biochemical pathways that characterize such conditions such as disease progression or organ development . TherefoDuring the past decade, serious research effort has been devoted in the development of methods for differential isoform expression analysis (DIEA) and alternative splicing (AS) while also trying to address some of its inherent biases, such as the gene-length bias (read accumulation over longer transcripts)  which maFinally, a quite common observation when using multiple tools for DIEA of an RNA-Seq dataset is that while the input data to each method are the same, the different tools tested usually do not reach the same conclusions, making it difficult not only to decide on the best methodology but also to aggregate the independent results towards a reliable consensus suitable for downstream analysis .p-value combination algorithm [p-values through weights estimated from real data, PANDORA was shown to optimize the result accuracy by ensuring the best precision-sensitivity relationship relative to the rest of the tested tools and other p-value combination methods. Furthermore, we recently showed evidence that PANDORA is insusceptible to biases introduced by different normalization frameworks, showed improved performance when applied to low count RNA molecules, such as lncRNAs and was able to control the propagation of gene length bias into downstream enrichment analysis [To tackle such problems for simple differential expression analysis at the whole gene level , we previously developed PANDORA, a weighted lgorithm . By combanalysis .p-value combination is no longer applicable in the case of DIEA due to the absence of compatible and comparable statistical scores reported by the examined DIEA tools and we propose metaDIEA, an integration approach based on established machine learning (ML) techniques. By using ML, we were able to successfully integrate six distinct DIEA tools and demonstrate the increased performance of our approach using simulated data across different organisms. Specifically, we provide evidence that the combined approach outperforms the application of single DIEA algorithms on simulated datasets and successfully overcomes the discussed output compatibility issues by reporting a single score constructed by the ML approach. The computational pipelines behind metaDIEA are available as docker containers which are also interfaced through an R Shiny application available at https://github.com/BiodataAnalysisGroup/metaDIEA (last access 21 September 2021).Driven by our previous results, we hereby extend the PANDORA concept towards DIEA in an effort to implement a unified framework to address the aforementioned algorithm compatibility and consistency issues. We argue that the approach of Although there is a plethora of published methods for DIEA based on RNA-Seq data and most of them are accompanied by the respective software tools, our research indicated that a significant portion of these tools are poorly maintained or documented, are designed to operate under specific non-flexible conditions, or are black-box, closed-source commercial solutions. Therefore, although we identified 25 potential DIEA tools for the purposes of the present study, we posed a series of additional criteria so as to keep only those tools that: (i) are open-source, (ii) have an active user community in the sense of usage and citations, (iii) are relatively user-friendly as non-experts should be able to adequately use them and (iv) are still under active development or support. Notably, out of the 25 identified tools, only 10 were found to still be in active maintenance or development and out of them, a subset of six different tools met all our additional criteria. These tools were sleuth , New Tuxp-values) and their runtime and execution models. The six selected tools are based on different execution models. Specifically, their runtime environments span several profiles, including Python, R packages and scripts, standalone executable binaries and also combinations of the above wrapped with a runtime script. However, the most important difference between them is that they produce different metrics of evidence and prediction from the underlying differential analysis procedures. For example, some provide for p-value and a q-value for each prediction , some provide the posterior probability of differentially expression (PPDE)  and others provide the probability of positive log ratio (PPLR) . The difference between the output metrics of all tools, renders them incompatible for comparison since no direct relation exists between the metrics returned by each tool. Hence, because of this heterogeneity of the final outputs, it is virtually impossible to combine all predictions and end up with a weighted meta-prediction such as the combined p-value proposed by the initial PANDORA algorithm. Last, when provided with the same input data, the different tools do not reach the same conclusions. Although this is expected to a certain extent, there may be cases of significant deviations.As already mentioned, many DIEA tools are not consistent with respect to a common output format including the significance metrics of the results  were trained and evaluated against the ground truth as derived from the simulated data. Once a model is built, it can be used with new data from the same species without the need for re-training or fine-tuning. In the case of another organism, the methodology of training the ML models presented in this article can be replicated based also on the accompanying pipelines and docker containers.To overcome the problem of combining and integrating the heterogeneous output of the selected tools, we developed a meta-analysis method that uses ML to compose a final decision on whether a transcript is differentially expressed or not, based on the output of the various different tools. This framework was tested with simulated RNA-Seq data (starting from raw reads) for four discrete species, namely In order to effectively combine the outputs of several DIEA algorithms and integrate the outcome of each individual tool towards a joint solution reflecting the advantages of each algorithm as well as mitigating the respective disadvantages, we developed an integration workflow based on established top-performing ML methodologies and tested it on simulated data. More specifically, the ML workflow integrates six separate DIEA workflows centered around the tools sleuth, New Tuxedo suite, the previously developed Tuxedo suite, BitSeq, RSEM and EBSeq. The rationale behind the selection of these tools is further explained in the Materials and Methods section.Our approach is depicted in All the pipelines resulted in lists of candidate differentially expressed transcripts accompanied by the evidence scores reported by each algorithm see . As thesShiny is an R framework for building interactive web applications purely written in R without the need of writing any additional web related code . A Shiny application is a user interface (UI) connected to a server that runs R. The main application script is composed of two separate functions, one containing all the necessary logic the program needs in order to run smoothly and the other containing all user-related features. In addition, Shiny deploys reactivity, i.e., when an input changes, the output changes automatically with minimal developer effort.A simple application was developed for the purpose of generating RF and XGBoost models automatically. The user can process and visualize the data obtained from the DIEA tools, combine all or a subset of them and train several ML models for the purpose of finding the best hyperparameters that fit the data. The execution time of this grid search varies due to both the number of models specified by the user as well as the number of data combined. Once the most suitable model is found, the user can make predictions about the data related to the levels of DIEA. Two already generated models (one for RF and one for XGBoost) are shipped with the application, based on our analysis, and can be used for prediction without the need of first training user-specified models.Upon launching the application, the user is asked to choose the mode in which they want to run the application. Currently, two modes are available, one for training ML models and one for making predictions about the DE level. If the model training mode is picked then the user can specify the tools to be used for processing in the first panel, followed by picking specific variables from each tool in the second one. The third panel in this mode consists of the actual training process where the type of model along with number of models to train can be specified. Once the grid search is over, the best hyperparams and some evaluation metrics are printed on the screen and the generated model can be saved as an .rds file. If the prediction mode is picked, the user is asked to provide the file to be used for prediction in the first panel followed by the model to be used  in the second panel. The outcome, i.e., the predicted values, can be saved in a .csv file. A combined snapshot of the metaDIEA Shiny application is depicted in In order to assess the overall performance of the metaDIEA combined approach, we evaluated the results produced by each method , across a number of objective criteria. These include five standard metrics utilized in machine learning , as well as three DE-specific characteristics, i.e., the classification of a transcript as DE (regardless of up/down regulation), the true positive classification of an up-regulated transcript and the true positive classification of a down-regulated transcript. These eight metrics, taken together, can provide a robust and objective assessment of the validity of each method towards the identification of DE transcripts, while avoiding overfitting of any given model. In all cases, the ground truth is provided by a simulated dataset, with a priori of defined characteristics.Mus musculus. The last row of the table lists the method that produced the best value for the particular metric. Although XGBoost outperforms all others in four out of the six metrics, all methods appear to perform closely. It is quite evident that there is no single method that clearly outperforms all others; as such, the ranking of each method was performed by aggregating the performance of all metrics as an average value, and ranking all methods based on this indicator as shown in Regarding the individual algorithms, Moreover, our results indicate that in principle, the combined approach using ML methods outperforms in most cases the application of individual algorithms for DIEA. Although not every ML integration outperforms all the single algorithms, XGBoost and RF seem to perform better both in terms of overall ranking  as well Mus musculus (mm10), we also performed the same analysis using simulated data for Homo sapiens (hg19), fruit fly Drosophila melanogaster (dm6), and Arabidopsis thaliana (tair10) with the detailed results available as Homo sapiens, genome version hg19), Drosophila melanogaster, genome version dm6) and Arabidopsis thaliana, genome version tair10). Overall, similar patterns of performance and ranking are observed, with the ML integrative approaches, especially XGBoost and RF, generally outperforming the individual DIEA pipelines. The only exception is the method ranking in human simulated data, where EBSeq appears to slightly outperform the ML methods when the individual performance metric values are averaged and sorted as a summary performance statistic.Apart from the application of our pipeline to the species In order to assess the stability of the produced models, a 10-fold cross validation (CV) method was used. The accuracy of the ensemble classifier was picked as a stability metric. During cross validation, the dataset was first randomly perturbed and then split into training and testing sets composed by 9- and 1-fold splits respectively. The training set was fitted to one of the corresponding models (one of RF or XGBoost) and the test data were predicted during the evaluation process. The results are depicted in Apart from the stability assessment we measured the execution time of a random grid search for both RF and XGBoost models. During random grid search, a sample combination was picked for training a model each time at random from the expanded grid space of hyperparameters. In this case, 10 executions were measured with a different number of models produced each time. There is evidence for a linear dependence between the number of trained classifiers and the time taken for the search to complete.Although the application of RNA-Seq protocols for detecting the abundance and differential expression of genes across many species (known model organisms as well as newly sequenced ones), biological conditions and mechanisms such as disease generation and progression has been established for many years now, very frequently researchers remain confused as to which analysis methodology is best suited for own data and the problems under investigation. The perplexity of the landscape is often further increased by the variety of questions that an RNA-Seq dataset can be interrogated for, apart from the standard differential expression analysis, such as alternative splicing detection, de novo transcriptome assembly and calling of variants on RNA-Seq data. As a result of the wellness of applications, several computational methods have been developed to tackle the aforementioned problems. However, a consequence of this wellness is also the scattering and dispersity of these methods across undocumented software libraries, scripts hidden in related articles and packages developed for specific problems and later left unmaintained. Another consequence is the little recorded effort to integrate the multiple existing solution in an effort not only to increase the accuracy of final outcomes by combining the advantages of individual packages but also to maintain the integration result under a common framework and render its application reproducible and documented.p-value combination approach named PANDORA and we showed among others that PANDORA effectively controlled the tradeoff between true positives and false hits [p-value combination was not suitable for the integration of individual DIEA tools, mostly because of the heterogeneity of the output of each tool. Therefore, we resorted to using established ML techniques to perform the combination of the different tools. Subsequently, our computational experiments using simulated datasets for various organisms suggested that not only the ML approach overall succeed in providing better outcomes than the usage of individual tools, but also provides a framework for applying ML to other similar issues in the future.Driven by the aforementioned facts, we previously integrated successfully individual differential expression detection algorithms using a lse hits  and thatlse hits . As PANDTo this end, we applied four distinct ML techniques using the outcomes of six established, widely-used and well-maintained pipelines for DIEA. In general, the ML methods presented better results than using a single DIEA method alone in most of our experiments, while two of them, random forest and XGBoost, managed to excel across all our experiments.Notably the performance of all six methods was consistent across the different species: EBSeq and RSEM were always among the best performing methods. On the other end, New Tuxedo suite and BITSeq were always found in the lowest rankings. Regarding the ML methods, XGBoost and random forest always outperformed all other ML methods and in total most of the times they also outperformed all six methods. The performance of the other two ML methods, i.e., rotation forest and support vector machines were always somewhere in the middle; not the best performance but neither among the last ones.Similar to the original PANDORA concept for integration of multiple algorithms for differential expression analysis of bulk RNA-Seq data, the metaDIEA integration workflow remains simplistic and intuitive at its fundamentals. While it presents these condensed characteristics, it nevertheless achieves a major computational goal, which is the partial remedy of benchmarking single methods for a given dataset. As already mentioned, the wealth of available DIEA methods often confuses biomedical researchers and even trained bioinformaticians as to which method fits best the particularities of an RNA-Seq dataset. As a result, a lot of valuable research time is spent on benchmarking many algorithms with sometimes completely different syntactic rules, runtime environments and heterogeneous outcomes. On top of that, many times this benchmarking procedure must be repeated for new datasets, losing even more valuable time in repetitive procedures while holding back from spending quality time in answering the original biological questions. MetaDIEA offers an effective framework which integrates many DIEA methods under a single framework while providing evidence regarding the increased performance and accuracy of the results and therefore may comprise a valued research companion for gene expression research across several domains, species and datasets.Furthermore, because of tremendous and fast-pace technological advances in computation, such as the continuous emergence and development of cloud-computing and the resulting portability demands, there is an increasing trend for containerizing applications and workflows, and bioinformatics . MetaDIEHomo sapiens (genome version hg19), Mus musculus (genome version mm10), Drosophila Μelanogaster (genome version dm6) and Arabidopsis thaliana (genome version TAIR10). More specifically, for each species, three distinct simulated datasets  with different characteristics regarding the number of differentially expressed transcripts and the number of up and down regulated transcripts were created from a subset of 20,000 transcripts. Set1 consisted of 500, Set2 of 1000, and Set3 of 2000 differentially expressed transcripts. In each set, half the transcripts were up-regulated and the other half down-regulated. For each set six paired-end FASTQ files were produced, three up-regulated and three down-regulated resulting in 18 FASTQ files for each species which comprised the starting points of the downstream processing.The basis of simulated datasets was simulated RNA-Seq read sets created with the polyester R/Bioconductor package . Polyestp-value combination [A support vector machine creates a hypersurface that divides data points plotted in a multidimensional plane, according to their (different) characteristics. The original space is divided into partitions, as homogeneous as possible, by what is called a hyperplane. SVMs are known to model highly complex relationships and have been successfully used in many fields, from text and image recognition  to bioinRandom forest  is a treRotation forest  is anothEXtreme gradient boosting (XGBoost)  is an enBecause of the heterogeneity of the output of each tool, previous approaches such as bination ,20 couldThe suggested simulation as well as analysis pipelines were implemented as a series of scripts in Linux shell and the R language. Each script ensures the required dependencies, and all the scripted components are glued together with the help of docker containers. Docker is a software platform that makes use of OS-level virtualization to deliver software applications in packages called containers. For an application to be packed as a container, there exists an initial configuration step which essentially takes place once and is described in a special file called a dockerfile. A dockerfile contains all the necessary software dependencies and instructions for the application to run smoothly. By executing the dockerfile script a docker image is created, which essentially contains the whole application. Dockerfile scripts are divided into layers, rendering the building clearer and more memory-efficient. The application execution is started by initializing a container based on this image. Docker containers are isolated from one another and bundle their own software, library dependencies, and configuration files. They can communicate with each other through well-defined channels.Therefore, a docker image was created for each one of the selected six DIEA analysis tools. As a base image, for all six images, a lean Ubuntu Linux image of a small size was chosen. By adding software dependencies and installing the necessary packages, all six images were constructed with varying sizes depending on each tool. Once all images are generated, containers based on any of them can be instantiated, thus executing the DIEA analysis offered by each tool.Although RNA-Seq revolutionized the study and monitoring of genome-wide gene expression, with a wider dynamic range of measurements and a large portfolio of applications, including the study of expression across single-cells, several challenges such as the golden standards regarding differential isoform expression remain unresolved. On top of that, existing methods are poorly documented or unmaintained. In the present study, we reviewed and evaluated several pipelines for differential isoform expression using simulated data. We proposed an integration framework that combines six individual pipelines using machine learning and provided evidence regarding the performance improvement in the detection of differentially expressed transcripts. The pipelines are available as docker containers for enhanced portability and a Shiny application was also developed to facilitate users of various computational experience levels.