Although many studies highlight the implication of circular RNAs (circRNAs) in carcinogenesis and tumor progression, their potential as cancer biomarkers has not yet been fully explored in the clinic due to the limitations of current quantification methods. Here, we report the use of the nCounter platform as a valid technology for the analysis of circRNA expression patterns in non‐small cell lung cancer (NSCLC) specimens. Under this context, our custom‐made circRNA panel was able to detect circRNA expression both in NSCLC cells and formalin‐fixed paraffin‐embedded (FFPE) tissues. CircFUT8 was overexpressed in NSCLC, contrasting with circEPB41L2, circBNC2, and circSOX13 downregulation even at the early stages of the disease. Machine learning (ML) approaches from different paradigms allowed discrimination of NSCLC from nontumor controls (NTCs) with an 8‐circRNA signature. An additional 4‐circRNA signature was able to classify early‐stage NSCLC samples from NTC, reaching a maximum area under the ROC curve (AUC) of 0.981. Our results not only present two circRNA signatures with diagnosis potential but also introduce nCounter processing following ML as a feasible protocol for the study and development of circRNA signatures for NSCLC. Aberrant circular RNA (circRNA) expression is present in lung cancer. Using nCounter with machine learning, we discovered two signatures able to discriminate FFPE lung cancer samples from controls even at early stage. Our results not only highlight the potential of circRNAs as lung cancer biomarkers but also introduce nCounter as a suitable platform for circRNA expression studies in these samples. AUCarea under the curvecircRNAcircular RNAFFPEformalin‐fixed paraffin‐embeddedGBMgradient boosting machinesKNNk‐nearest neighborsLOOCVleave‐one‐out cross‐validationmiRNAmicro RNAMLmachine learningNPVnegative predictive valueNSCLCnon‐small cell lung cancerPCRpolymerase chain reactionPPVpositive predictive valueRFrandom ForestRFErecursive feature eliminationRNAseqRNA sequencingROCreceiver operating characteristicRT‐qPCRquantitative reverse transcription PCRSDstandard deviation1circRNAs are a newly re‐defined type of endogenous RNA molecules originated by a noncanonical process called ‘back‐splicing’. Through this mechanism, the 5′ splice donor covalently links to the 3′ end of an upstream exon, resulting in a single‐stranded circular structure which can include one or different exonic/intronic regions . This paThe existence of circRNAs has been acknowledged for more than 45 years. First evidence was reported in 1976 with the first description of viroids as ‘single‐stranded and covalently closed circular RNA molecules’ , and theCircRNAs have been classified as noncoding RNA for many years, due to the lack of a 5′ cap structure and their inability to bind to ribosomes. However, recent studies reported that some circRNAs can be translated into small functional peptides in a cap‐independent manner . Other fHowever, the number of studies focusing on the development of circRNA signatures with either diagnostic or prognostic value in human malignancies is rather small, probably due to the lack of standardized circRNA quantification methods, which in turn is hampering the development of clinically applicable assays. RT‐qPCR is widely used as a quantification tool for circRNA expression studies. While its sensitivity and short turnaround time proves beneficial for circRNA research, several events such as template switching, rolling circle amplification, or the bias attached to this technique may hinder the results . In addiThe nCounter technology allows multiplex analysis of up to 800 transcripts by direct capture and counting of individual targets . With a In this proof‐of‐concept study, we retrospectively analyzed the circRNA expression profiles in NSCLC cell lines and FFPE tissues by using a custom‐designed 78 circRNA nCounter panel. Our data demonstrate that nCounter can be employed not only for basic circRNA research but also for the development of clinically useful circRNA signatures.22.1This study was carried out in accordance with the principles of the Declaration of Helsinki, under an approved protocol of the institutional review boards of Quirón Hospitals, and the IGTP‐HUGTP Biobank. FFPE lung cancer tissues were retrospectively collected from 27 early‐stage and 26 late‐stage cancer patients from the different Quirón hospitals Table . FFPE tiAll collected samples were assessed for tumor and lymphocyte infiltration by a pathologist Table .Written informed consent was obtained from all patients and further documented; samples were de‐identified for patient confidentiality. Clinical information collected from each patient was limited to gender, age, smoking status, tumor histology, driver mutation, and stage.A panel of seven human lung cancer cell lines harboring different mutations was selected along with two normal epithelial cell lines Table . Cell li2.2RNA extraction was performed following previously published methods , 19. RNA2.35 µg of total RNA was either treated or mock‐treated with RNase‐R . RNA samples were denatured at 95 °C for 30 s following addition of a master mix containing RNase‐R (or molecular grade water in the case of mock‐treated samples), 10× RNase‐R buffer adjusted to the final volume, and molecular grade water. Samples were incubated 160 min at 40 °C and kept at 4 °C prior RNA quantification and subsequent nCounter hybridization.2.4RT‐qPCR and Sanger sequencing of circRNA junction sites were performed as previously described . 10 µL oA 1 : 3 dilution of cDNA was performed, and 2.5 µL were added to the Taqman Universal Master Mix (Applied Biosystems) in a 12.5 µL reaction containing a specific pair of primers and probe for each gene. Three replicas of each sample were run for the quantification of the expression of each assessed circRNA. Three replicas of each sample were run for the quantification of the expression of each assessed circRNA. Divergent primers and probe sets were designed using Primer Express 3.0 Software  with the latter spanning the circRNA junction site .In all quantitative experiments, a sample was considered not evaluable when the standard deviation of the For Sanger sequencing, 10 µL of each PCR product was loaded on a Precast Agarose HT‐1gel and visualized under UV light  after electrophoresis .Five microliters of each cDNA sample were purified using the PCR ExoSAP‐IT Product Clean up Reagent (Applied Biosystems). Sequencing PCRs were set up using the BigDye Terminator v3.1 Cycle Sequencing Kit (Applied Biosystems), forward primer, cDNA and water in a final volume of 20 µL. Sequencing PCR was performed using a Verity 96 well thermal cycler (Applied Biosystems).After sequencing amplification, samples were loaded into a 96‐well plate and subjected to Sanger sequencing using the 3130 Genetic Analyzer (Applied Biosystems).2.5circinteractome tool (https://circinteractome.nia.nih.gov). circRNA‐miRNA interaction network was built using cytoscape . Association of miRNAs with cancer‐associated downstream signaling pathways was investigated using the miRCancer database (https://mircancer.ecu.edu).MiRNAs targeted by the differentially expressed circRNAs found in early‐stage FFPE lung cancer tissues were predicted using the 2.6A custom‐made panel of 78 circRNAs was produced, including both highly and lowly expressed circRNAs that could be related to lung cancer Table . Each prSample processing in the nCounter was performed as previously described  followin2.7Raw count values were exported to Microsoft Excel  using nSolver Analysis Software . For each of the circRNAs included in the panel, raw counts lower than the cut‐off value established as background were automatically excluded from further analysis. Background was calculated for each sample by using the mean of the negative probe counts plus two times the standard deviation. Only circRNAs with a value > 10 counts after background subtraction were considered as expressed. Subsequent circRNA‐specific counts were normalized by dividing this number by the total number of counts for this sample. Resulting number was multiplied by 10 000 (units expressed in counts per 10 000).r  and R studio . Technical variability correction, normalization, and differential expression analysis was performed using the ruvseq  and deseq2  packages . Firstly, the RUVg function was used to estimate the unwanted variation among samples based on the positive controls. The positive controls used in the NanoString panel are Spike‐In control sequences; therefore, analogous constant expression of positive controls is expected across all samples. Secondly. DESeq2 was used to perform the normalization of the data, while accommodating the estimated factors provided by the RUVg function. Finally, DESeq2 was used to perform hypothesis testing in order to identify differentially expressed circRNAs. Shrunken log2 fold‐change (log2FC) was then reported by DESeq2 along with adjusted P‐values. Batch effect was considered during normalization using RUVSeq‐DESeq2. The normalized data were employed for ML techniques.Further differential expression analysis of raw nCounter data was carried out with 2FC on the x‐axis and −log10 adjusted P‐values on the y‐axis.Volcano plots were used to visualize log2.8Recursive feature elimination (RFE) was used to perform feature selection and the LOOCV algorithm was applied on the full panel of circRNA transcripts. The number of features to select was set by default at 4, 8, 16, and 78. The number of features that yielded best performance after cross‐validation was automatically selected. To test whether generated data had enough discriminative information to build a robust model for the classification of cancer samples from controls, different paradigms of classification models were tested to provide the most accurate results. Under this context, three classification approaches were performed with the selected features: an ‘instance‐based’ model (KNN). This model uses the distances among samples to obtain a predictive label; and two different ensemble mechanisms with decision trees—bagging (RF) and boosting (GBM).For the analysis of early‐stage lung cancer samples versus control samples, GBM was excluded due to the high volume of samples is required for this model.The model with the highest ROC AUC value was then selected as the final model. A confidence threshold of 0.5 was considered for the calculation of PPV and NPV. Additional statistical indicators such as accuracy, sensitivity, and specificity were also calculated.33.1r of 0.82–0.88, P < 0.01  demonstrating the specificity of the assay . However, despite the suboptimal quality observed in the RNA extracted from the FFPE cells Fig. , a statiNext, we assessed the feasibility of RNase‐R treatment in FFPE samples. As a result, overall circRNA enrichment was not achieved, in contrast to what was previously observed in RNA extracted from fresh cells. Most circRNAs were found to be degraded to different extents in RNase‐R‐treated replicas when compared with the controls, indicating that such treatment should be avoided when working with FFPE samples Fig. B.Then, different concentrations of FFPE‐derived RNA  were tested assessing the effect on downstream nCounter analysis. As a result, saturation was not achieved with the highest concentration, suggesting that a greater RNA input could be applied. Analysis of normalized counts across all samples indicated similar performance of 250 ng compared to the rest of tested concentrations, with a Pearson's correlation between 0.99–1.00 Fig. . As a re3.3A set of seven lung cancer cell lines were selected according to their driver mutation, along with two normal epithelial cell lines Table . DuplicaOut of the 78 circRNAs included in the panel, 33 were expressed in all cell lines. Nineteen were expressed in epithelial cells and not in all lung cancer cells, while only one, circFUT8 was only expressed in all lung cancer cell lines Fig. . NineteeFinally, differential expression analysis revealed four circRNAs that allowed for differentiation between the seven‐lung cancer cells and normal epithelial cells. CircPIK3R1, circFARSA, and circCHST15 were found downregulated in the cancer cell lines, while circFUT8 was upregulated Fig. .3.4A total of 53‐lung cancer samples and 16 control tissue samples were selected and processed with the circRNA nCounter panel. Initial analysis included normalization of counts for each circRNA as described in the methods section, followed by unsupervised hierarchical clustering of patient samples based on total circRNA expression. A partial separation between cohorts was achieved, indicating a group of circRNAs with discriminatory potential Fig. .A differential expression analysis revealed a cluster of 10 differentially expressed circRNAs. CircEPB41L2, circBNC2, circSOX13, and circFOXP1 were downregulated in lung cancer tissues, while circRUNX1, circCHD9, circACACA, circFUT8, circRHOQ, and circC1ORF116 were overexpressed Fig. .P = 0.015, P = 0.043, P = 0.017, and P = 0.045 respectively (Student's t‐test).Additionally, we also investigated the possible differences in circRNA expression based on the smoking habits of the lung cancer cohort. As a result, four circRNAs  negatively correlated with smoking status with Next, a ML approach was used to develop a circRNA signature predictive of lung cancer.n = 59), we decided to use LOOCV as a validation model, which considers only one sample for testing in each interaction reducing the bias to the minimum when compared to other techniques such as stratified cross validation. As a result, a RFE algorithm selected an 8‐circRNA signature  as the final model, providing a ROC AUC of 0.965, 0.953, 0.983 with RF, KNN, and GBM classifiers, respectively  of our cohort were compared to the 26 late‐stage specimens (stages IIIB and IV) Table  to assesFrom the 41 circRNAs expressed in early‐stage samples, 39 were shared with late‐stage samples Fig. . Only 6 Additional ML analysis was performed in early‐stage lung cancer and control samples. RFE algorithm provided a signature that included four circRNAs  and provided a ROC AUC of 0.981, 0.918 with RF and KNN, respectively Fig. . PPV and3.5We then explored if certain patient characteristics could provide risk factors for lung cancer by performing a univariate analysis Fig. . Several3.6CircEPB41L2, circSOX13 and circBNC2 not only were significantly downregulated both in early and late stages showing the highest fold‐change, but also were selected by the RFE algorithm as part of the two predictive signatures. As a result, these 3 targets were selected to validate the nCounter performance using RT‐qPCR.Divergent primers and probes spanning the junction sites were designed for the specific amplification of cited circular transcripts Fig.  in 10 NSRT‐qPCR results correlated with the data previously obtained from nCounter, indicating downregulation of cited circRNAs in NSCLC samples Fig. . A gel e4Precision oncology currently relies on genomic, transcriptomic, or proteomic‐based features that serve as decision‐making support, predicting treatment outcome .The re‐discovered role of circRNAs as regulatory entities of miRNAs, affecting the occurrence and development of different malignancies, has been supported by the growing number of studies that highlight their potential as cancer biomarkers and therapeutic targets in future personalized medicine . InvestiThe nCounter platform allows multiplexed digital gene expression analysis by direct counting of RNA molecules. With a wide use for transcriptomic studies, nCounter has been recently adapted for the detection of circRNAs using a specific probe design where sequences span the circRNAs junction site . On thisHere, we prove the use of nCounter for circRNA studies in FFPE lung cancer tissues and cell lines, developing a protocol for their study.Due to the lack of any commercially available nCounter circRNA panel, we first performed an extensive literature research, looking for circular RNA candidates described to be differentially expressed in lung cancer cells, tissues, or liquid biopsies. Out of the 78 circRNAs conforming the panel Table , 40–51 cThe resulting 13 circRNAs were detected in the cell lines with three of them, circPIK3R1, circADAM22, circCHN1 being only present in the normal epithelial cell lines (AALE and HBEC30KT). These three were described in literature to be downregulated in NSCLC , 29, 30.Most genetic analyses performed in the clinic come from paraffined specimens with either very little material or compromised quality. CircRNAs are very stable, even in this type of samples, due to their circular configuration . Also, tRNase‐R treatment can efficiently degrade highly structured RNA in 3′ end‐dependent manner . Since mSince circRNA represents only 5–10% of total RNA , 34, difUsing the explained workflow and custom‐made circRNA nCounter panel, expression analysis in lung cancer cell lines was performed. Interestingly, an overall increase in the number of circRNA raw counts was found in normal epithelial versus cancer cells. This result is in agreement with a previous study, where a global reduction of circRNA expression in cancer compared to healthy specimens was found, along with a negative correlation of overall RNA abundance and proliferation .In addition, a group of differentially expressed circRNAs was discovered in the assessed cancer cell lines. Interestingly, although circPIK3R1 was downregulated in agreement with formerly published results , both ciFurthermore, a circRNA from the FUT8 gene which was found upregulated in cancer cells and further validated in FFPE lung cancer tissues, even at the early stage of the disease. In addition, circCHD9 and circC1orf116 were found highly expressed, while circEPB41L2, circBNC2, and circSOX13 were strongly downregulated in such material. These last three circRNAs could be further seen downregulated in NSCLC samples by RT‐qPCR validating previous nCounter results.Circinteractome was used to further elucidate possible miRNA targets of aforementioned circRNAs.Out of 28 predicted miRNAs for circFUT8, hsa‐mir‐186, and hsa‐miR‐1305 were the only ones presenting more than one potential binding site. Hsa‐miR‐186 was described downregulated in NSCLC, acting as an inhibitor of cancer proliferation, progression, and metastasis , 41, wheFor circCHD9, only one miRNA, hsa‐miR‐1229, was predicted. This miRNA was found upregulated in breast cancer activating β‐Catenin/Wnt signaling ; howeverInterestingly, among the several predicted miRNAs for circEPB41L2, circBNC2, and circSOX13, hsa‐miR‐942 was a common target of cited circRNAs with 4, 2, and 1 binding sites, respectively Fig. . This miLastly, further machine learning analysis of generated data using RF, GBM, and KNN algorithms provided not only a signature able to correctly classify lung cancer samples from the control specimens, with an AUC of 0.985 (RF), 0.955 (GBM), and 0.993 (KNN) using an 8‐circRNA signature, but also a 4‐circRNA signature for early‐stage lung cancer classification with comparable accuracy. These ML‐based signatures included circEPB41L2, circSOX13, circBNC2, adding evidence of the potential of mentioned circRNAs as early‐stage lung cancer biomarkers.Since we did not perform microdissection of the tumor samples nor single cell analysis, we could not verify whether presented signature‐based circRNAs came from cancer cells or tumor microenvironment. Although this was out of our research scope since we mainly focused on the diagnostic potential of such signatures, we believe it could be of interest for future investigations. Also, most samples included in this study were lung adenocarcinomas, except for one squamous carcinoma and nine NSCLC samples with unknown histological subtype. Inclusion of different histologies in forthcoming validation studies are recommended to assess the specificity of the presented signatures. Finally, the work presented here was a proof‐of‐concept study and the main purpose was to demonstrate the feasibility of using nCounter for the study of circRNAs in lung cancer specimens. In consequence, the number of samples included was small and the abovementioned signature should be validated in a larger cohort.5In summary, we have developed a circRNA nCounter panel and workflow that can be used for multiplex detection of circRNA in FFPE lung cancer specimens. A cluster of differentially expressed circRNAs have been presented and further investigation is warranted to explore their potential as therapeutic targets. In addition, a 4 circRNA signature has been found through ML proving effective for early‐stage lung cancer differentiation.These findings pave the way to future biomarker investigations and validation of liquid biopsy signatures for lung cancer detection.Chung‐Ying Huang and Sarah Warren were full‐time employees of NanoString Inc. at the time the study was performed. The rest of the authors declare no conflict of interests.https://publons.com/publon/10.1002/1878‐0261.13182.The peer review history for this article is available at CPV, JWPB, RR, and MAMV conceptualized and designed the experiments. MI and AAH were responsible of patient recruitment and sample collection. CPV, AGC, EA, and SRM performed the experiments. SPG, NP, CPV, performed data analysis with contributions of AFH and MH. CPV, RR and MAMV wrote the main manuscript and prepared the figures. CYH and SW contributed to reagents and materials. AFH, MH TB, MF, and CAE provided editing, comments, and experimental guidance. All authors reviewed the manuscript.Fig. S1. nCouter probe design allows specific recogition of the circRNAs included in the panel.Click here for additional data file.Fig. S2. nCounter workflow for circRNA expression studies in FFPE lun tissues.Click here for additional data file.Fig. S3. Reproducibility experiment comparing the log2 of normalized counts by nCounter from three independent RNA samples derived from the PC9 cell line.Click here for additional data file.Fig. S4. Bioanalyzer profiles of faired fresh (left) and FFPE (right) PC9 cell line‐derived RNA.Click here for additional data file.Fig. S5. nCounter analysis of FFPE PC9 cell line.Click here for additional data file.Fig. S6. Total RNA concentration assessment for circRNA analysis using the nCounter platform.Click here for additional data file.Fig. S7. Venn diagram showing circRNAs identified in all healthy cells (19) versus those only expressed in all lung cancer cell lines (1).Click here for additional data file.Fig. S8. Overall total number of raw counts in lung cells.Click here for additional data file.Fig. S9. Different miRNA binding sites of dysregulated circRNAs in early‐stage lung cancer tissues.Click here for additional data file.Fig. S10. Diagram showing the tracking of those circRNAs of the circRNA nCounter panel not detected in assessed FFPE tissues.Click here for additional data file.Table S1. Diagnosis and associated pathologies of the control cohort.Click here for additional data file.Table S2. Characteristics of FFPE samples included in the study. Tumor and lymphocyte infiltration is indicated.Click here for additional data file.Table S3. circRNA and mRNA candidates included in the nCounter panel.Click here for additional data file.


A Molecular Features Set (MFS), is a result of a vast diversity of bioinformatics pipelines. The lack of a “gold standard” for most experimental data modalities makes it difficult to provide valid estimation for a particular MFS's quality. Yet, this goal can partially be achieved by analyzing inner-sample Distance Matrices (DM) and their power to distinguish between phenotypes. The quality of a DM can be assessed by summarizing its power to quantify the differences of inner-phenotype and outer-phenotype distances. This estimation of the DM quality can be construed as a measure of the MFS's quality.  Here we propose Hobotnica, an approach to estimate MFSs quality by their ability to stratify data, and assign them significance scores, that allow for collating various signatures and comparing their quality for contrasting groups. A signature based on a predefined Molecular Features Set (MFS), which is designed to distinguish biological conditions or phenotypes from each other, is a crucial concept in bioinformatics and precision medicine. In this context, signatures typically originate from MFS from contrasting experimental data from two or more sample groups, which differ phenotypically. These MFS incorporate information on the differences between the groups. The nature of the MFS depends on the modality of the original data. For instance, the MFS provided by the Differential Gene Expression approach is a list of Differentially Expressed genes (DEG); Differential Methylation analysis provides Differentially Methylated Cytosines or regions (DMC and DMR) as MFS. MammaPrint), for pathways and gene enrichment analysis , and for drug re-purposing .A significant number of mutational, expression and methylation-based signatures have recently been published and they are actively used in research and translational medicine. Examples of expression-based signatures involve gene sets for clinical prognosis , the corresponding lower triangle matrix may be considered  in un-subsetted triangle matrix will beGiven the numbers of samples isLikewise, total number of ranks is subsetted to in-class values will beA if the subset procedure selects minimal values of ranks, or, in other words, ranks in the selected part of rank matrix are values from 1 toM:Σ reaches its min valueB is delivered when maximal values of Ranks Matrix are selected - from (N −M) toN:The minimal value is reached when the lowest ranks reside in the diagonal squares of the distance matrix that correspond to in-group distances, and, therefore, best groups separation. Similarly, max valueA,B]. Now performing a scaling of compact  to  finally allows us to retrieve valueα with requested properties:Thus, Σ ∈ ,Analysis codehttps://github.com/lab-medvedeva/Hobotnica-mainAnalysis code available from:https://doi.org/10.5281/zenodo.5656814Archived analysis code as at time of publication:GNU General Public License v2.0License: In my opinion the authors answer to all my criticism. In this version of the article, they correct some imprecision in the graphs and add figures that help to understand the method. I think that now is ready for indexing.Is the rationale for developing the new method (or application) clearly explained?YesIs the description of the method technically sound?PartlyAre the conclusions about the method and its performance adequately supported by the findings presented in the article?PartlyIf any results are presented, are all the source data underlying the results available to ensure full reproducibility?YesAre sufficient details provided to allow replication of the method development and its use by others?PartlyReviewer Expertise:Epigenetics. R-developrerI confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. I have gone through the responses provided by the authors and the updates of the paper. The author has addressed the main questions and revised the manuscript. I have no further questions. For the final acceptance, I approve it.Is the rationale for developing the new method (or application) clearly explained?PartlyIs the description of the method technically sound?PartlyAre the conclusions about the method and its performance adequately supported by the findings presented in the article?PartlyIf any results are presented, are all the source data underlying the results available to ensure full reproducibility?YesAre sufficient details provided to allow replication of the method development and its use by others?YesReviewer Expertise:Data science, Machine learning, network analysis, computational biology, gene expression data analysisI confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. The authors present an approach called Hobotonica for quantitatively evaluating (by assigning H score) MFS quality for given sample labels. This approach could be useful for analyzing samples, for e.g., quality comparison, filtering out poor quality samples, and comparing different phenotypical conditions and experiments. It is important that the authors should discuss a reasonable H-score interpretation in terms of various implications of data quality/outcome related to experimental conditions, sample size, data preprocessing, and the complexity of biological systems reflecting the non-trivial correlation structure.The author should add simulation studies providing a realistic understanding and interpretation of the H score.How is the current approach different from the clustering-based approach where the optimized number of clusters are compared to sample labels using rand-index (where a high rand score means the clustering solution and the sample labels are in agreement) or other measures.The analysis should consider experimental conditions (data derived from multiple experiments representing the same phenotype), data preprocessing methods, sample size, and gene expression data covariance structure.when MFS is differentially expressed genes.when MFS is randomly selected.When MFS is a predefined set .How does H-score vary with relation to the number of phenotype conditions and number of MFS. The authors should add analysis and interpretation of results:The author should add accurate descriptions of all the notations used.Add a definition of H-score. I highlight some of the recommendations to be discussed in the paper:Is the rationale for developing the new method (or application) clearly explained?PartlyIs the description of the method technically sound?PartlyAre the conclusions about the method and its performance adequately supported by the findings presented in the article?PartlyIf any results are presented, are all the source data underlying the results available to ensure full reproducibility?YesAre sufficient details provided to allow replication of the method development and its use by others?YesReviewer Expertise:Data science, Machine learning, network analysis, computational biology, gene expression data analysisI confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above. The author should add simulation studies providing a realistic understanding and interpretation of the H score. The Reviewer raises an important problem of parametric and nonparametric statistics. The nature of H-score distribution indeed was not discussed in detail. Yet, its distribution was not the focus of this study, for distribution of H-scores for randomly selected Molecular Feature Sets is only employed to compute empirical p-values. This procedure is nonparametric and, therefore, is not affected by the nature of H-score distribution. We agree the nonparametric nature of the statistic needs to be mentioned more explicitly in the manuscript, and add a paragraph to the Discussion section:The non-parametric statistic used in the approach not only allows for MFS of various types  and data modalities , but also for different structure of contrasted samples groups ”How is the current approach different from the clustering-based approach where the optimized number of clusters are compared to sample labels using rand-index (where a high rand score means the clustering solution and the sample labels are in agreement) or other measures. We agree the difference of our approach from clustering-based methods was not mentioned explicitly. We have added a paragraph to the Methods section discussing the difference of our approach. “ “The proposed approach is different from existing metrics, such as often used Rand index and other clustering-based measures, as it allows one to avoid clustering procedure, that itself may be carried out with various approaches and parameters. In contract to clustering-based methods, H-scores directly reflects the sample stratification quality.”The analysis should consider experimental conditions (data derived from multiple experiments representing the same phenotype), data preprocessing methods, sample size, and gene expression data covariance structure. The dataset details the reviewer mentions are crucial for understanding a particular dataset’s structure, and it is correct that they may affect some types of analysis. However, since the nature of the statistic is nonparametric, the details do not affect our approach. We agree this point needs to be stressed explicitly in the manuscript, and add a paragraph in the Discussion section:The non-parametric statistic used in the approach not only allows for MFS of various types  and data modalities , but also for different structure of contrasted samples groups ” “ The exploration of H-score distribution in relation to experimental conditions mentioned above certainly is an interesting fundamental question. However, it does not affect the practical implementation we introduce. Therefore, we believe that such study is beyond the scope of a current paper.How does H-score vary with relation to the number of phenotype conditions and number of MFS. The authors should add analysis and interpretation of results:when MFS is differentially expressed genes.when MFS is randomly selected.When MFS is a predefined set .  The nature of MFS indeed may be quite different. For this reason we have performed and discussed the analysis of following MFS types:differentially expressed genesrandomly selected genesMysigDB gene sets (predefined genesets) In the revised version of manuscript differentially methylated MFS are also considered.   In this way, we feel the analysis and the interpretation the Reviewer suggested can be found in the manuscript.The author should add accurate descriptions of all the notations used. To address Reviewer’s comment we carefully checked the manuscript to ensure all introduced notations are defined and explained, and added a subsection to the Methods section for more detailed description of H-score.  Add a definition of H-score. We agree the definition of the H-score statistic and notation we introduce should be expanded. To address Reviewer’s comment, we have added a subsection to the Methods section where we define H-score with more details.   omics techniques . In this short article the authors present an R package called Hobotnica, whose purpose is to evaluate the goodness with which different methodologies can stratify the results presented as Molecular Feature Sets (MFS). With MFS the authors point to all those types of data as a result of different -Major commentsThe authors present three examples in which it is demonstrated how this approach is able to evaluate the effectiveness of MFS stratification, but the examples considered are all based on expression data. To verify the statements presented in the article, it would be useful to test the methodology on different data (for example methylation arrays). The approach chosen for this evaluation is based on the calculation and comparison of Distance Matrices (DM). The example of figure 3 evaluates two different standard approaches for the analysis of RNAseq using Hobotnica and the H0 value as discriminant. It can be appreciated in this figure how the stratification quality of Deseq2 is decidedly more efficient than both random genes and top variant genes. Surprisingly, however, the H0 value calculated using the top 100 genes collected with edgeR is very similar to that calculated using random genes, this confused me. Authors should explain this similarity more in depth.    The data presented in this article do not seem to convince satisfactorily. The quality evaluation power obtained by applying Hobotnica does not seem to correspond to the premises made. While not in fact a slate on the methodology, my advice is to review the examples and try to improve in benchmarking, adding different types of data.Is the rationale for developing the new method (or application) clearly explained?YesIs the description of the method technically sound?PartlyAre the conclusions about the method and its performance adequately supported by the findings presented in the article?PartlyIf any results are presented, are all the source data underlying the results available to ensure full reproducibility?YesAre sufficient details provided to allow replication of the method development and its use by others?PartlyReviewer Expertise:Epigenetics. R-developrerI confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above. The authors present three examples in which it is demonstrated how this approach is able to evaluate the effectiveness of MFS stratification, but the examples considered are all based on expression data. To verify the statements presented in the article, it would be useful to test the methodology on different data (for example methylation arrays). The approach chosen for this evaluation is based on the calculation and comparison of Distance Matrices (DM). We thank the Reviewer for this suggestion. Indeed, the method we propose is applicable for Molecular Feature Sets evaluation of different nature, yet in the manuscript we demonstrated its work only for Expression based data.To improve the manuscript in a way the Reviewer suggested , we have performed additional analysis on Methylation based data for several datasets. We added this case study to the Validation and Results sections and comments to the Discussion section. The example of figure 3 evaluates two different standard approaches for the analysis of RNAseq using Hobotnica and the H0 value as discriminant. It can be appreciated in this figure how the stratification quality of Deseq2 is decidedly more efficient than both random genes and top variant genes. Surprisingly, however, the H0 value calculated using the top 100 genes collected with edgeR is very similar to that calculated using random genes, this confused me. Authors should explain this similarity more in depth.   The point the Reviewer mentions indeed raises concern. After thorough examination we identified a data analysis related problem that caused the wrong genes subset process and resulted in the incorrect depiction in the presented barplot. We have corrected this issue. The rest of our findings were not changed during our reevaluation and all the results hold. The results are depicted and Fig.5 in the current manuscript version. The data presented in this article do not seem to convince satisfactorily. The quality evaluation power obtained by applying Hobotnica does not seem to correspond to the premises made. While not in fact a slate on the methodology, my advice is to review the examples and try to improve in benchmarking, adding different types of data.   To address the Reviewer’s comments on the evaluation made in the manuscript we conducted additional analysis and validation carried out for methylation data modality on several datasets, and improved and fixed validation for differential expression analysis. Now we feel that the argument and claims we make regarding the H-score are compelling and plausible.


Outliers in the training or test set used to fit and evaluate a classifier on transcriptomics data can considerably change the estimated performance of the model. Hence, an either too weak or a too optimistic accuracy is then reported and the estimated model performance cannot be reproduced on independent data. It is then also doubtful whether a classifier qualifies for clinical usage. We estimate classifier performances in simulated gene expression data with artificial outliers and in two real-world datasets. As a new approach, we use two outlier detection methods within a bootstrap procedure to estimate the outlier probability for each sample and evaluate classifiers before and after outlier removal by means of cross-validation. We found that the removal of outliers changed the classification performance notably. For the most part, removing outliers improved the classification results. Taking into account the fact that there are various, sometimes unclear reasons for a sample to be an outlier, we strongly advocate to always report the performance of a transcriptomics classifier with and without outliers in training and test data. This provides a more diverse picture of a classifier’s performance and prevents reporting models that later turn out to be not applicable for clinical diagnoses. To alidation . It was lidation .One reason for the differences between accuracies obtained from within training data validation and from validation in external test data are divergences in the distribution of sample characteristics. Simple examples would be the male/female ratio or the age distribution is different in the training and test data. In addition to differences in the distribution of sample characteristics, batch and study effects in the expression data themselves can result in different outcomes of classifier evaluations from within the training data or from the test data ,14. The The difference in the distribution of sample characteristics in the training and test data can especially be affected by outliers in one or both datasets. Outliers, basically samples with extreme data, can either be caused technically  or represent real biological extremes. In the case of a classification problem with a pre-defined number of classes, an outlier can also represent an unspecified additional class or an unspecified subpopulation . In infeN significant genes will most likely be found in the differential gene expression analysis.Outliers also play a role in the detection of differentially expressed genes that are to be selected as predictor variables for the classifier models. Many microarray and RNA-Seq studies have focused on finding such subsets of genes, often referred to as ‘molecular signatures’ ,19. HoweMoreover, it is difficult to determine how much a data point must deviate from the remaining data points, based on a certain distance measure, to be considered an outlier, especially in high-dimensional data such as gene expression data . A numbeTo facilitate the handling of outliers, we present a new algorithm for determining outlier probabilities for each sample using a bootstrap approach. Bootstrap approaches have a long tradition in bioinformatics to study the distribution of analysis results ,26,27. Rwww.r-project.org (accessed 5 May 2022), version 4.1.2). In this section, we describe the standard procedures of training and evaluating the machine learning (ML) models, our new procedure for assessing outlier probabilities, as well as the simulated and real-world data examples. We performed all analyses using the R programming environment , random forest (RF), and linear discriminant analysis (LDA). These three ML models have widely been used in the context of high-dimensional expression data ,29,30). ,30. 29,3To determine how well the trained models generalize on new data, bootstrap validations were conducted, i.e., random subsets consisting of 2/3 of all samples were used for training a model, and the respective hold-out subset was used for testing. The training subset was constructed by sampling without replacement from the original data. Accordingly, the test dataset consisted of the remaining third of the observations. Model performance was assessed by classification accuracy, sensitivity , specificity, and positive (precision) and negative predictive values. In addition, the Brier score was assessed. Whereas the accuracy reflects the proportion of correct classifications, the Brier score reflects the mean difference between true classes and predicted class probabilities . This bop-value of 0.05 was used as the significance level.Outlier probabilities were quantified using a separate bootstrap procedure, specifically by resampling one dataset 100 times with replacement and detecting the outliers in each resampled dataset. For the outlier detection itself, we used first the bagplot algorithm , which iAlgorithm 1 Pseudo-code of bootstrap–bagplot algorithm for outlier detection for two study groups.1:D in the bootstrap runs and the outliers that are detected in each case:Initialize lists to report the samples that are randomly drawn from the original dataset         2:fordo3:D with replacement.    Create a bootstrap dataset 4:    Report the sample IDs present in         5:    Conduct a principal component analysis for 6:    Split 7:for do    8:k-th study group subset         Detect outliers of             9:        Report the samples being detected as outliers with            10:end for    11:    Aggregate 12:    Aggregate 13:    Compute relative outlier frequencies and conduct binomial test with 14:end forThe formal procedure for determining outlier probabilities and binomial test results can be summarized as shown in the following pseudo-code (Algorithm 1), here for the bagplot approach. The algorithm for the PCA-Grid method works analogously. A new R-function ‘bagplotol’, presented in the For evaluation, we first constructed simulated data to study the effect of outliers under a controlled setting and used additional real-world data for illustration. In all scenarios, a two-study-groups design was used. In general, the simulated and the real-world data represented scenarios with different numbers of transcripts, different sample sizes, and different proportions of outliers.We studied simulation scenarios for the cases of two and four study groups, respectively, and generated data for 1000 genes in both scenarios. For both cases of two and four study groups, we generated data representing gene expression values for 50 samples per study group. In addition, five outlier samples per group were generated. Hence, 110 and 220 samples were simulated in total for the two-study-group and the four-study-group design, respectively. In the following, we describe the procedure for generating the data for the case of two study groups.t  . In the In contrast to the original study, we pooled the samples of patients who had a previous asymptomatic course or a severe course of disease after infection with WNV. We then contrasted the PBMC expression profiles from the whole group for the two conditions ‘mock treatment’, i.e., negative control, and ‘WNV infection ex vivo’ to study the effect of outliers on the estimated model performance. Hence, the two-study-groups scenario was as follows: the class ‘mock’ contained expression data from PBMCs from all 39 donors that were incubated in medium alone, and the class ‘WNV’ consisted of expression data from PBMCs from all 39 donors that were infected with WNV ex vivo.In this section, we first describe the results of the simulation study and next the effects of outlier identification and removal on the classification performance in the two real-world datasets. In the two subsections on the results with the two real-world datasets, strategy B  was, of course, not studied.First, simulation data with typical characteristics of gene expression data were generated to test the outlier detection method, including calculation of outlier probability and its influence on classification performances in a controlled setting. To mimic data for the two-group study design, differential expression was set by providing a non-zero log fold change between the two phenotypes for some transcripts, as described in For each simulated dataset, i.e., for each of the 100 simulation runs, our proposed bootstrap approach for bagplot and PCA-Grid was used to assess the respective outlier probabilities, as described in In each of 100 simulation runs, we determined the performance measures for the three different classifier models, when either keeping all samples (strategy A), when removing the simulated outliers (‘true outliers’) (strategy B), or when removing only the samples detected as outliers by our proposed bootstrap method (strategy C). For the cases where no significant outlier was detected, the same data were used for strategy A and strategy C, and consequently the same performance values were reported. This was, for example, the case in 10 of 100 simulation runs when using bagplot for outlier detection.Performance values were always taken in dependence of the number of differentially expressed genes selected as predictors in the ML algorithms. The different ML models used for classification were also cross-validated by randomly drawing 2/3 of the original data as training samples without replacement for 100 times. In each cross-validation run, the remaining 1/3 of samples were used as test data to evaluate the performance of the previously trained classifiers.For all ML methods and for all performance measures, we observed that the performance was increased when outliers were removed. Furthermore, removing all true outliers yielded even higher performance than removing only the detected outliers. The plots in During the analysis of our simulation data, we considered that the presence of outliers, depending on where they are located relative to the regular observations of the two groups, can facilitate the classification. In other words, we assumed that the location of outliers can either increase or decrease the separability of two groups. Therefore, we studied the classifier performances within each simulation run in more detail. In particular, we inspected the distribution of the samples of each simulation run and the corresponding classification results.For the four-study-group scenario, the outlier probability estimation was conducted similarly to the two-study-group design. The proposed bootstrap method using both PCA-Grid and bagplot algorithm, respectively, successfully detected potential outliers regarding their occurrences during bootstrapping. When evaluating the classifiers, it could be observed that the accuracy increased until the number of genes was 30 and decreased with further numbers of genes included as predictor variables in the models. With an increasing number of genes above 30, slightly lower accuracies were observed when all simulated outliers were excluded from the dataset, compared to strategy A, where all samples were included in the models. Between strategy A and strategy C, i.e., only samples detected by our proposed method were removed, no substantial difference could be observed regarding accuracy and Brier score.For the SARS-CoV-2 dataset, the outlier probabilities and the classification performances for the different outlier handling strategies were assessed in the same way as for the simulation data. Similarly, the performances at different numbers of top significantly differentially expressed genes as predictors were reported.Supporting the findings of the simulation study, it could also be shown for the SARS-CoV-2 data that the detection of outliers was significantly affected by the composition of the samples present in the respective bootstrap dataset. Of all 258 samples, 77 were detected as outliers in at least one bootstrap run, but in total only 12 had an outlier probability significantly larger than 0.5. Using the bootstrap approach for PCA-Grid, 26 samples were identified as outliers, whereby 9 of the samples were also found within the aforementioned bagplot method.p-values from the binomial test. For example, the sample ‘GSM4972835’ occurred in 68/100 bootstrap runs and was detected as an outlier in all 68 runs. The estimated outlier probability therefore is 1.00. The 12 outlying samples with an outlier probability significantly larger than 0.5 include six samples from the group ‘other Virus’ and six samples from the group ‘SARS-CoV-2’. Analogously, the most frequently found outliers of the PCA-Grid method are reported as a The removal of samples with outlier probabilities significantly higher than 0.5 led to different classification results than when all samples were kept in the dataset. For the WNV dataset, the evaluation was carried out the same way as for the SARS-CoV-2 data. Analogous to the SARS-CoV-2 study, the bagplot–bootstrap procedure was used to detect any outliers and to assess outlier probabilities. Subsequently, the classification performances for the different classification models using SVMs, RFs, and LDAs were calculated for different numbers of genes included in the models.p-values for the binomial test. Overall, only one sample was found to be a significant outlier. The respective sample belonged to the group of PBMCs infected with WNV. All other samples shown in the table had an outlier probability of less than 0.5. Using PCA-Grid technique within the bootstrap runs led to two samples identified as significant outliers, i.e., one sample from each study group. The sample identified as an outlier when using bagplot was not counted as an outlier. A table of most frequently found outliers from the PCA-Grid method is included in the After cross-validation of the classifiers for the strategy where all samples remain in the dataset, the one outlier was removed from the WNV group accordingly and the classifier performances were re-evaluated. In addition to the effect on classifier performances for the simulated and real-world data, we were also interested in the influence of outlier removal on the differentially expression (DE) analysis, as this analysis is usually run to select predictor genes for the ML models. Therefore, we inspected the genes or transcripts that were most frequently found as belonging to the top DEGs within cross-validation and compared the findings of each strategy of outlier handling. For the SARS-CoV-2 dataset, the overlap between the top 200 DE genes of strategy A and strategy C is 71.7%. Looking only at the top 200 genes, 33 genes represented in the top 200 of one strategy are not found among the top 200 of the other strategy. Of the 167 overlapping DEGs in the respective top 200, the ranking did not change for 40 DEGs after removing the outliers. For three overlapping DEGs, the frequencies of being reported in the top 200 differed by more than 40 from those before the removal of the outliers. One gene, i.e., FAM92B, was found to be within the top100 DE genes after removal of the outliers but not even in the top 200 DE genes when all samples were included for DE analysis.As expected, the differences were smaller in the WNV dataset, where only one outlier was removed, i.e., the overlap between the top differentially expressed transcripts between the two strategies was larger, namely, 89,6%. In addition, for more overlapping DE genes, i.e., 115, the ranking of counts being reported in the top 200 did not change after removal of the outlier in the WNV group. Three overlapping DE genes were reported in the top 200 with a frequency that differed by more than 40 after the removal of the outlier. Notably, one transcript  was found in all 100 cross-validation runs for strategy C, where the outlier was removed, to be within the top 200 differentially expressed transcripts, but for strategy A, where the outlier was not removed, this transcript was not present in the top 200 after cross-validation.k runs for the cross-validation, where each of these runs involves as main steps the differential expression analysis, fitting of the classifier with m subsets of predictor genes, as well as B runs to assess and test outlier probabilities. Thus, the procedure has a run time complexity of The analysis of a single dataset with our proposed approach involves The evaluation of one simulated dataset with 110 samples and 1000 genes, including the outlier probability estimation with 100 bootstrap runs and cross-validation of the classifier performances with and without simulated outliers and detected outliers, respectively, for different numbers of genes took less than ten minutes on a standard PC with 16 GB RAM and using one core. The bootstrap procedure for outlier probability estimation itself using bagplot and PCA-Grid took 80 s and 25 s, respectively, without parallelization.The computational cost for the bootstrap procedure to assess outlier probabilities in the real-world datasets varied between 10 and 35 min. The individual run time depended on the used outlier detection method (bagplot or grid approach), the sample size, and the number of genes present in the data. Again, the times were obtained from analysis on a single PC with using only one core. Parallelization would, of course, decrease run times substantially.A large number of studies on transcriptomics classifiers has been presented in the last twenty-five years. Although many expectations were connected with these developments, transcriptomics classifiers ‘have so far not revolutionized clinical practice, and their clinical utility has not been satisfactorily established’ . Neverthk-fold, bootstrap) yield average values for accuracy and other measures of classifier performance. This also includes to some extent the effect of outliers, because some cross-validation subsets will contain or not contain outlying samples. However, a direct comparison of the classifier performance in absence or presence of outliers is not given. Therefore, we studied here in particular how incorporation or removal of outliers in the training and test datasets affect the performance of transcriptomic classifiers. This provides researchers a broader view of how a classifier can perform under extreme deviations between the distributions of the data used for training and evaluation and the distribution of data if the classifier is in practical use.One obstacle between reporting a high-performant classifier and its clinical usability for diagnosis or prediction is the different composition of the populations from which training and evaluation samples were taken and the population in which the classifier is to be applied. Outliers especially can strongly bias the estimation of a classifier’s performance. Typical cross-validation procedures , we present a bootstrap approach to also determine the outlier probability of each particular sample. This enables us to evaluate the performance of their models by including or excluding outliers based on these probabilities. In summary, our approach can help to assess a classifier’s performance with a higher robustness and to avoid a too optimistic estimate that later turns out as not tenable.


Estimation of isoform-level gene expression from RNA-seq data depends on simplifying assumptions, such as uniform read distribution, that are easily violated in real data. Such violations typically lead to biased estimates. Most existing methods provide bias correction step(s), which is based on biological considerations—such as GC content—and applied in single samples separately. The main problem is that not all biases are known.Xβ, where the design matrix X is known and is computed based on the simplifying assumptions. In contrast XAEM considers Xβ as a bilinear model with both X and β unknown. Joint estimation of X and β is made possible by a simultaneous analysis of multi-sample RNA-seq data. Compared to existing methods, XAEM automatically performs empirical correction of potentially unknown biases. We use an alternating expectation-maximization (AEM) algorithm, alternating between estimation of X and β. For speed XAEM utilizes quasi-mapping for read alignment, thus leading to a fast algorithm. Overall XAEM performs favorably compared to recent advanced methods. For simulated datasets, XAEM obtains higher accuracy for multiple-isoform genes. In a differential-expression analysis of a real single-cell RNA-seq dataset, XAEM achieves substantially better rediscovery rates in independent validation sets.We have developed a novel method called XAEM based on a more flexible and robust statistical model. Existing methods are essentially based on a linear model http://fafner.meb.ki.se/biostatwiki/xaem/.The method and pipeline are implemented as a tool and freely available for use at Bioinformatics online. Alternative splicing is a regulated process where inclusion and exclusion of exons from the same gene produce multiple transcripts. Alternative gene transcripts can also arise from alternative start or stop locations. Transcripts originating from the same gene have mostly similar sequences, but can have distinct biological functions . Thus, vAll methods of isoform expression estimation start with basic simplifying assumptions, such as uniform read distribution  to locate the potential transcript of origin of a sequenced read. This quasi-mapping is also implemented as an independent mapper, i.e. Rapmap, and applied to the updated versions of Sailfish  algorithm, alternating between estimation of X and β. For speed XAEM utilizes the quasi-mapping approach for ultra-fast read mapping. We compare XAEM with several state-of-the-art quantification tools including Salmon  from a recent paper  that are mapped to eqclass j. For a specific set of eqclasses J with T transcripts, let tβ be the expression level of transcript t. The key statistical problem is to estimate transcript abundances tβ from the read-count data j asw refers to the total number of mapped reads divided by 109 (per kilobase per million reads), jta is the isoform-specific bias/non-uniformity effect, tL is effective length and jtp is the proportion of reads in eqclass j under uniform distribution. For identifiability we set t. It is conventionally assumed that jy has Poisson distribution with mean jμ.We first summarize the number of reads that map to different eqclasses using Rapmap  a list of transcripts connected due to exon-sharing, (ii) a set of exons shared by these transcripts and (iii) the proportion of read counts from the eqclasses that contribute to the total reads of each transcript. As an illustration, suppose the corresponding design matrix for jtx in the matrix indicate the proportion of reads originating from eqclass j for transcript t. Note that the sum of proportions for each transcript The starting value of 2.2.1L. In practice, we used L = 2× transcript length, but with a minimum of 1000. For this simulated data, reads with uniform distribution are generated with base error rate set to 0.005. We then use Rapmap (transcript response profile (TRP), which defines a set of transcripts sharing exons with the source transcript, which is tx1 in this example.The eqclass shows the connection between a set of transcripts sharing the same exon. To investigate this connection, we use Polyester (transcript neighbors (TNs). To remove the false positives, the neighbors with reads less than a threshold H  are filtered out. Thus, in this example tx5 is discarded, and tx1 has three remaining neighbors: tx2, tx3 and tx4. We also evaluate the effect when we set the H as 0, which means all the neighbors are kept in a TRP. As shown in H = 2.5%. However, we would still suggest to use a threshold to filter false positive neighbors in the TRPs.transcription clusters (TCs). As shown in Similarly, we generate TRP and TN for each transcript . To expa2.2.2X. Particularly, we summarize the reads from the TRPs into one table .We combine the information of TC and TRPs to create a design matrix ne table . In this2.2.3X matrix, the columns of the corresponding transcripts can be extremely similar to each other, thus making the X matrix singular and the β parameter non-identifiable. To deal with this issue, we implement the k-means clustering to combine the paralogs into one transcript. Theoretically, exact paralogs will produce zero singular values, so the number of clusters k is the number of non-zero singular values in the singular-value decomposition of X. But in practice, we may get values very close to zero. As shown in Supplementary Section S2, based on our numerical studies, we set k as the number of singular values greater than 0.02. An illustrative example of the merging process can be seen in the Supplementary Section S3. The final set of processed X matrices is used as the starting X in the model (2).Paralogs are isoforms that have high similarity in their sequences. Paralogs complicate the quantification of isoform expression due to the difficulty in distinguishing them from each other. If there are paralogs in jny from input RNA-seq sample n. The RNA-seq reads are mapped to the reference transcriptome using the quasi-mapping algorithm. We extract the list of eqclasses and their read counts jny, then match them in accordance with the eqclasses in the design matrices. To be concise we simply denote jny as jy in the following text. We then summarize the read counts jy across multiple samples and construct matrix Y as described in X and β is done using the alternating expectation-maximization (AEM) algorithm, as shown in X, the transcript abundance tβ can be estimated from the model in formula (5):jtx, estimate tβ for multiple samples using (5)A. Given tβ, estimate jtx using model:B. Given C. Iterate steps A and B until convergence.jtx to estimate tβ, we calculate the value of tβ usingβ0 is a starting estimate for tβ, e.g. the mean read counts among T transcripts. In step B, for the updated X we standardize jtx for each eqclass to calculate the proportion of read counts from observed counts jy. The parameter tβ is then re-estimated using the proportion multiplied by jy. The iteration procedure is considered to be converged when both jtx and tβ have less than 1% difference between successive iterations. The 1% threshold is for maximal differences across the matrix elements, and if a matrix element is <0.01 then the threshold is set to 1% of 0.01.Specifically, in step A, given X during the isoform quantification in AEM, XAEM automatically adjusts the design matrix for bias correction using the variation information from multiple samples. We have tested the robustness of the X matrix using different tissue data from the genotype-tissue expression (GTEx) project  as the baseline distribution for isoform expression. The models are assigned to isoforms in the transcriptome. For each isoform, the expression level across samples is generated from the assigned beta-Poisson model.RNA-seq simulation is commonly used for the comparison of estimation accuracy among different tools. To investigate the estimation performance in a situation where the true expression levels are known, we use a simulator named Polyester , half of which were treated with metformin. Two independent cell batches were captured and prepared using the Fluidigm C1 system with 96-well plate. Each batch consists of 96 treated cells and 96 untreated cells. For batch 1, two Illumina HiSeq 2500 machines are used for sequencing of the treated and control cells separately. For batch 2 the order of machines is reversed. The length of sequence reads is 100 bp. We remove samples with 0 cells; the library sizes of the rest of the samples range from 1 to 10 million.A second real RNA-seq dataset with qPCR validated expression is obtained from the Sequencing Quality Control Consortium (SEQC) project  and non-uniform (with bias) settings. Under the uniform setting we expect the methods to perform similarly, while we would expect some differences between methods under the non-uniform setting.3.1.1To get a comprehensive comparison, the isoforms are divided into three categories according to the source genes: (i) singletons, (ii) multiple isoforms excluding paralogs  and (iii) multiple isoforms which are paralogs . ‘Singletons’ are isoforms originating from the genes with only one isoform, while ‘multiple isoforms’ are those from genes with more than one isoform. Paralogs, which have been discussed in more detail in Section 2.2.3, are isoforms with extremely similar sequences, which are difficult to distinguish from each other. The expression estimation for singletons is trivial, simply counting the fragments aligned to the gene, but the quantification for multiple isoforms is more complex due to exon-sharing. There is a total of 14 446 singletons and 31 950 multiple isoforms, where 25 838 are non-paralogs and 6112 paralogs.3.1.2We also compare the performance of XAEM and other methods under the non-uniform setting in We note that the number of non-paralogs (18 597) and paralogs (13 353) are different from the uniform setting. In XAEM we update the X matrix to learn the bias structure from the observed data. For different datasets, i.e. uniform and non-uniform, the biases are not the same, leading to different X matrices. Thus, the final X matrix might be different for different input datasets. When we merge the paralogs within the updated X matrix, the number of paralogs might vary between datasets. Compared to Salmon, XAEM achieves higher accuracy in 71% of the multiple isoforms (22 740 out of 31 950). The SEQC study  isoforms, we perform the analysis in a training set and then validate the results in an independent validation set. We use two different settings: (i) a simple setting, where the training and validation sets are random splits within one batch; and (ii) a more demanding setting, where we use batch 1 data for training and batch2 data for validation.et al., 2014). Since the quantification accuracy of Salmon, Sailfish and Kallisto is similar , which is the proportion of significant DE isoforms from the training set that are also significant in the validation set  singletons, in which we do not expect a difference between methods, and (ii) multiple isoforms, which are non-paralogs. For each method, we calculate the RDR based on the top 100 to 1000 significant DE isoforms in the training set. Xβ with known X, XAEM is based on a more flexible and richer bilinear model that treats both X and β as unknown parameters. Instead of adding a correction step of presumed known biases, as is commonly done in existing methods, XAEM automatically corrects for potentially unknown biases using an AEM approach. The AEM makes use of the variation of expression from the multiple samples. XAEM also utilizes the state-of-the-art quasi-mapping to speed up the overall quantification of isoform expression. We have reported a test of computational resource usage of XAEM in Supplementary Section S9, which shows that XAEM is fast and memory efficient. Overall, we show that better statistical modelling helps XAEM achieve significantly better results in comparison with recent advanced methods in both simulated and real RNA-seq datasets.We have developed a novel method called XAEM to estimate isoform expression from multiple RNA-seq samples. Compared to existing methods, which are implicitly based on a linear model X. Technically, as done in existing methods, it is possible to perform the EM algorithm without knowing X explicitly. However, with such an approach, it is not obvious how to generalize the linear model into a much richer bilinear model as we have described. The existing methods also use a transcriptome-wide X, which would be too high-dimensional to treat as an unknown parameter. XAEM uses the concept of transcription clusters, thus breaking the transcriptome into smaller units; this limits the number of X-parameters substantially, and makes the procedure computationally and statistically feasible. The initial X matrix in XAEM is generated using uniform assumption. We have also investigated the effects using the biased assumptions. As shown in Supplementary Section S10, the results are similar to those based on the uniform assumption. We have used the VBEM algorithm inside XAEM, but in principle any version of EM algorithm can be used, e.g. the alternative squared-EM . Without the paralog information, we do not have this option, and may unknowingly use inappropriate expression value. The merging of paralogs in XAEM at least makes the quantification process transparent.μ = Xβ, which, for example, looks appropriate to the data from tiling microarrays. As before, β represents the underlying transcript intensity; μ represents the probe intensities; the X matrix represents the mapping of the set of transcripts that theoretically can contribute to each probe. The starting X matrix must be derived from the assumed gene model in the transcriptome annotation file rather than learned computationally as we have described. One weakness is that the gene model may not tell us much about paralogs, which will likely lead to a lot of non-specific binding. Mueckstein et al. present an improved thermodynamic model for microarray hybridization, which may allow assessment of non-specific binding due to the cross-hybridization among paralogs ; for example, we can start with a clustering step to the samples, and assign one X for each cluster. Theoretically, distinct X matrices should produce distinct vector spaces, and these are likely learnable from the observed count data. Developing such an extension is a worthy project, but beyond the scope of the current paper. For now, in practice we recommend the use of the algorithm in batches of reasonably homogenous samples such as the same tissue or cancer type.Performances of XAEM depend on multiple samples for updating the design matrix In summary, we have introduced XAEM, an efficient and accurate method to estimate isoform expression from multiple RNA-seq samples. Compared with other competing methods, XAEM achieves better performance in estimation accuracy and rediscovery rate of DE isoform from RNA-seq data.This work is partially supported by funding from the Swedish Cancer Fonden, the Swedish Research Council (VR), the Swedish Foundation for Strategic Research (SSF) and China Scholarship Council (CSC).Conflict of Interest: none declared.btz640_Supplementary_DataClick here for additional data file.


Recent years have seen a surge of novel neural network architectures for the integration of multi-omics data for prediction. Most of the architectures include either encoders alone or encoders and decoders, i.e., autoencoders of various sorts, to transform multi-omics data into latent representations. One important parameter is the depth of integration: the point at which the latent representations are computed or merged, which can be either early, intermediate, or late. The literature on integration methods is growing steadily, however, close to nothing is known about the relative performance of these methods under fair experimental conditions and under consideration of different use cases.We developed a comparison framework that trains and optimizes multi-omics integration methods under equal conditions. We incorporated early integration, PCA and four recently published deep learning methods: MOLI, Super.FELT, OmiEmbed, and MOMA. Further, we devised a novel method, Omics Stacking, that combines the advantages of intermediate and late integration. Experiments were conducted on a public drug response data set with multiple omics data  that was obtained from cell lines, patient-derived xenografts, and patient samples. Our experiments confirmed that early integration has the lowest predictive performance. Overall, architectures that integrate triplet loss achieved the best results. Statistical differences can, overall, rarely be observed, however, in terms of the average ranks of methods, Super.FELT is consistently performing best in a cross-validation setting and Omics Stacking best in an external test set setting.https://github.com/kramerlab/Multi-Omics_analysisWe recommend researchers to follow fair comparison protocols, as suggested in the paper. When faced with a new data set, Super.FELT is a good option in the cross-validation setting as well as Omics Stacking in the external test set setting. Statistical significances are hardly observable, despite trends in the algorithms’ rankings. Future work on refined methods for transfer learning tailored for this domain may improve the situation for external test sets. The source code of all experiments is available under The online version contains supplementary material available at 10.1186/s12859-023-05166-7. Data analysis in the life sciences often involves the integration of data from multiple modalities or views. Integration is necessary to obtain models with improved predictive performance or explanatory power. One currently popular approach to integrate multiple views is to take advantage of latent representations as computed by neural network architectures. Views are usually defined as  groups of variables that originate from one measurement technology. In bioinformatics and computational biology, views often originate from different omics platforms, e.g., from genomics, transcriptomics, proteomics, and so forth.Frequently, views from different omics platforms are collected by different research groups, which then evaluate them separately. This independent analysis of views can lead to inconsistent conclusions, which are hard to interpret when combined . The proEach view provides distinct, complementary information about a phenomenon of interest, and integrating them can lead to an improved understanding of the underlying biology . PreviouGiven several views, there exist mainly three different paradigms of how to integrate them: early, late, and intermediate integration. They depend on the point at which information of distinct views is combined .early integration, the view’s features are concatenated into a single representation and analyzed with traditional methods. This has the advantage of being simple, but the disadvantage of creating a higher-dimensional and sparse feature space. Early integration methods assume a high dependency between omics. Furthermore, they are not suitable for combining non-tabular views as for example visual data or text.In Late integration, on the other hand, combines the results of models, which are trained independently on a single view, to obtain the final prediction. It models interaction between views only by weighting the single-view results differently. Late integration allows the combination of heterogeneous views, as only the output of a model is essential and view-specific models can be trained.intermediate integration represents a trade-off between the both. In intermediate integration, the views are first transformed to a lower dimensional representation with a suitable encoder. Only then the transformed representations are concatenated and evaluated.Whereas early integration assumes a high dependency between views and late integration a low dependency , intermeThe encoding of a view into the latent feature space is widely performed with neural networks. The neural network architectures include either just encoders or encoders and decoders, as in autoencoder-type architectures. A myriad of architectures is possible for the integration of multi-omics data: encoders only, encoders-decoders (autoencoders of various sorts), integration either early , intermediate (concatenating latent representations following their computation), or late  , and usiAs the literature on the topic is growing, one would expect that a more recent publication would improve upon a previous publication in terms of performance, or, at least, that a specific use case with superior performance has been identified then. However, as it turns out, various approaches have been optimized and tested with different sets of hyperparameters , with different values for hyperparameters, and with different test protocols. Further, performance can be very different for cross-validation (assuming a similar distribution of data for each test set) and so-called external test sets. Thus, it is at this point far from clear which method performs best overall, and, specifically, which method is to be preferred in which setting, e.g., in a cross-validation-like setting or with an external test set from an unknown distribution.In this paper, we level the playing field, establish a uniform, basic protocol for various existing methods, and test the methods with a state-of-art statistical method for the comparison of machine learning algorithms . As a siin vitro to in vivo [We study the prediction performance of the various algorithms on data sets for drug response prediction, which have been used widely in the literature in past few years . The ult in vivo .A fair comparison of recent deep multi-omics integration algorithms for drug response prediction,A new combined intermediate and late architecture for multi-omics integration, andA thorough validation study of the methods’ predictions on external test sets.The remainder of the paper is organized as follows: First, we interpret the methods’ results on the test and external sets and test significance in the observed differences. An in-depth discussion and the conclusions follow. Next, we give an overview of the used data sets and the design of the fair comparison framework. Towards the end of the article, we still give details of the included integration architectures.In summary, the contributions of this paper are as follows:At first, we analyzed the results on the five cross-validation test sets Tables , 2. Supevariational autoencoder (VAE).OmiEmbed ranks in the midpoint, showing the difficulty of optimizing its No method clearly outperforms the others, but Omics Stacking and Super.FELT performed slightly better, according to their mean ranks of 2.86 and 2.43, respectively.The AUPRC results are similar, but the three algorithms that use triplet loss achieved the best result for five out seven drugs and EI was the the worst performing algorithm. The positive trend for methods using triplet loss shows its regularization benefit on discriminating responders and non-responders.All architectures have a high standard deviation for AUROC and AUPRC, which underlines the importance of stratified cross-validation to alleviate the influence of data splitting.The visualization of the mean ranks and the critical differences Fig.  supportsin vitro response, but at the same time, has to learn general features that can predict drug response on an unknown in vivo distribution.Next, we analyzed the results on the external test sets Tables , 4. HereAgain, no single method performed best with all drugs, but Omics Stacking had the lowest mean rank in both metrics, achieving the best results in more than half of the data sets for AUROC and AUPRC. Additionally, it achieved the lowest mean rank , not individual features. In the following, we analyze the sum of attributions as well as the mean attribution of features from various views.For all of the above models and architectures, it would be desirable to compute the importance and contribution of individual variables to the outcome of a prediction model. This problem is addressed in the field of explainable AI (XAI) by so-called  infancy . TherefoTo train the final model we computed the final hyperparameter set for an architecture with a 5-fold cross-validation and the same hyperparameter optimization. The final model was trained on the complete GDSC data and the Shapley values  were comShapley values are a concept from cooperative game theory and can be used to interpret the predictions of a machine learning model. The Shapley value of a feature represents its attribution to the prediction. Shapley values are usually computed in a model-agnostic manner, by changing the inputs and measuring the changes in the models’ output.Shapley sampling.For the Shapley value combination, we employ a perturbation based approach, where features are added one by one to a baseline. The order in which features are added is changed randomly in every iteration, and the mean of all attributions represents the final attribution of a feature . In the The Shapley values of an architecture for one drug were normalized so that they sum up to one. We took the sum of an omics’ attributions to analyze its influence on the drug response prediction and the mean of an omics’ attributions to evaluate the average influence of its features on the result. Subsequently, the mean of both values over all drug data sets were calculated.It is important to notice that after the variance threshold, the number of features for mutation 10–15 thousand) and CNA (18–22 thousand) were one magnitude higher than for expression (around three thousand). Additional files  thousandFor EI, Shapley sampling assigns the highest attribution to the CNA view (0.642), and less to the expression (0.155) and the mutation view (0.203). However, the mean influence of an expression feature is greater than the mean influence of a copy number or mutation feature (0.00005 vs. 0.00003 resp. 0.00001). This indicates that EI learns that the gene expression is an important indicator for the drug response , 14, butMOMA focuses mostly on two omics , due its attention mechanism, and less on CNA (0.094). PCA is the only architecture that uses expression (0.496) and CNA (0.467) equally.The attributions for MOLI, Super.FELT and OmiEmbed show similar tendencies. The sum of the attributions of expression makes up more than half of the whole attribution. The mean attribution of an expression feature is up to approximately twenty times higher than that of a copy number or mutation feature. The intermediate integration does not favor views with many features and learns which inputs are not relevant for the drug response prediction.Omics Stacking distributes the attribution more evenly on expression (0.518), mutation (0.210) and CNA (0.272). Its summarized attribution has a high variance, indicating that the meta learner learns individually for each data set to focus on the most important omics. Similar to the other intermediate integration architectures, the averaged attributions for expression features are considerably higher than the others.The diagrams in Additional file The emerging interest in multi-omics integration with neural networks produces an increasing number of different architectures. In this work, we compared a subset of recently published methods as fairly as possible to validate their predictive capabilities. Our experiments focused on drug response prediction, a high-dimensional problem, with the added complexity of having few samples.The performance was only tested on data sets from cancer research mainly due to two reasons: First, the availability of multi-omics data, as cancer samples are more common compared to other diseases and data sets for other diseases with enough samples and a high enough quality are still rare. The second reason is that the extensive hyperparameter optimization with an inner and outer cross-validation increases the hardware requirements greatly and combined with the sheer amount of different, recently published algorithms made it necessary to focus on just a single field.We focused our research on the comparison of the predictive performance and the summarized attributions, but another important aspect is to analyze the attribution on a more fine-granular level, i.e., on the level of genes. As this is a research topic of its own, especially in the context of high-dimensional data and autoencoder-type architectures, we have to leave that for future work.in vitro to in vivo samples. However, larger in vivo or ex vivo data sets would be necessary for this to succeed.Current drug response methods are trained on cell lines only and applied directly to patient samples without adaptation. A different strategy is transfer learning, where information from source data is transferred to target data for improved prediction . This caOur experiments validate results that using EI results in worse predictions than intermediate integration, because concatenating the views creates a higher-dimensional sparse feature space that further increases the issue of having only few samples. EI is suitable to be used as simple and fast to implement baseline, but at least one intermediate integration architecture should be validated. However, making a recommendation for one of them is difficult, as none of them performed significantly better.In this paper we showed that none of the current multi-omics integration methods excels at drug response prediction, however, Early Integration and PCA performed significantly worse.Researchers should not rely on a single method, but rather consider more than one method for their task at hand. If the number of experiments is limited or translatability is wanted, we recommend to use the newly introduced method, Omics Stacking, as it achieved good results on test and external data. When faced with a new data set in a cross-validation like setting, Super.FELT is another good option.Our experiments also suggest that a fair experimental is necessary to see the strengths and weaknesses of various algorithms, which were not visible from the publications alone. We hope that this comparison has shed some light on the relative performance of multi-omics integration methods, has produced valuable insights for their application, and that it encourages further research.patient-derived xenograft (PDX) or patient data necessary to test the method’s translatability [For our experiments, we used a publicly available drug response data set containing responses for six drugs: Docetaxel, Cisplatin, Gemcitabine, Paclitaxel, Erlotinib, and Cetuximab . The dattability . Two extThe data set contains the drug response as target and data for three omics: somatic point mutations, somatic copy number profiles and gene expression profiles. Gene expressions are standardized and pairwise homogenized. Gene-level copy number estimates are binarized, representing copy-neutral genes as zeroes and genes with overlapping deletions or amplifications as ones. Somatic point mutations are also in a binary format: one for mutated genes and zero for genes without a mutation . For a fThe Cancer Genome Atlas (TCGA) [Genomics of Drug Sensitivity in Cancer (GDSC) [Sharifi-Noghabi et al. acquired the data from PDX mice models , The Cans (TCGA)  and Genor (GDSC) . GDSC coThe characteristics of the data set per drug are summarized in Table To compare the algorithms fairly, different precautions were taken: First, the same preprocessing was performed in all experiments to provide the same input data, and second, the hyperparameters of the algorithms were optimized with an equal number of iterations by random search from a fixed grid. All algorithms draw parameters from the same grids Table .Table 11area under the receiver operating characteristic (AUROC) was used as performance measure. The algorithm was retrained with the best hyperparameter set on the combined train and validation sets. The trained network was used to compute the final results on the test sets from cross-validation and the external test set.A area under precision recall curve (AUPRC) [The data sets are imbalanced as they contain only few responders. In the drug discovery process, researchers are mainly interested in the positive samples to find an effective drug. To account for this requirement, the  (AUPRC)  was addiOne requirement to use these methods is for them to work on patient data sets which are normally too small for transfer learning and still detect positive samples. This was covered by using the final model of an outer cross-validation iteration on an external patient or PDX data set.The comparison of multiple algorithms on multiple data sets can lead to ambiguous results, if none of them performs significantly better. We compute the mean ranking as single value for a better comparison. Additionally, we compute the critical difference (CD)  with theMost deep learning algorithms for multi-omics integration are based upon the concept of encoding the feature space into a lower-dimensional latent space. The encoded representation of features in the latent space is commonly called latent features. The subnetwork that computes the latent features is called encoder and computes a non-linear dimensionality reduction. The smaller dimension of the latent representation assures that the encoder does not simply learn the identity function. An autoencoder transforms an input first into the latent representation and then decodes it back into the input dimension. The reconstructed sample should resemble the input as far as possible. The difference between input and reconstruction is measured and used as loss function .Early integration (EI), which serves as baseline in our experiments. EI concatenates the omics before they serve as single input of a neural network, which consists of an encoding subnetwork and a classifier layer. The network is trained by minimizing the binary cross-entropy loss function, given that our target is to classify subjects into responders and non-responders.Next, we explain briefly the seven multi-omics integration architectures we compared. The first one is The schematic architecture for three omics in visualized in Fig. principal components analysis (PCA) for intermediate integration was tested. After computing the PCA for each view individually, the resulting principal components are concatenated to obtain a feature representation. Subsequently, this feature representation is used to train a classifier. The scheme of PCA intermediate integration is visualized in Fig. An architecture based on Multi-Omics Late Integration (MOLI) developed by Sharifi-Noghabi et al. [The next method is i et al. , which iDue to the small sample size, the neural network is prone to overfitting, so MOLI uses the triplet loss  for reguSupervised Feature Extraction Learning using triplet loss (Super.FELT) is a variation of MOLI, in which the encoding and classification is not performed jointly, but in two different phases [With them, the following loss function is minimized:t phases . The firSuper.FELT added a variance threshold in the feature selection to remove features with a low variance . It is bAdditionally, we developed a novel extension of MOLI, which we call Omics Stacking. It is inspired by stacking and a combination of intermediate and late integration. The method stacks the results of different neural network classifier layers that use different latent features as input, with a meta-learner. The meta-learner can be any classifier, but we opted for a fully connected layer to train the neural network end-to-end.The omics are transformed to the latent space with individual encoders, but instead of only classifying the concatenated features, Omics Stacking trains a separate classifier for each omics and the concatenated omics Fig. . The triOmics Stacking combines the advantages of intermediate and late integration. It models the interaction between omics and retains the weak signals of individual omics, hence, capturing shared and complementary information. It relies not only on the combined omics, but also on the individual omics and so fosters generalization.The former methods were developed and tested on drug response prediction, but in fact can be used for other tasks as well. Next we present two methods that were validated for the prediction of a tumor’s type or stage.Multi-task Attention Learning Algorithm for Multi-omics Data (MOMA) [The first of these methods, a (MOMA) , uses a a (MOMA) . It is cSecond, it focuses on important modules between omics using a module attention mechanism. This mechanism is designed to act as a mediator to identify modules with high similarity among multiple omics. The relevance between modules is measured by the cosine similarity and is converted to a probability distribution with the softmax function. The distributions are then used to create an attention matrix that stores the relationship information between modules of different omics. To highlight important modules, the module vectors are multiplied by the attention matrix Fig. .Subsequently, fully connected layers and the logistic function are applied to flatten the multidimensional vectors and to compute the final probabilities for each omics .Fig. 10SMOMA is trained with cross-entropy loss between the true label and the omics specific outputs. After the training of the neural network, a logistic regression is fit on the omics specific outputs to generate the combined prediction.The last architecture we tested OmiEmbed , which iBCE is the binary cross-entropy to measure the differences between the input x and the reconstruction M omics. The loss function of OmiEmbed is the sum of two parts: the reconstruction loss and the downstream task loss. The embedding loss function is used together with the loss of the downstream task, which is in our case classification:Instead of training all layers at the same time, the network learns in three phases: First, only the VAE is trained. In the second phase, the VAE weights are fixed and only the downstream network is trained and in the last phase the complete network is fine-tuned.Table The algorithms were implemented in PyTorch 1.11.0 and NumPy 1.22.4 was used to compute the AUROC and AUPRC. The critical differences were computed and visualized with Orange3 3.32.0. The Shapley values were computed with Shapley value sampling with 50 permutations of Captum 0.5.0.The triplets were generated online with an all-triplets scheme, which creates all possible triplets of a batch, as in the experiments of the corresponding paper of MOLI and Super.FELT. An open source implementation of triplet loss and triplet creation was used.Early Integration and Omics Stacking were implemented by the authors. For MOMA we used the provided example source codeTo reduce the optimization runtime, we applied an early stopping scheme in the inner cross-validation. If it was not possible anymore to improve on the currently best mean AUROC for a hyperparameter set – even with best results (an AUROC of 1) on all remaining sets – the inner cross-validation terminates early and the optimization continues with the next hyperparameter set.Additional file 1. Triplet loss ablation study: AUROC and AUPRC values for all architectures with and without triplet loss.Additional file 2. Attributions for Cetuximab: Gene level attributions defined by Shapley values.Additional file 3. Attributions for Cisplatin: Gene level attributions defined by Shapley values.Additional file 4. Attributions for Docetaxel: Gene level attributions defined by Shapley values.Additional file 5. Attributions for Erlotinib: Gene level attributions defined by Shapley values.Additional file 6. Attributions for Gemcitabine PDX: Gene level attributions defined by Shapley values.Additional file 7. Attributions for Gemcitabine TCGA: Gene level attributions defined by Shapley values.Additional file 8. Attributions for Paclitaxel: Gene level attributions defined by Shapley values.Additional file 9. Visualization of the aggregated attributions: Mean and standard deviation over all drug data sets of each omics mean and summarized attributions.