A scoring procedure is then applied to each module to further assess their contribution in terms of classification. An important output of miRModuleNet is that it provides a hierarchical list of significant miRNA-mRNA regulatory modules. miRModuleNet was further validated on external datasets for their disease associations, and functional enrichment analysis was also performed. The application of miRModuleNet aids the identification of functional relationships between significant biomarkers and reveals essential pathways involved in cancer pathogenesis. The miRModuleNet tool and all other supplementary files are available at https://github.com/malikyousef/miRModuleNet/Increasing evidence that microRNAs (miRNAs) play a key role in carcinogenesis has revealed the need for elucidating the mechanisms of miRNA regulation and the roles of miRNAs in gene-regulatory networks. A better understanding of the interactions between miRNAs and their mRNA targets will provide a better understanding of the complex biological processes that occur during carcinogenesis. Increased efforts to reveal these interactions have led to the development of a variety of tools to detect and understand these interactions. We have recently described a machine learning approach miRcorrNet, based on grouping and scoring (ranking) groups of genes, where each group is associated with a miRNA and the group members are genes with expression patterns that are correlated with this specific miRNA. The miRcorrNet tool requires two types of -omics data, miRNA and mRNA expression profiles, as an input file. In this study we describe miRModuleNet, which groups mRNA (genes) that are correlated with each miRNA to form a  The World Health Organization (WHO) reported in 2019 that cancer is the leading cause of death in three out of four countries in the world . ApproximiRNAs are non-coding RNAs, roughly 22–25 nucleotides in length  and are There are presently seven major techniques in literature for the integration of miRNA-mRNAs, as shown in Correlation-based techniques have the following disadvantages. These techniques assume that one miRNA affects only one mRNA, an assumption that is not entirely true . Linear Another approach used for the integration of miRNA and mRNA interactions is the Bayesian network technique. All events that take place in a living system happen within a specific biological organization. In other words, the events that occur at the molecular level are not random. This understanding has motivated the development of statistical solutions for miRNA and mRNA integration . Along tAnother method that generates miRNA-mRNA groups is the probability learning based technique. In this approach, the interaction probabilities of known miRNA-mRNA pairs are estimated . HoweverWith the advancements in technology we now have access to data which describes different levels of molecular regulation from the same individual. These rich and complicated data sets require the development of novel techniques to integrate and understand this data. All the tools that we have surveyed above are based on statistical approaches. To the best of our knowledge, there are only two available tools that can adequately address the classification problem using integrated miRNA-mRNA groups. These bioinformatics tools are maTE  and miRcIn this study, we present a novel bioinformatics tool named miRModuleNet. miRModuleNet differs from our two previous approaches in that miRNA-mRNA integration has been developed using statistical information. In this paper, we have comparatively evaluated these three different grouping methodologies and showed the superiority of miRModuleNet against state of the art methods.In this study, miRNA and mRNA expression profiles which have been obtained from the same individuals have been used. Due to the aforementioned reasons, in this study we focused on cancer. In this context, 11 different cancer datasets were downloaded from The Cancer Genome Atlas (TCGA) data portal . The detmiRModuleNet was developed based on the generic approach named G-S-M. This generic approach was adopted by different tools such as SVM RCE, SVM-RCE-R , maTE Y, CogNet 1. G Component: Detect groups of genes2. S Component: Score the groups.3. M Component: Creates the model by training a classifier (Random Forest)As illustrated in groups that contain the genes from the mRNA (gene expression) data. This operation is represented as the G component whose output is the set of groups. Group names are the names of the biological entity such as miRNA names, where a group of genes may be targeted by that miRNA, a KEGG pathway name, or a disease phenotype name. Note that, in most of the cases each group has an important biological meaning. The resulting set of groups is indicated in the Groups box in In the first component G, bioD is a biological database, or another prior biological knowledge that will be used to create the train and Ctest, where the Ctrain is used to score the groups and to train the classifier in the M component. The Ctest is used for testing and reporting the performance.Assume that we have n samples and k genes in our dataset C. The C data is split into two parts as Csize(Gr) be the number of groups generated by the G component and let Gr be the collection of all the groups as Gr = . From now on, we will refer to one group of Gr as bioD_grp.Let m = bioD_grp in Gr is scored, as shown in size (Gr) different sub_data sets which are the sub matrices of the gene expression matrix Ctrain  that contains genes that belong to the group of bioD_grp. In Component S, each f, scoref) f = 1,..,size(G)]. Then we sort this list based on score and obtain grp_scores_sorted = . Let S (sub_data) be the k-fold cross validation procedure that computes and returns some performance measurements such as accuracy, specificity, sensitivity and Area Under the ROC Curve (AUC). We used AUC as the major performance metric to score for the sub_data. Next, we score all the groups using the S function which produces scores for groups, named as grp_scores and grp_scores = [ model and report the cumulative performance of the algorithm, we implement the procedure presented in f is one of the groups of Gr . The Performance Table in #G is the number of genes in the cumulative group. The output of this step is the Performance Table shown in the right hand side of In miRModuleNet tool is developed as a specific application of our G-S-M approach on the -omics data integration problem including miRNA and mRNA expression profiles. Hence, miRModuleNet makes use of the above-mentioned G-S-M approach with further additions. Before utilizing the G-S-M method, miRModuleNet includes some preprocessing steps as explained in detail below. The main idea behind miRModuleNet is illustrated in st step of miRModuleNet, both miRNA and mRNA expression profiles are cleaned by removing the columns containing the missing data. For miRNA-seq profiles, raw read counts were normalized to reads per million mapped reads (RPM). For mRNA-seq profiles, the raw read counts were normalized to Reads Per Kilobase Million Mapped Reads (RPKM). Subsequently, whole data at different ranges were normalized using z-score normalization. Second step identifies statistically important miRNAs and mRNAs that were to be used in the following steps. In the 3rd step, using statistically significant miRNAs and mRNAs, differentially expressed miRNAs and mRNAs are detected using the edgeR package  Let C= {gene1, gene2, ...genek } be the profiles of the mRNAs from data Dgenes2) Let Str ←∅ be the “Star” group for the miRNAi = I of each mRNA genei in C.3) Compute Ii {Ii}, Select the gene with the highest value of mutual information4) Let gene*= maxGenerate the “Star shaped” module that contains single miRNA and multiple mRNAs.3 approach suggested by . At the end of this step a Scoring Table is produced containing the miRNA in rows and the score of the corresponding mRNA group in the columns. In the 7th step, a machine model is trained using the top ranked groups. In other words, the machine learning model which uses Random Forest is trained via only considering top f groups. This means that miRModuleNet is using all of the genes within top f groups in a unified manner. The default value of f is set as 10 and miRModuleNet generates 10 different machine learning models where each model is trained using a different number of groups from 1 to 10. The user can also change the value of the f. Classification strategy is explained more in detail in the following section. Then the last step is the evaluation step that uses the test part.The 6In this study, the Random Forest algorithm , which ip-Value to each element in the aggregated list, which describes how good each element/entity was ranked compared to the expected value.While generating the model, 100 fold Monte Carlo Cross Validation (MCCV) was used in the learning phase . In ordeThe KNIME Analytics platform is used for the implementation of miRModuleNet . The KNIAs it can be seen in Results are obtained after running the KNIME workflow, which is shown in The performance of machine learning models can be evaluated through several quantitative metrics. In this respect, statistical metrics such as Accuracy, Sensitivity, Specificity and Precision could be calculated by constructing the confusion matrix. For the problems involving imbalanced data, it is essential to prove the consistency of the results. In this regard, Area Under the Curve (AUC) metric is reported as an accurate metric in terms of evaluating the performance results in such problems .miRModuleNet tool uses (MI) to detect the relationships between miRNAs and mRNAs. In order to identify the optimal value of the MI threshold, we experimented with three different values . As stated above, we selected 0.25 as the optimal threshold. In our comparison, the AUC value versus the number of genes is taken into account. Such a comparison on THCA data is demonstrated in In this study, we have tested miRModuleNet using 11 different cancer datasets presented in In addition, in terms of performance, miRModuleNet has been compared with other existing tools i.e., SVM-RFE, maTE and miRcorrNet. These tools differ in terms of the data they use and the way they produce results. While miRcorrNet and miRModuleNet both use miRNA and mRNA expression profiles, SVM-RFE and maTE tools use only mRNA data. In addition, while miRcorrNet, miRcorrNet and maTE give the results on group level, the SVM-RFE tool gives the results directly at the gene level. In other words, miRcorrNet, maTE and miRModuleNet tools give their results by building a Random Forest model over the top 1 to 10 cumulative groups of genes. On the other hand, SVM-RFE tool gives its results using different levels of genes, i.e., 1, 2, 4, 6, 8, 10, 20, 40, 60, 80, 100, 125, 250, 500 and 1,000 genes. In order to make a fair comparison of the existing methods involving different approaches, it has become necessary to determine benchmarks at both the group level and the gene level. The comparison level for miRcorrNet, miRModuleNet and maTE, which produced results at the group level, was determined as two according to the number of genes criterion. When these three tools used two as the group level, the lowest number of genes was found to be 7.48, and the highest used number of genes was found to be 141.26. Therefore, it was decided to use gene levels 8 and 125 to be able to include the SVM-RFE tool in the comparison. In In order to better understand the disrupted mechanisms of the disease at the molecular level, functional enrichment analysis was carried out. Hence, we investigated whether the obtained results have biological meaning. For this purpose, GeneCodis  and DAVI14 and E30). The constructed pathway-pathway interaction network is presented in It can be observed from In order to check the robustness and reliability of miRModuleNet, an external dataset was considered. In this context, the GSE40419 dataset  was downTesting was carried out as follows. All genes for specific diseases in the train data and significant genes obtained by miRModuleNet are kept in separate files. To make a fair comparison, the number of random and significant genes was determined as 1, 5, 30, and 50. Subsequently, using the test KNIME workflow, the results were obtained both using these random genes and using the significant genes found by miRModuleNet. While the accuracy obtained using only 1 random gene was 51%, the accuracy reached 87% when the most important 1 gene found by miRModuleNet was used. Likewise, when comparing 50 genes, accuracy increased by approximately 11% with miRModuleNet, and reached 95%. Summary of these results are shown in In bioinformatics problems, the biological value that the tool is providing is as important as the comparative performance evaluation with existing tools. In this section, we explore those features and provide a biological validation of our tool.p-Values, which are assigned by the RobustRankAggreg method. In order to show the biological relevance of our findings, we refer to the miRNA - Disease association databases that are widely used in the literature. These databases are HMDD  imply that miRModuleNet maintains good performance metrics when applied to new independent data.Exploring the biological functions of differentially expressed genes through the integration of different types of -omics data such as miRNA and mRNA expression profiles remains an important research topic. However, the problems associated with how to best assess the repression effect on target genes using integrated miRNA/mRNA expression profiles are not fully resolved. To address this problem, we have proposed a novel tool, miRModuleNet, which conducts a machine learning-based integration of two-omics datasets to detect miRNA-mRNA modules that are most significant to the classification task. The tool detects the miRNA/mRNA groups, which are later subjected to Rank procedure. The strength of miRModuleNet is that the identified set of genes that are represented in groups are guaranteed to distinguish two classes (cases vs. controls) and may serve as a biomarker for the specific disease under investigation.


Raman Spectroscopy has long been anticipated to augment clinical decision making, such as classifying oncological samples. Unfortunately, the complexity of Raman data has thus far inhibited their routine use in clinical settings. Traditional machine learning models have been used to help exploit this information, but recent advances in deep learning have the potential to improve the field. However, there are a number of potential pitfalls with both traditional and deep learning models. We conduct a literature review to ascertain the recent machine learning methods used to classify cancers using Raman spectral data. We find that while deep learning models are popular, and ostensibly outperform traditional learning models, there are many methodological considerations which may be leading to an over-estimation of performance; primarily, small sample sizes which compound sub-optimal choices regarding sampling and validation strategies. Amongst several recommendations is a call to collate large benchmark Raman datasets, similar to those that have helped transform digital pathology, which researchers can use to develop and refine deep learning models. Biomedical applications of Raman Spectroscopy (RS) have been steadily growing over the decades as technology matures. Applications range from the label-free staining of histology slides to determine the biochemical composition of a sample to classification tasks to determine the presence and grade of disease, including determining tumour margins during cancer surgery . RS has Machine learning (ML) has long been used to classify Raman spectral data. The key feature of ML is that a model in some way learns from data. It can therefore be described as a data-driven approach to modelling. Although there are many variations, the most common ML model used in biomedical RS is Principal Component Analysis–Linear Discriminant Analysis (PCA-LDA) . PCA redp-values, it is becoming increasingly important to become literate in ML [However, ML, traditional or otherwise, is not without its limitations. Just as medical researchers need to understand something of the statistical science of hypothesis testing, and the debate and misunderstandings regarding te in ML . One of te in ML . Indeed,te in ML . Prominete in ML . Though k-fold cross-validation (CV). This repeats the training/test split k times such that all of the data are sequentially used in the test set, thus producing k estimates of the models performance. The average performance is then given, sometimes with an accompanying measure of variance. Taken to the extreme is leave-one-out CV (LOOCV), in which the test set comprises a single data point, thus training the model on the maximum possible amount of data.The pertinent point of interest in how well an ML model performs is how it would cope with previously unseen data in the clinical setting: its generalisability. It is possible for a model to simply memorise data, thus giving a perfect classification for data it has seen with no ability to generalise to unseen data. This is known as over-fitting. Ideally, performance would be tested with a newly created dataset. However, there are many practical limitations to collecting new data, particularly in clinical research, which can be expensive and time-consuming. A common compromise is to split the existing dataset into a training and a test set; the former being used to create the model, the latter simulating the process of collecting a new dataset and being used to test the models performance. This requires holding out a proportion of the data during training. This can compound the problem of small datasets. Therefore, an extension of single train/test splitting is In addition to the training/test split, sometimes an additional split is made called the validation set. The validation set is used to optimise the model hyper-parameters (or even guide the choice of ML model): choices about the model that a researcher makes which influence its classification ability. Similar to how a model can over-fit the data, the hyper-parameters can be selected such that it performs well for a given test set, but fails to generalise well. This has been described as over-fitting at the second level of inference . The valRegular CV splits the data such that the same test data are never used twice. Repeated CV iterates this process several times such that many permutations of possible test sets are used . It is not commonly used as it is computationally expensive, though it can reduce the variance of a model’s estimated performance when sample sizes are small .This literature review follows the principles set out in the PRISMA  guidelines . The datMany studies used several ML models, conducted analyses on different subsets of their data and/or compared several pre-processing techniques, producing a multitude of results. For instance, several studies compared the performance of different machine learning models, often traditional ML models such as LDA against deep learning ML models, such as CNNs. For ease of comparison, and to mitigate selection bias on our part, the best-performing model and/or dataset is presented in From each study, we extracted: (1) authors, institution and year of publication; (2) type of cancer and sample substrate; (3); ML models used; (4) validation strategy; (5) number of patients and samples; (6) number of spectra; (7) how the data were split during validation; (8) number of classes classified; (9) performance metrics.A total of 25 studies were identified , 18 of wXia et al.  probed tThe same team used a similar set-up to collect two datasets taken under conditions of ‘illumination’ and ‘no light’ . These dThe last publication from this team used a similar set-up and dataset to compare the performance of a custom-built CNN against PCA-LDA and PCA-SVM  . They foIt is not clear if the tissues used in these three studies are the same. However, in all cases the diseased and healthy samples were obtained from the same subjects.k-fold versus a LOOCV strategy. Finally, they compared a PCA-LDA and a PCA-QDA  classifier. Using these methods, they found that taking the average spectrum of a sample yielded a better performance than a point-wise approach and with PCA-QDA typically performing better than PCA-LDA, though not across all sub-group analyses. LOOCV resulted in lower error rates compared to k-fold CV for the ‘all cancer’ versus ‘healthy’ analysis, but this was reversed for the sub-group analysis, which consisted of smaller sample sizes.Also investigating oral cancers, Jeng et al. interrogated cryopreserved samples, seeking to discriminate between healthy and cancerous tissues . They fuTwo studies focused on nasopharyngeal cancers. Zuvela et al. used an in vivo set-up to collect data during endoscopy . They emThe same team expanded their investigations with a similar study which included many more subjects, samples and spectra with a similar recruitment protocol . Of all Qi et al. adopted a novel approach to classify Raman spectra of lung tissue as adenocarcinoma, squamous cell carcinoma or normal in a pairwise fashion . They trChen et al. also discriminated between lung cancers, as well as glioma, a common brain cancer, using spectra taken from blood serum . They coTwo studies from the same team also explored gliomas. Riva et al. took fresh-tissue biopsies and classified healthy versus cancerous tissue using the traditional models of Random Forest (RF) and Gradient Boost Tree (GB) . The latBury et al. also analysed brain tissue, attempting to discriminate the primary source of metastatic brain cancers . Seven sMehta et al. used 35 serum samples from meningioma patients and compared them to 35 samples from healthy controls in an attempt to develop an approach to diagnose brain tumours using minimally invasive techniques . ApproxiKoya et al. created Raman maps from ex vivo breast tissue and classified spectra as cancerous or healthy . It is tMa et al. also classified breast tissue using a CNN . They coZhang et al. interrogated five breast cancer and one healthy breast cell lines with RS , using PLee et al. explored Raman spectra of extracellular vesicles derived from blood serum samples as a biomarker for prostate cancer in combination with a CNN . This waWu et al. interrogated biopsy samples taken during endoscopy, classifying spectra as normal, adenomatous polyps or adenocarcinomas . They foIto et al. developed a boosted tree model from serum samples taken from suspected colorectal cancer patients, classifying them into four categories, colorectal cancer, adenoma, hyper-plastic polyps and neuro-endocrine tumours, in a pairwise fashion . They ack-nearest neighbours and logistic regression. Instead of selecting the best single model, the outputs of all models were used to create a soft voting classifier, allowing each model to ‘vote’ on an outcome, and the consensus across all models was taken. Splitting the data 50/50 into train/test sets, this was repeated 1000 times to build a spread of estimates. This method achieved an accuracy of 90.5%.Serzhantov et al. used an ensemble of traditional ML models to classify skin tissue as cancerous or normal . The modBaria et al. compared PCA-LDA and PCA-ANN for the task of classifying spectra taken from cultured cell lines to distinguish between three skin melanoma genotypes . The LDASantos et al. classified skin samples with spectra from the high wavenumber region using PCA-LDA, distinguishing between melanoma and not-melanoma . They acDaniel et al. compared a PCA-LDA model to a PCA-ANN model in classifying cervical tissue as healthy, neoplastic or malignant . In addiChen et al. used RS on serum to classify ovarian samples as normal, cystic or cancerous in a two-step binary classification regime . The firHe et al. interrogated ex vivo renal tissue seeking to identify cancerous tissue and demarc surgical boundaries as well as classify those tissues . Althougk-fold CV, which were either 5-fold or 10-fold. These are common default values in ML as they have been found to produce a good balance between bias, variance and computational cost [k-fold CV strategy may be preferable, though the precise number of k depends on a number of interacting factors such as the sample size, signal-to-noise ratio of the data and the model used, which are not easy to reconcile. Most of the reviewed studies had low sample sizes, with an average of 82 subjects. When the sample size is small, LOOCV has been shown to have a high bias and variance, while k-fold strategies had a low bias and their variance can be further reduced by performing repeated splits [k-fold CV strategy. Jeng et al. found that LOOCV yielded higher accuracies than k-fold CV  for a binary classification task, but this was reversed in a three-class problem. The study did not compare the variance or bias of these performances.Nearly all of the reviewed studies conducted some kind of splitting of the data in order to produce train, test and, in some cases, validation sets. Several partitioning strategies were used . The mosnal cost . Howevernal cost . Howeverd splits . One of k-fold CV splits data into folds at random. Shu et al. employed a variant called Venetian Blinds, which systematically assigns data to folds [Standard to folds . There ito folds , and verRepeating CV several times has been shown to be effective at reducing the variance of generalisation estimates . SerzhanSingle-split validation refers to the method of dividing the data into different sets just once, as opposed to CV methods which iteratively split the data to give several performance estimates. Although single splits have been shown to be unbiased, they have a high variance, particularly with small datasets . Single Most of the reviewed studies classified individual spectra, while some classified average spectra ,28. ManyOf those studies that split the data at the level of spectra, the best accuracies of 90%, 96.6%, 97.7%, 93.8% and 94.8% were achieved ,26,29,35For many studies, one sample was taken per subject. However, some studies took multiple samples from the same subjects, which introduces an additional strata into the the hierarchy. For instance, Zuvela et al. took 113 samples from 60 patients  and Shu Two studies took only the average spectrum from a single sample, thus flattening the hierarchical structure of the data and bypassing this issue ,28. BothThere are two methods by which sample heterogeneity is currently incorporated in the literature. Averaging spectra provides one means, while including multiple spectra from a single sample provides another. There has been no direct comparison of these methods in the reviewed literature. A priori, there is no reason to suspect one will work better than the other and will likely depend upon the intended application. For instance, a mean spectrum typically has a higher signal-to-noise ratio than individual spectra, and a model trained on the former may not generalise to applications requiring individual spectra to be classified, such as post-surgery cancer edge detection. It is also an open question whether averaging spectra from a sample before analysis would be an effective use of data for deep learning models, which are notoriously data intensive. Overall, it is not clear that averaging spectra will always provide a benefit, and the decision to do so should take into account the nature of the application and the ML model being used.Many studies took healthy and diseased samples from the same patient ,36,37,38Some studies were able to obtain healthy spectra from in vivo tissue, usually via endoscopy ,35,40. TWu et al. bypassed this potential problem by obtaining independent samples by including patients who had biopsies taken of suspicious lesions detected during endoscopy, which were later determined to be normal by traditional histopathology .There are other potential sources of label noise. It could occur when Raman maps are taken of an area which are given a single label, as Koya et al. did, but it is possible that the area is not homogenous in its class, thus mislabelling some spectra. This is only a problem if spectra, rather than the entire map, are being classified, and might be mitigated if average spectra are taken, to the degree that one class dominates the sample. Santos et al., as well as taking the average sample spectra for training, also explicitly controlled for this by designating certain samples as heterogeneous during histopathological assessment : samplesAnother source of label noise that is perhaps more well understood comes from mislabelling amongst domain experts such as histopathologists. The negative impact of inter-observer variation upon ML models has been demonstrated in the context of circulating cancer cells . A way tData augmentation is a technique by which the amount of spectra is increased by adding replicated spectra to the data and adding noise and/or other alterations to them. This is a technique used in deep learning to both increase the sample size and to inject noise into the data so that the model is less likely to overfit to the training set. Not only does this increase the sample size, which is important for data-intensive deep models, but it also makes the model robust against irrelevant features in the data by forcing it to pay attention to what is not being distorted. This has the effect of regularising the data, smoothing out the learning process. This is well established and widely used in other fields using deep learning, particularly image recognition . This prOf the studies reviewed here using deep learning, six employed data augmentation. This was achieved via several strategies. Chen et al. added white Gaussian noise of varying levels to the spectra, increasing the training data by a factor of five . Lee et  spectra , increas spectra . Shu et  spectra .Only two studies assessed the impact data augmentation had upon classification performance. Chen et al. found it consistently increased performance across multiple subset analyses , and Ma Augmentation is traditionally only performed upon the training data, as data inflation and its regularisation effect is only pertinent during training. The test data are a representation of the general population of interest and adding noise could be considered dangerous as it may then become less representative. Four of the reviewed studies which performed data augmentation did so on the entire dataset before splitting into training and test sets ,27,35,51There are numerous pre-processing techniques involved in the analysis of Raman spectra, such as baseline correction, smoothing and normalisation. However, one of the putative benefits of CNNs is that they can automatically perform feature selection and pre-processing at the same time. There is a degree of arbitrariness to many pre-processing steps, with evidence suggesting that most pre-processing techniques, and their numerous parameters, actually worsen the subsequent classification . FindingLee et al. compared ‘baseline corrected’ data to raw data and found that for traditional ML models baseline correction improved performance . HoweverThis suggests that CNNs can classify data without pre-processing. There is even a suggestion that the architecture is able to exploit diagnostic information present in raw data too subtle for traditional models to detect and usually discarded. However, the results are not ubiquitous.Yan et al. pre-processed data by smoothing with a Savitsky-Golay filter and baseline removal via asymmetric weighted penalty least squares . Comparek-nearest neighbours (KNN), RF and SVM, but the largest increase in performance was observed in the CNN (accuracy: pre-processed = 81.3% vs. raw = 75.0%).Wu et al. found similar results. They performed baseline correction and spectral smoothing using the Vancouver Raman Algorithm and compared this regimen to raw data . The proAlthough there is some suggestion that using CNNs could preclude an explicit pre-processing stage, this is not clearly established in the literature. There are a plethora of pre-processing techniques and it may be that some techniques were better suited to particular datasets than others—i.e., the pre-processing step could itself introduce over-fitting and CNNs trained on raw data would generalise better, even though their performance would be worse during training and testing. This is just speculation until it is more thoroughly explored. In all the above cases, where traditional ML models were used, they were improved by pre-processing. If pre-processing is to be conducted, the choices made to select the best method should be part of the model building process, essentially becoming another model hyper-parameter. The best pre-processing method should then be selected based on a validation set, and only tested on the test set once the final pre-processing method has been selected.Due to the heterogeneity of the reviewed studies, no attempt was made to statistically aggregate the performance of traditional and deep learning models between studies. However, many studies performed such a comparison themselves. Prima facie, deep models consistently outperform traditional models, sometimes improving accuracy only by a few percent, but often by a large amount. However, given the methodological limitations discussed above, together with the propensity of more complex models to over-fit to data, particularly small datasets, this observation needs to be taken with caution. It should also be noted that deep learning models have more hyper-parameters to exploit, which can make them easier to overfit during hyper-parameter selection.ML studies, particularly deep learning, are often criticised for lacking interpretability, providing results without alluding to the underlying theory ,54. In tTraditional models, being more simple, are often more amenable to interpretation. Riva et al. identified 19 Raman shifts as biochemically pertinent to classification via GB, as the technique involves an interpretable feature selection step . Zuvela There currently exists no explicit standard for conducting and reporting clinical applications of machine learning. However, reviews in adjacent medical domains have highlighted similar problems. A review of the literature focusing on deep learning applications assessed against clinicians found several areas to improve . First, In addition, studies should detail enough of their methodology to be reproducible, which has been found to be lacking in adjacent domains . In the If the full potential of deep learning for biomedical RS is to be exploited, then large, publicly available datasets need to be curated. Despite the complexity of curating medical databases due to ethical concerns, such databases do exist for the development of deep learning models, such as the Cancer Genome Atlas . No suchWhen data are split into training, validation and test sets, the level of the split should be conducted at the highest level . The chosen level should be explicitly stated.If model hyper-parameters are explored, including pre-processing, these should be selected based on a validation set and once selected only then tested against the test.All pre-processing and data augmentation techniques should be described in sufficient detail to allow for replication. While there is doubt regarding the need for pre-processing with CNNs, it would be beneficial for studies to include results from both raw and pre-processed data.All the hyper-parameters used in the model should be stated and the hyper-parameter search strategy should be reported.Where possible, consensus pathology should be used to confirm class labels.Where possible, attempts should be made to relate the results to the underlying biochemistry by interrogating model outputs, in addition to the traditional methods of comparing average spectra per class, peak comparisons and obtaining difference spectra.k-fold and/or repeated CV. As these methods provide many generalisation estimates, standard deviations of performance metrics should be reported.Resampling validation strategies should be used, such as Taking all this into consideration, together with the literature reviewed here, we make the following recommendations:This review has found several common methodological issues in the recent literature that may be leading to an over-estimation of the ability of ML models to classify Raman oncology samples. Chief amongst them is the perennial problem in medical research of small datasets. This exacerbates several other issues including sub-optimal validation and sampling strategies such as using single splits instead of CV and failing to use a validation set when selecting hyper-parameters, including pre-processing methods and their attendant parameters. Regardless of the sample size, data are split into training/validation/test sets, but the highest hierarchical level in the data results in over-optimistic performance estimates. Thus, it is likely that the generally high performances seen in the literature are over-estimating true model performance. This is consistent with other medical domains employing ML .This review is limited by searching English-language publications and by not by considering the grey literature, in particular conferences which provide rich sources of discussion. The review has also omitted several enhanced RS techniques such as SERS, sacrificing depth for brevity. Due to the heterogeneity of the studies, it was not possible to conduct a meta-analysis to provide a quantitative exploration of the literature. Contrary to the PRISMA guidelines, publication bias was not assessed. This is particularly difficult to assess in non-randomised trials and when meta-analysis has not been performed.Future research could explore whether deep learning with sufficient sample sizes could make some of the numerous pre-processing techniques obsolete, or whether hand-crafted feature selection is an important junction in which to insert domain knowledge. Additionally, the use of data augmentation when using deep learning is relatively unexplored in the context RS and studies specifically looking into the facet would benefit the field. Regardless, most recommendations concern methodological issues. The reviewed papers are generally at the proof-of-concept stage, justifying their small sample sizes, but this necessitates more methodological rigour in order to draw accurate inferences. If the technique is ever to progress into accepted clinical practice, much larger studies need to be conducted, preferably populating a publicly accessible dataset, with the transferability between settings being established.


AUC) can be achieved on imbalanced data sets using DOB-SCV instead of SCV. We investigated the effect of the oversamplers on this difference. The study was conducted on 420 data sets, involving several sampling methods and the DTree, kNN, SVM, and MLP classifiers. We point out that DOB-SCV often provides a little higher F1 and AUC values for classification combined with sampling. However, the results also prove that the selection of the sampler–classifier pair is more important for the classification performance than the choice between the DOB-SCV and the SCV techniques.Nowadays, the solution to many practical problems relies on machine learning tools. However, compiling the appropriate training data set for real-world classification problems is challenging because collecting the right amount of data for each class is often difficult or even impossible. In such cases, we can easily face the problem of imbalanced learning. There are many methods in the literature for solving the imbalanced learning problem, so it has become a serious question how to compare the performance of the imbalanced learning methods. Inadequate validation techniques can provide misleading results , which leads to the development of methods designed for imbalanced data sets, such as stratified cross-validation (SCV) and distribution optimally balanced SCV (DOB-SCV). Previous studies have shown that higher classification performance scores ( One of the crucial challenges in machine learning and data mining is how to achieve the desired classification accuracy when handling data with significantly skewed class distributions. In such a case, the accuracy of the classification of the majority class elements (samples) is better than the classification of the minority class samples in most cases. This challenge eventually led to “learning from imbalanced data”, the birth of a new field of statistical learning.The importance of this research area is continuing to grow because the problem can appear almost anywhere where the data belongs to more than one category. A popular example from the financial field is bank card fraud detection ,2, whereThere are roughly two main approaches for handling the imbalance problem at the data level—undersampling and oversampling. Undersampling techniques are based on removing samples from the majority class, but it can lead to information loss. The oversamplers augment the data sets with replicas of the minority samples or with similar synthetic ones. Oversampling usually gives a better result than undersampling , but it k groups randomly. Nevertheless, the problem of data shift can weaken the validation [folds) will be similar to the initial distribution. This method is the so-called stratified cross-validation (SCV) [In a real application, it is very important to know about the performance of the given classifier. In order to validate the performance, the so-called k-fold cross-validation is one of the most commonly used methods in the literature. The basic idea of this method is to split the elements into lidation . If we rlidation  since thon (SCV) . Howeveron (SCV) .A validation technique called “Distribution optimally Balanced SCV” (DOB-SCV) can be used to avoid the covariate shift problem. The main idea of this validation is to select the closest neighbors and put them in different folds to keep the distribution in the folds close to the original distribution .In this paper, we investigate the SCV and the DOB-SCV methods. The difference between the two techniques is known from the literature , but theTo design a complex experiment, we selected the commonly used oversampling methods , classifThe rest of this paper is organized as follows. In this section, we summarize the basic concepts necessary to understand the results of the experiments. Interested readers can read about the validation methods, the classifiers and samplers involved in the study, and the data sets and measures used.k-fold cross-validation. This method randomly shuffles the samples of the data set and divides them into k parts (folds) of (nearly) equal size. Then, for each fold f, the other k − 1 folds are used to train a classifier, and the fold f is used to validate the obtained model according to a suitable measure (The methods that are the subject of our study were developed from the  measure . The perk folds such that every fold has (nearly) the same percentage of samples from minority and majority classes as the complete set. One fold is selected for the test, and the rest is used for training (Algorithm 1). The further steps of the procedure are the same as for cross-validation.Algorithm 1 Fold generation of SCV (based on Ref. [ on Ref. )Require: k            // number of foldsRequire: // classesEnsure: // generated foldsfor n do      if k) then                  end if      for k do      n samples from                         S            end for      end for The stratified k-folds cross-validation splits the data set on k nearest neighbors into different folds and repeats this until the samples from the original set run out (Algorithm 2). After partitioning the data set, the procedure continues as specified for cross-validation.Algorithm 2 Fold generation of DOB-SCV (based on Ref. [ on Ref. )Require: k             // number of foldsRequire: // classesEnsure: // generated foldsfor n dowhile do                                          for k do                                                               if then                 // end for j                      end if                 end for            end while      end for Zen and Marinez pointed out that the distribution of the folds (and thus the training and test sets formed from them) can change strongly even if the proportions of samples from different classes are preserved by the SCV, especially if the number of samples in one or more classes is small. They suggested that nearby points belonging to the same class should be placed in different folds , but theIn this section, we briefly introduce the classification procedures involved in the experiment.The k-nearest neighbor classifier (kNN) is a very simply supervised machine learning method. The classification of each sample is based on the vote of its nearest neighbors. All neighbors vote that the element belongs to their class. In the simplest case, the decision is made by majority rule, but it is also customary to weight the votes by the reciprocal of their distance from the sample to be classified. It is easy to see that this learning method is sensitive to the training set and selection of metrics. More detailed information can be found in Refs. ,24.The support vector machine (SVM) is a supervised machine learning method developed by V. Vapnik. The main idea of this classifier is to find the margins between two classes based on the support vectors determined from the training set. In the linear case, the separating hyperplane (whose task is to separate the samples of the different classes) can be determined based on the support vectors by maximizing their distance from the margins. More detailed information can be found in Refs. ,25.Multi-Layer Perceptron (MLP) is a neural network with hidden layers and backpropagation training. One of the important advantages of this classifier is its ability to classify non-convex sets. However, finding the global optimum is not guaranteed. More detailed information can be found in Ref. .The decision tree is one of the well-known classifiers that reduces complicated decisions to a series of simple ones. The trained model can be considered a tree-shaped flowchart of elementary decisions, where the vertices have an attribute, the edges between the vertices with the result of this test  are labeled, while the leaf tips the decision itself  ,27.The research was carried out using sampling methods that have proven their effectiveness on a wide range of imbalanced data sets , and theseed) with its nearest minority neighbors (co-seeds). Unfortunately, some synthetic samples may be more similar to the majority samples than to the minority ones, which may harm the classification. The SMOTE-TomekLink [named samplers).The SMOTE (Synthetic Minority Over-sampling Technique)  generateomekLink  handles omekLink  also appomekLink . CCR [33omekLink  connectsomekLink . The puromekLink  is similomekLink  uses a somekLink . The G-SIn addition, we also included automatically generated oversamplers in the study, some of which do not use clustering, some of which use DBSCAN clustering , and somWe examined the effect of the two validation techniques on data set collections of the KEEL repository ,42, whicminority and the majority classes are composed of 1–4 clusters. The samples were randomly drawn from a normal distribution around the vertices of a D-dimensional hypercube, where D is the number of features. The length of the sides, which affects the separability of the classes, was set to 1.4. The imbalance ratio (ratio of the number of majority and minority samples) is around 8 and 16  A total of 320 data sets were created.To observe the effect of within-class clusters, we also generated synthetic data sets using scikit-learn . Each daAccording to our experience, these values result in data sets that cannot be classified too easily with the classification methods included in the study, which is important because neither the effect of the oversamplers nor the effect of the validation methods could be observed in easy problems.F1-scores  and TN (true negative) as the total number of correctly classified minority and majority samples, respectively, over the validation folds. Similarly, the FN  and FP  are defined as the total number of wrongly classified minority and majority samples. In this paper, we primarily use F1 and AUC scores.The evaluation tool we used  defines The experiment was conducted as follows. First, we created the training and test sets required for the 5-fold, three-repeat SCV and DOB-SCV methods for each KEEL and synthetic data set mentioned in k = 5 and 7, using both voting schemes mentioned in D, D is the number of the features of the samples. From the family of decision tree classifiers, the Classification and Regression Trees [F1, AUC, G, and Acc) among the different parameterizations of the classifiers was taken into account for each oversampled data set.Then, we trained the different classifiers on the original training sets (for a baseline) and the oversampled training sets and evaluated the obtained model on the corresponding test sets according to the rules of repeated cross-validation. Among the classifiers, the kNN was performed with on Trees  were useThe statistical analysis of the results was performed separately for the data sets belonging to KEEL and the synthetic data sets, and in some of the tests, we also treated the samplers known from the literature and the generated samplers separately.AUC values can be achieved with DOB-SCV than SCV because DOB produces more homogeneous folds. They also showed that the differences become stronger as the level of imbalance increases [According to Moreno-Torres et al., DOB-SCV is slightly more effective than SCV . The comncreases .Our first experiment aimed to check whether differences between DOB and SCV can be observed even with the use of different samplers, and, since randomness is an essential element of the samplers, we also wanted to check whether the oversamplers alone—using the same validation method—do not cause significant differences in the estimated performance of the classification models. For this purpose, we experimented with each sampler–classifier pair using the same DOB folds twice. The results of the two runs are referred to as DOB and DOB2.F1, AUC, G, and Acc measures. Next, we applied Nemenyi’s post-hoc test [With a Friedman test, which we chose based on the work of Demšar , we examhoc test , which sF1, AUC, G, Acc) achieved by the sampler–classifier pairs on the data sets. The distributions cannot be considered normal, so we performed Spearman’s rank correlation analysis [N_min) and majority samples (N_maj), the imbalance ratio (D), and two measures to characterize the level of overlap between classes in the data sets (R and AUG_R). The R-value of a data set consisting of two classes is the proportion of elements in the data set that have more than k nearest neighbors. The augmented R-value (AUG_R) is a version of the R-value that considers the size of the classes. The analysis was performed with We examined the correlation between some properties of the data sets and the results diffIn IR and the relative difference of the Acc values. However, the relative differences show a moderately strong positive correlation with the R or AUG_R values in the case of DTree, kNN, and MLP.In the case of the decision tree and the SVM, there is practically no correlation between the IR and the IR and the In the case of the synthetic data sets, the correlations between negative . HoweverAdditionally, we can see again that the relative differences show a positive correlation with the degree of overlap between the classes to a greater or lesser degree, and also with the number of clusters within the classes.Although the observation of Lopez’s et al. on the relationship between IR and To see behind the numbers, we visualized the relationship between the validation methods and the performance scores of the sampler–classifier pairs on violin plots. The shape of the “violin body” is determined by the distribution of the results obtained for a particular data set by a certain classifier combined with different oversamplers. The measure was also fixed. The red and the black lines in the diagrams show the mean performance of the classifier without sampling (ws) and with sampling, respectively.winequality-white-3-9_vs_5 (Based on the literature, the nature of the data determines which classifier can be used more successfully without applying any oversampling ,57. It a3-9_vs_5 .The difference in the effect of validation is less striking; however, there is a slight but relatively stable difference between them . Stable,AUC values achieved by variously named samplers and DTree classifier differ the most using the two validation methods.However, let us see some examples where differences arose. R_AUG values are quite high for most of these data sets, and there are a few cases where the number of minority samples is extremely small. When these two things happen simultaneously, it is difficult to predict the outcome. For example, on the lymphography-normal-fibrosis data set, the SCV partitioning provides such folds that oversampling could only worsen the initial result. On the other hand, we can see the poker-9_vs_7 data sets where the classification performance improved in more cases when we sampled the folds generated by SCV than by DOB. In our opinion, the location of the samples plays a crucial role, which is not described by many measures. Furthermore, selecting the most representative samples from the clusters has a significant role in overlapping classes.Note that the For each classifier, we can find examples where there are larger differences between the two types of validation, but these differences of a few percent are, in many cases, insignificant compared to the effect of the oversampling.Earlier, we saw that the number of clusters within the classes shows a negative correlation with the classification result and a positive correlation with the relative differences of the validation methods (We again used violin-plots to represent the results of our experiments, but the data has now been plotted in a grouped form. The values on synthetic data sets with the same number of minority and majority clusters were placed on the sample plot. The plots belong to different classifiers, and different metrics were placed in different figures .We can see the results obtained with the generated samplers from the synthetic data sets. It can be noticed that even despite the large number of oversampling methods, the distribution of the performance scores estimated by DOB and SCV is very similar.F1 of the classifications combined with oversampling.Compared to the classification without oversampling, the classification combined with oversampling is significantly better when the number of clusters of the minority class increases. On the other hand, the change in the number of clusters of the majority class has less influence on the efficiency in terms of F1 decreased radically, regardless of the validation method when the number of the minority clusters was increased and smaller clusters were formed. The large IR alone does not cause problems in classification, but difficulties can arise if classes overlap. It is worth noting that we found a stronger relationship between the R values and the F1 values—for both validations—than between the number of clusters and the F1 value.It is interesting to note that the performance of the SVM and the MLP concerning Based on our experiments, we cannot confirm the statement  that theThe classification combined with oversampling is significantly better when the number of clusters of the minor class increases. See F1, AUC, G, Acc achieved by the sampler–classifier pairs on the data sets, and conducted visual data analysis.We have published the result of our extensive study involving 420 data sets about the SCV and the DOB-SCV methods when oversampling is used before classification. We have verified the differences between DOB-SCV and SCV with a suitable statistical test, examined the correlation between some properties of the data sets and the values F1, AUC G, Acc values can be achieved with DOB-SCV than SCV in combination oversamplers and classifiers;In general, slightly higher Based on our experiments, we can not confirm the statement  that theWe can state that there is a difference between SCV and DOB-SCV in favor of DOB-SCV when the number of clusters within the classes or the volume of overlapping between the clusters increases;The selection of the sampler–classifier pair is much more critical for the classification performance than the choice between these two validation techniques.We would like to summarize the most important results below:Our results could help researchers to focus on the part of the training process that can significantly impact classification performance and to choose the right validation method for the given situation.


Accurate classification of cancers into their types and subtypes holds the key for choosing the right treatment strategy and can greatly impact patient well-being. However, existence of large-scale variations in the molecular processes driving even a single type of cancer can make accurate classification a challenging problem. Therefore, improved and robust methods for classification are absolutely critical. Although deep learning-based methods for cancer classification have been proposed earlier, they all provide point estimates for predictions without any measure of confidence and thus, can fall short in real-world applications where key decisions are to be made based on the predictions of the classifier. Here we report a Bayesian neural network-based model for classification of cancer types as well as sub-types from transcriptomic data. This model reported a measure of confidence with each prediction through analysis of epistemic uncertainty. We incorporated an uncertainty correction step with the Bayesian network-based model to greatly enhance prediction accuracy of cancer types (> 97% accuracy) and sub-types (> 80%). Our work suggests that reporting uncertainty measure with each classification can enable more accurate and informed decision-making that can be highly valuable in clinical settings. Existence of heterogeneity in cancer assumes significant importance for the therapeutic interventions. Different types and sub-types of cancer are driven by distinct molecular factors and often require specific anti-cancer treatment8. Therefore, an accurate classification method can greatly aid in choosing appropriate treatment strategies specifically targeted towards these types and sub-types15.Recent explosion in genomic, epigenomic and transcriptomic data has provided us a glimpse of the extent of molecular heterogeneity in cancer and has led to classification of cancer into different types and sub-types based on their molecular signatures17. This, in turn, can greatly enable accurate prediction of disease type and progression19. There are, however, several challenges associated with the use of transcriptomic data for classification of cancer types and sub-types. These datasets are high-dimensional and the expression values of many genes are intertwined in a highly complex manner20. In addition, the measurements from two different samples are rarely obtained under the same conditions, thus adding noise to the data.Among all the omics datasets, the transcriptomic data holds a lot of promise for classification of cancer types. This is because of the fact that diverse genomic and epigenomic changes often eventually impact the same cellular processes and this is reflected in the gene expression program of the cell21. Machine learning technique has been applied to predict inactivation of a tumor suppressor gene in glioblastoma and to predict patient response to chemotherapeutic drugs with good accuracy23. Artificial Neural Networks (ANN) or Deep Neural Networks (DNN), consisting of complex network of simple information propagating units (neurons), can learn the patterns ingrained in complex datasets and thus, are increasingly being applied for modelling complex and high dimensional biological datasets27.In this scenario, machine learning and deep learning techniques can greatly aid in accurate classification of cancer type and sub-types as these techniques can capture the complex and non-linear relationships within the data31. One of the first studies applied ANNs for prediction of small, round blue-cell tumors (SRBCTs) and could achieve high accuracy of prediction28. Further, Lyu and Haque29 utilized a convolutional neural network that could predict 33 different cancer types from transcriptome data with an accuracy of ~ 95%. Kim et al.31 used a neural network-based method to classify 21 different types of cancers using bulk as well as single cell RNA-Seq data and could achieve accuracy of ~ 90%. Xiao et al.25 developed a semi-supervised deep learning method that could predict three cancer types with accuracy varying between 96%-99%. Gao et al.32 devised a deep learning-based cancer classification method that could be applied to single samples with ~ 90% accuracy. However, the accuracy declined with a reduction in the number of feature genes.Several methods based on artificial neural networks and deep learning have already been developed for classification of cancer types and they show good accuracy33. Second, one does not have any measure of confidence in the prediction values which could be especially problematic for test data falling outside the distribution of the training dataset34. In addition, the eventual goal of all such cancer classification techniques is to devise a classifier that can classify individual patient samples into cancer types and sub-types. Here a measure of confidence or uncertainty with each prediction is important to ascertain the reliability of class predictions and can greatly benefit clinical decision-making35.However, all these methods used point estimates for model parameters. This has several drawbacks. First, this can result in overconfident decisions in case of limited data and when there is an imbalance in the number of samples of different cancer types37. The second form of uncertainty arises from choice of the type of model, and the selection of model parameters and is referred to as epistemic uncertainty37. Thus, running the same model multiple times on the same dataset can lead to different predictions. Uncertainty can be estimated through various methods. Bayesian framework is an ideal way to measure epistemic uncertainty as distributions of values for the model parameters are obtained. However, Bayesian neural networks are often computationally intractable to train and thus require approximations such as variational inference, Markov Chain Monte Carlo (MCMC) method and Laplace transformation41. In addition, Gal and Ghahramani34 showed that dropouts in non-Bayesian neural networks can estimate uncertainty values in predictions and these are equivalent to Bayesian inference. Further, Lakshminarayanan et al.42 showed that use of ensembles of neural networks can also enable estimation of predictive uncertainty.The performance of a deep learning model is heavily dependent on the quality of the dataset on which the model is trained on. Thus, uncertainty in predictions by a deep learning model can be of two different types. The first type of uncertainty is aleatoric uncertainty which arises due to quality of data where datapoints with imprecise measurements or labels are includedIn this work, we developed a deep learning-based cancer classification method called Epistemic Invariance in Cancer Classification (EpICC). This method utilized a Bayesian neural network (BNN) and analysed the uncertainty in the classification of cancer types and subtypes. Addressing the issue of aleatoric uncertainty requires acquiring new better-quality datasets which is beyond the scope of model fitting. Epistemic uncertainty, however, can be accounted for through model fitting. Thus, with EpICC, we incorporated a model-based correction of uncertainty at the output of the BNN that greatly enhanced classification accuracy. We applied this method to classification of 31 cancer types from their transcriptomic profiles. The BNN alone could achieve an overall accuracy of 93.7%. With the incorporation of Model-based uncertainty correction, EpICC could classify cancer types with an accuracy of 97.83%. In addition, EpICC could classify sub-types of four cancer types with an accuracy of > 80%. Thus, we believe that uncertainty correction can greatly aid in making more informed, accurate and reliable decisions. This is critical in clinical prediction tasks where an accurate prediction can significantly improve patients’ well-being.41 for optimization of the model parameters.To build a classifier that reports confidence measures associated with each prediction, we utilized a Bayesian Neural Network (BNN) for classification. The weights of the connections were determined from prior probability distributions and their values were gradually improved from learning on new data to generate posterior distributions Fig. . In cont43 that arose due to variations in the model structure and parameters. Since the Bayesian approach in neural networks was not computationally tractable44, we utilized variational distribution or variational posterior that was assumed to approximate the true posterior. This was done by minimizing the Kullback–Leibler (KL) divergence45 between the variational posterior and the true posterior.BNN, in addition to providing regularization, paved the way for the analysis of uncertainty of the predictions. We added a level of confidence with each prediction through modelling Epistemic uncertaintyWe estimated epistemic uncertainty by performing multiple iterations of testing and quantifying the variation in the predictions. To do so, during inference, we performed Monte Carlo sampling of weights from the approximate variational distribution and obtained the prediction class. We repeated this over 500 iterations and calculated the variation in output by estimating the average of the difference between the actual softmax probability in individual iterations and the mean softmax probability over all iterations see . This gaUncertainty enabled us to reduce incorrect predictions in cancer classification. We tested two approaches incorporating uncertainty for improving cancer classification. First, in the filtering approach, we chose a threshold in the uncertainty value obtained as above for distinguishing between correct and incorrect predictions and discarded all predictions showing higher uncertainty than the threshold as wrong predictions Fig. . HoweverWe applied EpICC for classification of 31 different cancer types from > 10,000 cancer samples for which we obtained the expression profiles from The Cancer Genome Atlas (TCGA) data portal. We first applied a two-step principal component analysis (PCA) and logistic regression to select the genes (or features) that had the highest power to distinguish different cancer types from the transcriptomic data (Supplementary fig. We next investigated whether these 103 genes (features) , two subtypes of BRCA , two subtypes of ESCA (Adenocarcinoma and Squamous Cell Carcinoma), and two subtypes of THCA (Papillary Cell Carcinoma and Follicular Cell Carcinoma) were selected. For each cancer type chosen for subtype classification, separate feature genes were identified by performing PCA analysis. In this case too, 80% of the data was used for feature selection, hyperparameter tuning, and training while 20% data was used for testing.To reduce the risk of overfitting and to get rid of redundant genes, Principal Components Analysis (PCA) was used on the training split of the TCGA data. This was done in two steps: (i) Selecting a set of genes from the original high dimensional RNA-seq data (ii) Selecting an even smaller set of genes from the gene set selected in the first step. To determine the number of optimal genes to be selected in steps (i) and (ii), a supervised feature selection technique was used using a combination of Principal Components Analysis (PCA) and Logistic Regression. For each component and up to 10 components, a certain number of genes having the highest absolute value of factor loadings were selected. The optimal number of genes selected in the first and the second steps were determined by Logistic Regression, which was used as a classifier to identify the minimum number of genes required to achieve a high accuracy, beyond which the accuracy did not increase much with further addition of the number of genes.A typical Deep Neural Network (DNN), if viewed probabilistically, can be considered as a maximum likelihood estimate of the model parameters These models concentrate on finding out point estimates which may lead to over-confident decisions for imbalanced classes. To overcome this situation, accounting for uncertainty in the neural networks and estimating weights in the form of probabilistic distributions can lead to a more generalized model that is more robust to imbalanced datasets. Keeping this perspective, BNN was used to predict the various cancer types. According to Bayes theorem, the likelihood 44, so a simplified distribution called variational distribution  consisted of 31 neurons as there were 31 different types of cancers in our dataset. Sigmoid activation function was used except for the output layer in which Softmax activation function was used. Normal Initialization was used for the weights and Adam’s optimizer was used to update them.The DNN also comprised of three layers. The first layer consisted of 250, neurons, the second layer consisted of 55 neurons and the output layer consisted of 31 neurons. In DNN too, sigmoid activation before the output layer and Softmax activation in the output layer was used. DNN is l2-regularized. Xavier’s initialization was used to initialize the weights in case this case and Adam optimizer was used to update them.60. During inference, Monte Carlo sampling of weights from the approximate variational distribution for 61 was used:tth Monte Carlo iteration, Uncertainty estimation in predictive modelling provides an idea about the confidence of predictions by a model. In a multi-class classification setting, the Softmax activation function in the output layer of the neural network returns the probability value of each classIn the matrix representing Epistemic Uncertainty from Eq. , the diaAfter calculating the uncertainty values, the predictive outputs were corrected for the uncertainty associated with them. For each cancer type, a linear model between the log odds ratio of the expected value of the predicted output from The function After estimating the coefficients, the corrected probability values For evaluating the performance of the classifiers, the following evaluation metrics were used:MYO1F, COL16A1, ANTXR1, TMEM54, PCGF2 and PARVA) for Acute Myeloid Leukaemia (LAML) and one gene (TARP) was selected for Prostate Adenocarcinoma (PRAD). These genes had unique and distinctive expression signature in the corresponding cancer types and thus, were able to classify them accurately (Supplementary fig. Whether a small subset of 103 feature genes identified by PCA could classify specific cancer types was also tested. To do so, performance of each individual gene in cancer type classification was evaluated (Supplementary fig. Supplementary Legends.Supplementary Information 2.


Methods for the global measurement of transcript abundance such as microarrays and RNA-Seq generate datasets in which the number of measured features far exceeds the number of observations. Extracting biologically meaningful and experimentally tractable insights from such data therefore requires high-dimensional prediction. Existing sparse linear approaches to this challenge have been stunningly successful, but some important issues remain. These methods can fail to select the correct features, predict poorly relative to non-sparse alternatives or ignore any unknown grouping structures for the features.We propose a method called SuffPCR that yields improved predictions in high-dimensional tasks including regression and classification, especially in the typical context of omics with correlated features. SuffPCR first estimates sparse principal components and then estimates a linear model on the recovered subspace. Because the estimated subspace is sparse in the features, the resulting predictions will depend on only a small subset of genes. SuffPCR works well on a variety of simulated and experimental transcriptomic data, performing nearly optimally when the model assumptions are satisfied. We also demonstrate near-optimal theoretical guarantees.https://github.com/dajmcdon/suffpcr. Package documentation may be viewed at https://dajmcdon.github.io/suffpcr.Code and raw data are freely available at daniel@stat.ubc.caBioinformatics Advances online. Global transcriptome measurement with microarrays and RNA-Seq is a staple approach in many areas of biological research and has yielded numerous insights into gene regulation. Given data from such experiments, it is often desirable to identify a small number of transcripts whose expression levels are associated with a phenotype of interest . Indeed, projects such as The Cancer Genome Atlas have aimed to generate massive volumes of such data to enable molecular characterization of various cancers. While these data are readily available, their high-dimensional nature (tens of thousands of transcript measurements from a single experiment) makes identification of a compact gene expression signature statistically and computationally challenging. While the identification of a minimal gene expression signature is valuable in evaluating disease prognosis, it is also helpful for guiding experimental exploration. In practical terms, a set of five genes highly associated with a certain disease phenotype can be characterized more rapidly, at lower cost, and in more depth than a set of 50 or 100 such genes using genetic techniques such as CRISPR knockout and cancer biological methods such as xenotransplantation of genetically modified cells into mice. Therefore, this article prioritizes selecting a small subset of transcript measurements, which still provide an accurate prediction of phenotypes.With these goals in mind, supervised linear regression techniques such as ridge regression , the lasIn genomics, one may collect expression measurements for thousands of genes from microarrays or RNA-Seq with the goal of predicting phenotypes or class outcomes. In these settings, the number of patients is much smaller than the number of gene measurements and researchers are interested in (i) the accurate prediction of the phenotype, (ii) the correct identification of a handful of predictive genes and (iii) computational tractability. Among these properties, the correct identification of a small number of predictive genes is of crucial importance in practice, since it can lead biologists to further investigate specific genes through CRISPR knockout or other techniques. It is this genetic pattern discovery for which our proposed methodology is intended: data with many more measurements than observations; the potential that some of the measurements may be grouped or correlated; the existence of either a continuous or discrete outcome we wish to predict; and the belief that these predictions only depend on some small collection of groups rather than the entire set of measurements.PCA has two main drawbacks when used in high dimensions. The first is that PCA is non-sparse, so it uses information from all the available genes instead of selecting only those which are important, a key objective in omics applications. That is, the right singular vectors or ‘eigengenes’ . This meModern approaches specifically for pattern discovery in the genomics context such as supervised gene shaving , though In this paper, we leverage the strong theoretical properties associated with sparse PCA to improve predictive accuracy for regression and classification problems in genomics. We avoid the strong assumptions necessary for SPC, the current state-of-the-art, while obtaining the benefits associated with sparse subspace estimation. In the case that the phenotype is actually generated as a linear function of a handful of genes, our method, SuffPCR, performs nearly optimally: it does as well as if we had known which genes were relevant beforehand. Furthermore, we justify theoretically that our procedure can both predict accurately and recover the correct genes. Our contributions can be succinctly summarized as follows:We present a methodology for discovering small sets of predictive genes using sparse PCA;Our method improves the computational properties of existing sparse subspace estimation approaches to enable previously impossible inference when the number of genes is very large;We demonstrate state-of-the-art performance of our method in synthetic examples and with standard cancer microarray measurements;We provide near-optimal theoretical guarantees.Our methodology can be used in a variety of genomic pattern discovery settings. One such example is a modified version of traditional differential expression analysis. If we have treatment and control measurements, the logistic version of our method is appropriate with the advantage that it examines the impact of one gene adjusted for the contributions of others. In addition, with a continuous treatment, the detection power can be increased relative to using an artificial dichotomization.sufficient PCR relative to previous approaches and present details of SuffPCR. Section 2.2 illustrates performance in simulated, semi-simulated and real examples (Section 2.3) and discusses the biological implications of our methods for a selection of cancers. Section 2.4 theoretically justifies our methods, providing guarantees for prediction accuracy and correct gene selection. Section 3 concludes.In Section 2.1, we motivate the desire for Notation. We use bold uppercase letters to denote matrices, lowercase Arabic letters to denote row vectors and scalars and uppercase Arabic letters for random variables. Let Y be a random, real-valued n-vector of independent variables iY, and X be the row-wise concatenation of i.i.d. draws iX from a distribution on Y as X is an n × p matrix where each row is a set of transcriptomic measurements from RNA-Seq or microarrays for a patient while iy is an observed phenotype of interest for the ith patient. Because X is a matrix, this symbol represents both a random matrix and its realization. In the following, the meaning should be clear from the context. We assume, without loss of generality, that X have been centered. The singular value decomposition of a matrix A is X, we suppress the dependence on X in the notation and write d columns of the matrix A and ja to denote the jth row. In the case of the identity matrix, we use a subscript to denote its dimension when necessary: A while A. A, that is the number of rows in A that have non-zero a, taking value 1 if a is true or 0 if not.iY is a real-valued scalar phenotype, iX is a real-valued vector of genes, iϵ is a mean-zero error. Defining as above SPC  a low-dimensional linear manifold. It finds the best d-dimensional approximation of the span of X such that the reconstruction error in d right singular vectors, and the estimator of the first d principal components is PCA is a canonical unsupervised dimension reduction method when it is reasonable to imagine that V is sparse in some sense. Many of these methods achieve this goal by adding a penalty to V. This intuition is justified by the following result.Proposition 1. Consider the linear model with . Let be the eigendecomposition of with for . Then As discussed in Section 1.1, standard PCA works poorly in high dimensions. Much like the high-dimensional regression problem, estimating high-dimensional principal components is ill-posed without additional structure. To address this issue many authors have focused on different sparse PCA estimators for the case when j, if j given a gap between the dth and The proof is immediate. For any d-dimensional linear manifold such that Solutions q is a deterministic function of d principal components simultaneously and nearly achieves the lower bound:Many authors have developed different versions of sparse PCA. For example, p > n) for two reasons. First, the original ADMM algorithm has per-iteration computational complexity p is large. Second, because of the convex relaxation using q is a function of this ratio). We give explicit formulas for the SNR under this model in the d-dimensional subspace through the relative magnitude of p. In genomics applications with low SNR, which is common, estimates For these reasons, FPS is known as the current state-of-the-art sparse PCA estimator with the best performance. However, despite its theoretical justification, FPS is less useful in practice for solving prediction tasks, especially in genomics applications with Y and gene expression measurements X by making use of the (near) low-dimensional manifold that supports X. In broad outline, SuffPCR first uses a tailored version of FPS to produce a row-sparse estimate Y on the derived components to produce sparse coefficient estimates. SuffPCR for regression is stated in Algorithm 1 and summarized visually in Y and X are standardized so that In this section, we introduce SuffPCR. The main idea of SuffPCR is to detect the relationship between the phenotype Algorithm 1SuffPCR (regression version)Input: X, S, y, d, λ.1: 2: while not converged do3: 4: 5: 6: end while7: d eigen decomposition8: Decompose 9: Compute t by applying Algorithm 2 to l10: Choose t as 0, and get 11: Set rows in 12: Solve Return:13: 5. 6. The first issue is the time complexity of the original FPS algorithm. Essentially, FPS uses the same three steps depicted in Lines 4–6 in Algorithm 1.p2 elements. The second and third steps are computationally simple (element-wise soft-thresholding and matrix addition). But the first, Q, an Q through the solution of a piecewise linear equation in τ: τ such that p, the authors suggest the number of features should not exceed one thousand. But typical transcriptomics data have many thousands of gene measurements, and preliminary selection of a subset is suboptimal, as illustrated above. Due to the form of the piecewise solution, most eigenvalues will be set to 0. Thus, while we will generally require more than d eigenpairs, most are unnecessary, certainly fewer than p. If we compute enough to ensure that some The only difference here between our implementation and that in FPS is in Step 4. Each of these steps takes a matrix and produces another matrix, where the matrices have n AIRLB;  as implep = 5000, though the same experiment on a high-performance Intel Xeon E5-2680 v3 CPU with 12 cores, 256 GB of memory, and optimized BLAS were somewhat less dramatic (improvements of three-fold and four-fold respectively). For large RNA-Seq datasets  whenever this is meaningful. Additional experiments are given in the d and the impact of different SNR choices.We compare SuffPCR with a number of alternative methods. The Oracle estimator uses OLS on the true features and serves as a natural baseline: it uses information unavailable to the analyst (the true genes) but represents the best method were that information available. We also present results for Lasso , Ridge , ElasticAlgorithm 3 Generate synthetic dataInput: n = 100, p = 1000, r = 5, d = 3, 1: 2: Generate i.i.d. 3: Set 4: Generate i.i.d. r times (s = rd).5: Extend 6: Set 7: Generate i.i.d. 8: Set 9: Set 10: Set 11: Set 12: Set 13: Set d = 3, and we use the correct dimension for methods based on principal components.The first setting is designed to show the advantages of SuffPCR relative to alternative methods, especially SPC. We note that other methods that employ screening by the marginal correlation  will havX, we create a semi-synthetic analysis wherein only the phenotypes are generated. We first performed PCA on the NSCLC data  10 times to reduce any bias induced by the random splits. In all cases, all methods were tuned to optimize validation-set MSE.We randomly split each dataset into 3-fold for training, validation and testing with proportions 40%, 30% and 30% respectively. We set the number of components λ via 5-fold cross-validation on all the data and then train SuffPCR with this λ. Our model selects 87 features corresponding to 32 unique genes and 2 expressed sequence tags (ESTs) for DLBCL. Seventeen of the identified genes encode ribosomal proteins, overexpression of which is associated with poor prognosis  corresponds to a portion of ZBTB44, which encodes an uncharacterized transcriptional repressor, while EST Hs.343870 (GenBank accession AA804270) does not appear to be contained within an annotated gene. The To assess the potential relevance of the genes selected by SuffPCR to the cancer type from which they were identified, we further explored the DLBCL data and extracted the selected genes.  that is independent of n and p but may depend on any of the other values given in A1–A6. Theorem 1 gives a convergence rate for the prediction error of SuffPCR comparable to that of Lasso though with explicit additional dependence on d. Under standard assumptions with fixed design, this dependence would not exist for Lasso. On the other hand, our results are for random design with d small, along with different constants absorbed by the big-t is chosen correctly. We note that this result is a direct consequence of p and n grow.In both results above, High-dimensional prediction methods, including regression and classification, are widely used to gain biological insights from large datasets. Three main goals in this setting are accurate prediction, feature selection and computational tractability. We propose a new method called SuffPCR which is capable of achieving these goals simultaneously. SuffPCR is a linear predictor on estimated sparse principal components. Because of the sparsity of the projected subspace, SuffPCR usually selects a small number of features. We conduct a series of synthetic, semi-synthetic and real data analyses to demonstrate the performance of SuffPCR and compare it with existing techniques. We also prove near-optimal convergence rates of SuffPCR under sparse assumptions. SuffPCR works better than alternative methods when the true model only involves a subset of features.The authors gratefully acknowledge support National Science Foundation (grant DMS–1753171 to D.J.M.), the National Institutes of Health (grant R35GM128631 to G.E.Z.) and the National Sciences and Engineering Research Council of Canada (NSERC) (grant RGPIN-2021-02618 to D.J.M.).Conflict of Interest: none declared.vbac033_Supplementary_DataClick here for additional data file.


Heart failure (HF) is the main manifestation of cardiovascular disease. Recent studies have shown that various RNA molecules and their complex connections play an essential role in HF’s pathogenesis and pathological progression. This paper aims to mine key RNA molecules associated with HF. We proposed a Prior-knowledge Driven Joint Deep Semi-Negative Matrix Factorization (PD-JDSNMF) model that uses a hierarchical nonlinear feature extraction method that integrates three types of data: mRNA, lncRNA, and miRNA. The PPI information is added to the model as prior knowledge, and the Laplacian constraint is used to help the model resist the noise in the genetic data. We used the PD-JDSNMF algorithm to identify significant co-expression modules. The elements in the module are then subjected to bioinformatics analysis and algorithm performance analysis. The results show that the PD-JDSNMF algorithm can robustly select biomarkers associated with HF. Finally, we built a heart failure diagnostic model based on multiple classifiers and using the Top 13 genes in the significant module, the AUC of the internal test set was up to 0.8714, and the AUC of the external validation set was up to 0.8329, which further confirmed the effectiveness of the PD-JDSNMF algorithm. Heart failure (HF) is a common type of cardiovascular disease, and its morbidity and mortality are increasing yearly . The patWith the development of RNA sequencing data, researchers can effectively identify and mine disease-related RNA molecular biomarkers. In addition to messenger RNA (mRNA), long non-coding RNA (lncRNA) and microRNA (miRNA) have also been shown to be extensively involved in the pathological progression of HF . MiRNAs Due to their complementary information, multi-omics data can capture and mine disease-related and biologically meaningful biomarkers. Exploring multi-omics integration algorithms is a hot topic in bioinformatics research. Joint non-negative matrix factorization (JNMF) has attracted the attention of researchers due to its low time complexity and strong clustering performance. Zhang et al. first proposed this algorithm, applied it to cancer genomics , and ideTo this end, this paper proposed a prior knowledge-driven joint, semi-nonnegative matrix factorization algorithm (PD-JDSNMF) to integrate miRNA, mRNA, and lncRNA. The algorithm adds PPI interaction information based on the JDSNMF algorithm. The aim is to drive the algorithm to obtain more biologically meaningful co-expression modules. Since the algorithm based on matrix factorization is more sensitive to noise, we add Laplacian matrix constraints on the three kinds of data to enhance the anti-noise performance of the algorithm. We used the PD-JDSNMF algorithm to obtain multiple biologically significant co-expression modules. For the modules with strong correlation, we performed correlation analysis on the three types of data and mined multiple pathways significantly related to the disease. In addition, we constructed a heart failure diagnosis model using multiple classifiers for the top 13 features with high importance in this module, and its AUC in the test set reached the highest of 0.8714. We use an independent external validation set for diagnostic model validation, which achieves the highest AUC of 0.8329. It suggests that the selected features have diagnostic significance for heart failure.This section describes the framework of the PD-JDSNMF algorithm, which combines three types of RNA-seq data to identify co-expression modules. Then, the three RNA expression matrices are decomposed into a common basis matrix Let JDSNMF adopts the principle of multilayer NMF and nonlinear activation functions to represent nonlinear manifolds. Also, it uses regularization to prevent overfitting. Its objective function is as follows:Among them, This section proposed a prior knowledge-driven Joint Deep Semi-Non-Negative Matrix Factorization. Specifically, we added PPI interaction information to the feature latent matrix Among them,  is high . Its expHere, http://string-db.org) database was used to predict the PPI network in the module and analyze the interactions between proteins. Pairs of nodes in the PPI network were screened, and interactions with a combined score >0.4 were considered statistically significant. Molecular interaction networks were visualized using Cytoscape. Next, we identified core genes from the PPI network using the MCC method in cytoHubba (Cytoscape plugin).The STRING  enrichment analysis and Kyoto Encyclopedia of Genes and Genomes (KEGG) enrichment analysis in selected modules. The DO enrichment analysis as well as the KEGG enrichment analysis were performed by the “clusterProfiler” package. The “ggplot” package is used to draw bubble plots. In DO analysis, q-values less than 0.05 were considered statistically significant. Pathways with https://www.ncbi.nlm.nih.gov/geo/). This paper used the human genome assembly GRCh38 file to convert the ensemble ID of GSE141910 data into gene name and genotype annotation and extracted 14914 mRNAs, 3,134 lncRNAs, and 30 miRNAs. Moreover, we downloaded the GSE116250 data from the GEO database as a validation dataset for ROC analysis. 14 normal samples and 50 HF samples were included in the GSE116250 data. Then randomly select 80% of the samples as the training set and 20% as the test set. Finally, 292 training samples and 74 testing samples are obtained.We downloaded RNA-seq data (GSE141910) containing 200 HF samples and 166 normal samples from the GEO database    and 162 DEmRNAs) .In this paper, the sigmoid function is used as the activation function so that the decomposed feature latent matrix has non-negative nonlinearity. Furthermore, for the initialization of the basis and coefficient matrices, we use the singular value decomposition (SVD) algorithm, which has been shown to produce better local optima for matrix classification algorithms.Furthermore, this section will introduce the hyperparameter settings of PD-JDSNMF and classification models. PD-JDSNMF model has four key hyperparameters: number of layers, dimensionality reduction We selected the parameters based on the mean value in subgraph D of  library  to evaluaterials .In all matrix factorization models, dimensionality reduction It can be seen from the above table that there are three sets of bination . It can After performing the PD-JNMF algorithm on the HF RNA-seq dataset with the optimal parameter combination, we obtained 30 co-expression modules, each of which contained an average of 5.47 miRNAs, 118.33 mRNAs, and 28.73 lncRNAs. The mean correlations of the reconstructed mRNA expression data, lncRNA expression data, and miRNA expression data with the original data were 0.8362, 0.8758, and 0.8383, respectively.We counted the Pearson correlation coefficients of the original and reconstructed matrices in all modules, and the To meet the needs of subsequent analysis, we draw Venn diagrams for the three elements in these modules, as shown in As shown from the Subsequently, to explore the biological functions of genes in module 12, we perform DO enrichment analysis and KEGG enrichment analysis on module 12. The results of DO enrichment analysis show that the genes in module 12 are mainly associated with cardiovascular diseases, such as atherosclerosis, arteriosclerotic cardiovascular disease, arteriosclerosis, myocardial infarction, and coronary artery disease . These dNext, we constructed a PPI network for the genes in module 12 and evaluated the network using the NetworkAnalyzer plugin in Cytoscape . Next, wAs can be seen from In order to evaluate the genes in module 12 as having diagnostic significance for HF, we first used the RF algorithm to rank the feature importance in this module. Next, based on four classifiers , we used Top 1 to Top 50 genes in module 12 to classify whether there was HF or not and compared the changes in AUC as shown in There are two points marked in each subplot in As can be seen from As can be seen from This study identified 13 genes  that were closely related to HF. Myocardial infarction leads to the death of cardiomyocytes, leading to cardiac fibrosis, cardiac remodeling, and heart failure. In a human fibroblast model, researchers found that MIR590 inhibits cardiac fibrosis after myocardial infarction . ChronicIn order to verify the ability of the proposed algorithm to reconstruct the original matrix, we use the features selected by the proposed PD-JDSNMF and the three algorithms of JDSNMF, MCJNMF, and JNMF, and use four classifiers to compare the classification accuracy. To fairly compare the feature selection ability of several algorithms, we first select the module with the most robust reconstruction performance from all co-expression modules of several algorithms. Then, 50 features were randomly selected from the saliency module and repeated ten times to calculate the AUC for classification using the four classifiers, respectively, and the following violin plot was drawn as shown in As can be seen from This paper proposed a PD-JDSNMF algorithm to integrate prior information in genetic data, extract nonlinear features in genetic data, capture the underlying expression patterns of multiple data, mine heart failure-related biological markers, and build diagnostic models. Specifically, we identified module 12 as a key module and the genes in the module as inputs for subsequent analyses through functional enrichment analysis. Using multiple classifiers, we constructed a heart failure diagnostic model and validated the diagnostic model on an external dataset, which achieved an AUC of 0.8329. Compared with several other NMF-based algorithms, the proposed algorithm has a stronger matrix reconstruction ability. Furthermore, the PD-JDSNMF algorithm is confirmed to have a more stronger feature selection ability by using elements for classification.