It is crucial to accurately categorize cancers using microarray data. Researchers have employed a variety of computational intelligence approaches to analyze gene expression data. It is believed that the most difficult part of the problem of cancer diagnosis is determining which genes are informative. Therefore, selecting genes to study as a starting point for cancer classification is common practice. We offer a novel approach that combines the Runge Kutta optimizer (RUN) with a support vector machine (SVM) as the classifier to select the significant genes in the detection of cancer tissues. As a means of dealing with the high dimensionality that characterizes microarray datasets, the preprocessing stage of the ReliefF method is implemented. The proposed RUN–SVM approach is tested on binary-class microarray datasets (Breast2 and Prostate) and multi-class microarray datasets in order to assess its efficacy . Based on the experimental results obtained from analyzing six different cancer gene expression datasets, the proposed RUN–SVM approach was found to statistically beat the other competing algorithms due to its innovative search technique. For the past few years, gene expression profiling has allowed scientists to simultaneously examine thousands of genes. As a result, significant knowledge has been gained regarding the processes that take place within cells. In situations involving cancer, having access to such particular information is helpful for both diagnosing the disease and evaluating the patient’s prognosis. Microarray data features such as low sample size, high noise levels, and high dimensionality persist in making gene selection a challenging task. Identifying the best candidates for gene classification is essential to resolving this problem. This method will not only cut down on computational costs, but it will also enable physicians to focus on a small subset of physiologically significant genes that are linked to specific cancers, allowing for the formulation of more cost-effective studies . In addiThe “curse of dimensionality”, manifested by small sample sizes and the presence of genes that are unrelated to one another, plagues the microarray data ,4,5. DueThere are a few different ways to go about selecting genes, but they can all be broken down into three broad categories: embedding, wrapping, and filtering. Using statistical methods like minimum redundancy maximum relevance, information gain, and ReliefF, the filter strategy collects intrinsic gene properties in order to differentiate the targeted phenotypic class . This apSelecting the significant genes from gene expression data has been implemented using the wellknown machine learning approaches, including Neural Networks (NN), support vector machines (SVMs), and K-nearest neighbor (KNN). There is some evidence that gene selection can improve the accuracy of sample classification . AccordiHere we provide RUN—SVM, a novel approach to classifying gene expression data that combines the SVM classifier with the RUN feature selection optimizer.Overcomes the “feature dimensionality curse” in order to eliminate unnecessary features and focus on the most informative ones when classifying cancers based on gene expression data.Assesses the effectiveness of the introduced classifier in binary and multiclass classification tasks using six benchmark datasets.Compares the capability and feasibility of the proposed classification algorithm to existing approaches through a comprehensive evaluation.To emphasize the significance of this study, the following are the contributions that it makes:The remaining sections of the article are presented in the following order: The review of the available literature can be found in A number of different gene selection optimization approaches have been presented in the literature as a way to improve the accuracy with which cancers are diagnosed. The following is a list of the most recent studies that have been conducted for the purpose of making suggestions as to which genes should be employed in the classification and diagnosis of cancer.Li , proposeIn order to analyze microarray datasets, it was suggested that combining SVM with the artificial bee colony approach might be beneficial . The proIn , the autIn , The resMohamad et al., , suggestWrapper strategies combining the SVM classification method and the FireFly algorithm (FF-SVM) have been proposed as a means to classify cancer microarray gene expression profiles . The expThe authors of  concentrA hybrid model, SVM-mRMRe, was presented for gene selection by the authors of . The mosIn , a numbeA new hybrid feature selection technique was introduced in . It invoHere, we provide an overview of the methods that would be employed in the proposed RUN—SVM classification technique. These methods include the SVM classification method, the RUN optimization method, and the ReliefF feature selection method for microarray data.The first Relief algorithm  has beenThe goal of optimization algorithms is to enhance the efficiency with which they solve various optimization problems by striking a balance between exploiting and exploring the problem space. This is done in order to improve the efficiency with which they tackle various optimization problems. Iman Ahmadianfar and her colleagues recently presented the RUN, which is one example of such an optimization method . This alv of dimensions 1,2,…, and M are, respectively, The decision variables r is an integer number that may be 1 or −1. g is a random number in the range of . The well-known RUN mechanism is applied to the following equations in order to modify the solutions Additionally, using r is an integer that can be either 1, 0, or −1. v is equal to 2 × w gets smaller as the number of iterations rises.Using the ESQ technique, RUN incorporates better solution quality, escape from the local optima, and quick convergence. In order to use this approach, three new solutions—Algorithm 1 The RUN algorithm1:Input: Population size, Population initialization, and MaxIter of iterations.2:Output: The best Solution.3:Evaluate Objective Function using Equation .4:Calculate Equation .5:while Stop condition not met do6:for n = 1 ... N do     7:           Evaluate Objective Function using Equation .8:if then           9:                calculate Equation 10:if then               11:if rand < w then                   12:                         Calculate Equation .13:if then                         14:                             15:end if                         16:end if                   17:else               18:                   19:end if               20:end if           21:end for     22:end whileThe pseudo-code steps of the RUN method is depicted in Algorithm 1.In the realm of supervised machine learning, SVMs are a helpful tool for dealing with classification and regression issues . DespiteC regulates the compromise between accurate training-data classification and a consistently smooth resolution limit. Meanwhile, Algorithm 2 The SVM algorithm1:Input: Test and training points.2:Output: Best fitness.3:C and Select the SVM’s expense factor 4:while stop condition not satisfied do5:    For each data point, Apply the SVM training phase.6:    For testing data points, Apply the SVM classifying phase.7:end while8:return fitnessThe results of a support vector machine are controlled by two variables. In order to partition the data into K separate sets, a statistical technique known as K-fold cross-validation is utilized. As a result, only the training subsets were employed at first; however, more recently, other subsets have been used to test or validate the training subsets. When testing the model, a pending subsample will be used for validating the data, and K-1 sets would be employed for training points.The combination of ReliefF, RUN, and SVM into a single framework for the effective classification of microarray datasets is novel, as far as we are aware. Not only is RUN being used for the first time as a gene selection method for cancer gene expression datasets, but it is also being used in a novel way. Here, we offer an overview of the hybrid cancer classification method we’ve proposed for picking the most precise SVM-classified genes from cancer gene expression datasets. In order to offer a set of genes that are predictive as well as informative, the proposed RUN–SVM approach, which is depicted in Applying algorithms directly to microarray datasets is a real challenge due to the fact that these datasets are of a high dimension and contain thousands of genes that need to be filtered. As a result of this, using these kinds of datasets without first filtering them makes it difficult to accurately train the SVM classification algorithm. For its excellent performance, great efficiency, and widespread application in the field of resolving problems with huge dimensions, we decided to make use of the ReliefF algorithm in this particular piece of work. As a result, the first stage of the proposed RUN–SVM methodology is called ReliefF. It can be utilized for the purpose of removing redundant genes, which contribute to the background noise and lower the classification accuracy , worst fitness function, and mean fitness function. In general, these are the four accuracy measurements that were used. The best fitness score at a run I is 1.Mean: is calculated by multiplying the number of times the algorithm is run by its fitness value (N).2.Best fitness function: is a value that, after running the algorithm N times, outperforms all other best fitness values.3.Worst fitness function: the smallest value for each maximum fitness metric that can be achieved by repeated iterations of the method N times4.The standard deviation (STD): is a quantity that can be obtained by running the algorithm N times and measuring the amount of change in the fitness values. Data points with a smaller STD value show that they cluster tightly around the mean, while data points with a larger STD value suggest that they may deviate greatly from the mean.The cross-validation method was implemented in this investigation in order to enhance classification performance, minimize bias in point selection for testing and training, and improve overall accuracy. Therefore, 10-fold cross-validation was utilized during the process of training the SVM classifier. In addition, the ReliefF filter and SVM classifier-improved comparison algorithms were each subjected to 30 separate runs with each run consisting of 100 iterations.According to the results displayed in We compared the proposed RUN–SVM approach to some other popular optimization techniques and showed how it converges after 100 iterations. Based on the results shown in In this section, we draw attention to the results of the proposed RUN–SVM approach. The proposed RUN–SVM approach outperformed the competition in this evaluation, as is shown by the boxplot curves in The datasets that were utilized for the analysis of this study contain feature sizes that range anywhere from 4869 to 12,600 features, which makes for an excellent testing environment for an optimization method.Any chance for improving RUN–SVM can easily be implemented due to its straightforward architecture.The purpose of this section is to examine the numerous systems now in use for classifying cancers. The goal of this research is to propose a search strategy for the FS problem that takes into account the challenges posed by high-dimensional datasets. The research suggested using a ReliefF filtering approach for preprocessing and then combining RUN and SVM for the FS and classification stages. The effectiveness of the proposed RUN–SVM approach is demonstrated by experimental analysis and a comparison study. The following is a list of benefits that come with using our proposed RUN–SVM technique:On the other hand, the proposed RUN–SVM method offers certain restrictions, and these are listed below.Due to the fact that RUN–SVM uses a randomization-based optimization technique, the features it uses may vary from run to run. The feature subset picked in one run may not be present in another.As most genes are related to one another either directly or indirectly, microarray data processing is quite complex.The limited number of samples available in microarray datasets, the presence of noise and genes that are not relevant to the study, and the “curse of dimensionality” all add to the difficulty of classifying a particular sample.Using the Runge Kutta optimizer (RUN) approach, this investigation presents an improved way for choosing significant genes from cancer microarray patterns. The first primary goal was to pick the genes from the microarray datasets that were the most relevant and helpful with the least amount of effort, and the second primary goal was to select the genes that were the most accurate with the least amount of effort. In this paper, a wrapper gene selection approach called RUN—SVM is proposed. RUN—SVM combines the RUN algorithm with the support vector machine (SVM) classifier in order to identify the most relevant genes from a small dataset after removing redundant genes and selecting related genes from gene expression datasets. Six standard microarray datasets were used to evaluate the proposed RUN–SVM approach, namely Brain Tumor1, Brain Tumor2, Breast2, Breast3, Lung Cancer, and Prostate. The findings of the experiments showed that the RUN-SVM methodology obtains the highest level of accuracy in comparison to other comparative methodologies, such as HHO, MRFO, GJO, HGS, WOA, and ABC. The effectiveness of recent deep learning classifiers, such as transformer-based approaches and self-attention methods, could be examined for cancer classification using microarray datasets as part of future work that is associated with this research. Also, the performance of those classifiers could be evaluated in order to determine whether or not it would be enhanced by making use of the optimizer that was proposed in this study.


The accurate prediction of patients with complex diseases, such as Alzheimer’s disease (AD), as well as disease stages, including early- and late-stage cancer, is challenging owing to substantial variability among patients and limited availability of clinical data. Deep metric learning has emerged as a promising approach for addressing these challenges by improving data representation. In this study, we propose a joint triplet loss model with a semi-hard constraint (JTSC) to represent data in a small number of samples. JTSC strictly selects semi-hard samples by switching anchors and positive samples during the learning process in triplet embedding and combines a triplet loss function with an angular loss function. Our results indicate that JTSC significantly improves the number of appropriately represented samples during training when applied to the gene expression data of AD and to cancer stage prediction tasks. Furthermore, we demonstrate that using an embedding vector from JTSC as an input to the classifiers for AD and cancer stage prediction significantly improves classification performance by extracting more accurate features. In conclusion, we show that feature embedding through JTSC can aid in classification when there are a small number of samples compared to a larger number of features. However, in most cases, the amount of biomedical data is small owing to the high cost of data collection. Thus, several computational efforts have been made to overcome this lack of data. In the case of image data, data augmentation methods have been used to prevent overfitting and train more complex models5. Generative adversarial networks (GAN) is a leading data augmentation technique based on the zero-sum principle6. GAN trains two neural networks, a generator, and a discriminator. While the discriminator distinguishes the generated data from the real data, the generator strives to generate synthetic data that closely adheres to the original data distribution. Recently, deep metric learning with triplet loss has been proposed to overcome the limitations of small datasets. Deep metric learning is an approach for learning metric embedding7. It learns the representation of the input data in a low-dimensional vector space by calculating the distance between vectors. Embeddings have been trained to obtain similar representations for the same class of data through the model. Recently, many loss functions have been developed for deep metric learning, such as contrastive, triplet, and quadruplet losses9. The Euclidean distance and cosine similarity are widely used as distance functions to bring samples of the same class closer together and others further apart. A triplet network passes three instances of anchor, positive, and negative samples10. It calculates pairs of positive and negative distances from the anchor and gives an advantage to training by generating a large triplet structure from a relatively small amount of data.Deep learning methods have improved prediction accuracy for a wide range of tasks in the biomedical field11 and Viñas et al.12 proposed GAN-based data augmentation studies for gene expression data. Moreover, Moreno-Barea et al.13 developed a conditional GAN method using gene expression data, and Ahmen et al.14 proposed a GAN architecture that integrates two omics datasets to generate omics data from the other omics dataset. However, compared to GAN, deep metric learning has been less applied to gene expression data.Chaudhari et al.15. Accumulation of plaques (beta-amyloid) and tangles (tau) are generally identified as causes of the disease16, but the detailed pathogenic mechanisms are unknown. In the characterization of gene expression values in patients with AD, several studies have identified AD-related genes and discovered other potential candidates18. However, these studies used gene expression in brain tissues, which is invasive and cannot be used for the early diagnosis of AD. However, in the blood, there are differentially expressed proteins similar to those found in the brains of patients with AD19. Therefore, gene expression and protein levels in the blood have been investigated for the early diagnosis of AD21.In the present study, we developed a new deep metric learning based model and applied it to gene expression profiles from Alzheimer’s disease (AD) and The Cancer Genome Atlas (TCGA) datasets. AD is a complex disease that causes memory loss. The number of patients with AD has increased in recent decades with the increase in life expectancy24. Aouiche et al.25 used a pathway network to extract stage-specific genes by constructing gene modules. Park et al.26 applied deep learning to stage prediction in gene expression profiles, and Rahimi et al.27 improved the performance of cancer stage prediction and identified gene sets that are commonly related across different cancer cohorts in TCGA. Kwon et al.28 applied GAN to data augmentation to address the problem of the small number of clinical samples and increase prediction accuracy.With regard to cancer, several studies have predicted patient survival and identified biomarkers for predicting cancer type and biological changes based on gene expression29. It can better capture features from unbalanced data with a larger number of features than from the number of samples in biomedical data classification. The model detects global features with a minimal kernel stride by making the CNN stride equal to the kernel size30. Therefore, the 1D-CNN model was used as the embedding layer in our experiments using gene expression data.As convolutional neural networks (CNN) are used in numerous applications, simplified CNN as 1D-CNN have been proposed for use in modeling 1-dimensional featuresWe developed a new approach, called joint triplet loss with semi-hard constraint (JTSC), to mine more accurate triplets and identify improved performance by overcoming the lack of data. First, we trained a 1D-CNN model with a vector representation by generating sufficient triplets for training and obtained a classifier with improved prediction performance using these embedded vectors. Second, we propose a semi-hard constraint for sophisticated semi-hard mining and joint loss for training the vector representations. Finally, while learning the embedding function, we analyzed the change in the number of triplet categories to check whether the training of the embedding function was performed properly. When we measured the prediction performance of the proposed method using AD and cancer datasets from TCGA, the performance was improved by overcoming the lack of data.https://www.ncbi.nlm.nih.gov/geo/). These expression profiles were generated using the Illumina HumanHT-12 v3.0 Expression BeadChip for the ANM1, and Illumina HumanHT-12 v4.0 Expression BeadChip for the ANM2. The other peripheral blood AD dataset was downloaded from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) (http://adni.loni.usc.edu). We used all samples in ANM1, ANM2, and ADNI; however, samples without clinical information were excluded. We converted the probe ID to the Entrez Gene ID with the information from GPL6947 and GPL10558 for ANM1 and ANM2, respectively. Probe IDs not assigned to Entrez gene IDs were removed. We selected protein-coding genes in the assigned Entrez Gene ID based on the Homo_sapiens.GRCh38.94 database (http://asia.ensembl.org/Homo_sapiens/Info/Index); Ensembl IDs of protein-coding genes in the database were converted into Entrez Gene IDs using the “biomaRt” package in R software. The expression values for the duplicated Entrez gene IDs in protein-coding genes were treated as mean values. Subsequently, 16,730, 14,957, and 20,384 gene expression values were selected from ANM1, ANM2, and ADNI, respectively.We used peripheral blood gene expression profiles, such as GSE63060  and GSE63061 , downloaded from the Gene Expression Omnibus (GEO) , and the samples without clinical information were removed, identical to the data processing of the AD dataset. Note that we did not use the KIRP dataset, which does not include the cancer stage of the samples in the clinical information.27  is defined as follows:Metric learning maps the data into a new vector space using an embedding function and helps a model train with better performance than when using the original features36. Therefore, we constructed random N triplets out of all possible n data samples, and selected only semi-hard samples for training. We constructed a half of N triplets containing disease anchors and the other half of N triplets containing control anchors for balanced anchor training. An online semi-hard mining strategy was used to prevent poor training and converge quickly in each mini-batch. Unlike offline mining, which defines triplet categories from samples at the beginning of the epoch at once, we calculated distances from all N triplets and defined easy, semi-hard, and hard samples at each epoch to determine which semi-hard samples to use for weight updates.Moreover, there are three types of triplet samples: easy, semi-hard, and hard, depending on the relationship between 37. The angular loss compares the relative ratio of edges and treats all three edges, which is different from the traditional triplet loss, which deals with only two edges from P is perpendicular to the edges of P and circle C with the edges of An angular loss can achieve better similarity than the traditional triplet lossAs previously mentioned, the semi-hard sample is defined according to entclass1pt{minimaMoreover, we added the loss term 30. The model sets the stride size of the kernel equal to the kernel size in order to extract important features from non-overlapping genes. The training batch size was set to 300. We then constructed 50,000 triplets of three data points: anchor, positive, and negative samples for ANM1 and ANM2. For training with a balanced number of anchors, data consisted of 25,000 triplets of random AD anchors and 25,000 triplets of random control anchors. However, considering the low performance of the baseline from other machine learning methods on the ADNI dataset, we constructed a larger dataset with 100,000 triplets, including 50,000 AD anchors and 50,000 control anchors. For each anchor in two classes, the positive sample was extracted from the same label as the anchor, and the negative sample was extracted from the opposite label of the anchor. We loaded all triplets that we constructed and calculated the Euclidean distance between the output vectors of three data points  through the 1D-CNN model. Owing to the anchor dependency in the triplet category definition, we selected strict semi-hard samples for training the 1D-CNN model, additionally considering the semi-hard constraint. The Adam optimizer with mini-batch was employed in the training step. Subsequently, the joint loss was used as a combination of triplet loss, angular loss, and switching loss for training. After embedding the training, validation, and test sets, a fully connected network was added to classify the disease and controls using these features. The fully connected one-layer neural network was initialized with Xavier initialization and was trained using the Adam optimizer. All experiments were performed at a margin m (> 0) of 0.5, where a margin indicates the degree to which The 1D-CNN model consists of a 1-dimensional convolutional layer, a max pooling layer, and a fully connected layer to embed in the vector space used for deep metric learning. We adopted the learning rate For JTSC, hard and semi-hard samples whose triplet categories changed when switching between the anchor and positive samples were filtered out. Strict semi-hard samples that satisfied the semi-hard constraint were used for training. Thus, training the embedding functions with these semi-hard samples could converge more quickly than training with all possible triplets. We calculated the area under the curve (AUC), area under the precision-recall curve (AUPRC), and F1 score to measure the performance of the AD prediction model using ten-fold cross-validation.38. Then, an SVM classifier was trained using both the training and generated data.The performance of the proposed method was compared with simple machine learning-based models: support vector machine (SVM), random forest (RF), XGBoost (XGB), single layer neural network (NN), 1D-CNN, and other augmentation methods using GAN: SVM with GAN and 1D-CNN with GAN. These machine learning methods were implemented with the sklearn and xgboost python packages. For the SVM with GAN, GAN was applied to the data augmentation using a one-layer neural network for both the generator and discriminator. The input was embedded into the embedding vector with a dimension of 512 and a latent variable was randomly extracted from a normal distribution. We generated the same number of augmented data points as that of training data in each fold through the conditional GAN modelWhen measuring the prediction performance with AD gene expression data Table , the RF In JTSC, even when training with large triplet data, the model quickly arrived at an optimized model with strict semi-hard mining. During the training epoch, we counted the number of triplets in the easy, semi-hard, and hard categories to verify whether vector embedding through an embedding function was appropriately trained. Figure In addition, when hard samples were included in the training, learning ended before a sufficient number of easy samples were generated, and an oscillation interval occurred  biological process terms were used to group genes into pathways. Genes within the same pathways were positioned as neighboring elements in 1D-CNN. Note that genes appearing in multiple pathways were handled without duplication. Table Furthermore, in the ablation study, we opted for cosine similarity to compute the loss instead of Euclidean distance. Unlike Euclidean distance, distance from cosine similarity computes relative comparisons. However, in JTSC using cosine similarity, performance degradation was observed as shown in Table 28, GAN5 (G5)28 and multiple kernel learning (MKL)27, as shown in Table 28 additionally used DNA mutation data from the GDC Data Portal. Common protein-coding genes between the mutation and gene expression data were selected, which resulted in 16,804 common genes in STAD and 16,391 common genes in the remaining cancer type data. In each fold, the RandomForestClassifier from scikit-learn was performed in Python and genes with higher feature importance were selected using all the parameters described in the paper. A generator model G, which consisted of a single layer encoder and decoder and discriminator with two layers, generated multiples of the training data with selected features from FS28. From the training set, G5 generated five times the number of samples for gene expression data, which generally performed well in the paper28. Then, SVM was used as a classifier. MKL is an SVM-based method for finding the weighted combination of kernels by solving the inner optimization problem27. MKL was performed and a validation set was used to search for parameters that optimize the weight of the kernel.To evaluate our method in another task, an experiment was performed to classify early- and late-stage cancers using their gene expression profiles. To train the embedding function through 1D-CNN, 30,000 triplets were constructed for the training dataset and 10,000 triplets for the validation dataset, and the number of anchors in the early and late stages was set to be a half the number of the triplet. Clinical information, such as age and sex, was concatenated for the experiment, and specifically included in every kernel calculation in the 1D-CNN model. Five-fold cross-validation was performed with Xavier initialization. Hyperparameters, such as embedding feature dimension, patience for early stopping, and degree of angular constraint, were selected using the validation set. Then, the performance of JTSC was compared with SVM, RF, XGB, NN and other recent cancer stage classification methods; we also used an original dataset with selected features only (FS)The predictive performance for nine cancer types, COAD, ESCA, HNSC, KICH, KIRC, PAAD, STAD, TGCT, and THCA, was the best in JTSC. In particular, for cancer types with less than 100 samples in a class , JTSC performed the best, with the exception of READ (Supplementary Table 39. However, for tasks with a small number of samples and a large number of features, such as gene expression data, the semi-hard mining method can improve the prediction performance. To obtain a model with better performance, the semi-hard samples that were converted to hard or easy samples when the anchor labels and positive labels were exchanged were removed. Unlike other triplet mining methods that use all triplet data for training, the embedding function was trained in a more sophisticated manner by considering the distance between embedded samples in certain conditions. In addition, angular loss was added to alleviate the scale dependence of a triplet loss, along with a distance-based objective function in the training. As a result, the sampling method and joint loss improved prediction performance on AD and TCGA datasets.In the present study, a novel strict semi-hard mining method with constraint and joint triplet loss for deep metric learning is proposed. In distance metric learning, for tasks that train with a large number of data points, such as image classification, hard mining is more efficient than using whole triple training dataSupplementary Information.


The analysis of whole genomes of pan‐cancer data sets provides a challenge for researchers, and we contribute to the literature concerning the identification of robust subgroups with clear biological interpretation. Specifically, we tackle this unsupervised problem via a novel rank‐based Bayesian clustering method. The advantages of our method are the integration and quantification of all uncertainties related to both the input data and the model, the probabilistic interpretation of final results to allow straightforward assessment of the stability of clusters leading to reliable conclusions, and the transparent biological interpretation of the identified clusters since each cluster is characterized by its top‐ranked genomic features. We applied our method to RNA‐seq data from cancer samples from 12 tumor types from the Cancer Genome Atlas. We identified a robust clustering that mostly reflects tissue of origin but also includes pan‐cancer clusters. Importantly, we identified three pan‐squamous clusters composed of a mix of lung squamous cell carcinoma, head and neck squamous carcinoma, and bladder cancer, with different biological functions over‐represented in the top genes that characterize the three clusters. We also found two novel subtypes of kidney cancer that show different prognosis, and we reproduced known subtypes of breast cancer. Taken together, our method allows the identification of robust and biologically meaningful clusters of pan‐cancer samples. We propose subgroup identification for pan‐cancer studies via novel rank‐based Bayesian clustering, illustrated on RNA‐seq samples from 12 tumor types. The method provides probabilistic assessments of cluster stability and transparent biological interpretation of identified clusters through their top‐ranked genes. We identify robust tissue‐of‐origin clusters but also pan‐cancer clusters including three pan‐squamous clusters with different biological functions over‐represented in top genes. ASCATallele‐specific copy number of tumorsBLCAbladder cancerBRCAbreast cancerCOADcolon adenocarcinomaERestrogen receptorFEfold enrichmentGBMglioblastomaGSEAGene set enrichment analysisHNSChead and neck squamous cell carcinomaIQRinterquartile rangeKIRCrenal clear cell carcinomaLAMLacute myeloid leukemiaLUSClung squamous cell carcinomaMCMCMarkov Chain Monte CarloOVovarian cancerQ1first quartileQ3third quartileREADrectum adenocarcinomaSFEselected functional eventsSNFsimilarity network fusionTCGAthe Cancer Genome AtlasUCECendometrial cancer1Whole‐genome multi‐omics profiling of tumors has brought unprecedented information and an increased understanding of the characteristics of cancer, allowing the comprehensive investigation of the underlying DNA  and of the phenotype as represented by gene expression profiling and protein profiling. However, the analysis and interpretation of these data are still a challenge. Pan‐cancer molecular studies aim to identify key molecular characteristics that distinguish subtypes of cancer and allow exploring differences and commonalities of tumors originating from different tissues. Hence, emerging knowledge may highlight key cancer‐driving alterations and elucidate the difference between alterations stemming from the cell of origin and alterations stemming from environmental or otherwise external influence. Ciriello et al.  performeFor the purposes of performing a pan‐cancer analysis, clustering methods need to be capable of handling heterogeneous data sources across different tissues. Moreover, fully probabilistic (Bayesian) methods may help to interpret detected subgroups, to allow the integration and propagation of data and model uncertainties into the final results, and to be able to draw probabilistic conclusions that are useful for biological interpretation. A typical drawback of Bayesian methods, and the reason why their use in large‐scale integrative genomics has often been limited, is the heavy computational burden associated with such procedures.U test as an alternative to the t‐test), we expect the use of a statistical model for rankings to add robustness to the analysis. In addition, the use of ranks eases model assumptions regarding the distributions of the data and allows the joint analysis of heterogeneous data from different tissue types, since ranks are insensitive to scale heterogeneity.We propose to use a Bayesian rank‐based approach , which iOur rank‐based Bayesian method allows for (a) fully probabilistic uncertainty propagation via prior and modeling choices, (b) data integration from heterogeneous sources, and (c) joint analysis of RNA‐seq data for thousands of tumor samples and genomic features. Here, we apply the method to a pan‐cancer cohort comprising 2617 samples from 12 cancer types from the Cancer Genome Atlas (TCGA). We identify 16 distinct and robust clusters (named RankClusters) and confirm that the tissue of origin is a strong determinant of the transcriptional programs behind each cluster. In addition to confirming known breast cancer subtypes, with this method, we identify three biologically distinct pan‐squamous clusters and two novel subgroups of kidney cancer with different survival prognosis. We demonstrate the precision and robustness of our rank‐based unsupervised model, together with the computational feasibility and usefulness of the Bayesian approach, e.g., for providing easy interpretation of the results, on a previously analyzed benchmark pan‐cancer dataset.22.1N = 2617 of these patients, and the data were downloaded from the corresponding Synapse.org repository (DOI: 10.7303/syn2468297). For computational reasons, and to focus on genes already known to be important for cancer, we selected all gene expression variables, which correspond to genes that are affected by any of the SFEs identified in Ciriello et al. [In this manuscript we reexamined the ‘Pan‐Cancer‐12’ tumor set of 3299 patients, which were also analyzed by Ciriello et al. . Pre‐proo et al. . The fin2.2n), where n indicates the total number of genes in the data set (n = 1247). Only the RNA‐seq rank among genes within each sample is used in the analysis. Moving from the original measurement scale to a rank scale can be beneficial, especially when the objective is to jointly analyze samples originating from different tumors: Different tissues might produce data on different scales, which are not necessarily linked to each other linearly, RNA‐seq rankings  can allow a more meaningful joint analysis since they are insensitive to scale heterogeneity. Moreover, being able to easily use multiple datasets jointly in one single analysis can lead to more powerful inference.The aim of the statistical analysis is to perform clustering of the samples. The employed clustering method is based on ranks: RNA‐seq gene measurements are ordered for each sample from the largest (rank 1) to the smallest  the model parameters represent long‐term frequencies, and they are definite, fixed numbers, even though unknown. The aim of the statistical analysis is to provide their estimate from the data at hand. However, such an approach does not take into account any previous information that might make certain parameter values more likely, even before observing the current set of data. An alternative inferential framework, named Bayesian paradigm from the central role played by the Bayes theorem, provides instead this possibility thanks to a quite radical change of perspective. In the Bayesian approach, model parameters are random variables, whose distribution ‘a posteriori’ is the target of inference. Inferential efforts are not any longer focused on a single unknown value of the model parameters, but a variety of values is allowed, each with its own associated uncertainty. Probability distributions are therefore used to express our knowledge about the parameters even before observing the data, in the so‐called prior distributions. Data are then used to update this knowledge via the likelihood, thus obtaining a new parameter distribution (the posterior), which can be described and used in different ways to summarize the analysis results. In conclusion: ‘The Bayesian paradigm is a practical approach where prior and posterior distributions are used as models of our knowledge before and after collecting some data and making an observation. It is particularly useful for integrating or combining information from different sources’ [In the classical approach to statistics  distribution Pθ|data. The posterior distribution is therefore a combination of prior and likelihood, in the sense that the a priori distribution of model parameters is updated with data information, as expressed in the likelihood. This has some important implications: The posterior of model parameters is generally close to the likelihood for big data; when instead data are low‐dimensional or sparse, clinical knowledge as embedded in the prior becomes very relevant. For a thorough introduction to Bayesian inference, we suggest to consult the first chapters in Bernardo and Smith [One can perform inference about nd Smith . For a vnd Smith . A more nd Smith , in Sect2.3Rij be the rank given to gene Ai by sample j, for i = 1, … n and j = 1, … N . A Mallows model [Rj defined on the ranking space (the permutation space Pn of dimension n), which is very similar to a generalized Gaussian distribution for continuous data. If we assume that the N observed rankings follow the Mallows model, the data likelihood then takes the formLet ws model  is a disρ, called consensus ranking, which is a ranking describing the center of the distribution , and a scale parameter α, which is a scalar describing its variation . Therefore, if the Mallows model is assumed to describe the distribution of the observed rankings of genes in the sample, the probability of observing each patient's specific ranking of genes will be larger the closer the ranking is to the consensus ranking, i.e., the location parameter, quite the same as for normally distributed data in the continuous case. The closeness is measured by the chosen distance between rankings, d.., and weighted by the scale parameter α. In other words, given a set of patients, each providing an observed ranking of genes, the Mallows consensus ranking estimates (with uncertainty) the gene ranking that best describes the agreement within the set (see [C components (clusters) is here used to jointly estimate the groups and the group‐specific consensus gene rankings. Note that ‘group’ and ‘cluster’, as well as ‘grouping’ and ‘clustering’, will be used interchangeably throughout the paper. Each cluster c = 1, … C gets a specific location ρc and scale ac parameter, and each sample is assigned to a cluster via the latent variables z1,…zN∈1…C. In the clustering framework, the model likelihood then takes the formThe Mallows model depends on two parameters: a location parameter set . We thus obtain a co‐localization N×N dimensional matrix S whose elements take the form:z¯iC and z¯jC are the majority vote cluster assignments of samples i and j, respectively, when using a mixture with C clusters, and where Cm=10 and CM=30. Values in S range from 0 to 1, with 1 indicating pairs of samples that are always grouped together, and 0 indicating pairs that are never grouped together. Therefore, observing a block structure in S indicates good stability of the clustering across values of C.Concerning the dynamics of the clustering results, since our mixture‐based Bayesian clustering approach is nonhierarchical, if samples are often assigned to the same group when varying 2.5ρc of the genes, which is estimated with uncertainty by the MCMC algorithm. Therefore, in order to inspect the gene‐wise commonalities of samples belonging to the same group, it is sufficient to summarize the marginal posterior distribution of the corresponding cluster‐specific consensus ranking ρc, for c=1,…C.A secondary but equally important aim of the analysis is to describe the commonalities in the RNA‐seq measurements of the samples assigned to the same cluster in terms of the estimated cluster consensus gene ranking, i.e., the cluster‐specific ranked list of genes. Each cluster in the mixture model is characterized by the consensus ranking c=1,…C, we compute a cluster‐specific list of genes by ranking the genes according to their posterior probability of being top‐ranked in the cluster‐specific consensus ranking, with ‘top‐k’‐ranked indicating a rank not larger than k, for k ranging from 10 to 100. Specifically, this entails computing, for each cluster c and each specific value of k, the marginal posterior probability: P for all genes i=1,…n. Ranking such probabilities allows for obtaining several cluster‐specific gene lists, which can be used for different purposes. For instance, we observe that very few genes show a significant posterior probability of being top‐10 ranked, and thus choosing k = 10 results in stable lists of few genes, which can easily be studied comprehensively. They represent sparse and easily interpretable molecular characterizations of the clusters, which still capture the most important characteristic features of each cluster. On the other hand, we find that larger numbers of genes show a large posterior probability of being top‐100 ranked, and the resulting larger gene lists can be assumed to be a robust choice of relevant genes, which can for example be used in a subsequent gene set analysis.Specifically, for each cluster 2.6gsea) was performed using the hallmark (H) and gene ontology (C5) gene set collections obtained from the Molecular Signatures Database v7.1 [P‐values for over‐representation were calculated by one‐sided hypergeometric testing (R function phyper), where each gene list of interest was tested for association to each gene set. P‐values were corrected for multiple testing using the Benjamini–Hochberg procedure (R function P.adjust), and the enrichment was considered significant if the corrected P‐value was smaller than 0.05. To minimize the bias caused by the 1247 SFE‐associated genes, this gene set was used as background when the gsea was performed.Gene set enrichment analysis  for all genes. The difference between groups was considered statistically significant if the two‐sided FDR‐corrected P‐value according to the method of Benjamini and Hochberg [Differential gene expression analysis was performed using the nonparametric Mann–Whitney Hochberg  was smal2.9To assess the robustness of the Bayes Mallows method and the importance of the choice of gene set, we also performed the Bayes Mallows clustering on a random selection of 1247 genes (see Table 2.10r package pheatmap v. 1.0.12 [hclust method ward.D2).To compare the Bayes Mallows clustering to traditional clustering methods, we performed unsupervised hierarchical clustering using both the SFEs‐associated gene set and the random gene set. The heatmaps and dendrograms were produced with the . 1.0.12 , and the2.11r software version 4.2.0 [All statistical analyses were performed using the on 4.2.0 .33.1S obtained by counting the samples' co‐localizations in the same cluster for the groupings obtained with C (the number of groups) varying from 10 to 30. Note that we chose to not consider grouping structures obtained for C < 10 since they clearly under‐represent the variation in the data, and thus do not provide a reasonably meaningful clustering of the samples. Figure C even if the hierarchy is not enforced by the mixture‐based clustering procedure: Indeed, most values in S are either close to 0 or 1, indicating that the clusters represent a biologically meaningful grouping, not influenced by the model choices and model adequacy. Samples from the same tissue are often seen stably clustered together , and a pan‐cancer cluster composed of BLCA, HNSC, and LUSC appears for many values of C. Some cancer types are occasionally found together, indicating that they cluster together at lower values of C .We applied the Bayes Mallows approach to the ‘Pan‐Cancer‐12’ tumor set of 2617 patients interrogating the gene expression of 1247 genes representing selected functional (cancer‐driving) events [C) we inspected the boxplots of the posterior distributions of the within‐cluster sum of distances , while RankClusters 4, 8, and 16 all showed co‐occurrence counts stochastically smaller  than average , thus allowing easy investigation of upregulated pathways characterizing clusters. Specifically, each RankCluster can be characterized by the group of genes showing the largest probability of being top‐ranked: In Fig. 3.4RankCluster 3 contained 66 HNSC, 27 LUSC, and eight BLCA samples, RankCluster 5 contained 75 HNSC, 116 LUSC, and eight BLCA samples, and RankCluster 6 contained 150 HNSC, 16 BLCA, and two LUSC samples, illustrating that subgroups of squamous cell carcinomas share more commonalities across the tissue of origin than to other carcinomas of same tissue of origin , it was clear that the two clustering methods did not capture precisely the same variation. Hierarchical clustering of squamous cell carcinomas using mRNA expression has been reported by Campbell et al. [P‐value = 1.81e‐86). The mRNA clusters 2 and 3 mostly contain samples, which were clustered into RankClusters 5 and 6, respectively. However, by splitting mRNA clusters 1, 4, and 6, RankCluster captured additional and complementary information.Pan‐cancer clusters of squamous cell carcinoma have been discovered previously  where thl et al. , and a q3.5gsea showed that the genes in both clusters were enriched in gene sets related to transducer activity and plasma membrane transport , showing that the two subgroups of KIRC share key biology and simultaneously differ with regard to transcriptional programs. Survival analysis showed that patients in RankCluster 15 had better progression‐free survival compared with patients in RankCluster2 , and ort Fig. . InteresP‐value = 1.71e‐34), with mRNA cluster 1 containing almost exclusively samples from RankCluster 15. Among the three other mRNA clusters, the samples were quite evenly distributed between RankCluster 2 and 15 ‐estimated tumor purities  across R4We have presented a pan‐cancer analysis performed using a rank‐based Bayesian mixture model, i.e., a clustering method based on the Bayes Mallows model, which allows for drawing probabilistic conclusions. Bayesian inference allows for expressing the uncertainty associated with all results, which is useful in biological interpretation. The use of ranks instead of original measurements gives greater flexibility in combining heterogeneous data sources (across tissues) as pre‐processing to match measurement scales is not required, which in turn allows analyses of combined datasets, e.g., for a pan‐cancer analysis. A parametric mixture‐based model .The application of the Bayes Mallows method to a random subset of genes resulted in a very similar clustering of cancer samples compared with the selection of 1247 gene expression features associated with the SFEs identified in Ciriello et al. . HoweverC = 16 was 9 h12’, while mixtures with a smaller (larger) number of groups require slightly less (more) computing time. We are currently working on a Variational Bayes alternative to the MCMC algorithm [The rank‐based Bayesian Mallows clustering method currently presents some weaknesses, which we hope to address in future to bring the model even closer to the needs of integrative genomics applications. The first and most important aspect concerns computational costs: The Bayesian estimation procedure is currently based on an MCMC algorithm, which is known to be inefficient and can be slow, particularly for high‐dimensional applications. The current implementation limits the number of features to be ~ 1000 (like in the current pan‐cancer application) and requires hours to run in such a situation. The average computing time for Bayes Mallows with lgorithm  that samgsea. We are currently working on a dimension reduction version of the method, which will address this limitation [Secondly, the method is limited in how many features  it can handle, not only for computational but also for modeling reasons. While it is reasonable to assume that the pre‐selection of 1247 genes associated with the SFEs are all expressed and would rank differently in different clusters, this would not be a reasonable assumption for a genome‐wide data set, because differences across clusters in the top‐ranked genes would affect the likelihood in the same way as differences way down in the ranking would do. A biologically more sound assumption could be that only the genes that are expressed in the tissues of interest are the ones driving cluster differences. The method's limitation to ~ 1000 genes also puts limitations on the biological interpretation of cluster‐associated gene lists, as different selections of genes will give different results in a mitation .snf [iclusterplus [Finally, it would be very relevant to propose a generalization of the Bayes Mallows method to perform data integration across multiple omics data types (multi‐view clustering). With such a generalization, we would be able to provide a rank‐based clustering of the samples that jointly account for several data views. The gold standard for multi‐view clustering is still snf  and iclusterplus  for a Basterplus . A Bayes5We have presented the novel use of a Bayesian rank‐based clustering method for the purposes of a pan‐cancer analysis of a gene expression data set from TCGA. As with any other Bayesian approach, the method has the advantage of providing proper uncertainty quantification of all unknowns, together with probabilistic interpretation of all results. This is particularly helpful in the analysis of large molecular databases, to allow straightforward assessment of the stability and reliability of conclusions, and to provide biological characterization and interpretation of the identified clusters. In addition to tissue‐specific clusters, we identified three pan‐squamous clusters composed of a mix of lung squamous cell carcinoma, head and neck squamous carcinoma, and bladder cancer, with different biological functions enriched in the lists of top‐ranked genes. We also identified two subtypes of kidney cancer with very different survival prognosis, and reproduced known subtypes of breast cancer. Taken together, our method allows robust gene expression‐based clustering of pan‐cancer samples, providing novel biologically meaningful insight.The authors declare no conflict of interest.TF, VV, MZ, AF, EA, and VNK conceptualized the manuscript. VV and MZ performed the rank‐based Bayesian cluster analysis. TF and JA performed the additional statistical analysis. TF, VV, and MZ interpreted the data. TF and VV wrote the manuscript with contributions from all other authors. All authors read and approved the final version of the manuscript.https://publons.com/publon/10.1002/1878‐0261.13354.The peer review history for this article is available at Fig. S1. Explanation of the Bayesian rank‐based clustering method.Fig. S2. Bayes Mallows clustering method on a random selection of 1247 genes.Fig. S3. Hierarchical clustering and comparison with RankClusters and random gene selection.Fig. S4. Density of the counts of how many times a top‐ranked gene (for a given RankCluster) was top‐ranked also for other RankClusters for genes with at least 1% probability of being ranked top‐100 in each cluster.Fig. S5. Kaplan–Meier plot of progression‐free survival for patients in the three pan‐squamous RankClusters.Fig. S6. Average allele‐specific copy number of tumors (ASCAT)‐estimated tumor purities for the 16 RankClusters.Click here for additional data file.Table S1. Random genes used for clustering.Click here for additional data file.Table S2. Gene rankings (probability of being top 10).Click here for additional data file.Table S3. Probabilities of genes (Table S2) being top 10.Click here for additional data file.Table S4. Gene rankings (probability of being top 100).Click here for additional data file.Table S5. Probabilities of genes (Table S4) being top 100.Click here for additional data file.


With the rapid development of the Internet of Things (IoT), the frequency of attackers using botnets to control IoT devices in order to perform distributed denial-of-service attacks (DDoS) and other cyber attacks on the internet has significantly increased. In the actual attack process, the small percentage of attack packets in IoT leads to low accuracy of intrusion detection. Based on this problem, the paper proposes an oversampling algorithm, KG-SMOTE, based on Gaussian distribution and K-means clustering, which inserts synthetic samples through Gaussian probability distribution, extends the clustering nodes in minority class samples in the same proportion, increases the density of minority class samples, and improves the amount of minority class sample data in order to provide data support for IoT-based DDoS attack detection. Experiments show that the balanced dataset generated by this method effectively improves the intrusion detection accuracy in each category and effectively solves the data imbalance problem. According to the IoT OS Security 2022 White Paper , the gloThe proliferation of IoT devices has also led to a significant rise in cyber attacks, particularly Distributed Denial of Service (DDoS) attacks perpetrated by attackers leveraging IoT devices. One notable example is the Mirai attack in 2016, which resulted in the infection of millions of IoT devices and the launch of DDoS attacks against DYNs, causing substantial economic losses. The CNCERT Internet Security Threat Report reveals that, as of 2022, over 4.78 million botnet agents were utilized in DDoS attacks, some of which were IoT devices [Traditional classification models assume a balanced sample distribution when addressing classification problems . HoweverThe data-level approach involves modifying the original dataset through data preprocessing and resampling in the sample feature space in order to obtain a balanced dataset. The algorithm-level approach focuses on enhancing or integrating classification algorithms in order to improve data differentiation and classification accuracy. The integrated learning approach combines data-level and algorithm-level methods with integrated learning techniques to adapt to the requirements of unbalanced data classification and enhance the learning of minority class samples.In this paper, the focus is on data-level methods as they are implemented during the data preprocessing stage and can be applied generally. Specific strategies are used to increase the density and diversity of minority class samples without altering the sample distribution, aiming to reduce or solve the issue of imbalanced data. The methodology includes undersampling techniques , oversamUndersampling techniques involve balancing the dataset by removing some majority samples in order to achieve balance. However, removing majority samples may cause the classification algorithm to lose important information about the majority classes, leading to a decrease in intrusion detection accuracy. Therefore, this paper focuses on the oversampling technique to address the data imbalance problem . The basThis paper proposes a novel oversampling method for unbalanced data called KG-SMOTE, which is based on Gaussian distribution and K-means clustering. The main objective of this method is to improve the prediction accuracy of minority categories, particularly the minority categories with limited sample sizes. The proposed approach aims to achieve balance among all categories by adjusting the distribution of the minority categories, ensuring that their sample sizes are not significantly different from the majority categories. To achieve this, the KG-SMOTE algorithm is employed to generate new samples for the minority category.The motivation behind developing the KG-SMOTE method is to address the challenge of extremely unbalanced datasets. The utilization of the K-means clustering algorithm allows for the division of the minority class into distinct clusters with similar attributes. This approach enables the extraction of additional information from each cluster, revealing hidden patterns or groupings in the data. Moreover, it helps to reduce the generation of noisy samples. The adoption of Gaussian distribution is another key aspect of the KG-SMOTE method. This choice ensures that the synthetic data generated maintain similar probability distributions to the original minority data. It also addresses the issue of overfitting that can arise from an excessive concentration of synthetic samples produced by traditional SMOTE in extremely unbalanced datasets.The remaining sections of this paper are structured as follows. There are different approaches in the literature for dealing with the class imbalance problem. This section reviews recent methods and techniques used for balancing data. Random Oversampling (ROS) is the most straightforward oversampling technique. Since the ROS method reaches the equilibrium point by replicating a few samples in order to expand the dataset, it does not generate new samples. Therefore, it is prone to overfitting problems during the training of classification algorithm models and has some limitations in practical applications.Chawla  proposedYehui et al.  proposedLast F  used theZhu Shen et al.  proposedZhang Z et al.  proposedJiang Zetao et al.  proposedKamalov et al.  proposedZhang H et al.  proposedHassan et al.  proposedLi Tao et al.  proposedZhang H et al.  proposedThe aforementioned work has made significant contributions to addressing class imbalance in various domains, such as network datasets, financial data, and medical data. However, most of these methods focus on algorithmic improvements using existing datasets and do not specifically target algorithmic enhancements for DDoS attack scenarios in the context of the Internet of Things.This paper aims to fill this gap by proposing the KG-SMOTE oversampling algorithm, designed specifically for IoT devices launching DDoS attacks against internet servers and clients. Taking into consideration the characteristics of IoT traffic and highly imbalanced datasets, this method greatly improves the detection effectiveness for DDoS attacks initiated by different virus protocols.This section reviews some existing well-known resampling methods.SMOTE is a widely used oversampling method for addressing the issue of imbalanced datasets with minority class samples. This algorithm aims balance the sample distribution among different classes by generating synthetic samples.The SMOTE algorithm follows a series of steps. First, it iterates through all minority class samples and calculates the Euclidean distance between each minority class sample and other minority class samples. Next, for each minority class sample, the algorithm randomly selects a sample from its k-nearest neighbors. Using the distance information between these samples, new synthetic samples are generated.By adding these synthetic samples to the original dataset, the number of minority class samples is increased to match that of the majority class samples. This helps to alleviate the class imbalance problem. The formula for synthesizing samples in SMOTE is as follows:The SMOTE method balances the class distribution in the dataset by synthesizing new minority class samples in order to increase the number of minority class samples. This reduces the classifier’s over-preference for majority class samples and improves the recognition of minority classes.The K-Means SMOTE algorithm incorporates the k-means clustering step the traditional SMOTE algorithm in order to address the data imbalance issue. This algorithm first clusters the samples using k-means and calculates the Euclidean distance between them. It then selects the centroid of each cluster for SMOTE processing, generating new synthetic samples.By including the k-means clustering step, the K-Means SMOTE algorithm effectively preserves the features of the minority class samples. This approach reduces the generation of noisy points and outliers compared to the traditional SMOTE algorithm. As a result, the performance of the classifier is enhanced.The Gaussian probability distribution, also known as the normal distribution, is a continuous probability distribution that is commonly observed in everyday life. It is considered the most important distribution, as it applies to a wide range of events. The principle behind the Gaussian distribution is based on the central limit theorem, which states that the sampling distribution of means tends to converge to a normal distribution, regardless of the true underlying distribution.When applying the Gaussian probability distribution for data analysis, there are typically five steps involved. First, relevant data related to the subject of study are collected. Next, the data are pre-processed to ensure accuracy and consistency by removing outliers or other necessary steps. The parameters of the Gaussian distribution, such as the mean and standard deviation, are then estimated using methods such as maximum likelihood estimation. These estimated parameters are used to obtain the Gaussian distribution curve through distribution fitting methods. Finally, the data are analyzed and inferences are made based on the properties of the Gaussian distribution, such as calculating probabilities and determining confidence intervals.The probability density function of the Gaussian distribution is represented by the formula shown in :(2)f(x)=To address the imbalance in the dataset, it is necessary to generate a large number of synthetic data samples. However, this approach can result in excessive overlap of synthetic data, leading to duplicated features in the minority sample set. This, in turn, can cause the classification algorithm model to overfit the data. Overfitting occurs when the model learns too much from a few classes of samples, leading to a decrease in detection accuracy. Moreover, due to the characteristics of IoT DDoS attacks, each device issues attack protocols and formats that differ from one another. This further increases the data imbalance ratio, which negatively impacts the accuracy of detecting the same attack type.To address these challenges, this study proposes an improved oversampling algorithm called KG-SMOTE, which is based on K-Means clustering and Gaussian probability distribution. The KG-SMOTE algorithm enhances the intrusion detection of imbalanced IoT attack data from a data-level perspective.The KG-SMOTE algorithm first employs the K-Means clustering algorithm to divide the minority classes into different clusters with similar attributes. This step helps reduce interference from noise and anomalous samples. Secondly, the algorithm utilizes a Gaussian probability distribution to generate synthetic data with a similar probability distribution as the original minority class data. This approach improves the quality of the generated samples in datasets with high imbalance ratios, reduces the generation of redundant and noisy samples, and mitigates the occurrence of overfitting in the classification model.The KG-SMOTE algorithm is based on the K-Means SMOTE algorithm, which combines a Gaussian distribution to generate new samples during synthetic data generation. The K-Means algorithm is used to cluster the minority classes in the dataset, calculate the number of minority class samples in each cluster, and increase the density of minority class nodes in equal proportion. The number of synthetic samples is determined by treating each attribute of the training data as a random variable, and all attributes are independent of each other. The given m attributes, denoted as n is the number of instances of the minority class. Inspired by (5), given the values N, we obtain the following equation:In (6), In (7), We will refer to (8) as the normal distribution model. The flow chart of its KG-SMOTE algorithm is shown in The determination of the K-value in this study was carried out using the Elbow Method, which is a commonly employed technique in the K-means algorithm. The Elbow Method evaluates the intra-cluster structure by analyzing the sum squared error (SSE) between each cluster and the samples within it.The degree of line distortion in the plot represents the spatial variation of the samples within the cluster. A higher degree of distortion indicates a looser cluster structure, while a lower degree of distortion suggests a tighter cluster structure. The Elbow Method aims to identify the point at which the distortion degree starts to exhibit a significantly slower rate of change. This point is typically considered the optimal number of clusters .As illustrated in The algorithm employed in this study utilizes a clustering approach to synthesize new samples within clusters. This technique effectively reduces the interference from noise and anomalous samples, enabling the extraction of more valuable information from each cluster. The primary goal is to uncover hidden patterns or groupings within the data.To generate synthetic data with similar probability distributions as the original minority class data, a Gaussian distribution is applied to each minority cluster. This process involves calculating the normal distribution of all minority-class samples. New samples are then inserted within the range of this calculated distribution. By simulating the distribution characteristics of real data, the generated samples possess higher realism and usability. This approach helps prevent the generation of redundant samples and mitigates the risk of overfitting.Furthermore, compared to the traditional SMOTE algorithm, KG-SMOTE does not require the execution of the nearest neighbor algorithm in order to obtain the k-nearest neighbors before creating synthetic samples. This design choice reduces the impact of the parameter settings. In this paper, we utilized the MBB-IoT dataset , which iThe specific experimental scenario involved the creation of an IoT botnet by infecting various devices such as smartphones, webcams, smart speakers, smart gateways, and a Raspberry Pi. The targets of these attacks were servers located outside the local area network (LAN). The network environment consisted of a blend of both benign and botnet traffic. The raw PCAP file obtained from this environment contained a staggering 54 million records. For this study, a subset of the dataset was selected, comprising 1% of the total records, amounting to over 540,000 records. Data pre-processing is an essential step in ensuring the quality and usability of a dataset. Raw data often contain irregularities and incompleteness, such as missing values and outliers. Modeling with such data can result in distorted and inaccurate models. Therefore it is necessary to pre-process the raw data before conducting any modeling and analysis.Data pre-processing typically involves various steps, including data cleaning and data transformation. In this study, the dataset was iterated to identify and handle missing values. Fortunately, no missing values were found in the dataset.Additionally, features that had values consistently equal to a certain value were removed during the pre-processing stage. The specific features that were removed are listed in The experimental environment and associated hardware are illustrated in Commonly used metrics to evaluate the performance of a classifier include accuracy, precision, recall, F1-score, and AUC (Area Under the Curve). In the case of unbalanced multi-classification problems, accuracy is not a reliable measure of classifier performance. Given the complexity of unbalanced multiclassification, this paper adopts precision, recall, F1-score, and AUC as the evaluation metrics. These metrics are widely used to assess the classification performance of unbalanced data and provide a more comprehensive evaluation of the classifier’s effectiveness .TP denotes the number of true positive samples, FN denotes the number of false negative samples, and FP denotes the number of false positive samples. By calculating precision, we can evaluate how well a classifier is able to accurately identify positive cases from the total number of samples predicted as positive.Precision is a commonly used measure to assess the classification accuracy of a classifier for positive class samples. It represents the proportion of samples that are truly positive among all samples predicted to be positive by the classifier. Precision is calculated using the following formula:TP represents the number of true positive samples and FN represents the number of false negative samples. By calculating the recall, we can assess how well a classifier is able to correctly identify positive cases from the total number of positive instances in the dataset.Recall, also known as sensitivity or true positive rate, measures the proportion of samples that are correctly classified as positive out of all samples that are truly positive. It is calculated using the following formula:The F1-score is a widely used evaluation metric for assessing the performance of classifiers on unbalanced data. It is a harmonic mean of precision and recall, combining both measures into a single value. By considering both precision and recall, the F1-score provides a balanced assessment of a classifier’s performance in scenarios where class distribution is imbalanced.In binary classification problems, the AUC (Area Under the Curve) is a widely used evaluation metric to assess the performance of a classification model. It is calculated by determining the area under the Receiver Operating Characteristic (ROC) curve, which ranges between 0 and 1. The AUC value indicates the classifier’s ability to rank positive and negative cases correctly based on the ROC curve.An AUC value of 1 suggests that the classifier perfectly distinguishes positive cases from negative cases. Conversely, an AUC value of 0.5 implies that the classifier’s performance is equivalent to random guessing. If the AUC is less than 0.5, it indicates that the classifier’s performance is worse than random guessing.In this paper, we conducted experiments to compare the performance of classical oversampling algorithms, namely Borderline SMOTE, SMOTE, and K-Means SMOTE, combined with classical decision tree classification algorithms. The MBB-IoT dataset was used for the experimental evaluation, and the results were compared accordingly. Additionally, to validate the effectiveness of oversampling, we included the classification results of the decision tree classification algorithm without any sampling method as a basis of comparison.The experimental process involved preprocessing the dataset and employing five-fold cross-validation as the evaluation method. We ran the experiments ten times independently and calculated the average results. The experimental findings, as shown in In However, it is important to note that high precision does not necessarily mean that samples are correctly identified. Precision only considers the ratio of correctly classified positive samples to all samples classified as positive. It does not take into account the number of false negatives, thus not providing a comprehensive reflection of the classifier’s performance.In highly imbalanced datasets, if the classifier predicts all samples as the majority class, the precision may be high because the majority of samples are correctly classified as the majority class. However, this does not mean that the classifier can correctly identify minority class samples, and the recall rate may be low. Therefore, relying solely on precision to evaluate the performance of the classifier is not comprehensive.In conclusion, although KG-SMOTE performs well in terms of precision and achieves high accuracy in the BASHLITE TCP attack type, relying solely on precision is not sufficient to comprehensively evaluate the performance of the classifier. It is important to consider other metrics such as recall, F1 score, etc., in order to obtain a more comprehensive and accurate assessment.In The F1 score is a comprehensive performance evaluation metric that combines precision and recall. From In the attack types where KG-SMOTE did not achieve the top performance, the algorithm only slightly lags behind the best algorithm by 0.4% and 0.38% in the Benign and BASHLITE TCP attack types, respectively. This indicates that the performance gap of KG-SMOTE in these attack types is very small.According to The results depicted in When comparing the classification results of unsampled data and SMOTE oversampling, it is apparent that the classification results after employing SMOTE oversampling are significantly lower. This can be attributed to the fact that the synthesized data points are distributed throughout the intervals of multiple classes, giving rise to a large number of noise samples and anomalous samples. This blurs the classification boundaries, making it challenging for the classification algorithm to accurately distinguish between different classes, thereby reducing recognition accuracy.In contrast, when using the Borderline SMOTE method, synthetic samples are generated along the line connecting minority samples in the feature space. However, in scenarios where there is a substantial disparity between the majority and minority classes, a large number of synthetic data points are required. Consequently, these synthetic samples tend to be concentrated along the same line, resulting in overlapping synthetic samples. This can lead to overgeneralization issues in the model, thereby reducing recognition accuracy.Furthermore, the Borderline SMOTE algorithm classifies minority class samples into three categories. While this approach may help address the class imbalance in the dataset, it can also result in fewer minority class samples remaining in an already imbalanced dataset. This reduction in minority class samples may cause important features to be overlooked, further contributing to a decrease in recognition accuracy.From From To accurately simulate real-world IoT attack scenarios, we have utilized the MBB-IoT dataset, which was collected in the context of constructing an IoT botnet by infecting various IoT devices such as smartphones, webcams, smart speakers, smart gateways, and a Raspberry Pi. The attacks were targeted towards servers that were not on the same local area network. Due to the diverse nature of these IoT devices, the dataset exhibits variations in data formats, posing challenges in balancing the dataset.Unbalanced data present a challenging task for many classification algorithms. Resampling the training data to achieve a more balanced distribution is an effective method to address this issue, independent of the choice of classifier. However, simply replicating instances of the minority class to balance the classes can lead to overfitting, thus reducing the performance of the model on unseen data. On the other hand, techniques that generate artificial samples often tend to produce noisy samples, hindering the inference of class boundaries. Furthermore, most existing oversamplers fail to offset the imbalance within the minority class, which is typically a major problem when classifying imbalanced datasets. In order to effectively assist the classifier through oversampling, it is important to avoid amplifying noise by detecting safe regions in the input space where class regions do not overlap. Additionally, any imbalances within the minority group should be identified and samples generated to achieve a balanced distribution.In response to the problem of data imbalance in the field of intrusion detection based on the Internet of Things (IoT), this paper proposes a KG-SMOTE algorithm that combines Gaussian probability distribution and the K-Means clustering algorithm. This algorithm utilizes k-means clustering to group the data and concentrate the generation of synthetic samples in the critical regions of the input space. A high proportion of minority observations are used as indicators of safe regions. Oversampling is performed only on the safe clusters, which allows the k-means SMOTE to avoid generating noise. Additionally, the average distance between minority samples within a cluster is used to identify sparse regions. More synthetic samples are generated for the sparse minority groups, which helps alleviate the imbalance within the class. Considering the protocol diversity of IoT devices in DDoS attacks and the heterogeneity of data structures, this approach provides a more effective way of synthesizing samples, thus addressing the class imbalance issue in IoT intrusion detection datasets to some extent. Building upon the K-Means SMOTE oversampling algorithm, new samples are generated using a Gaussian probability distribution. This preserves the sample distribution, increases sample diversity, reduces the overlap ratio of synthetic samples, reduces the risk of classifier overfitting, and improves the detection accuracy of minority class samples.The main advantages of the KG-SMOTE method are its simplicity, efficiency, and flexibility in generating synthetic samples for the minority class. The KG-SMOTE algorithm performs particularly well for attack types with higher imbalance ratios. For BASHLITE UDPHex (1:768), it outperforms SMOTE, Borderline SMOTE, and K-Means SMOTE in terms of accuracy, precision, recall, F1-score, and AUC. In mirai syn (1:249), it exhibits advantages in accuracy, precision, F1-score, and AUC. However, for BASHLITE TCP (1:164), it does not show any advantage in any of the metrics. The KG-SMOTE algorithm demonstrates strong performance in scenarios with high imbalance. Conversely, it has some limitations in situations with lower imbalance levels. It should be noted that IoT devices have become new actors in launching DDoS attacks, while the targets of these attacks are still in the early stages of internet applications. One future research direction is to further explore and study imbalanced classifiers in this context.


One important characteristic of single-cell RNA sequencing (scRNA-seq) data is its high sparsity, where the gene-cell count data matrix contains high proportion of zeros. The sparsity has motivated widespread discussions on dropouts and missing data, as well as imputation algorithms of scRNA-seq analysis. Here, we aim to investigate whether there exist genes that are more prone to be under-detected in scRNA-seq, and if yes, what commonalities those genes may share. From public data sources, we gathered paired bulk RNA-seq and scRNA-seq data from 53 human samples, which were generated in diverse biological contexts. We derived pseudo-bulk gene expression by averaging the scRNA-seq data across cells. Comparisons of the paired bulk and pseudo-bulk gene expression profiles revealed that there indeed exists a collection of genes that are frequently under-detected in scRNA-seq compared to bulk RNA-seq. This result was robust to randomization when unpaired bulk and pseudo-bulk gene expression profiles were compared. We performed motif search to the last 350 bp of the identified genes, and observed an enrichment of poly(T) motif. The poly(T) motif toward the tails of those genes may be able to form hairpin structures with the poly(A) tails of their mRNA transcripts, making it difficult for their mRNA transcripts to be captured during scRNA-seq library preparation, which is a mechanistic conjecture of why certain genes may be more prone to be under-detected in scRNA-seq. Single-cell RNA-sequencing (scRNA-seq) allows the dissection of gene expression heterogeneity at single-cell resolution , which cMany computational methods and pipelines for scRNA-seq include components of gene selection and dimension reduction to address the high sparsity of the data. Selection of highly variable genes enables subsequent analysis to focus on genes whose zeros counts are more enriched by biologically meaningful zeros and less affected by the technical limitations . In dimeIn the literature, the high sparsity in scRNA-seq is often referred to as dropout. The term dropout was introduced to describe technical failures that may cause a highly expressed gene to be undetected  However,In order to develop methods to address dropouts or under-detection in scRNA-seq, it is important to understand the factors that contribute to the dropouts. Recent studies have suggested that 3′-UTR length, compartment, transcript count, and differential expression levels  may playThrough extensive literature search, we have identified eight datasets with paired bulk RNA-seq data and scRNA-seq data available for the same samples. A summary of these datasets is listed in the Materials and Methods. In total, we have paired bulk RNA-seq data and scRNA-seq data for 53 samples. The samples originated from diverse biological contexts, including fibroblasts, trachea, women reproductive system, breast cancer, and cancer cell lines.For each GEO bulk RNA-seq dataset, median-of-ratios normalization was performed followed by log transformation. The scRNA-seq data for each sample was preprocessed separately, with library size normalization followed by log transformation. Then, a pseudo-bulk RNA-seq profile was calculated for each sample, by averaging the scRNA-seq expression data across all cells in the sample. Next, for each sample pair, the normalized bulk RNA-seq and single-cell based pseudo-bulk expression of overlapping genes among the two data types were identified. With these preprocessing steps, for each of the 53 samples, we obtained one bulk RNA-seq profile and one pseudo-bulk RNA-seq profile for the overlapping genes. Then quantile normalization was performed for the bulk RNA-seq profiles.The preprocessed paired bulk and pseudo-bulk data were visualized using the scatter plots in In order to identify genes that are more prone to dropout or under-detection in scRNA-seq, we examined whether there exist genes that repeatedly appeared in the upper-left corner of the scatter plot in The top 15 genes that most frequently appeared in the upper-left gate are listed in Given the diversity of biological contexts of the 53 samples and the possibility that any given gene may only be expressed in a subset of those contexts, we broadened our criterion for under-detected genes in scRNA-seq and considered all genes that occurred more than once in the upper-left gate in To examine sequence-based commonalities among the genes that appeared more than once in the upper-left gate in As a comparison, we manually drew another gate in the upper-right corner of the density plot in For the top 15 genes with highest frequency of occurrences in the upper-right gate, their numbers of occurrences ranged from 29 to 50, which were more than half of the 53 samples. Given the diverse biological contexts of the 53 samples under consideration, such consistency of highly expressed genes was interesting. Meanwhile, since many housekeeping genes are known to be involved in diverse fundamental biological processes, such consistency of highly expressed genes was also expected. The top 15 genes with highest frequency in the upper-right gate included MALAT1, RPLP1, EEF1A1, RPL10, RPL13, RPS18, FTH1, B2M, TMSB4X, RPS4X, RPL13A, RPL32, RPS12, RPS27A, and RPL11. B2M encodes a protein which is associated with MHC class I heavy chain on the surface of nearly all nucleated cells . TMSB4X Using the MEME Suite, we searched for enriched motifs among the 96 genes that appeared more than once in the upper-right gate in For completeness, we also attempted to identify genes that are frequently over-detected in scRNA-seq compared to bulk RNA-seq. We manually drew a third gate in For genes that appeared more than once in the bottom gate in To demonstrate the robustness of the enriched sequence motifs for the genes that are under-detected in scRNA-seq, we repeated the analysis of the 53 paired samples with four choices of scRNA-seq normalization algorithms, including DESeq2 , SCTransTo examine the robustness of our comparison between paired bulk RNA-seq and scRNA-seq expression data, we randomly shuffled the gene expression profiles to create 53 random pairs, where each pair of bulk RNA-seq profile and pseudo-bulk profile from scRNA-seq were generated from different biological samples. With the randomly paired data, we performed the same analysis as above, and examined whether the randomly paired data would produce similar results.The randomly paired bulk and pseudo-bulk data were visualized using the scatter plots where each dot represents expression data of one gene in one randomly paired expression profiles . In FiguSimilar to the analysis above, we manually drew three gates in As a further comparison between the paired and randomly paired RNA-seq and scRNA-seq data, we examined genes that frequently appeared in the three gates, ≥20 times out of the 53 samples under consideration. For each gate in The percentage for frequently appearing genes in the upper-right gate was above 20%, indicating great consistency of highly expressed genes across the 53 samples from diverse biological context, which agreed with our observation that the upper-right gate was enriched for housekeeping genes required for diverse fundamental cellular processes. The percentage for frequently appearing genes in the upper-left gate was around 4%–5%, which was lower than the upper-right gate but much higher than the bottom gate, indicating that the pattern of under-detection in scRNA-seq is more consistent than over-detection in scRNA-seq.In addition to comparing the number or percentage of frequently appearing genes, we also examined whether those frequently appearing genes were the same between paired and randomly paired data. For each gate, we computed the average intersection-over-union ratio between the sets of frequently appearing genes in the paired and randomly paired analyses, averaging across the 100 iterations of random pairing. The average intersection-over-union ratios were 0.75 and 0.66 for the upper-left and upper-right gate, indicating high overlap for those two gates in the paired and randomly paired analyses. In contrast, the average intersection-over-union ratio for the bottom gate was only 0.17, showing that frequently appearing genes in the bottom gate were quite different between the paired and randomly paired analyses, which further indicated that the pattern of over-detection in scRNA-seq is weak.In this study, we analyzed paired bulk RNA-seq and scRNA-seq data from 53 samples from various biological contexts. Comparison between bulk RNA-seq and scRNA-seq data revealed genes that were consistently under-detected in scRNA-seq, and this result was robust to random pairing of the data. In addition, we observed that the frequently under-detected genes in scRNA-seq were significantly enriched by the poly(T) motif. In contrast, enrichment Tof poly(T) motif was not observed in genes consistently highly expressed in both technologies or genes that appeared to be over-detected in scRNA-seq. The motif-based observation led to our hypothesis that the poly(T) motif in genes may be able to form hairpin structures with the poly(A) tails of their mRNA transcripts, making it difficult for their mRNA transcripts to be captured during the capturing step of scRNA-seq library preparation, which is a mechanistic conjecture of why those genes may be more prone to be under-detected in scRNA-seq compared to bulk RNA-seq.The datasets analyzed in the study not only reflected a variety of biological contexts, but also contained technical variations in experimental and computational analyses. These technical variations include choices of alignment tools, choices of reference genome, library preparations and other experimental factors. All these factors could impact the data and subsequent analyes, including the results presented in our study. An ideal situation for our study is that all samples were processed using the same experimental protocol, the same reference genome, and the same alignment software with identical version. However, this is infeasible because almost all previous datasets involved some unique details in their experimental protocols. In addition, since raw FASTQ files were unavailable for many of the bulk RNA-seq and scRNA-seq samples in this study, we were unable to obtain the raw reads to run a standardized pre-processing pipeline to derived the gene expression data for all the samples. One experimental factor in scRNA-seq, the choice between 3′ vs. 5′ library preparation protocol, presents an interesting discussion, because our motif observations and mechanistic conjecture are both relevant to the 3′ end. Among the 8 datasets included in this study, 7 were generated using 3′ protocol. The remaining dataset contained a mix of 3′ and 5′ scRNA-seq data, but did not provide information on which samples were profiled by which protocols. Therefore, we did not distinguish 3′ vs. 5’ in our analysis. Since the above-mentioned factors were ignored in our analysis, we were effectively embracing the variations caused by those factors. Even with such variations in the data, we still observed a robust motif for the upper-left gate of genes under-detected in scRNA-seq. Therefore, these technical variations strengthened the robustness of our results.Although the poly(T) motif was significantly enriched among genes that were frequently under-detected in scRNA-seq, there were consistently under-detected genes that lacked this motif. For example, among the top 15 genes that were most frequently under-detected in scRNA-seq as shown in For the genes that are repeatedly under-detected in scRNA-seq, they are less likely to be considered as highly variable genes, and thus, are less likely to drive clustering or trajectory analysis results in downstream analyses. However, recognizing such genes is important. If the goal of a research project is to investigate a specific gene which happens to be more prone to be under-detected in scRNA-seq, scRNA-seq may be a less reliable experimental strategy compared to bulk RNA-seq. When developing imputation analysis of scRNA-seq data, more attention should be paid to genes with enriched poly(T) motif. As another future direction, the bulk vs. single-cell comparison in this study can be extended to other genomic data types, such as paired ATAC-seq and scATAC-seq data for a common set of samples.Expression data for 53 paired bulk RNA-seq and scRNA-seq samples were obtained from 8 published GEO datasets. The paired samples were from either the same individuals, the same cell lines, or the same tissue sources. The scRNA-seq for majority of the 53 samples were generated using 10X Chromium single-cell 3’ v2 or v3 protocol. Some of the samples in one GEO dataset (GSE176078) were processed using 10X Chromium single-cell 5’ protocol, but the identity of those samples was not available. A summary of the 8 GEO datasets and their references is available in For each GEO bulk RNA-seq dataset, median-of-ratios normalization was performed by DESeq2 which accounts for factors including sequencing depth and RNA composition. Next, log transformation was performed on the normalized data. Then the overlapping genes among bulk RNA-seq and scRNA-seq of each paired sample were identified for the 53 sample pairs from the GEO datasets, and a matrix representing the normalized bulk RNA-seq expression of all overlapping genes of the 53 paired samples was created. Finally, quantile normalization was performed on this matrix.For scRNA-seq samples, library size normalization was performed by Seurat followed by natural log transformation and calculation of average expression of each gene across all the cells to get pseudo-bulk RNA-seq data. Then, a matrix representing the single-cell based pseudo-bulk expression of the 53 samples was created.Pearson correlation and Spearman correlation were used to calculate the relationship between normalized bulk RNA-seq and pseudo-bulk RNA-seq expression profiles of the paired and randomly paired data.MEME Suite was used to find significant enriched sequence motifs of last 350bp of cDNA sequences of longest transcripts of candidate genes. For motif site distribution, any number of occurrences was selected for the analysis. MEME Suite reports E-value which serves as an indicator of the statistical significance of a motif. A motif with an E-value smaller than 0.05 is considered to be significant.DAVID  was used to identify the enriched KEGG pathways, and biological process, cellular component, and molecular function GO terms. FDR value was reported by DAVID for each significantly enriched pathway.