Advancements in genomic sequencing continually improve personalized medicine, and recent breakthroughs generate multimodal data on a cellular level. We introduce MOSCATO, a technique for selecting features across multimodal single-cell datasets that relate to clinical outcomes. We summarize the single-cell data using tensors and perform regularized tensor regression to return clinically-associated variable sets for each ‘omic’ type.Robustness was assessed over simulations based on available single-cell simulation methods, and applicability was assessed through an example using CITE-seq data to detect genes associated with leukemia. We find that MOSCATO performs favorably in selecting network features while also shown to be applicable to real multimodal single-cell data.MOSCATO is a useful analytical technique for supervised feature selection in multimodal single-cell data. The flexibility of our approach enables future extensions on distributional assumptions and covariate adjustments.The online version contains supplementary material available at (10.1186/s12864-022-08759-3). Classic bulk genetic sequencing involves averaging signature levels across all cells. Different sequencers may sequence different types of molecules such as ribonucleic acid (RNA), proteins, DNA methyl groups, etc. Disease progression, therapy success, and other clinical outcomes often vary among individuals suffering from complex diseases , 17, 35,Methods for identifying graphs and gene regulatory networks within a single molecular type has been well studied , 20, howmulti-modal (or multi-omic) single-cell sequencing. For example, CITE-seq simultaneously sequences both cell surface proteins and RNA on each cell of a sample [In bulk sequencing experiments, rare cells or smaller cell-types will be diluted due to the averaging across all cells within the sample. This motivated single-cell sequencing techniques where molecular information could then be sequenced on a cell-by-cell basis. While initial protocols were limited to RNA , 29, newa sample . Althouga sample . Figure Many bulk sequencing problems utilize matrix decomposition techniques for various analytical goals where the dimensions correspond to different feature sets . However, single-cell sequencing essentially creates an added dimension for cells that did not previously exist in bulk sequencing. Tensors provide a general framework for organizing high dimensional data, and a matrix is a special case of a two dimensional tensor. Therefore, leveraging tensors for single-cell sequencing is an interesting approach to accommodate the added cellular dimension. For example, ScLRTC and scLRTD utilize tensor decompositions to impute dropouts or missing data found in single-cell sequencing data , 28. scTnetwork, within multimodal single-cell data that relates to a disease or phenotypic outcome. Current single-cell methods are often either limited to one subject/experiment, limited to just RNA, unsupervised with no clinical outcome associations, or intended for cell clustering and not feature selection. MOSCATO uses regularized tensor regression to address each of those common limitations found in existing methods. “This manuscript proposes a novel method, Multi-Omic Single-Cell Analysis using TensOr regression (MOSCATO), for identifying the superstructure of a semi-directed graph, or MOSCATO utilizes regularized tensor regression, and this section describes existing and relevant tensor concepts. “d contains pd variables for d=1,...,D. For example, D=1 denotes a vector and D=2 denotes a matrix. Many mathematical operations for tensors build on mathematical operations used in matrices. For example, Definition outer products between D vectors to create a D-dimensional tensor, where ∘ denotes the Khatri-Rao product.High dimensional data may be organized into a tensor, and a matrix may be thought of as a 2-dimensional tensor. Utilizing familiar tensor notation as provided by Kolda and Bader , we let mentclass2pt{minimb1,b2,...,bD, denote vectors where outer product of those vectors, b1∘b2∘⋯∘bD, creates a D-dimensional tensor of size p1×p2×⋯×pD, and each th element equals Let vectorizing or mode-d matricizing the tensor. Definitions 2 and 3 describe these reorganization techniques.It may also be convenient to reorganize a tensor into a lower dimensional space by vec operator i1,...,iD)th value in Let mode-d matricization operator id,j)th element within i1,...,iD)th value within Let inner products, as described in Definition 4.Similarly as done in matrix operations, it may be of interest to multiply two tensors with comparable dimensions via inner product may be obtained by Suppose two tensors dth dimension of a tensor through d-mode products as described in Definition 5.Furthermore, it may be of interest to multiply a matrix along the d-mode product between U may be expressed as i1,...,id−1,j,id+1,...,iD)th value equals Suppose a tensor The rank of a matrix denotes the maximum number of linearly independent rows/columns in the matrix. Building on those concepts, the rank of a tensor may be thought of as the maximum number of vectors that can be multiplied and added to replicate the tensor, as shown in Definition 6.R if no smaller R exists such that Assume a D-dimensional tensor Zd matrices for d=1,...,D.where Definition 7 defines d=1,...,D. Furthermore, let rth column of Zd. Then Let The true rank of a tensor may often be difficult to determine due to the high dimensionality, motivating decomposition techniques that estimate vectors for a given rank that approximate the tensor, as shown in Definition 8.rank-R CP decomposition aims to use R vector sets (one vector per dimension) to approximate Assume a D-dimensional tensor where Kolda and Bader present additional details on decomposition and other tensor operations .y for each subject that follows some exponential family with link function g(·) and mean μ. Classic GLM uses univariate independent variables to predict the outcome, but tensor regression extends those concepts by additionally allowing a predictor tensor. This is accomplished by multiplying the dimensions of the tensor through coefficient vectors that convert the dimensions to a univariate value that may then predict the outcome.Building on notation covered in “D=2 dimensions for the predictor tensor. Each dimension in the predictor tensor corresponds to a different feature set, and each subject will contain its own D=2 predictor tensor to be used to predict their univariate outcome y.Figure Imaging modalities such as magnetic resonance imaging (MRI) present an excellent application for tensor regression models. For example, a particular brain MRI slice may be of interest for an outcome, say, disease status. This image slice can be expressed in an array structure  where the value at a given pixel denotes the MRI signal associated with that location. Referring to the model shown in Fig. Tensor regression may also involve higher rank problems with the more formal representation U contains the univariate independent variables . A rank-R tensor regression estimates R coefficient vectors for each dimension in the predictor tensor, but for simplicity, in this manuscript we will assume rank-1. The Block Relaxation Algorithm is used to estimate the coefficient vectors with additional details described by Zhou et al. [where u et al. . Zhou etu et al.  also claτ, we could derive an equally optimal model If one naively vectorized the predictor tensor and fit a classic GLM model, it would require estimating u et al.  describep-values or whether the AUC was less than 0.3 or greater than 0.7.MOSCATO was applied to both simulated data and real data. Results from the simulations are presented in “Two ‘omic’ types denoted as maxG and maxX. In a perfect execution of MOSCATO, maxG would be tuned to 10 and maxX would be tuned to 15 due to known network sizes in the simulated data. All simulations tuned maxG and maxX to values greater than the true number of network features, regardless of the simulation setting. For maxX, smaller values were tuned as technical noise decreased and number of cells increased, but this trend did not persist when tuning maxG.Figure p-values, and AUC using cutoffs). Sensitivity measures the probability that network features are properly included in the selections, and specificity measures the probability that non-network features are properly excluded from the selections. As shown in Fig. p-values as the number of cells increases and the technical noise improves. AUC based on p-values not only produced counterintuitive results where the performance actually degraded as the technical noise reduced, it also selected too many features such that the results were not remotely sparse. This explains that while the sensitivity remained high for all simulations, this is simply due to the fact that nearly all features were selected using that criteria. AUC selections based on cutoffs resulted in opposite issues where it did not select nearly any features and produced poor sensitivity with nearly perfect specificity.Figure MOSCATO reproduced the superstructure of the graph  reasonably well with generally high sensitivity and also limited false positives present. This is especially true when comparing against approaches using the AUC. However, when it is expected that high levels of technical noise are present with limited cells per subject, caution should be used when considering MOSCATO.Leukemia is a broad disease that encompasses all cancers that occur in blood cells. The 5-year survival rate is about 65% according to data from 2011 to 2017, and about 459,000 people were living with leukemia in 2018 in the United States . LeukemiTo assess MOSCATO in practice, we applied it to real single-cell data with multiple data types. Limited data is currently available due to the infancy of multimodal single-cell sequencing, so data across multiple studies that all used CITE-seq protocols  on bone The data was pre-processed and integrated using Seurat version 4.0.3 , and FigIn this application after pre-processing, each subject contained 8 scRNA-seq matrices (one for each cell cluster) where each matrix contained 17991 columns and rows corresponding to the cells within that cluster. In addition to the scRNA-seq datasets, each subject contained another 8 matrices (one for each cell cluster) for their single-cell ADT abundance where each matrix contained 5 columns and rows corresponding to the cells within that cluster (same cells as in the scRNA-seq datasets). Finally, each subject had a binary disease status . The feature selection techniques were applied to the 8 pairs of matrices separately , resulting in 8 different MOSCATO and AUC selection results. The first step of MOSCATO estimates the correlation between each subject’s scRNA-seq and single-cell ADT features, resulting 21  correlation matrices, each of size 17991×5.p-values, and AUC selections based on the AUC cutoffs for each of the 8 cell clusters are provided in Additional file p-values resulted in nonsparse feature sets. DAVID [p-values resulted in RNA selections well over this 3000 restriction for most cell clusters, we only focused on gene clusters from the MOSCATO and AUC cutoff selections.Analysis was performed on each cell cluster separately, and the selections were later reviewed for biological relevancy. The rest of this section will focus on the results obtained from cell cluster 0, but the complete results from MOSCATO, AUC selections based on s. DAVID , 31 was p-values, the AUC selections based on p-values selected nearly half of all RNA features considered. AUC selections based on cutoffs selected 11 RNA features and 0 ADTs for cell cluster 0, and DAVID was not able to discern any gene clusters based on the genes selected.MOSCATO selected 96 RNA features within cell cluster 0, and DAVID identified 2 gene clusters. The strongest gene cluster (based on highest enrichment score) used 7 of these 96 genes. This was the most notable gene cluster which included the genes CD3D, CD3E, and CD3G which are part of the KEGG pathway for Human T-cell Leukemia Virus type 1 (HTLV-I) infection (KEGG pathway hsa05166), and HTLV-I infections are a known risk factor for developing adult T-cell leukemia/lymphoma . Additiop-value selections did not produce sparse solutions. Also, AUC selections were based directly on normalized expression/sequenced values, but MOSCATO performs selections based on the similarities between data types. This possibly helps reduce batch effects found in individual subjects by standardizing each value between -1 and 1.In conclusion, the selections made by MOSCATO under cell cluster 0 resulted in a concise gene cluster discovered by previously known annotations and functionalities. These functionalities not only related to the disease of interest , it also related to cell surface functionalities. This highlights that MOSCATO not only considers supervised information , it also considers the relationships across data types . Performing feature selection using solely AUC not only neglects the between data type relationships, it also was not able to return a concise set of genes that related to leukemia. Furthermore, it is arbitrary to select pre-specified AUC cutoffs for selections, and the MOSCATO was performed on both simulated and real data. The simulations produced fairly accurate results with a sensitivity and specificity close to 1 for many of the simulations, although MOSCATO did not perform as well in situations with high technical noise or low cell counts per subject. Since MOSCATO calculates its predictor tensor to be the estimated correlation of the datasets per individual, it is unsurprising that cell count would contribute to more accurate correlation estimates. Although not used in this manuscript, covariate adjustments could easily be made in MOSCATO by simply adding them to the tensor regression.The real data application involved 21 subjects, and given the highly variable nature of single-cell sequencing data, the features selected should be interpreted with some caution. Furthermore, since the data was collected across different studies, only 5 proteins were used that merged across all 21 subjects. As single cell technologies become more available and cost feasible, future experiments should be considered with more consistent proteins across experiments and larger sets of subjects.MOSCATO currently assumes all cells come from the same cell type, but it might be more interesting to accommodate situations in which multiple cell types are present. We are currently working on higher dimensional applications of MOSCATO in a future manuscript. A reasonable solution could incorporate another step which estimates a similarity matrix for each cell type and includes another dimension to the predictor tensor for cell type. Additionally, MOSCATO was only tested on experiments with two data types, but extensions should be considered in situations where more than two data types are present. This could be accomplished by extending the predictor tensor to accommodate a dimension per data type extracted, although higher dimensional summary measures would need to be considered.MOSCATO was only tested using Pearson’s correlation as the summary measure to construct the predictor tensor al lasso  is a popThis study only used rank-1 tensors, but higher ranks could be considered that may unveil other patterns and networks available in the data. The proper rank could be obtained using typical model selection criteria such as cross validation, although interpreting the results may not be as straight forward.Zhou et al. discuss hypothesis testing via asymptotic normality results , and thebalance. This concept has been considered in bulk level analyses with a single ‘omic’ type [Although MOSCATO returns the superstructure for a graph, it does not provide information on directionality and does not currently consider directional consistency. For example, suppose two genes are positively correlated with each other, but they contain conflicting correlative directions with the outcome. This inconsistency in directionality makes interpreting the results more difficult, and may be mitigated by additionally tuning based on optimizing ic’ type , and it MOSCATO is a statistical technique for analyzing multimodal single-cell data where the study goals are to identify which features within the ‘omic’ datasets relate to each other and a clinical outcome. MOSCATO was found to perform favorably through a series of simulations and a real data application. Multimodal single-cell data continues to grow in popularity, and feature selection techniques such as MOSCATO may be critical to fully leverage the potential in using the data for highlighting biological markers in complex diseases.n subjects with two data types, bulk sequencing studies would contain two data sets , i with two data types, their single-cell data would contain two datasets mi denotes the number of cells for subject i. Since the number of cells typically differs across subjects , we organize each subject’s data into separate datasets  method [ϕ is met. Instability is estimated based on subsamples from the data by performing the feature selection under each subsample and summarizing the consistency in results across different subsamples. Although ϕ may initially be thought of as an arbitrary cutoff between 0 and 1, it may be easily interpreted as the amount of allowable instability. In essence, a smaller ϕ would imply a more sparse but stable result. The motivation behind allowing some instability as opposed to fixing ϕ to 0 is to allow some noise to be selected in order to ensure that no true signal is missed in the final feature selection. In statistical terms, this means that StARS prioritizes reducing type II errors.MOSCATO assumes fixed values for ) method . Tuning ) method . The mosαX,αG,maxX, and maxG will be tuned using similar logic. Focusing on tuning one dimension at a time, we initialize to a sparse solution with some small maxX. Fixing maxX, we estimate the instability for a range of αX values between 0 and 1. Select the αX value resulting in the lowest instability, and if that instability is less than ϕ, increase maxX and repeat the process. This continues until the ϕ instability is hit to select maxX and αX. Using the highest maxX and corresponding optimal αX with instability less than ϕ, a similar process is then repeated for tuning maxG and αG. In this case with two dimensions, one for αX and maxX for some fixed αG and maxG, and then use αG and maxG. The initial fixed αG and maxG may be kept large, suppose αG=0.5 and maxG=floor(p/2) such that an overly sparse Although the StARS method was developed for tuning a single sparsity parameter, the four hyperparameters Splatter  is a popTo accomplish this, we leverage latent structures for multimodal supervised networks detailed in Zhang et al. . Figure mentclass2pt{minimS, H, X, G, G′,X′, and y) for each subject using a multivariate normal distribution with zero mean and the latent variables’ covariance matrix. Since the latent values were simulated from a multivariate normal, they will each be normally distributed with mean 0. The Splatter simulation assumes the initial gene mean comes from a gamma distribution, so we take the square of the latent value divided by its standard deviation. By doing so, the transformed latent values then become gamma distributed with shape equal to 1/2 and scale equal to 2 times its variance. These transformed latent values will be used as the initial gene means for each subset of features, and the latent value for y will be used as the mean to simulate an observed outcome from a normal distribution. Initial gene means for the noise subset of features are randomly simulated independently. Additional theoretical details may be found in the Additional file To extend Splatter for supervised multimodal networks, we will first simulate the latent values  and amount of technical noise . Simulations were replicated 50 times under each simulation setting and each simulation had 100 subjects. gridα={0.2,0.5,0.7},ϕ=0.02,R=50, and c=50. ϕ for its initialization. Ideally, MOSCATO would tune maxG to 10 and maxX to 15 in order to select the proper network size according to Table For tuning the hyperparameters, we set mentclass2pt{minimp-value was less than the nominal significance level (set to 0.05) under the null hypothesis that the AUC equals 0.5. Additionally, selection criteria using cutoff values where features with an AUC either less than 0.3 or greater than 0.7 were selected. Since AUC requires categorical outcomes, we use the median of the outcome to binarize it .In addition to applying MOSCATO, we also applied competing methods using the AUC. Seurat provides a popular single-cell sequencing workflow, and following similar methods used by the authors of Seurat , this AUData was combined across multiple studies for healthy and leukemia subjects. The studies used to obtain the data are summarized in Table Seurat version 4.0.3  was usedgridα={0.01,0.05,0.1}. Since Seurat clusters cells by maximizing correlation between features, the multicollinearity across features would consequently be high and require more weight on the L2-norm .We applied MOSCATO to each of these cell clusters separately, each with ϕ to 0.05 for most applications [ϕ was set to 0.001.Due to the modest sample size, we tuned the hyperparameters using a subsampling size of 20  to estimate stability based on a “leave one out” scheme. Although StARS suggests setting the instability threshold ications , in thisp-value was less than 0.05 under the null hypothesis that the AUC equals 0.5 or whether the AUC was less than 0.3 or greater than 0.7. Since the p-value would likely be small in situations with many cells , both a p-value approach and an approach based on the AUC values were considered.To compare selections with another method, the MOSCATO results were compared to selections based on the AUC. The AUC approach was done using the presto version 1.0.0 R package . Similarhttps://github.com/lorinmil/MOSCATOLeukemiaExample.The real data application may be reproduced by following the steps provided at Additional file 1 Additional simulation and application details.Additional file 2 Gene and protein selections.


Transcriptome profiling of blood cells is an efficient tool to study the gene expression signatures of rheumatic diseases. This study aims to improve the early diagnosis of pediatric rheumatic diseases by investigating patients’ blood gene expression and applying machine learning on the transcriptome data to develop predictive models.RNA sequencing was performed on whole blood collected from children with rheumatic diseases. Random Forest classification models were developed based on the transcriptome data of 48 rheumatic patients, 46 children with viral infection, and 35 controls to classify different disease groups. The performance of these classifiers was evaluated by leave-one-out cross-validation. Analyses of differentially expressed genes (DEG), gene ontology (GO), and interferon-stimulated gene (ISG) score were also conducted.Our first classifier could differentiate pediatric rheumatic patients from controls and infection cases with high area-under-the-curve (AUC) values . Three other classifiers could distinguish chronic recurrent multifocal osteomyelitis (CRMO), juvenile idiopathic arthritis (JIA), and interferonopathies (IFN) from control and infection cases with AUC ≥ 0.8. DEG and GO analyses reveal that the pathophysiology of CRMO, IFN, and JIA involves innate immune responses including myeloid leukocyte and granulocyte activation, neutrophil activation and degranulation. IFN is specifically mediated by antibacterial and antifungal defense responses, CRMO by cellular response to cytokine, and JIA by cellular response to chemical stimulus. IFN patients particularly had the highest mean ISG score among all disease groups.Our data show that blood transcriptomics combined with machine learning is a promising diagnostic tool for pediatric rheumatic diseases and may assist physicians in making data-driven and patient-specific decisions in clinical practice.The online version contains supplementary material available at 10.1186/s12969-022-00747-x. Pediatric rheumatic diseases encompass a spectrum of autoimmune and autoinflammatory diseases that can affect the joints, muscles, bones, and other organs in children under the age of 16–18 years. Although many pediatric rheumatic diseases typically present with joint manifestations, other organs, including the eyes, skin, muscles, and gastrointestinal tract, may also be affected. Children who present with rheumatic symptoms often pose several challenges to their physicians. First, typical symptoms such as fever, rash, redness, pain and/or swelling at joints are common in many rheumatic diseases: (i) a transient/self-limiting process such as (reactive) infectious arthritis, (ii) a relapsing process like auto-inflammatory diseases (AID), (iii) a chronic condition like vasculitis, chronic recurrent multifocal osteomyelitis (CRMO), or juvenile idiopathic arthritis (JIA), (iv) an interferonopathy (IFN) such as dermatomyositis or systemic lupus erythematosus (SLE) characterized by dysregulation of type I interferon, or (v) diseases related to the human leukocyte antigen B51 serotype (HLA-B51). Second, once a disease has been confirmed, it is still difficult to make a rapid and definitive classification amongst the different subtypes within the disease due to its rarity and clinical presentation variability. This has challenged physicians’ efforts in making specific diagnoses and assigning proper treatments.Transcriptome profiling of blood cells has proven to be useful and efficacious in identifying gene expression signatures in rheumatic diseases , 2, enabBy investigating the whole blood gene expression of children with rheumatic diseases in comparison with reactive/post-infection controls, we aim to develop computational classifiers based on the obtained transcriptome data that allow us to identify pediatric patients with rheumatic diseases and distinguish different rheumatic groups , and thus, ultimately, to improve the diagnosis of future patients.After obtaining written consent, 48 children (1 to 16 years old) with rheumatic diseases  were recruited, prior to any treatment except non-steroidal anti-inflammatory drugs (NSAIDs), from May 2016 until August 2020, at the Divisions of Pediatric Rheumatology of four hospitals in Belgium . Venous blood was collected into PAXgene® Blood RNA tubes . Clinical details of the patients can be found in Table S1. As controls, 46 children with PCR-confirmed viral (mainly enterovirus) infections were also recruited and requested to provide blood twice: first while being actively infected, and second following remission . Only 35PAXgene® Blood RNA tubes were kept at -80 °C within 72 h after blood collection until use. RNA extraction was performed via a column-based RNA extraction using the PAXgene® Blood RNA extraction kit . To optimize RNA concentrations, we used the RNA Clean & Concentrator™-5 kit . We verified the RNA quality using the RNA ScreenTape Analysis on the Tapestation .All RNA samples were prepared with the QuantSeq3′ mRNA-Seq Library Prep Kit for Illumina  following the standard supplier’s protocol for long fragments. During the RNA removal step, we also added globin blockers, so none of the abundant globin mRNA was copied to double stranded cDNA. The resulting amplified cDNA libraries were equimolarly pooled and sequenced on NextSeq 500   with up to 40 samples per batch. This gave us an optimum of 10 million reads for each sample. All samples were prepared and sequenced in 4 batches.Raw data from the NextSeq was demultiplexed and further processed per batch through an in-house pipeline. The quality of all reads was evaluated using FastQC (v0.11.9) before and after processing with Trimmomatic (v0.36). Trimmomatic removed the leading 20 bases from reads, ensure a minimum quality score of 15 over a sliding window of 4 bases and required a minimum read length of 30 bases. As usage of oligodT primers might cause poly-A stretches at the 3′ end, the latter end was trimmed with our own in-house poly-A removal script. All sequences that remained after trimming were mapped against the human reference genome build 38 (polymorph variants excluded) with HISAT2 (v2.0.4). HTseq (v0.6.1) was used to count all reads for each gene and generate a readcount table.Clustering is an unsupervised learning algorithm that is used to discover patterns in high-dimensional data that would not be easily identified by conventional statistics. The readcount table obtained after raw data processing was normalized by median scaling. Then, the t-distributed stochastic neighbor embedding (t-SNE) method and hierarchical clustering algorithms were applied on the normalized readcount data to assign data into different clusters based on the (dis)similarity in the gene expression between samples.Preliminary filtering of the normalized readcount data was performed by removing genes with fewer than 10 readcounts over all samples. Differential gene expression analyses were performed using the DESeq2 Bioconductor package in the open-source statistical software R. To account for batch effects, the batch number was included in the DESeq2 design. Differentially expressed genes with log2 fold change ≥ 2 (either up- or down-regulated) and p-value < 0.01  were passed on to gene ontology enrichment analysis using the topGO package in R. Fisher’s exact test was performed to determine significantly enriched/depleted gene ontology terms relating to biological processes.Due to the larger number of genes in the dataset, a feature selection step was performed using the Boruta package in R  with a BP < 0.05; ** P < 0.005.First, relative expression (RE) was calculated based on the readcounts of interferon-stimulated genes (ISG) and the housekeeping gene GAPDH: able S4) . StatistWe compared the transcriptome profiles of six rheumatic disease groups  with viral infection and convalescent controls. Clustering analyses using t-SNE and hierarchical algorithms were displayed in Fig. The Random Forest algorithm was used for classifier development because it uses the ensemble learning technique that is robust to outliers, stable with new data, and can handle non-linear correlations. The first classifier was developed to distinguish between control, infection, and pediatric rheumatic cases based on normalized transcriptome data. Leave-one-out cross-validation results in Fig. More specific classifiers were then developed per disease group. As the number of rheumatic patients in our dataset was limited, these classifiers focused only on CRMO, IFN, and JIA groups, which had more subjects for model training and validation than the other disease groups. Three classifiers were developed to distinguish patients with CRMO (n = 6), IFN (n = 6), and JIA n = 20) from control (n = 35) and infection (n = 46) cases. They worked quite well as their AUC values are above or equal to 0.8  of CRMO, IFN, and JIA versus controls. The resulting DEGs were translated to corresponding Gene Ontology (GO) categories to understand which pathways were involved in the disease pathophysiology. Many of the top 10 GO categories of CRMO, IFN, and JIA groups are related to innate immunity including myeloid leukocyte and granulocyte activation, neutrophil activation and degranulation  to guide molecular diagnostics and to formulate targeted therapy approaches . QuantitDespite demonstrating that blood RNA sequencing can distinguish autoimmune/autoinflammatory diseases from viral infection/post-infection cases, as well as reveal their key genes and pathways, our study is limited by the sample size of each rheumatic disease, due to which cross-validation was done to validate the classifiers’ performance instead of training and testing on independent datasets. The limited number of rheumatic cases may also be responsible for the modest cross-validation performance of the CRMO-IFN-JIA classifier although rheumatic diseases in general could be identified well from controls and infections. Another limitation that challenges the predictive performance of our classifiers is disease heterogeneity. IFN and JIA are examples of heterogeneous groups that pose great difficulties for the classifiers to differentiate from other groups because they contain several disease subgroups . The heterogeneity of diseases often obstructs explicit modelling of underlying distributions of individual features, which can be even more problematic when the sample population is small . FinallyTo incorporate the expression-based assessment into clinical use, it is necessary to demonstrate that the reliability and accuracy of this approach is comparable or superior to the currently used methods. The development of diagnostic criteria is challenging in rheumatology due to the heterogeneity of many rheumatic diseases, variable clinical presentations, and complex pathophysiology. Given the lack of optimal diagnostic criteria, physicians must rely on a complicated decision-making process based on a combination of symptoms, physical examination, exclusion of competing diagnoses, and geographic prevalence . MoreoveOverall, our study indicates that blood transcriptomics is a promising tool to improve the diagnosis of pediatric rheumatic diseases. The ease of sample collection as well as the continuous enhancement and affordability of sequencing techniques can overcome the challenges of patient heterogeneity and allow for further fruitful research.Below is the link to the electronic supplementary material.Supplementary Material 1Supplementary Material 2Supplementary Material 3Supplementary Material 4Supplementary Material 5Supplementary Material 6


This software automates the validated multiclass classifier applicable to single analyte tests and multiplexing assays. The g3mclass achieves automation using the original semi-constrained expectation–maximization (EM) algorithm that allows inference from the test, control, and query data that human experts cannot interpret. In this study, we used real-world clinical data and gene expression datasets  to provide examples of how g3mclass may help overcome the problems of over-/underdiagnosis and equivocal results in diagnostic tests for breast cancer. We showed the g3mclass output’s accuracy, robustness, scalability, and interpretability. The user-friendly interface and free dissemination of this multi-platform software aim to ease its use by research laboratories, biomedical pharma, companion diagnostic developers, and healthcare regulators. Furthermore, the g3mclass automatic extracting information through probabilistic modeling is adaptable for blending with machine learning and artificial intelligence.The analytes qualified as biomarkers are potent tools to diagnose various diseases, monitor therapy responses, and design therapeutic interventions. The early assessment of the diverseness of human disease is essential for the speedy and cost-efficient implementation of personalized medicine. We developed  However, molecularly targeted therapies and immunotherapies are expected to benefit only a subset of treated patients. Identifying individuals likely to benefit from the targeted treatments and monitoring resistance to new cancer therapies depends heavily on biomarkers. This broad category of analytes, including biochemical, genomic, and proteomic measurements, may indicate an underlying disease mechanism or predict a response to a drug2.Targeted therapy is a crucial focus of drug development and a cornerstone of precision medicine4. To discover biomarkers, researchers often start by comparing references  and tests (a sick population), assuming the normal distribution of measurements in each group. However, this assumption is frequently not tenable as the real-world readouts of laboratory tests rarely fit into one normal (Gaussian) distribution6. The best-known example of such biomarker in oncology is a human epidermal growth factor receptor 2 (HER2) overexpressed in a fraction of breast (15–30%), gastric and gastroesophageal (10–30%), ovarian (20–25%), endometrial (14–80%), bladder (23–80%), and lung (up to 20%) cancers8. Furthermore, two distributions of continuous values from test and reference are rarely fully separated9. The overlapping distributions create a methodological dilemma for choosing a diagnostic cutoff value that impacts the diagnostic accuracy and clinical decisions10. Although an arbitrary cutoff between the positive and negative results remains a routine clinical practice, there is uncertainty (and hence concern) that dichotomization on biomarkers reflects the continuous clinical risk from a measured test value11. Overdiagnosis may cause unnecessary morbidity and cost, whereas underdiagnosis may increase the risk of disease progression. Innovative statistical approaches that can address these deficiencies are needed.With the rapid adoption of high-throughput assay technologies, discovering “promising/potential/candidate” biomarkers and therapeutic targets has exploded, producing an enormous volume of raw data needing human and robotic intelligence assessment13. Gaussian Mixture Model (GMM) is the flexible probabilistic approach that can characterize multimodally distributed continuous variables14. Each class is modeled according to a different Gaussian distribution. In addition, Bayesian classifiers can be used to predict class membership probabilities, such as the probability that a given value belongs to a particular class. The diagnostic applications of Bayesian analysis in medicine have been evolving over the past few decades17. We recently introduced the method for multiclass cancer classification on biomarkers by applying a Bayesian approach19. The proposed method allows the discovery of previously unknown groups with different levels of biomarkers by extracting information from molecular assay data using probabilistic modeling. The performance of this method has been validated over datasets of more than 300 clinical samples. In addition, it has been shown to improve the binary classification of clinical markers (HER2 and steroid hormone receptors).The probability approach is a statistical concept that measures the likelihood of something happening. It has been exploited successfully to analyze the underlying complex structure of text documents, image objects, and voice signals in many subject areas, including biology and medicineg3mclass, a practical Gaussian mixture modeling software for molecular assay data classification. It is intended for tests where the random variation of the parameter-of-interest is an essential component of the modeled situation. In this article, we introduce and employ g3mclass using a real-world example of human breast tissues. Samples were classified on clinical markers to help distinguish patients most likely to benefit from the Food and Drug Administration (FDA)-approved targeted therapies while sparing others that need different treatment.Given these successes and the need to improve disease diagnosis and personalized treatment options, we developed The g3mclass is a probabilistic modeling-based classification and visualization software purpose-built for analyzing laboratory assay data. Our development of g3mclass was motivated by the outstanding problem of classifying test results in the research laboratories and clinical settings, considering the prior knowledge of the reference. In this regard, Bayesian statistical methods to update pre-existing information about the likelihood of the change provided a robust system for our development of the g3mclass. The g3mclass core function requires two kinds of data entries for each analyte: test  and reference . Additionally, we built a complementary capability in this software to classify new data  obtained by the same assay but from an independent source. These incoming unknown data are queries. Thus, the g3mclass workflow includes data preparation and input, probabilistic modeling, automated classification of the test, reference, and optional queries, followed by the analysis and archiving of the results  algorithm20. To detect test classes distinct from the reference, we constrained the position , spread , but not the weight of the test class 0 to be equal to the corresponding reference values. All other test classes have all parameters adjustable. While our semi-constrained approach preserves parameters of the reference class, it does not affect the convergence and high speed of the analysis regardless of the overlap between the reference and the test values. For example, using a laptop computer, the g3mclass runtime for classifying ten analytes with thousands of measurements takes less than 20 s. In the test classes labeled with negative  and positive  integers, the mean values are lower and higher than class 0. The greater the absolute integer value, the further the class is positioned relative to class 0, as illustrated in a plot created by the g3mclass software  that measures the model's fit to the test data and outputs the test model parameters. The classification of references and queries relies on the selected GMM for the test. Individual models could be optimized for each analyte across the different study populations. The g3mclass additional functionalities include the output of spreadsheets with summary statistics  and classification heatmaps.The g3mclass performs consecutively three classification types—proba, cutoff, and stringent cutoff (a.k.a. s.cutoff) for a given test model to maximize the automated classification accuracy. The g3mclass proba utilizes the Bayesian approach to classify data. It classifies each value based on a maximum a-posteriori probability estimate of class membership. There is no a priori assumption about the number of classes. However, during this initial step, some analyte values in the test may be incorrectly assigned by the proba classification. This situation may occur when a component of GMM has a wide dispersion with its tails picking up values that otherwise belong to a different class . During s.cutoff classification, more values are assigned to class 0 by the expansion of cutoff intervals. If the weight of class 0 in the test GMM is close to null, the proba classification of reference is invalid  up to 100 times. Resampling is done randomly without replacement to limit the risk of creating false classes based on repeated events from far-tailed distributions. The independent GMM is learned for each resample, and parameter estimates are provided. If, for example, a user selects the option of variable bin number, the software will apply it to each resample. The optimal number of classes for each resample will be based on BIC and may or may not differ from sample to sample. The user may compare the variability of the estimates of the biomarker classifier parameters: the number of classes, the mean values, and the diagnostic cutoffs separating the reference-like values from the disease-related test values . Hence the user may base the judgment not only on the original model but also on the potential outcomes resulting from resampling. One way to conceptualize the utility of g3mclass in the field of biomarkers is to consider its value in updating the existing knowledge about biomarkers in reference samples  and assessing the biomarker change in the test .The g3mclass to automatically classify samples on the gene expression data for the established diagnostic/drug-response biomarker, the ERBB2 encoding HER2. The mRNA levels were measured by a QuantiGene Plex 2.0 (QG2) assay in the intended-use population . Among the models learned by g3mclass on varying histograms, the software picked GMM with the lowest BIC value = 1650.6 and the bin number k = 10, as illustrated in Fig. g3mclass-selected model, the ERBB2 mRNA test values fit into the 4-class GMM. Class 0 is like a reference ; classes 1, 2, and 3 have increased mean values of ERBB2 mRNA compared to a reference. The software learns new models by selecting a higher fixed number of bins in a histogram. For example, with k = 20 and BIC = 1660.8, the ERBB2 mRNA test values fit into 5 class-GMM. Class 0 is like reference (as above); classes 1, 2, 3, and 4 have the increased mean value relative to the reference. This model allows a more detailed classification of ERBB2 mRNA's test values and may be preferred in some clinical applications. With the increased number of bins, k = 30 and BIC = 1672.0, the ERBB2 mRNA test values fit into 8-class GMM. This model provides an even more detailed classification; however, the model overfitting may occur, and reference may be represented by two classes − 1 and 0.First, we demonstrated the capabilities of ERBB2 mRNA  and HER2 negative (HER2−) IBC based on the pathology reports. To test how well the measurements of ERBB2 mRNA predict HER2+, we dichotomized the ERBB2 mRNA data into groups using g3mclass-calculated cutoffs for three models. Choosing a model-predicted cutoff value, i.e., 186.6 (4-class GMM), 147.0 (5-class GMM), and 113.9 (8-class GMM), allows the transformation of multiclass into binary classification, at the same time eliminating equivocal results  and specificity (the fraction of HER2− cancers that are correctly identified by ERBB2 mRNA assay as not being HER2+). Considering the prevalence of HER2+ human breast cancer 20%, the estimated positive predictive values were 96%, 90%, 79%, and negative predictive values were 93%, 97%, 98%. The accuracy of the ERBB2 mRNA test was 93%, 96%, 94% in 4-class, 5-class, and 8-class GMM, respectively.Going a step further, we examined how well a HER2 status may be predicted from g3mclass modeling solutions, we calculated the ERBB2 mRNA diagnostic test parameters for query—an independent cohort of patients diagnosed with DCIS , the positive predictive value was 100%, while the negative predictive value was 91.53% . Thus, the results of g3mclass data analyses show the accuracy of identifying breast cancer potentially sensitive to anti-HER2 therapy and the robustness of the software in making the correct classification without equivocal results in the test and independent query.Additionally, using the g3mclass can easily be upscaled to analyze data from multiplexing assays. The data processing steps are the same as those for a single biomarker, i.e., preparation and entry of input data, modeling, classification, and analysis of the output results. We ran g3mclass to concurrently classify breast tissue samples on ERBB2, ESR1, and PGR mRNA measurements obtained in the validated and highly reliable multiplex QG2 assay  from pathology reports  for each model. Finally, it summarized data into spreadsheets and heatmaps. Using the resampling feature of the g3mclass, we found that diagnostic cutoff estimates for ESR1 were stable despite the appearance/disappearance of far-tailed classes  and with physiological levels of the ESR1 transcript (class 0) based on a 5-class test GMM for ESR1 mRNA. In cancer, three other groups emerged with either slightly (class 1), moderately (class 2), or highly increased (class 3) levels of ESR1 mRNA  were found in 9% of IBC, 13% of DCIS, and 0% of noncancer. In sharp contrast, low/undetectable levels of PGR mRNA (class − 1) were in 48% of IBC, 25% of DCIS, and 21% of reference. Thus, the g3mclass revealed low/or loss of PGR mRNA expression in IBC and the upregulation of PGR mRNA in DCIS in our study populations. Finally, we queried an independent set of mRNA data from the human breast cancer cell lines with the know expression levels of HER2, ER/PR24 and found them classified according to the established status  and unknown  information. This computational approach is the opposite of supervised classifications requiring the predefined knowledge of the number of the mixture components. For example, a two-component mixture model sorts differentially expressed genes in microarray experiments12. The principal innovation of g3mclass is embedding pre-existing experts’ knowledge of reference parameters into the test GMM. As a result, it substantially improves the differentiation of new-to-test versus reference-like values and provides biological and clinical context for interpretation outcomes. This approach defines the significant difference of g3mclass from other powerful software packages handling Gaussian finite mixture modeling as their clustering capabilities, including the most popular R package mclust14 and the Addinsoft XLSTAT (https://www.xlstat.com/en/company). Overall, there are two critical applications of the g3mclass in the biomedical field. First, it enables the discovery of previously unknown groups with different levels of biomarkers, including those that are not part of the reference and thus are more likely linked with disease. Second, it allows individual patient classification in line with personalized clinical decision-making.In the present article, we demonstrated how https://www.who.int/news-room/fact-sheets/detail/breast-cancer). Experts recommend that every primary IBC be tested for the presence of HER2 and ER and re-tested in subsequent recurrences and metastases by semi-quantitative immunohistochemistry (IHC) and/or fluorescence in situ hybridization (FISH)28. These tests may produce equivocal results that could not be interpreted as positive or negative. The challenge remains to define either at the protein29 or the mRNA33 the HER2/ER expression cutoffs that segregate patients who may derive meaningful clinical benefit from endocrine and targeted therapies from those who will not. We have previously demonstrated that dichotomization of structurally mixed mRNA data with a single cutoff, e.g., using a ROC model or two-component GMM, may result in the loss of reliable information about the patient groups, as well as misclassification of some individuals19. This article presented practical statistical software to help remedy such a problem and demonstrated how multiclass classification with g3mclass may help fine-tune stratification on clinical biomarkers. In our study cohorts, g3mclass automatically recognized cancers unlikely to be present in the reference, i.e., ERBB2 mRNA + (class 2 and higher) and ESR1 mRNA + (class 1 and higher). Likewise, recognizing by g3mclass the group of ERBB2 mRNA + (class 1) may help define HER2-low positive breast cancer in clinical trials34. Clinical studies adopting g3mclass are warranted to investigate whether the groups with differentially increased levels of ERBB2 mRNA and ESR1 mRNA have different sensitivity to the targeted therapy.This article focused primarily on validated biomarkers because the quality of HER2 and ER diagnostics affects millions worldwide. According to World Health Organization, with an estimated 2.3 million new annual cases reported globally, female breast cancer is the most diagnosed cancer type  human breast tissues were obtained from the Department of Pathology and Laboratory Medicine, Tumor Tissue and Biospecimen Bank, and the Cooperative Human Tissue Network at the University of Pennsylvania. The study was performed with 256 samples, including 34 mammoplasties from women with no history of breast cancer, 75 diagnostic biopsies of ductal carcinomas in situ (DCIS), 142 surgical excisions of primary invasive breast cancer (IBC), and 5 human breast cancer cell lines. The cell lines MCF-7T-47D, MDA-MB-231, SK-BR-3, and BT-474 were purchased from the American Type Culture Collection and cultured accordingly. MycoAlert Assay (Cambrex) confirmed that mycoplasma-free cells were used in the experiments. The tissue samples were accrued randomly from the same geographic region. Summaries of the characteristics of the study populations have been published19. QG2 is a highly reliable and validated molecular assay that uses amplified branch DNA (bDNA) technology for parallel gene expression profiling42. We analyzed measurements of specific probes with the QG2 assay kits for quantitation of multiple target specific RNAs directly in lysates from FFPE tissue and cell lines. Our human Plex Set 12988 included 14 target-specific and two housekeeping gene probes described in detail40. Here, we analyzed mRNA for ERBB2 (probe set region 1203–1621), ESR1 (probe set region 5671–6292), and PGR (probe set region 2609–3194).For mRNA data collection, we ran QuantiGene Plex 2.0 (QG2) assay  and read on FlexMap 3D (Luminex/Merck Millipore) according to manufacturers’ protocol and as described in detail40.The status of steroid hormone receptors (ER and PR) and HER2 were determined by IHC and/or FISH, the FDA-approved methods. The status of ER, PR for all tumors, and HER2 for primary IBC were obtained from surgical pathology reports. In addition, HER2 status in DCIS was assessed based on the FDA-approved method for IBC as describedg3mclass v.1.2 on macOS Mojave v.10.14.6. with Python v.3.9.5 and wxPython v. 4.1.1. Additionally, we used MedCalc, a diagnostic test evaluation calculator at https://www.medcalc.org/calc/diagnostic_test.php  and GraphPad Prizm 9.3.0  to visualize group comparisons.In this article, modeling and analyses of datasets were performed with Studies were conducted in accordance with recognized ethical guidelines. We used approval from the University of Pennsylvania Institutional Review Board committee with a waiver of written informed consent to analyze patients’ tissue and records.Supplementary Information 1.Supplementary Information 2.


With the rapid expansion of data, the problem of data imbalance has become increasingly prominent in the fields of medical treatment, finance, network, etc. And it is typically solved using the oversampling method. However, most existing oversampling methods randomly sample or sample only for a particular area, which affects the classification results. To solve the above limitations, this study proposes an imbalanced data oversampling method, SD-KMSMOTE, based on the spatial distribution of minority samples. A filter noise pre-treatment is added, the category information of the near-neighbouring samples is considered, and the existing minority class sample noise is removed. These conditions lead to the design of a new sample synthesis method, and the rules for calculating the weight values are constructed on this basis. The spatial distribution of minority class samples is considered comprehensively; they are clustered, and the sub-clusters that contain useful information are assigned larger weight values and more synthetic sample numbers. The experimental results show that the experimental results outperform existing methods in terms of precision, recall, F1 score, G-mean, and area under the curve values when the proposed method is used to expand the imbalanced dataset in the field of medicine and other fields. Minority categories in these fields have smaller sample sizes and poorer sample data quality; however, they typically carry more important information. Thus, we focus on a model’s ability to correctly classify the minority classes of samples, such as a complex network system, where it is more important to accurately diagnose the network fault types and maintain the normal operation of the system than to diagnose the network as normal.Imbalanced learning is a basic problem in machine learning. When the number of samples from different categories in a classification task dataset differs significantly, the dataset is called imbalanced. Typically, the category with the larger number of samples is called the majority category, and that with the smaller number of samples is called the minority category. Imbalanced data are widely available in various real fields, such as financial fraud detection9, oversampling11, and mixed sampling12. The second scheme is primarily from the perspective of the algorithms. By analysing the variability of the algorithm cost in different misclassification cases, the classification algorithm is optimised to improve the performance of the learning algorithm13. Data-based approaches are more popular in existing literature than approaches that improve a specific classification algorithm for a specific imbalanced dataset.However, in the classification process, traditional classification methods, such as support vector machines, decision trees, Bayesian networks, and k-nearest neighbours, are designed to maximise the overall classification accuracy. This often leads to higher classification accuracy for the majority category and lower classification accuracy for the minority category, which may be limiting for practical applications. The methods used to solve the imbalanced data problem and improve classification accuracy for minority class samples are divided into two main aspects. The first scheme is primarily from the perspective of the data. The main method comprises sampling, sampling through a strategy, and changing the category of the sample distribution to achieve the purpose of balancing the majority and minority class samples. There are three types of sampling strategies: undersampling14. The synthetic minority oversampling technique (SMOTE) is considered the most influential data pre-processing and sampling algorithm16. The basic idea of the SMOTE algorithm is to analyse the minority class samples, and based on them, manually synthesise new samples to add to the dataset17. SMOTE, as an undersampling method, balances the proportion of sample classes in the dataset to some extent by increasing the number of minority class samples, thereby improving the classifier's ability to classify minority classes while leaving its ability to classify majority classes unchanged18.Among the three sampling methods, oversampling is used more frequently than the other two. Undersampling reduces the number of samples, which may lead to the loss of important data information. In addition, Batista et al. demonstrated that oversampling generally performs better than undersampling, as measured by the area under the ROC curve (AUC)Although SMOTE is favoured in existing literature for its ability to solve imbalanced data problems and its implementational simplicity, it still has some limitations. SMOTE treats all minority class samples equally and does not consider the class information of the neighbouring samples. This often leads to the phenomenon of sample aliasing, which results in a poor classification effect. In addition, if the selected minority class sample is surrounded by minority class samples, the newly synthesised sample will not provide substantial useful information. If the selected minority class samples are surrounded by majority class samples, such samples may be noise; therefore, the newly synthesised samples will overlap more with the surrounding majority class samples, thereby resulting in classification difficulties.Increasing the filtering noise pre-treatment. Before the new sample synthesis operation, the isolated minority class sample points that are surrounded by majority class samples are screened out as noise. Thus, overlap between the newly synthesised samples and majority class samples is avoided, and the differentiability between sample classes is enhanced.Design of a new sample synthesis method. In order to solve the problem of how to no longer treat all minority class samples equally, this study divides minority class samples into several regions, and allocates different numbers of synthetic samples according to the sample clusters in different regions to solve this problem. Specifically, K sub-clusters are generated for a small number of classes of samples using the k-means method. With the total number of newly synthesised samples constant, the cluster is considered a unit, and the sub-clusters that contain more useful information are assigned larger weight values and a greater number of synthesised samples. This method comprehensively considers the spatial distribution of minority samples, allocates different numbers of composite samples to sample clusters in different regions, optimises the spatial distribution structure of new synthetic samples, and thereby improves the classification ability of the classifier for minority class samples.Construction of calculation rules for the weight values. In order to solve the problem of how to use the useful information contained in the sample clusters in different regions to realize the synthesis of new samples in different regions. This study solves this problem by constructing a calculation rule for the weight value. Specifically, by calculating the Euclidean distance from the cluster centre of each cluster to the class centre of the majority class sample, the clusters that are closer to the majority class sample are assigned a higher weight value. Samples at boundary locations are more likely to be misclassified in the actual modelling process; therefore, using the sample information at boundary locations to generate new samples can give the model a larger sign and provide sufficient information for classification, which can effectively reduce the false positive rate of the classifier for the majority class sample.In response to the above limitations of SMOTE, this study proposes an imbalanced data oversampling method, SD-KMSMOTE, based on the spatial distribution of minority samples. In contrast to SMOTE, SD-KMSMOTE considers the distribution of majority class samples around minority class samples. In addition, SD-KMSMOTE no longer treats all minority class samples equally, and completes the synthesis of new synthetic samples by considering the spatial distribution structure of minority samples. Specifically, the main contributions of this study are as follows:The imbalanced data problem is a research hotspot in the field of machine learning. Recently, an increasing number of studies have attempted to address this problem. The section only summarises the SMOTE-based improvement and extension methods in oversampling techniques.19, where the noise generated by newly synthesised samples can also be filtered. Both methods have been applied in various scenarios21. Another noise-filtering operation is to enhance SMOTE by adding filters. Puntumapon and Waiyamai proposed an algorithm, TRIM, to solve the problem of the over-generalization of SMOTE. That is, the synthetic data is generated to majority class regions, and the purpose of removing noise from minority class regions is achieved by iteratively filtering irrelevant majority class data from the exact minority class regions22. However, the method only targets minority class samples; therefore, the following situation may arise. The algorithm searches the minority class region located at the boundary; if the number of minority class samples is significantly less than the number of majority class samples, that is, the minority class samples are the noise at this time, the majority class samples are still filtered out as noise. This will erroneously remove useful information from the boundary region and blur the demarcation line, which in turn affects the classification performance of the data as a whole (including minority class samples and majority class samples). In addition, other methods to enhance SMOTE by adding filters include rough set-based filtering24 and ensemble-based filtering25. In summary, although many studies have explored the filtering of sample noise, implementing this for SMOTE still requires further research.One of the SMOTE noise filtering operations is denoising at the end of the SMOTE process, and two typical techniques for this are SMOTE-TomekLinks and SMOTE + ENN26. Ref.26 considered that samples on the boundary (Danger) are more likely to be misclassified than samples far from the boundary (Safe); thus, boundary samples are more important for classification. Therefore, the algorithm improves the class distribution of samples by using only minority-class samples on the boundary to synthesise new samples. Abdi et al. proposed an oversampling method based on the Marcian distance to synthesise samples in only minority class-dense regions, which can effectively overcome the sample overlap problem; however, this method cannot guarantee the boundary sample information of minority classes27. Both of the two abovementioned methods one-sidedly synthesise new samples in a specific region of the minority class without considering the general influence of data from other regions. In addition, if there are few samples in the boundary region, it will be difficult to select candidate samples for the synthesis of new samples. In Refs.28 and29, the initial selection of samples was performed using a machine learning approach. In Ref.28, initial points were selected in support vectors that were obtained using support vector machines. In29, an adaptive semi-unsupervised weighted oversampling method was proposed using a semi-unsupervised hierarchical clustering approach, which implements the sampling of hard-to-learn instances near decision boundaries. The two abovementioned methods provide new ideas on how to select candidate interpolation samples.Unlike SMOTE, which randomly selects objects to synthesize new samples, the proposed method is based on the initial sample selection. Typically, the objects for the best synthesised samples need to be selected by a method before the interpolation operation. This strategy aims to reduce overlap and noise in the final dataset. Based on SMOTE, Han and Wang designed two new minority class-based oversampling methods, Borderline-SMOTE1 and Borderline-SMOTE230. This ADASYN learning mechanism has led to a wide range of applications in different fields32.He et al. proposed a new adaptive synthetic sampling method (ADASYN) for the learning of imbalanced datasets. The basic idea of ADASYN is to use weighted distributions for different minority classes of samples based on their difficulty of learning; this generates more synthetic data for the minority samples that are more difficult to learn26, which only targets the boundary region for new sample synthesis, this study comprehensively considers the spatial distribution of minority class samples and removes noise without destroying the intra-class and inter-class distributions. Moreover, the computational complexity is low, which can be directly applied to most classifiers to further improve the classification ability of the sample dataset as a whole with the premise of improving the classification accuracy of minority class samples.In summary, although the abovementioned improved oversampling method based on SMOTE enables the synthesis of new samples and achieves good classification results in some specific cases, there are still some associated limitation. The spatial distribution structure of minority samples is not considered comprehensively, and only a single new sample is synthesised for a specific region, thereby neglecting some minority samples that contain important information. The method has no ability to mitigate intra-class and inter-class distribution imbalances while avoiding noise generation. No time or space complexity is considered; however, the respective improvements are achieved at the cost of high complexity, which makes it difficult to apply them in practice. To this end, this study proposes a new version of SMOTE based on the spatial distribution of minority class samples; it divides the minority class samples into several clusters by clustering and filtering the isolated sample points that are not in the clusters or are far from the clusters as noise. According to the distance between each cluster and the majority of samples, weight is assigned to each cluster. The number of new samples synthesised in different proportions is allocated according to the weight given to each cluster. Unlike Ref.26, a minority class of samples is classified into three categories during the sampling process: safe, dangerous, and noise. When more than half of the samples around a minority class are minority class samples, the sample is considered to be in a safe region; when more than half of the samples around a sample are majority class samples, the sample is considered to be in a border (dangerous) region; and when all samples around a sample are majority class samples, the sample is considered noise. Because the study in Ref.26 considered that minority classes of samples in the boundary region contain the most useful information, only samples from that region are sampled, and new samples are synthesised. However, the boundary region contains the most important samples that are beneficial for classification, which does not imply that there are no important samples in other regions.According to the definition in Ref.26, this study proposes an imbalanced data oversampling method, SD-KMSMOTE, based on the spatial distribution of minority samples. The basic idea of SD-KMSMOTE is that, unlike Ref.26 which only samples samples from edge regions, in this study, by comprehensively consider the spatial distribution of minority class samples by utilizing ideas from Ref.26 and assigning more synthetic new samples in the boundary region, while also assigning a particular number of new samples to be generated in the safe region, And by designing a new sample synthesis method and constructing the calculation rules of the weight value, the allocation of the number of new samples to be synthesized in different regions is completed.Therefore, inspired by Ref.SD-KMSMOTE is specifically implemented by introducing the k-means method to generate K sub-clusters by clustering minority class samples, thereby removing isolated points (noise) away from or not in the clusters before oversamling. The cluster is considered a unit, and different weight values are assigned to each cluster. With a constant total number of newly synthesised samples, clusters with higher weight values are assigned a higher number of synthetic samples. The calculation rules for weight values are constructed by calculating the Euclidean distance from the cluster centre of each cluster to the class centre of the majority class sample; the clusters that are closer to the majority class sample are assigned a higher weight value. The flow chart on the SD-KMSMOTE oversampling method is shown in Fig. Let the original training sample be Filtering isolated sample points According to the K-neighbor algorithm, the k-neighbourhood of all sample points in sample  Formula . If therCalculating the number of samples to be inserted. The balance rate Calculating the Euclidean distances Among them, in order to avoid differences in clustering results caused by the selection of initial clustering centers. We use the K-means +  + algorithm to improve the initial clustering center selection part of the K-means algorithm. The initial clustering center selection idea of the improved algorithm is as follows: The distance between the initial clustering centers should be as far as possible; Assuming that n initial cluster centers have been selected (n < k), when selecting the n + 1st cluster center, the farther away from the current n cluster centers, the higher the probability of being selected as the n + 1st cluster center.In addition, in order to avoid the k-means method falling into local optimization, we use the sum of the squared errors (SSE) to select the K value:The closer E is to 1, the closer the number of majority class and minority class samples in sample S.In the formula, (3.1)The class centre of class M is calculated. The class centre of sample M defined in this study is obtained by finding the mean of all sample points in the class using the following method:Let there be In the formula, (3.2)The distance between each cluster and M-class is calculated. Let the cluster centre of K subclusters obtained using the k-means method be (4)Calculating the number of samples to be inserted in each clusterLet the number of samples to be inserted in K sub-clusters be (5)The weight values assigned to each cluster is calculated. The sub-clusters with a shorter distance from class M, which are closer to the boundary, carry more useful information; thus, they receive higher weight values and allocate more synthetic samples. The sub-clusters with a longer distance from class M, which are further from the boundary, carry less useful information; thus, they will receive less weight values and allocate less synthetic samples. According to the above rules, the established formula to calculate the weight value W isIn the formula,(6)The number of samples to be inserted in each cluster is calculated based on weight values. The number of samples to be inserted in the i-th sub-cluster (7)Linear interpolation to synthesise a new sample Specific steps are as follows:(8) i be ri, the initial value of  ri is 0;Select a subcluster, randomly select one of the samples, and the k-nearest neighbour samples of this sample are determined based on the K-nearest neighbour algorithm. Let the number of new samples already synthesized by the current subcluster(9)If (10)If (11)Randomly select one of the remaining samples of the current sub-cluster (12)Repeat the above steps to complete the synthesis of new samples for other clusters.In the sub-cluster, a random sample selected based on the K nearest neighbors is The new samples generated by all subclusters are added to the original training sample S, and a balanced dataset The simulation process of the SD-KMSMOTE method proposed in this study is shown in Fig. 26, Borderline-SMOTE226 , and ADASYN30, as comparison methods, and explained and compared them in two parts: evaluation metrics and the analysis of experimental results. It is proved that the method proposed in this study is superior for imbalanced data processing and learning.In this section, the performance of the proposed SD-KMSMOTE oversampling method is demonstrated. We selected several classic over-sampling methods, including SMOTE, Bordline-SMOTE1Generally, we evaluate the good performance of an oversampling method by combining the classification algorithms. However, owing to the nature of the imbalanced classification problem, the accuracy and error rates that are generally used to evaluate classifier performance may no longer be applicable. This is because in an imbalanced classification problem, we are primarily concerned with whether minority classes are correctly classified; however, the accuracy of the overall performance in the classifier basically depends on the number of majority classes that are correctly classified. If all the majority classes are correctly classified and all the minority classes are incorrectly classified, the classifier can still easily achieve a high accuracy rate, but in reality, the classifier is ineffective.Therefore, for the imbalanced data problem, this study selects some representative metrics to evaluate the performance of the imbalanced data to assess the classification results. These metrics include precision, recall, F1-score, G-mean, and the AUC. In the binary classification problem, we consider the categories with small numbers as positive examples and those with large numbers as negative examples. Table The horizontal coordinate of the ROC curve is the false positive rate (FPR), the vertical coordinate is the true positive rate (TPR), and the AUC is the area under the ROC curve.We first use the method proposed in this paper to synthesise the new dataset by setting different balance rates using seven imbalanced datasets from the Kaggle. The basic information of the seven datasets used in this study is presented in Table The experimental environments in this section were all conducted in MATLAB 2020 (MathWorks). For the method described in this study, it is necessary to perform clustering and calculate the number of samples inserted in each cluster before synthesising new samples. For this purpose, Table As shown in Table To verify the performance of the proposed method in this study with datasets of different balancing rates, SMOTE is introduced as a comparison group in this section, and the first three datasets  are selected for experiments. After obtaining the synthetic dataset at different equilibrium rates using the methods of this study and SMOTE, we use decision trees and random forests as classifiers to obtain the classification results for the original and synthetic datasets at different equilibrium rates. By comparing the original dataset with the evaluation metrics under different balance rates, it is proved that it is necessary and effective to balance the imbalanced dataset.MATLAB 2020 is still implemented as the experimental environment for this part of the study. To make a fair comparison, the parameters of the classification algorithm are set as the default values. In addition, we divided each data set into 10 groups on average for a total of 30 experiments. Each time, 3 subsets of the data set were randomly selected as the test set and the remaining 7 subsets were selected as the training set. Finally, the average of five evaluation indexes obtained from 30 experiments was selected as the result of performance comparison of different classifiers. The number of samples from each of the three datasets used for training and testing is shown in Table In the classification experiments, we use decision trees and random forests to represent single and combined classifiers, respectively, to verify the performance of the proposed method.Figures 26, borderline-SMOTE226, ADASYN30, and SMOTE, for further comparison. We use the seven imbalanced datasets in Table To validate the performance of the proposed SD-KMSMOTE method, we select four classical algorithms, namely borderline-SMOTE1In this part of the experiments, we use Borderline-SMOTE1, Borderline-SMOTE2, and ADASYN methods from Python's imblearn library to obtain a synthetic dataset with a balance rate of 0.9. The remainder of the experimental environment is still performed using MATLAB 2020 with the same specific parameter settings and other instructions as in 4.2.1. Figures Figures The current expansion in the field of artificial intelligence has led to an imbalanced learning problem that is still receiving increasing research attention. Unlike most existing oversampling methods that randomly sample or sample only for a particular region, this study proposes an imbalanced data oversampling method, SD-KMSMOTE, based on the spatial distribution of minority class samples. In this study, we focus on the spatial distribution of minority class samples, allocate more synthetic new samples in the boundary region, allocate a particular number of new samples to be generated in the safe region, and improve the performance of the imbalanced learning problem in three main ways. (1) Adding filtering noise preprocessing. Before the new sample synthesis operation, the isolated minority class sample points, where the surrounding samples all belong to the majority class samples, are screened as noise. (2) K sub-clusters are generated for a small number of sample classes using the k-means method. The cluster is taken as a unit, and the sub-clusters that contain more useful information are assigned larger weight values and a greater number of synthesised samples. (3) Calculation rules are constructed for weight values. By calculating the Euclidean distance from the cluster centre of each cluster to the class centre of the majority class sample, the clusters that are closer to the majority class sample are assigned a higher weight value. In this study, five evaluation metrics that are commonly used in imbalanced learning are utilized, and two experiments are conducted to analyse the method’s performance under different balance rates and the performance of different sampling methods. The experimental results show the effectiveness, stability, and superiority of the proposed method in dealing with imbalanced learning.In the future, our research will focus on two aspects. The dataset selected for this experiment has some noisy data, which has not been elaborated on. Therefore, more noise will be subsequently introduced to address noisy samples in imbalanced data using the method presented in this study. Furthermore, the method presented in this study will be extended to the study of multiclassification imbalance learning.


The problem of drift in the electronic nose (E-nose) is an important factor in the distortion of data. The existing active learning methods do not take into account the misalignment of the data feature distribution between different domains due to drift when selecting samples. For this, we proposed a cross-domain active learning (CDAL) method based on the Hellinger distance (HD) and maximum mean difference (MMD). In this framework, we weighted the HD with the MMD as a criterion for sample selection, which can reflect as much drift information as possible with as few labeled samples as possible. Overall, the CDAL framework has the following advantages: (1) CDAL combines active learning and domain adaptation to better assess the interdomain distribution differences and the amount of information contained in the selected samples. (2) The introduction of a Gaussian kernel function mapping aligns the data distribution between domains as closely as possible. (3) The combination of active learning and domain adaptation can significantly suppress the effects of time drift caused by sensor ageing, thus improving the detection accuracy of the electronic nose system for data collected at different times. The results showed that the proposed CDAL method has a better drift compensation effect compared with several recent methodological frameworks. Electronic noses (E-noses) are sensor intelligence systems in the field of artificial olfaction that mimic the olfactory system of mammals to identify or measure gas samples. A complete E-nose system typically consists of three components: a gas sensor array, a data pre-processing unit, and a pattern recognition algorithm module. An E-nose has sensitivity to complex gas mixtures/compounds as well as analytical capabilities, thus allowing it to accurately identify complex gas samples. With the development of artificial olfaction, E-noses are also becoming increasingly important in fields such as , environDue to drift, it will lead to an inconsistent distribution of source and target domain samples, so the model trained with source domain samples will not obtain good classification performance for target domain samples. Modelling by continuously acquiring a large number of calibration samples with known labels at different times is a time-consuming and laborious task. In fact, a small number of labeled target domain samples contain knowledge of the target domain sample distribution. Joint modelling with a large number of labeled source domain samples and a small number of labeled target domain samples can discern information about the differences in the distribution of target and source domain data due to sensor drift, thus greatly improving the classification accuracy of the model for the target domain samples. Therefore, acquiring some of the labeled target domain samples at a small additional cost, thereby greatly improving the prediction accuracy of a large amount of unlabeled target domain data, is an acceptable low-cost method to address electronic nose drift.In this study, we aimed to mitigate the effects of distorted data distribution and degraded performance of the E-nose due to sensor drift. We sought to combine an active learning approach and domain adaptation to complementarily enhance the sensor’s performance in the identification of drifting gases. Specifically, we designed a semi-supervised cross-domain active learning (CDAL) model for drift compensation of the E-nose. As shown in (1)The framework consists of a combination of active learning and domain adaptation, which makes full use of the distribution differences in migration distributions of samples between different domains while considering the degree of disagreement among committee members on the classification results of the sample.(2)HD is used as a measure of the divergence of the enquiry committee’s output and is more appropriate for sample selection than the usual discrete judgement criteria.(3)The MMD is able to measure the distributional variability between the source and target domains, greatly facilitating the selection of more representative samples for modification of the classification model in active learning.(4)The mapping of Gaussian functions allows us to represent the distance between distributions by the inner product of points, which can be used to assess the impact of differences in the distribution of data due to drift.(5)Experiments show that CDAL is significantly effective in both drift suppression and pattern recognition.The advantages of CDAL over other methods can be summarized as follows:The paper is divided into the following sections: Due to the influence of the external environment or the ageing of the sensor itself, poisoning and other generated E-nose drift can produce irregular interference with the response of the gas sensor, thus reducing the recognition accuracy of the E-nose system. In recent years, much work has been done to solve the drift problem of E-nose sensors. These methods are roughly divided into three main types: component correction strategies, domain correction strategies, and classifier strategies.The component correction strategies aim to identify drift components in the original signal and then remove them before training the final discriminative classifier. This approach is primarily based on feature or component removal from sample data. Haugen et al.  proposedThe domain correction strategy is a model that learns from the source domain data distribution and yields good performance on different (but related) target domain data distributions and is well suited for use as drift compensation for the E-nose. Since E-nose drift is nonlinear and unstable, Tao et al.  proposedThe aim of classifier strategies is to design robust classifiers to achieve a better discriminative output in E-nose drift compensation. In turn, classifier methods are divided into single classifier methods and classifier integration methods. The single classifier approach is a drift compensation model with a specific method using a single classifier as the discriminant model. Zhang et al.  proposedL and U, which is shown in U denotes unlabeled target domain data (sample query set) for drift compensation sample selection, while L indicates the drift compensation set (drift compensation samples with labels) for “machine learning model” C updating. The “selected samples” L are selected from the “query set” U, which is full of target domain drifting instances, and the “labels” of the “selected samples” are sent to the “expert” S query. The “labels” of the “selected samples” are labeled by “expert” queries. The most critical of these is the “instance selection strategy” Q, which requires a suitable rule to select the most representative instances from the data pool for retraining the “machine learning model” C. Finally, the “machine learning model” C is updated with the selected instances and labels for the next recognition. Therefore, the AL framework is a distinct closed-loop structure that updates the “machine learning model” C with “selected instances” L and their “labels”, thus improving the model’s recognition accuracy.The AL process is a closed loop consisting of two sample data sets Q, which aims to reflect the maximum information with the minimum number of samples. Liu et al. proposed an active learning method based on dynamic clustering to balance the labels of different categories of labeled samples by dynamic clustering [In recent years, active learning methods with finite labeled samples have also started to be applied to the drift compensation of E-nose sensors. The core of the AL approach is its “instance selection strategy” ustering . Liu et ustering . More reustering  proposedustering . Consideustering .However, AL methods are still in their infancy, and most of the AL frameworks proposed only consider the amount of information contained in the samples while ignoring the effect of misaligned sample data distribution between domains due to drift. In this regard, we proposed the CDAL framework based on the HD and MMD for E-nose drift compensation.D represents the number of features and Suppose Unlike the traditional pool-based AL method, the CDAL method requires the calculation of the information value of the sample and the distribution difference from the source domain from two separate perspectives.First, we calculated the HD to measure the divergence of the selected samples. For two discrete probability distributions For computational purposes, we can also think of this as the Euclidean distance between two vectors of square roots of probability distributions as Equation (2).K denotes the number of committee members, and K.By definition, the HD is a metric satisfying triangle inequality. The We used HD to measure the information value of the selected samples, and next we introduce the MMD to solve the problem of interdomain data distribution differences when selecting samples.Since we need to calculate the migration distance MMD between each target domain sample and the center of the source domain, the formula for calculating the MMD in this paper is defined as Equation (5).The key to calculating the MMD is to find a suitable mapping function According to kernel function theory, the inner product of two vectors in a high-dimensional eigenspace can be found from the kernel function in the original space without knowing the mapping function Considering the mapping space dimensionality of the data distribution, we use the Gaussian kernel function as the kernel function for the MMD calculation:The larger MMD between the target domain samples and the center of the source domain indicates the larger difference between the two-domain sample distributions. In other words, the larger the MMD is, the more difficult it is to distinguish. Therefore, we need to calculate the MMD between each target domain sample and the source domain to select the instance with the largest MMD for labeling.The MMD represents the difference in data distribution between the sample and the source domain, and the HD represents the ease of sample differentiation. A larger HD means that the sample is more difficult to classify correctly, which means that the sample drifts more and reflects the target domain information. A larger MMD indicates a greater degree of migration between the target domain data and the center of the source domain. Therefore, we need to consider the impact of HD and MMD on sample selection in combination. We used an adjustable parameter N maximum weighted sums for labeling based on the pool-based approach and added them to the training set. Then, we removed them from the target domains. This cross-domain active learning mode can select the most representative samples and greatly improve the recognition accuracy of the classifier.Since we needed to maximize both HD and MMD, we aimed to obtain the samples with larger Algorithm 1 Proposed CDAL AlgorithmInput: sample query set U, training set n: number of samples in each query set U.N: number of selected samples.Output: updated training set 1: Initialize U = 2: fori = 1:do3: Calculate the HD U by the probability distribution of the committee members’ predictions through Equation (3).4: Calculate the MMD between each sample in the query set U and the entire training set 5: Calculate the weighted sum 6: end for7: Select U as the selected sample set 8: Label 9: Update 10: Update 11: Return training set For a more intuitive representation of our model, we summarized and reorganized CDAL in Algorithm 1.We used a public benchmark dataset of the time drift of the gas sensor array from UCSD  for the To visualize the data drift, we performed principal component analysis on 10 batches of data, mapped the data into a two-dimensional principal into molecular space and plotted the corresponding scatter plots. The PCA for each batch is shown in To test the compensation effect of our method on sensor drift with the dataset, we took the following two experimental setups:Setting 1: For the long-term drift of the sensor, we used batch 1 data as drift-free training data, and each remaining batch as drift data for testing.Setting 2: For the short-term drift of the sensor, we took batch k  as the training data without drift and batch k + 1 as the test data with drift.To validate the effectiveness of our CDAL method, we used the QBC method as the active learning query method. For the classifiers, we used the SVM with good classification performance as the chair and the SoftMax classifiers, which can output probabilities, as committee members to evaluate the information of samples. Considering the recognition accuracy, we used the Gaussian kernel for the SVM classifier and performed a grid search for the two parameters In Setting 1, we used 12 of the latest drift compensation methods for comparison. First, two single classifiers, SVM and ELM, were used to demonstrate the performance of the classifiers without drift compensation. Note that we used SVM as the baseline classifier for other drift compensation methods. Next, four component correction models, DS ,9, CCPCA(1)Our CDAL method has the best recognition results among all the compared methods under the same experimental conditions, and the average recognition accuracy is almost 10% higher than all the other methods.(2)The direct use of SVM and ELM classification was the worst. Among them, the migration learning methods CDSL and DRCA were able to achieve recognition accuracy of 69.59% and 58.72%, which was slightly better than that of the baseline methods. This indicates that the introduction of domain adaptation considering the interdomain distribution problem can improve the E-nose drift compensation effect.(3)The recognition accuracy of the AL-KLD, AL-JSD, AL-HD, and CDAL methods that used sample selection methods were all above 65%, which was significantly better than the other methods. This also shows that using a small amount of target domain data for labeling can greatly improve the classification accuracy, which is worthwhile and effective.To make the results more visible, we marked in bold the best results for each batch. Based on the data in We also optimized the SVM parameters (1)The CDAL method we used still has the best results in dealing with short-term drift, and with a selected sample of 30, the average recognition accuracy can reach over 82%, which far exceeds the other methods. This indicated that our CDAL method has better identification and robustness than other methods and was very effective in dealing with the short-term drift of the E-nose.(2)AL-KLD, AL-HD, AL-JSD, and CDAL, as the methods using sample selection, achieved an average recognition accuracy of over 70%, which indicates that selecting an appropriate sample selection criterion for E-nose drift compensation can greatly improve recognition performance. This also shows that the method of obtaining a very small number of target domain labels is worthwhile and effective.(3)The recognition accuracy of CDSL as a migration learning method can also reach 75.58%, which indicates that in the process of drift compensation of the E-nose system, the difference in data distribution caused by drift needs to be solved, which also provides a reference for our CDAL model.For the short-term drift pattern of the E-nose (Setting 2), we used the same 13 methods as for Setting 1 as a comparison. From the results in For short-term drift, we still optimized the SVM parameters CDAL requires a weighting rule based on MMD and HD to select samples from the target domain for labeling, so the size of the number of labeled samples, Based on the data in In this study, we used an approach based on active learning and domain adaptation to solve the sensor drift problem in the E-nose. We proposed a new cross-domain active learning framework based on HD and MMD, called CDAL. We discriminated the amount of information contained in the selected samples by using HD and measured the interdomain distribution differences of the selected samples by using MMD. Finally, we maximized the weighted sum of the two as the selection criterion for the target domain samples and used a pool-based QBC method to obtain the most representative labeled samples. The proposed CDAL method inherits the advantages of active learning, which can greatly improve the classification accuracy by consuming a small labeling cost; additionally, it is the first time that active learning has been combined with domain adaptation, which was also the focus of this study. The ability to introduce the domain adaptation framework into active learning, balancing the impact of interdomain distribution differences on labeled samples, is an important contribution, providing new ideas for future active learning and domain adaptation.