There is an urgent need to reuse and recycle textile fibers, since today, low recycling rates are achieved. Accurate classification methods for post-consumer textile waste are needed in the short term for a higher circularity in the textile and fashion industries. This paper compares different spectroscopic data from textile samples in order to correctly classify the textile samples. The accurate classification of textile waste results in higher recycling rates and a better quality of the recycled materials. The data fusion of near- and mid-infrared spectra is compared with single-spectrum information. The classification results show that data fusion is a better option, providing more accurate classification results, especially for difficult classification problems where the classes are wide and close to one another. The experimental results presented in this paper prove that the data fusion of near- and mid-infrared spectra is a good option for accurate textile-waste classification, since this approach allows the classification results to be significantly improved. The textile industry is currently one of the main economic activities worldwide. It amounts to approximately EUR 178 billion in Europe, and about 1.7 million jobs are generated throughout its value chain . HoweverEuropean institutions are implementing strict environmental regulations in the textile sector. In fact, in 2018, a European regulation (Directive (EU) 2018/851)  was apprMeasurements based on optical methods are appealing because they avoid any contact with the sample . SpectroData-fusion methods are based on combining data acquired by multiple sensors to generate more accurate, useful and consistent information than that obtained with the use of a single sensor. Multi-sensor data also combine the advantages of using different sensors , becauseAlthough several publications related to NIR spectroscopy, as a method to identify different textile fibers  ,42 and sThus, the aim of this paper is to develop a sorting system able to separate the most common pure textile fibers , as well as binary mixtures of viscose–polyester and cotton–polyester using data fusion of NIR and MIR spectra. The classification system needs to be as accurate as possible to assure a high quality of the recycled materials. Therefore, data fusion of near and mid infrared spectra is used to obtain an accurate classification.k-nearest neighbor (kNN) classifier.The mathematical treatment of the spectra consists of three steps. First, principal-components analysis (PCA) is applied, followed by the canonical-variate analysis (CVA) algorithm, and finally, the obtained data are classified by applying the The paper is structured in seven sections, with the first being the −1) is energetic enough to cause transitions between the rotational and vibrational levels of the molecular bonds. The absorption of this radiation is highly sensitive to the type of bonds in a certain molecule. This is why this region of the spectrum is widely used in both the qualitative and quantitative analyses of organic molecules and polymers.The middle infrared (MIR) electromagnetic radiation  to avoid any pre-treatment of the solid and liquid samples. ATR equipment consists of an optic crystal (with a high refraction index) that receives the IR beam from one side, while the other side is in contact with the sample. Through the crystal, the sample receives an IR beam, absorbs energy in specific regions and reflects the rest to the crystal again. Several reflections can take place until the beam arrives to the detector.−1, having 1 cm−1 resolution and averaging four scans, thus obtaining 3551 data points per sample.The MIR spectra of the textile samples were acquired using a PerkinElmer spectrometer  equipped with an ATR module. The FTIR spectrometer provides spectral information within the wavenumber interval of 4000–450 cmThe near infrared region (NIR) has a range from 750 to 2500 nm, and the radiation absorbed is due to overtones and combinations from fundamental vibrations produced in the medium infrared (MIR). These bands provide useful information but have important limitations due to their low intensity, which is two or three times lower than those that appear in the medium infrared. Only absorption bands due to vibrations of high frequency are observed in the NIR spectra. In addition, in the NIR region, bands and overtones overlap, thus making it difficult to interpret the spectrum.x groups . There are few bands apart from those due to the overtones of CH, OH and NH groups, although in some cases, PH and SH bands can be observed.Hydrogen, being the lightest of the atoms, vibrates more widely during stretching vibrations. Consequently, almost all the absorption bands observed in the NIR come from overtones of the stretching vibrations of AHTM OptiProbe Analyzer, FOSS AnalyticalA/S, Hillerod, Denmark) equipped with a fiber-optic probe to perform reflectance measurements. The instrument was controlled with Vision SoftwareTM . The NIR spectrometer provides spectral information within the wavelength interval of 400–2499.5 nm with a resolution of 0.5 nm, using an average of 32 scans, thus providing a total of 4200 data points per sample. The shorter wavelengths correspond to the visible spectrum, with the NIR spectrum corresponding to the 1100–2500 nm wavelength interval. However, the signal near 2500 nm contains more noise, thus making the results less accurate. Therefore, it was decided to analyze the 1100–2200 nm interval, so each spectrum included a total of 2201 wavelengths.The near infrared (NIR) spectra of the textile samples were acquired by using an FOSS spectrometer  region, while the absorption in the NIR (near infrared) region is due to the overtones and band combination of the previous fundamental vibrations that occur in the middle infrared.Data fusion allows one to comprehensively combine original data from the MIR and NIR, taking advantage of the synergistic and complementary information provided by both techniques.Therefore, studies based on the fusion of NIR and MIR spectra can provide more accurate and complete information about the samples under analysis. Data-fusion strategies are successfully implemented in different areas, improving the results obtained with the use of individual techniques, which provide only limited, partial information.NIR + MIR spectroscopy refers to the combined use of NIR and MIR spectra for specific purposes, in the form of data fusion. This approach increases the amount of information of a given sample, which allows better classification results to be obtained. The specific case of merging MIR and NIR data is beginning to be applied in various fields, such as food technology , medicalIn this case, NIR and MIR are combined because some problems were found using NIR alone when dealing with dark-colored samples (due to the low reflectance of the radiation) and with wet samples . These problems may be solved using MIR spectra due to the complementarity of both techniques, although MIR is more difficult to apply on an industrial scale. However, MIR sensors are likely to evolve in the coming years to be more easily industrially applicable.This paper analyzes textile samples that were collected from the catalogs of several companies. These textile samples include natural fibers , synthetic and artificial fibers  and mixtures between both types. It is noted that artificial fibers come from the transformation of natural products , whereas synthetic fibers are made from polymers that are derived from petrochemicals.The selection of textile fibers aimed to include maximum variability, so different presentations (fabric or yarn) and multitude of colors (from dark to light) were included in the analyzed set of samples. For an accurate identification, all textile fibers had a code and a description including the color, presentation type, manufacturer and catalog reference.Samples were provided by different companies  and were from different years (from 2016 to 2019). A total of 52 different commercial catalogs were used to obtain individual samples for each of the three studies performed, which are described in 2 in size, and the thickness widely varied because there were winter and summer pieces of textile. When the thickness was not sufficient, the sample was folded 2–3 times before spectral acquisition to obtain the desired thickness.Spectral acquisition was conducted via the contact between the sensor and the sample. All samples were at least 9 cmk-nearest neighbor (kNN) classifier.To assess the accuracy of the analyzed mathematical classification algorithms, it is a common practice to divide the entire set of samples into two subsets, i.e., the calibration and prediction subsets. This approach enables the behavior of classification models to be assessed by using a different set of samples from the ones used to train or calibrate the model. Thus, the model is calibrated/trained by means of the calibration sample subset. The classification accuracy is assessed by classifying the samples of the prediction set, which are not used in the calibration stage.The data analyzed in this work were randomly divided according to the 50–50% proportion, i.e., 50% of the samples were assigned to the calibration set, whereas the remaining 50% of samples were assigned to the prediction set, as shown in It is noted that the samples of the calibration set were labeled according to the group or class they belonged to. This meant that the class, i.e., its composition, was already known. This is the basis of supervised classification approaches. However, the composition of the prediction set samples was unknown; thus, the classifier had to provide an accurate estimation of their pertinence class, i.e., their composition.To improve the classification results, it is common to preprocess the spectral data prior to the application of the algorithms detailed in Before any preprocessing, all spectra were transformed into absorbance mode. The first treatment consisted of calculating the first or the second derivatives of the spectral data. The derivatives were calculated by applying the Savitzky–Golay differentiation algorithm, which applies a moving average of five/ten points  at each point of the spectrum, thus preventing a reduction in the signal-to-noise ratio due to the derivative operation. Therefore, the spectral data were analyzed in three modes, i.e., raw (without applying any derivative), and first- and second-derivative modes.n × m data matrix to all the m variables of each row (every row contained the spectrum of a particular textile sample). The columns of the mean-centered data matrix had zero mean value, but their variances were not modified.Next, to further improve the classification results, spectral data  were mean-centered. This operation was performed by subtracting the mean value of each column of the x = , where x represents the spectral-data vector of each sample, either NIR or MIR, with m components (wavelengths or wavenumbers). The normalized vector, normx, was calculated as follows:To balance the weights of NIR and MIR spectra for data fusion, the NIR and MIR spectra of all samples were normalized within the 0–1 interval by applying a min–max normalization, Finally, both vectors containing the min–-max normalized NIR an MIR spectra of each sample were merged in only one vector, as shown in Dimensionality is the number of input variables for a dataset. Dimensionality-reduction methods reduce the number of input variables in a dataset by obtaining a reduced set of new variables, which are combinations of the original variables and essentially include the same information as in the original variables.Under the experimental conditions of this study, NIR spectrometry provided over two thousand data points per spectrum (textile sample), so it was required to apply dimensionality-reduction techniques, since large numbers of variables tend to cause poor performance of machine learning algorithms.This paper applied PCA and CVA dimensionality-reduction algorithms, because they condense the analytically relevant data into a reduced set of inferred or latent variables ,47, whicThe CVA is a supervised multiclass dimensionality-reduction method that was conceived to maximize the distances among different data classes and simultaneously minimize the distances among samples of the same class for an accurate classification . To thisXn,m)T;Calculation of matrix s-first-normalized eigenvectors (ia) of C, ranked so that the respective eigenvalues are in decreasing order. From them, matrix Am,s) of Cs ia;Calculation of the Vm,s)( = (LT)−1m,m)(: svd(X*) = Un,n) among seven different classes . In the second study, textile samples were composed of a mixture of viscose and polyester in different percentages. Finally, the third study had the aim of classifying mixtures of cotton/polyester in three classes from high to low cotton percentages.The NIR and MIR spectra of the samples were registered and then mathematically treated according to the sequence of algorithms described above. The figures below show the absorbance NIR and MIR spectra of the different types of samples.In addition, the NIR and MIR spectra of the binary mixtures of fibers are shown in In this study, 210 textile samples were studied. These samples came from pure fibers according to the distribution shown in The samples in kNN algorithms, with different preprocessing options being applied to the raw spectral data. It is noted that the number of retained PCs was that explaining at least 99.99% of the total variance.The results presented in ymeas,i is the i-th component of a true vector, whereas ypre,i* is the value predicted by the algorithm, which is commonly used to evaluate the performance of mathematical models [It is worth noting that the RRMSE is the relative root mean squared error, which was calculated as:l models .Although the initial NIR spectra contained 2201 points each, after the PCA-based dimensionality-reduction stage, only the first PCs explaining 99.99% of the total variance were retained.kNN algorithms, this approach correctly classified 104 out of the 105 samples of the prediction set.The results presented in kNN algorithms, it was possible to correctly classify all samples of the prediction set.The results summarized in This second study dealt with 73 textile samples. These samples included pure viscose fibers and viscose mixed with polyester according to the distribution shown in The objective of this study was to correctly classify the three classes defined in In this study, the calibration and prediction data matrixes had 37 and 36 rows, respectively, whereas the number of columns depended on the study performed .kNN algorithms, it was possible to correctly classify 35 out of the 36 samples of the prediction set.The results presented in kNN algorithms, it was possible to correctly classify 34 out of the 36 samples of the prediction set.The results in As seen in This third study analyzed 90 samples that came from pure cotton mixed with polyester in different proportions as detailed in The objective of this study was to correctly classify the three classes summarized in kNN algorithms to the NIR spectra.The results in kNN algorithms to the MIR spectra.kNN algorithms, it was possible to correctly classify 40 out of the 45 samples of the prediction set.The results presented in kNN algorithms, it was possible to correctly classify 43 out of the 45 samples of the prediction set.The results presented in This section summarizes the results attained in the three studies performed in this paper for a fair comparison of the attained results. The results presented in The sources of errors were most probably the number and nature of the samples. In general, the more samples used in the calibration are, the better the prediction is. Nevertheless, really good prediction results were obtained in this case, at least with pure samples (study No. 1). Samples with a mixture of fibers were more difficult to classify, especially when classifying very wide groups (the composition of the samples belonging to the same group widely varied around 20–35%) and groups close to each other (around 5% difference in composition), as in the case of study No. 3. In this case, a greater number of samples in the calibration set would be advisable to obtain a better prediction rate.The main conclusion of the present work is that the data fusion of NIR and MIR spectra provides more accurate classification results than NIR or MIR separately, especially for the classification of similar samples .The classification of pure fibers into seven types  took place with 100% accuracy (no misclassified samples) using either NIR or data-fusion spectra as input information. On the other hand, with binary mixtures of fibers, data fusion always provided the best classification results . The three groups defined for the cotton/polyester blends (cotton/polyester) were very broad and close, i.e., cotton ≥ 97%, cotton at 70–90% and cotton at 30–65%. In contrast, the viscose/polyester mixtures were three groups that were narrower and more separated from each other, i.e., viscose at 100%, viscose at 90% and viscose at 70–75%.Data fusion is a very promising alternative when the classes that need to be accurately distinguished have narrow borders between them. This is expected to be the case in textile-waste classification for recycling purposes.These are excellent results for a possible application of the proposed method in textile-waste classification to ensure the quality and composition of the recycled materials.


The early symptoms of lung adenocarcinoma patients are inapparent, and the clinical diagnosis of lung adenocarcinoma is primarily through X-ray examination and pathological section examination, whereas the discovery of biomarkers points out another direction for the diagnosis of lung adenocarcinoma with the development of bioinformatics technology. However, it is not accurate and trustworthy to diagnose lung adenocarcinoma due to omics data with high-dimension and low-sample size (HDLSS) features or biomarkers produced by utilizing only single omics data. To address the above problems, the feature selection methods of biological analysis are used to reduce the dimension of gene expression data (GSE19188) and DNA methylation data . In addition, the Cartesian product method is used to expand the sample set and integrate gene expression data and DNA methylation data. The classification is built by using a deep neural network and is evaluated on K-fold cross validation. Moreover, gene ontology analysis and literature retrieving are used to analyze the biological relevance of selected genes, TCGA database is used for survival analysis of these potential genes through Kaplan-Meier estimates to discover the detailed molecular mechanism of lung adenocarcinoma. Survival analysis shows that COL5A2 and SERPINB5 are significant for identifying lung adenocarcinoma and are considered biomarkers of lung adenocarcinoma. Lung adenocarcinoma (LUAD) is one of the malignant tumors threatening human health and life, accounting for 40–50% of the total number of lung cancers . The occIn recent years, the rapid development of modern bioinformatics technology can easily collect high-throughput omics data of various cancers, providing a new direction for cancer diagnosis. It is of great significance to understand the changes in cancer at the molecular level and find out biomarkers through omics data . With thAt the moment, many researchers have studied the pathogenesis of various cancers  based onIn this paper, we develop a model based on a deep neural network (DNN) by which one can predict LUAD using gene expression and DNA methylation data. At present, researchers have used multi-omics data sets to predict Alzheimer’s disease and confirmed the feasibility of this method . HoweverWith the development of sequencing technology, the public genome database is becoming more and more complete. Through the analysis of data, genes play an important auxiliary role in the mechanism and prediction of cancer. The dimension of cancer gene data is much higher than the number of samples. The main work to deal with this problem is to reduce the dimension.Feature transformation is a method to extract the features in the original space by some mapping transformation, such as PCA, PLS, etc. Using the new combined data to learn the classification model has achieved good classification results, but the biological significance is not clear. Feature selection is to evaluate the original genes to a certain extent and take out the features with good judgment ability to form a feature subset. The selected features have good explanatory power. Therefore, feature selection is a very effective method in high-dimensional data such as gene expression.Liyingxin  put forwIn this section, our work is to introduce the specific process of predicting LUAD and identifying biomarkers. As shown in https://www.ncbi.nlm.nih.gov/geo/, enter the keyword “lung adenocarcinoma” and click search. 2) Select “Expression profiling by array” in the “Study type” option, “Homo sapiens” in the “Top Organisms”. DNA methylation data is collected by the following standards: 1) Complete step one of gene expression data. 2) Select “Methylation profiling by array” in the “Study type” option, and “Homo sapiens” in the “Top Organisms”. 3) To expand the sample size, two DNA methylation data sets with sufficient sample sizes and the same platform are selected. According to the above criteria, gene expression data GSE19188  is a typical method to process gene expression data, and the selected genes have clear biological interpretation . Selectit-test is to calculate a t-statistic for each gene to measure the difference in gene expression between the two types of samples, and then calculate the significance p-value according to the t-distribution to measure the significance of the difference. In addition, to prevent some genes from being misjudged as differential genes, we also calculate the false discovery rate (FDR) through the significance analysis of the microarrays algorithm to control the error rate of multiple tests and reduce the false-positive rate of results.The principle of the FC algorithm is to calculate the multiple of the average expression level of genes in two types of samples. If the value reaches the preset threshold, it will be determined that the genes are differentially genes. The principle of So far, many researchers have applied the method of identifying DEG and DMP to process gene expression data and DNA methylation data and identify cancer biomarkers more accurately . Faced wp-value<0.05, and FDR<0.01. In DNA methylation data, the same way is used to identify DMPs, and the threshold should meet | logFC | > 2.5, p-value < 0.05 and FDR <0.01. Genes are differentially expressed, hypermethylation and hypomethylation in different samples occurred. This gene may have a potential relationship with LUAD. As a result, it is reasonable to believe that the overlap between DEG and DMP has a potential relationship with LUAD, so we take the intersection of DEG and DMP , K-Nearest Neighbor (KNN), Naïve Bayesian (NB), DNN model together with conventional machine learning algorithms have also been used to predict various biomedical phenotypes . At presTo verify the rationality of the data, we do principal component analysis (PCA) and Pearson correlation analysis, which respectively show the distribution of the data and the correlation between the samples to judge whether the data is feasible. In the PCA diagram, a point represents a sample. The farther the distance between two points, the greater the difference between the two samples. p-value less than 0.05. From the vertical axis, the farther away from the horizontal axis, the smaller the p-value, and the more significant the gene difference. As can be seen from the figure, the gene expression data includes 88 up-regulated genes and 118 down-regulated genes, which were the focus of our attention. In A volcano figure can help us intuitively identify genes with large changes and statistical significance. As shown in In this paper, we propose the following three hypotheses: The accuracy of predicting LUAD using multi-omics datasets is higher than that using a single omics dataset; In the prediction of LUAD, the feature selection method using biometrics (DEG + DMP) is better than the ordinary dimensionality reduction algorithm; The performance of using DNN classifier to predict LUAD is better than that of the traditional classifier.We divide the experiment into three sections to test the aforementioned hypothesis. I) Compare the accuracy of the prediction model by inputting different types of datasets. The DNN prediction model’s input dataset is separated into three types: gene expression, DNA methylation, and integrated dataset. II): Using MI, t-SNE, and biometric feature selection method to reduce the dimension of data. It is worth noting that the data should be reduced to the same dimension while utilizing different dimensionality reduction methods. III): When predicting LUAD, the performance of the traditional classifier (RF、KNN、NB) is compared with the DNN model. We used the dimension reduction algorithm provided by Scikit-learn package and the traditional machine learning algorithm.The input layer of the DNN classifier in this article is the integrated gene expression data and DNA methylation data. Because it is dealing with binary classification problems, the output layer consisted of a node. It can be seen from the result of Bayesian parameter determination that the hidden layer is composed of eight layers, ReLU is introduced as the activation function, and each layer is composed of 300 nodes and a bias node. When training the model, adds one dropout to each layer, and the dropout rate is 0.85. DNN is compared with three classifiers of random forest (RF) , K-NeareA total of nine combinations are offered to evaluate the effects of different feature extraction methods on prediction results: MI-gene expression, t-SNE-gene expression, DEG-gene expression; MI-DNA methylation, t-SNE-DNA methylation, DMP-DNA methylation; MI-integrate multi-omics data, t-SNE-integrate multi-omics data, DEG + DMP- integrate multi-omics data.To avoid overfitting on the training set, when using single omics data set as the input dataset, we do five-fold cross validation of gene expression data and DNA methylation data. For each fold, the number of DEGs and DMPs identified from the training set is different. Therefore, when performing MI and t-SNE on each fold gene expression and DNA methylation dataset, we reduce the dimension to the same number of features as those obtained by DEG or DMP methods (as shown in When a multi-omics dataset is used as an input dataset, the same as the process of using single omics data, MI and t-SNE algorithms are used to extract the same dimensionality as the feature algorithm proposed in this research, and the average value of the simplified dimensionality is 173.4. Traditional classifiers such as RF, NB, and KNN are selected for prediction. To achieve the optimal performance of the DNN model, the Bayesian optimization algorithm is used to determine the optimal hyperparameters of the DNN model. Before model training, Bayesian optimization is performed on the integrated data set to obtain a parameter combination with the highest training accuracy. Finally, the average value of each parameter in the five-fold is applied to our model, and the optimal hyperparameters are shown in The selected gene set is analyzed by using the DAVID database. As shown in After literature retrieving, the gene ABCA3, studies have shown that it is usually highly expressed in the damaged lung , and mayThe Kaplan-Meier survival curve analysis of the 12 potential biomarkers identified in 492 LUAD patients using the TCGA database shows that COL5A2 and SERPINB5 are significantly correlated with the prognosis of LUAD .In this study, we use the DNN model to predict LUAD and identify biomarkers based on integrated multi-omics data. By comparing different feature selection methods and different prediction models, the results show that the method of this research is better. The advantage of the method is that it used integrated multi-omics data; there is no blind dimensionality reduction in feature selection, but biologically significant features are selected. The results of biological correlation analysis and literature verification also show that the selected genes can be used as biomarkers of LUAD. Although we have done careful bioinformatics analysis, there are still some limitations. In the future, we can continue to explore the application of this method to other similar omics data, and we will continue to improve our method.


It allows recovering draft genomes from a pool of mixed raw reads, yielding longer sequences that offer contextual information and provide a more complete picture of the microbial community.The de novo assemblers for metagenomic analysis, LMAS (Last Metagenomic Assembler Standing) was developed as a flexible platform allowing users to evaluate assembler performance given known standard communities. Overall, in our test datasets, k-mer De Bruijn graph assemblers outperformed the alternative approaches but came with a greater computational cost. Furthermore, assemblers branded as metagenomic specific did not consistently outperform other genomic assemblers in metagenomic samples. Some assemblers still in use, such as ABySS, MetaHipmer2, minia, and VelvetOptimiser, perform relatively poorly and should be used with caution when assembling complex samples. Meaningful strain resolution at the single-nucleotide polymorphism level was not achieved, even by the best assemblers tested.To better compare de novo assembler depends on the computational resources available, the replicon of interest, and the major goals of the analysis. No single assembler appeared an ideal choice for short-read metagenomic prokaryote replicon assembly, each showing specific strengths. The choice of metagenomic assembler should be guided by user requirements and characteristics of the sample of interest, and LMAS provides an interactive evaluation platform for this purpose. LMAS is open source, and the workflow and its documentation are available at https://github.com/B-UMMI/LMAS and https://lmas.readthedocs.io/, respectively.The choice of a  These contigs are longer sequences that offer better contextual information than reads alone and provide a more complete picture of the microbial community than the species composition. Despite efforts for the development, standardization, and assessment of software for metagenomic analysis, both commercial and open source [de novo assembly process still represents a critical point in these analyses.Short-read shotgun metagenomics has the potential to offer comprehensive microbial detection and characterization of complex clinical or environmental samples. Despite becoming an increasingly used approach, it comes at the cost of producing massive amounts of data that require expert handling and processing, as well as adequate computational resources. The n source , the de The assembly of draft genomes has become a central step when analyzing pure bacterial cultures, for instance, allowing genomic comparisons through single-nucleotide polymorphisms (SNPs) or gene-by-gene methods, such as core-genome multilocus sequence typing (cgMLST). de Bruijn graph (dBg) algorithms are currently the most widely used approaches in modern assembly software. dBg handles unresolvable repeats by essentially fragmenting the sequence, that is, forming multiple contigs for each of the possibly contiguous sequences present in the sample. Additionally, the inherent heterogeneity of complex samples, potentially containing a multitude of replicons, could make traditional genome assemblers, implementing optimizations based on the assumption of having a single genome in the sample, not suitable for metagenomics .Several dedicated metagenomic assembly tools for short-read data are available . These tde novo assembly validation methods relying on features of the created contigs themselves, such as QUAST [With an ever-increasing range of both traditional and metagenomic assemblers becoming available, choosing the best-performing tool can be an arduous and time-consuming task since the choice may vary depending on the purpose of the analysis, organism of interest, complexity of the sample, and computational infrastructure available. Additionally, the evaluation of the resulting contigs is not straightforward since one metric is not sufficient to classify an assembly, particularly with complex samples , 9. Despas QUAST , being uas QUAST , a modifThe use of mock communities, with known composition, abundance, and genomic information, provides a ground truth against which the success of the assembly of a complex sample can be evaluated. Furthermore, this can be done in circumstances in which the errors introduced by sample processing and the sequencing and associated methods used approximate, as much as possible, their effects in real samples. Such mock communities facilitate the identification of misassemblies, such as chimeric sequences generated from the improper combination of 2 distinct replicons, indels, or single-nucleotide variants improperly created by the assembler. On the other hand, the comparison of the performance of 2 assemblers is only possible if the input data are the same and if the same evaluation metrics are applied .de novo assembly software using defined mock communities. The results of LMAS are presented in an interactive HTML report where selected global and reference replicon-specific performance metrics can be explored. The mock communities can be provided by the user to better reflect the samples of interest. New assemblers can be added with minimal changes to the pipeline so that LMAS can be expanded to include novel algorithms as they are developed. The portability and ease of use of LMAS are intended to provide users with a continuous benchmarking platform to easily evaluate the performance of assemblers, in mock communities mimicking as closely as possible their samples of interest.To tackle these challenges, we developed LMAS (Last Metagenomic Assembler Standing), an automated workflow to enable the benchmarking of traditional and metagenomic prokaryotic de novo assembly software using defined mock communities  and relative to each of the replicons present in the sample (see Per reference metrics). In both, the tabular presentation in the reports allows the comparison of exact values between assemblers, and the interactive plots allow a more intuitive overview and easy exploration of results. In addition to the assembly success metrics, computational resource statistics are registered for each assembler  and the overall size of gaps in all reference replicons not covered by any contig. A more detailed description of all global metrics is available in For the computation of the reference-based metrics, only the filtered set (FS) contigs are considered, for each reference replicon in the sample. These contigs are the ones exceeding the user-defined minimum sequence length, filtered using BBTools (version 38.44). After this initial step, the contigs are mapped to the reference replicons with minimap2  , EMS (in silico generated evenly distributed with Illumina MiSeq error model), ERR2984773 , LNN , LHS , and ERR2935805   see . Detaileple) see . To evalMuricauda sp. ES.050) and the lowest only 0.1× coverage (Micromonospora coxensis)  >0.84 and 0.98, respectively  . This saoxensis) . For theectively . The 2 MBifidobacterium longum (ANIb = 0.95) possibly being the most challenging. It is worth noting that only draft genomes are available for 8 of the strains, including one of the B. longum subspecies  in this sample.To represent more realistic samples of a human microbiome study, the Gut-Mix-RR and Gut-Mix-HiLo standards, including 20 species known to be present in the human gut, were used as reference  [bspecies . The Rosde novo prokaryotic assemblers included in LMAS, 4 stand out as having an overall poor performance in the ZymoBIOMICS Microbial Community Standards dataset: ABySS, MetaHipmer2, minia, and VelvetOptimiser. Both ABySS and MetaHipmer2 performed inconsistently with differing resource requirements for the same sample in different runs, namely, runtime and memory allocation  and therefore a low N50   . AdditioDue to their poor performance discussed above, the following assemblers have not been included in subsequent analyses: ABySS, MetaHipmer2, minia, and VelvetOptimiser.de novo prokaryotic assemblers, all implementing multiple k-mer dBg algorithms. As observed in Fig. After excluding the poorly performing assemblers, LMAS includes 3 genomic  and 4 labeled as metagenomic-specific  Marinobacter replicons  were successfully recovered by all genomic and metagenomic assemblers with >0.87 breadth of coverage  were also recovered to a breadth of coverage >0.95 by most assemblers, except the genomic assemblers SKESA and Unicycler  with a breadth of coverage of >0.84 , despite their relatively modest (12.6%) difference in estimated coverage in the sample and very significant depth of coverage (>500×)  having more SNPs relative to the reference than the one with lower coverage (135×)  . Howevere (135×) . This inRoseburiaspp., the replicon with the lowest coverage  had a consistently higher number of SNPs, with the exception of the SKESA assemblies, where the opposite was observed. This is similar to what was observed for the Halomonas replicons in the BMock12 community standard.For the Gut-Mix-RR and Gut-HiLo-RR mock communities, the same pattern was observed as with the other mock communities, with the differences between the metagenomic and genomic assemblers being attenuated after filtering for contigs <1 kbp . ParticuSeveral factors contribute to suboptimal performance of the assembly process, including DNA isolation and library preparation protocol; sequencing technology, depth, and read length; and possible contamination and inherent characteristics of the sample composition.Lactobacillus fermentum is the least represented genome (92.2%–94.9%). Most replicon sequences are recovered in <100 contigs, except for Pseudomonas aeruginosa, Escherichia coli, and Salmonella enterica, and not considering IDBA-UB, which frequently produces a larger number of contigs when compared to other assemblers. However, in other mock samples, this worse performance of IDBA-UD in terms of number of contigs is not so clear. The absolute values of other metrics of assembly quality, such as LSA, misassembly events, or uncalled bases, are also different between bacterial genomes , variations in the assembly metrics were observed Fig. . For all genomes . The facl genome . SimilarIn the BMock12 and the Gut-Mix samples, which have pairs of much more similar replicons, it is true that the closely related replicons do have higher numbers of contigs. However, it is possible that the properties of the individual replicons also have an impact on the number of contigs generated by the assemblers . AnotherThe Pls metric, which measures the error rate of a contig relative to the reference, shows that for every replicon, longer contigs have higher Pls Fig. . This coE. faecalis and IDBA-UD, MEGAHIT, and metaSPADES for L. monocytogenes, but such performance is not consistent for all reference replicons. For instance, SKESA fails to assemble 2 large regions of the S. enterica genome that all other assemblers successfully cover.Some genomic regions in several replicons are consistently a challenge for all assemblers. As observed in Fig. Halomonas replicons present a very dissimilar gap pattern, with a high number of gaps  distributed throughout the replicon sequence, which possibly reflects the difficulty of assembling closely related replicons in the same sample.For the BMock12 community standard, the same pattern of consistency of difficult regions across replicons can be observed for all replicons, with the exception of the lowest abundance replicons see . InteresThe logarithmically distributed samples  of the ZymoBIOMICS community standard dataset showed greater variation in the assembly success metrics than the evenly distributed samples , reflectMicromonospora coxensis, 2,623,620,609, 0.02× coverage) fails to assemble in all tools  and Micromonospora echinofusca  fail to assemble with SKESA and Unicycler, and the Propionibacteriaceae replicon  fails to be assembled with SKESA. In the Gut-Mix-RR standards, there are several replicons with <1× to 20× depth of coverage. Significant breath of coverage (>0.7) was obtained with the contigs created by most assemblers, including successful assemblies of some of the replicons by SKESA and Unicycler that had failed with higher-coverage replicons in the BMock12 standard. Taking together these data and that of the ZymoBIOMICS standard, these suggest that it is hard to establish a universal breakpoint at which each of the assemblers is able to generate high breadth of coverage contigs, with the actual genome of interest and the composition of the sample possibly playing a role. Nevertheless, coverages >15× result in high breadth of coverage contigs, albeit in the lower range with many contigs and a significant number of SNPs.For the BMock12 community standard, the very low replicon abundance  or of artificially generated reads simulating their samples of interest to guide their choice of assembler. LMAS was developed to be an easy-to-use and flexible tool for this purpose. From the results that we obtained with various mock communities, the following assemblers performed consistently well : MEGAHIT, metaSPAdes, SKESA, SPAdes, and Unicycler. From our assessment, we conclude that these assemblers are the most likely candidates to perform well in other complex samples.The choice of LMAS was built with modularity and containerization as keystones, leveraging the parallelization of processes and guaranteeing reproducibility across platforms. The modular design allows for new assemblers to be easily added and existing assemblers to be easily updated, allowing LMAS to function as a continuous benchmarking platform and ensuring its future relevance as improvements in assembly software are proposed. LMAS will also support evaluating the gains of any cumulative improvements to existing assemblers using the same benchmark set adapted to a specific project or goal. Such reproducibility, capacity to easily add assemblers of interest not included in the current version, and flexibility for future extensions are important principles in computational method benchmarking. Moreover, in addition, by lowering the barriers to perform comparisons between assemblers, LMAS will encourage users to compare software performance against mock communities of special interest, depending on their operational focus.The interactive report provides an intuitive platform for data exploration, allowing the user to easily sift through global and reference-specific performance metrics for each sample, as well as providing information on the assemblers executed to allow traceability of the results. Producing an extensive, metric-rich report allows users interested in different aspects of assembler performance to make informed decisions, particularly when choosing among the top-performing assemblers, which show only minor differences.LMAS applies several well-known assembly metrics and proposes 2 more: LSA, which represents the fraction of the longest single alignment between a contig and the reference, and Pls, a scoring function based on the identity of each aligned contig to the reference replicon. The entire set of assembly quality metrics used in LMAS allows not only the assessment of quality based on statistics inherent to a set of assembled contigs but also a comparison to a ground truth provided using samples of known composition and reference sequences. The LMAS report provides an interactive and intuitive platform for the exploration of these results, allowing users to easily test assemblers in mock samples with species composition and distribution relevant for their own studies.de novo assembly process, LMAS is the only software-integrating assembly and its evaluation into a single pipeline, guaranteeing the same conditions are met for all tools. With LMAS, it is now possible to continuously evaluate which de novo assembler produces the most relevant results for a given community of interest. The LMAS workflow is open source, and its code and documentation are available at https://github.com/B-UMMI/LMAS and https://lmas.readthedocs.io/, respectively.Although computationally intensive due to the complex nature of the Project name: LMASProject home page:https://github.com/B-UMMI/LMASOperating system(s): UNIX-like systems.Programming languages: Nextflow, Python, Bash, JavascriptOther requirements: Java version 8 or highest. Docker/Singularity/ShifterLicense: GNU GPL v3RRID: SCR_022251Supplementary Fig. S1. LMAS misassembly classification. Misassembled contigs are classified into 6 main categories: chimera, insertion, deletion, inversion, rearrangement, translocation, and duplication, according to the mapping orientation, the distance between blocks in the contig, and the mapping coordinates in the reference replicon. If a contig is classified as being chimeric, no further classification is performed. The other categories are classified independently of each other, with combinations being possible, to better reflect the differences in comparison to the reference. If a contig is broken into multiple sequence blocks but fails to be classified in any of the previous categories, it is reported as being inconsistent.Supplementary Fig. S2. Computational resources used by each assembler for the evenly and logarithmically distributed samples. Each plot describes the distribution of resource consumption for 3 LMAS runs for the ZymoBIOMICS Microbial Community Standards dataset for the following metrics: (A) CPU/hour, (B) maximum memory in GB, (C) data written to disk in GB, (D) data read from dist in GB, and (E) runtime in hours. The mean for all samples and all assemblers is indicated in red. The samples are indicated as follows: ENN: dark blue, EMS: teal, ERR2984773: green, LNN: light green, LHS: yellow, ERR2935805: light orange.Supplementary Fig. S3. Performance per reference of genomic and metagenomic assemblers for the evenly distributed samples in the ZymoBIOMICS Microbial Community Standards dataset. For each sample in the dataset and for the 3 runs, the best and worst scores for each assembler category were selected: genomic (in blue) and metagenomic (in red). The results for each global assembly metric were normalized, with 1 representing the best result and 0 the worst.Supplementary Fig. S4. Assembler performance per reference for the ZymoBIOMICS Microbial Community Standards dataset for sample ENN. The best score for each assembler was selected for 3 LMAS runs. The results for each global assembly metric were normalized, with 1 representing the best result and 0 the worst. The following assemblers are represented: GATBMiniaPipeline: dark blue, IDBA-UD: light blue, MEGAHIT: dark green, metaSPAdes: light green, SKESA: yellow, SPAdes: orange, Unicycler: red.Supplementary Fig. S5. Assembler performance per reference for the ZymoBIOMICS Microbial Community Standards dataset for sample EMS. The best score for each assembler was selected for 3 LMAS runs. The results for each global assembly metric were normalized, with 1 representing the best result and 0 the worst. The following assemblers are represented: GATBMiniaPipeline: dark blue, IDBA-UD: light blue, MEGAHIT: dark green, metaSPAdes: light green, SKESA: yellow, SPAdes: orange, Unicycler: red.Supplementary Fig. S6. Assembler performance per reference for the ZymoBIOMICS Microbial Community Standards dataset for sample ERR2984773. The best score for each assembler was selected for 3 LMAS runs. The results for each global assembly metric were normalized, with 1 representing the best result and 0 the worst. The following assemblers are represented: GATBMiniaPipeline: dark blue, IDBA-UD: light blue, MEGAHIT: dark green, metaSPAdes: light green, SKESA: yellow, SPAdes: orange, Unicycler: red.Supplementary Fig. S7. Genome fragmentation for each reference replicon of the ZimoBIOMICS community standards dataset for the logarithmically distributed samples. Genome fragmentation for the 3 LMAS runs is represented by the number of contigs and breadth of coverage of the reference per assembler for the logarithmically distributed samples: LNN , LHS , and ERR2935805 . Each assembler is identified with the following color scheme—dark blue: Unicycler, light blue: SPAdes, dark green: SKESA, light green: metaSPAdes, yellow: MEGAHIT, orange: IDBA-UD, red: GATBMiniaPipeline.Supplementary Fig. S8. Genome fragmentation for each reference replicon of the BMock12 community standards dataset sample. Genome fragmentation is represented by the number of contigs and breadth of coverage of the reference per assembler. Each assembler is identified with the following color scheme—dark blue: Unicycler, light blue: SPAdes, dark green: SKESA, light green: metaSPAdes, yellow: MEGAHIT, orange: IDBA-UD, red: GATBMiniaPipeline. Each reference replicon is identified by its IMG Taxon ID: 2615840527, Muricauda sp.; 2615840533, Thioclava sp.; 2615840601, Cohaesibacter sp.; 2615840646,Propionibacteriaceae bacterium; 2615840697, Marinobacter sp. LV10R510-8; 2616644829, Marinobacter. sp LV10MA510-1; 2617270709, Psychrobacter sp.; 2623620557, Micromonospora echinaurantiaca; 2623620567, Micromonospora echinofusca; 2623620609, Micromonospora coxensis; 2623620617, Halomonas sp. HL-4; and 2623620618, Halomonas sp. HL-93.Supplementary Fig. S9. Location of gaps in comparison to the reference sequence, per assembler, for each reference replicon of the BMock12 community standards datasets. The resulting plot contains the gaps obtained for GATBMiniaPipeline, IDBA-UD, MEGAHIT, metaSPAdes, SKESA, SPAdes, and Unicycler assemblers.Supplementary Table S1. Tools available for the de novo assembly of prokaryotic genomes. For each tool, its publication is indicated, if available, as well as the assembly algorithm implemented if it was developed explicitly to handle metagenomic datasets. The tools are ordered by the date of the last update, with the source code indicated when available. The tools incorporated in LMAS are indicated as such.Supplementary Table S2. Comparison of metrics and features of LMAS with QUAST and MetaQUAST.Supplementary Table S3. The ZymoBIOMICS Microbial Community Standards datasets. Set of raw sequence reads used as input in LMAS of mock communities with an even and logarithmic distribution of species, from real sequencing runs and simulated read datasets, with and without error, matching the intended distribution of species in ZymoBIOMICS Microbial Community Standards.Supplementary Table S4. Microbial composition of the ZymoBIOMICS Microbial Community Standards dataset with even and logarithmic distribution of species. Theoretical microbial composition of the standards and the corresponding number of reads generated for each replicon.Supplementary Table S5. Global quality metrics variation in 3 LMAS runs for sample ENN per assembler. The average calculated for all samples in the dataset for the 3 independent LMAS runs, followed by the minimum and maximum values obtained, is presented for each metric for each assembler.Supplementary Table S6. Global quality metrics variation in 3 LMAS runs for sample EMS per assembler. The average calculated for all samples in the dataset for the 3 independent LMAS runs, followed by the minimum and maximum values obtained, is presented for each metric for each assembler.Supplementary Table S7. Global quality metrics variation in 3 LMAS runs for sample ERR2984773 per assembler. The average calculated for all samples in the dataset for the 3 independent LMAS runs, followed by the minimum and maximum values obtained, is presented for each metric for each assembler.Supplementary Table S8. Global quality metrics variation in 3 LMAS runs for sample LNN per assembler. The average calculated for all samples in the dataset for the 3 independent LMAS runs, followed by the minimum and maximum values obtained, is presented for each metric for each assembler.Supplementary Table S9. Global quality metrics variation in 3 LMAS runs for sample LHS per assembler. The average calculated for all samples in the dataset for the 3 independent LMAS runs, followed by the minimum and maximum values obtained, is presented for each metric for each assembler.Supplementary Table S10. Global quality metrics variation in 3 LMAS runs for sample ERR2935805 per assembler. The average calculated for all samples in the dataset for the 3 independent LMAS runs, followed by the minimum and maximum values obtained, is presented for each metric for each assembler.Supplementary Table S11. Inconsistent contigs produced by the assemblers in 3 LMAS runs. For each assembler, the total number of contigs produced over the 3 runs of the LMAS workflow is indicated, as well as the contigs present in only 2 runs and a single run.Supplementary Table S12. Global assembly metrics for single and multiple k-mer dBg assemblers. The median and the minimum and maximum values obtained are presented for each metric for all samples in 3 runs of LMAS. Single k-mer bBg assemblers: ABySS and minia. Multiple k-mer bBg assembler: GATBMiniaPipeline, IDBA-UD, MEGAHIT, MetaHipMer2, metaSPAdes, SKESA, SPAdes, Unicycler, and VelverOptimiser.Supplementary Table S13. Global assembly metrics for genomic and metagenomic multiple k-mer dBg assemblers. The median and the minimum and maximum values obtained are presented for each metric for all samples in 3 runs of LMAS. Genomic assemblers: SKESA, SPAdes, and Unicycler. Metagenomic assemblers: GATBMiniaPipeline, IDBA-UD, MEGAHIT, and metaSPAdes.Supplementary Table S14. Per reference quality metrics variation in 3 LMAS runs for sample ENN per assembler of the ZymoBIOMICS Microbial Community Standards dataset. The average calculated for all samples in the dataset for the 3 independent LMAS runs, followed by the minimum and maximum values obtained, is presented for each metric for each assembler.Supplementary Table S15. Per reference quality metrics variation in 3 LMAS runs for sample EMS per assembler of the ZymoBIOMICS Microbial Community Standards dataset. The average calculated for all samples in the dataset for the 3 independent LMAS runs, followed by the minimum and maximum values obtained, is presented for each metric for each assembler.Supplementary Table S16. Per reference quality metrics variation in 3 LMAS runs for sample ERR2984773 per assembler of the ZymoBIOMICS Microbial Community Standards dataset. The average calculated for all samples in the dataset for the 3 independent LMAS runs, followed by the minimum and maximum values obtained, is presented for each metric for each assembler.Supplementary Table S17. Inconsistent gaps produced by the assemblers in 3 LMAS runs. For each assembler, the total number of gaps consistently produced in relation to the reference replicons over the 3 runs of the LMAS workflow is indicated, as well as gaps present in only 2 runs and a single run.Supplementary Table S18. Annotation of consistent gaps produced by the assemblers in 3 LMAS runs.Supplementary Table S19. Number of transfer RNA and ribosomal RNA coding sequencing, as well as mobile elements in ZymoBIOMICS Microbial Community Standards reference replicons. The average calculated for all samples in the dataset for the 3 independent LMAS runs, followed by the minimum and maximum values obtained, is presented for each metric for each assembler.Supplementary Table S20. Taxonomic classification of the ZymoBIOMICS Microbial Community Standards dataset. The classification was performed with Kraken2, using the Standard Database. The results are presented as the percentage of classified reads for the 8 bacterial species in the community, as well as unclassified reads and the group of reads that are classified as species not contained in the community standard.Supplementary Table S21. Global assembly metrics for dBg assemblers with single and multiple k-mer algorithms.Supplementary Table S22. Reference-based quality metrics in 3 LMAS runs for the ZimoBIOMICS community standards dataset.Supplementary Table S23. Pairwise comparisons of the ZymoBIOMICS Microbial Community Standards reference replicons. All pairwise comparisons among the set of genomes were conducted using Average Nucleotide Identity through BLAST as a proxy for DNA–DNA hybridization.Supplementary Table S24. Microbial composition of the BMock12 microbial community standard dataset.Supplementary Table S25. Pairwise comparisons of the BMock12 microbial community standard reference replicons.Supplementary Table S26. Global quality metrics for the BMock12 sample SRX4901583 per assembler.Supplementary Table S27. Per reference quality metrics for the BMock12 sample SRX4901583 per assembler.Supplementary Table S28. Microbial composition of the Gut-Mix-RR microbial community standard dataset.giac122_GIGA-D-22-00108_Original_SubmissionClick here for additional data file.giac122_GIGA-D-22-00108_Revision_1Click here for additional data file.giac122_Response_to_Reviewer_Comments_Original_SubmissionClick here for additional data file.giac122_Reviewer_1_Report_Original_SubmissionAdrian Fritz -- 6/7/2022 ReviewedClick here for additional data file.giac122_Reviewer_1_Report_Revision_1Adrian Fritz -- 10/6/2022 ReviewedClick here for additional data file.giac122_Reviewer_2_Report_Original_SubmissionRayan Chikhi -- 6/10/2022 ReviewedClick here for additional data file.giac122_Reviewer_2_Report_Revision_1Rayan Chikhi -- 10/25/2022 ReviewedClick here for additional data file.giac122_Supplemental_FilesClick here for additional data file.bp: base pair; cgMLST: core-genome multilocus sequence typing; dBg: de Bruijn graphs; FS: filtered set; GB: gigabytes; HPC: high-performance computing cluster; LMAS: Last Metagenomic Assembler Standing; LSA: longest single alignment; OLC: overlap-layout-consensus; Pls: Phred-like score; SNP: single-nucleotide polymorphism.The datasets analyzed during the current study are available in the Zenodo repository, under . All suphttps://github.com/B-UMMI/LMAS) is also available via the GigaScience database, GigaDB [An archival copy of the GitHub repository .C.I.M. and M.R. designed the workflow. C.I.M. implemented and optimized the workflow, created the Docker containers, generated mock shotgun metagenomics data used to test and validate the workflow, contributed to the development of the HTML report, and analyzed the data. C.I.M. and M.R. wrote the manuscript. P.V.C. contributed to the development of the HTML report. M.R., J.A.C., Y.M., and J.M.G critically revised the manuscript. All authors read, commented on, and approved the final manuscript.


Three different Flow Cytometry datasets consisting of diagnostic samples of either peripheral blood (pB) or bone marrow (BM) from patients without any sign of bone marrow disease at two different health care centers are provided. In Flow Cytometry, each cell rapidly passes through a laser beam one by one, and two light scatter, and eight surface parameters of more than 100.000 cells are measured per sample of each patient. The technology swiftly characterizes cells of the immune system at the single-cell level based on antigens presented on the cell surface that are targeted by a set of fluorochrome-conjugated antibodies. The first dataset consists of N=14 sample files measured in Marburg and the second dataset of N=44 data files measured in Dresden, of which half are BM samples and half are pB samples. The third dataset contains N=25 healthy bone marrow samples and N=25 leukemia bone marrow samples measured in Marburg.The data has been scaled to log between zero and six and used to identify cell populations that are simultaneously meaningful to the clinician and relevant to the distinction of pB vs BM, and BM vs leukemia. Explainable artificial intelligence methods should distinguish these samples and provide meaningful explanations for the classification without taking more than several hours to compute their results. The data described in this article are available in Mendeley Data  However, it is an essential question whether bone marrow (BM) samples of patients with hematological diseases are diluted with peripheral blood (pB) in order to assess Minimal Residual Disease status (MRD) in the correct manner c.f. .•Researchers can use the data in two difficulty levels for three types of assessment of explainable AIs (XAI) or supervised interpretable machine learning methods: performance, interpretability, and processing time.•Performance e.g. : The two•Interpretability e.g. : Explana•Processing time: Due to the large number of cases within a clinical setting, the computations of any algorithm have to finish within an acceptable time (hours) on standard personal computers.•Researchers can use cases of each sample to benchmark unsupervised machine learning methods c.f.  because 1Data is provided as text files separated by tabular. In addition, there are two overview files per dataset storing the classification information of the sample , CD34 FITC (Fluoresceinisothiocyanate) (8G12), CD13 PE (Phycoerythrin) (L138), CD7 PerCP-Cy5.5 (Peridinin chlorophyll protein-Cyanine5.5) (M-T701) CD56 APC (Allophycocyanin) (NCAM16.2), CD33 PE-Cy7 (Phycoerythrin Cyanine7) (D3HL60.251), CD117 AlexaFluor750 (104D2D1), HLA-DR Pacific blue (Immu357), and CD45 Krome Orange (J33).https://github.com/Mthrun/dbt.DataIO) with extensive documentation is provided for the programming language R in order to read the data. Alternatively, any *.csv reading function can be used.Each file can be read-in with any text editor, Microsoft Excel, or any programming language. In addition, the R package on GitHub  for the Marburg data, and BD FACSCanto II™, BD Biosciences (Heidelberg) for the Dresden data, for which ten features were measured. Both measure forward and side light scatter (FS and SS) and use the same panel of fluorescent antibody clones against the same antigens CD34, CD13, CD7, CD56, CD33, CD117, HLA-DR, and CD45. Hence, an event or case consists of these ten features measured as optical characteristics for a cell in the flow cytometer as follows (c.f. The Marburg dataset consists of n = 7 data files (samples) containing event measures from peripheral blood (BP) and n = 7 data files for bone marrow (BM). The next dataset from Marburg consists of N=25 healthy bone marrow samples and N=25 leukemia bone marrow samples.The Dresden dataset comprises of n = 22 sample files for peripheral blood and n = 22 samples for bone marrow. Each sample file containes more between 130.000 and 880.000 events. The samples described as BM represent the first ml of aspiration into the first syringe.Additionally, overall samples of the Marburg data with Data was generated during typical diagnostic procedures and data processing was according to the local ethics committees in Marburg and Dresden (Vote 91/20). Data is anonymized and does not contain any personal information or images of patients.Michael Thrun, Alfred Ultsch: Conceptualization, Methodology; Jörg Hoffmann, Maximilian Röhnert, Malte von Bonin, Uta Oelschlägel, Cornelia Brendel: Data curation; Michael Thrun: Writing – original draft preparation; Cornelia Brendel: Writing – review & editing.The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.


Single-cell RNA-seq overcomes the shortcomings of conventional transcriptome sequencing technology and could provide a powerful tool for distinguishing the transcriptome characteristics of various cell types in biological tissues, and comprehensively revealing the heterogeneity of gene expression between cells. Many Intelligent Computing methods have been presented to infer gene regulatory network (GRN) with single-cell RNA-seq data. In this paper, we investigate the performances of seven classifiers including support vector machine (SVM), random forest (RF), Naive Bayesian (NB), GBDT, logical regression (LR), decision tree (DT) and K-Nearest Neighbor (KNN) for solving the binary classification problems of GRN inference with single-cell RNA-seq data (Single_cell_GRN). In SVM, three different kernel functions  are utilized, respectively. Three real single-cell RNA-seq datasets from mouse and human are utilized. The experiment results prove that in most cases supervised learning methods  perform better than unsupervised learning method (GENIE3) in terms of AUC. SVM, RF and KNN have the better performances than other four classifiers. In SVM, linear and polynomial kernels are more fit to model single-cell RNA-seq data. Human diseases, especially polygenetic genetic diseases, mainly including heart disease, hypertension, diabetes, asthma and cancer, are caused by the interaction of multiple gene loci and environmental factors –4. ThereFor gene regulatory network modeling, the existing learning methods could be divided into two categories: supervised learning and unsupervised learning . SuperviThe gene expression data used in supervised learning algorithms are all obtained by traditional sequencing technology, such as DNA microarray. However, biological tissue is composed of a variety of heterogeneous cells, and the differences between single cells may have a profound impact on the functions of multicellular organisms. In recent years, single-cell RNA-seq technology has been developed, which can be used for unbiased, repeatable, high-resolution and high-throughput transcription analysis of single cells –20. CompIn order to investigate the performances of supervised learning methods for GRN inference with single-cell RNA-seq data, we proposed a hybrid supervised learning method (Single_cell_GRN), which utilizes SVM, random forest, Naive Bayesian (NB), GBDT, logical regression (LR), decision tree (DT) and K-Nearest Neighbor (KNN) to infer gene regulatory network separately. For SVM, linear kernel, polynomial kernel and radial basis function are utilized and investigated. Three real single-cell RNA-seq datasets from mouse and human are utilized to test the supervised learning methods.Support vector machine is a kind of machine learning method based on statistical theory, which was proposed by Vapnik . It is mThe mechanism of SVM is to search the optimal hyperplane to meet the classification requirements. Two restricted conditions need to be considered, such as classification accuracy and maximizing the blank area on both sides of the hyperplane. So the learning process of SVM is an optimization problem.xi, yi), i = 1,2,…,N, N is the number of data, xi is feature vector, and yi is classification label  . Hyperplane is labeled as (w·x) + b = 0) w and b are coefficients and deviation term). The optimal hyperplane problem is constructed as follows.Give the training dataset , which are defined as followed.N trees could create N classification results. RF integrates all the classification results by voting method and specifies the category with the most voting times as the final output. The principle of RF is given as follows.Random forest (RF) is a flexible and easy-to-use machine learning algorithm . CompareM samples are randomly selected from the sample set by bootstrap algorithm. For each sample, K features are selected randomly from all attributes. According to the selected K features, a decision tree is established.Firstly N times in order to obtain N decision trees.Repeat step (1) N decision trees could get N classification results.Input variables are given to each decision tree, which could output a result. Calculate the number of votes of all classes and select the classification result with the highest number of votes as the final category.Naive Bayesian is built on Bayes' theorem and is a typical generative learning method . The maiBayesian theorem is expressed as follows:c denotes class, P(c) represents a priori probability, P(c | x) indicates a posteriori probability, P(x | c) denotes the class conditional probability and P(x) is the edge probability of x.where Based on the assumption of attribute conditional independence, Eqs.  can be rd is the number of attributes and xi is the value on the i-th attribute.where Because the denominator in Eqs. GBDT is a type of ensemble learning method  and an aT= {,,…, }, and the loss function is L) , where xi represents the feature vector and y is the label. The main flowchart of the algorithm is indicated as follows.The training dataset is (1) Initialize weak learner.m= 1,2,…,M, where M is the number of iterations.Calculate the negative gradient of the loss function in the current tree, the residual is written as:Fit a regression tree to the target For The strong learner in this iteration is obtained.(2) For (3) After all the iterations, the strong learner is obtained.Logistic regression  is an imx is the input vector, θ is the parameters to be solved, and y represents the prediction result of linear regression, the linear regression model is given as follows.(1) Firstly, assuming that (2) In LR, the logic function is Sigmoid function, which is defined as follows.According to Sigmoid function, the output of model is constructed as follows.The parameter Decision tree is a machine learning method used to solve classification problems . It is aIn the generation process of decision tree, the key step is the measurement of feature selection. The feature selection is based on the principle that the samples contained in the branch nodes belong to the same category as much as possible. At present, there are three main algorithms for the construction of a decision tree, namely ID3, C4.5 and CART. In this paper, we select CART algorithm , 52.Gini index is utilized to select the features, determine the optimal partition points and measure the purity of dataset D, which is defined as follows.CART algorithm utilizes binary recursive segment method to divide the sample set into two sub sample sets, which contains feature selection and tree pruning. p(xi) is the probability of category xi , and n is the number of categories in D. Gini (D) reflects the probability that the category labels of two samples are inconsistent, which are randomly selected from dataset D. Therefore, the smaller the Gini (D) is, the higher the purity of the dataset D is.where K training samples of the known categories that are most similar to the test sample based on some distance measures in the sample space. Then judge the category of the sample based on the information of K neighbors. The main algorithm flowchart of KNN is given as follows.K-Nearest Neighbor is a commonly used machine learning algorithm , 54. It (1) Build the training sample set and calculate the distances between the test sample and the training samples based on the distance measurement.(2) Sort the training samples in ascending order according to the distances.K training samples closest to the test one as the K neighbors of the test sample.(3) Select the K neighbors, and select the category with the highest frequency as the category of the test sample.(4) Count the category frequencies of NTF. With the regulatory networks verified by biology experiments from the well-known databases, for each regulatory factor i, all the target genes set can be divided into two categories. The target genes regulated by the regulator factor i are marked as positive gene set, while the target genes not regulated by the regulator factor i are marked as negative gene set. The single-cell RNA-seq data of two kinds of gene sets are constructed. K-fold cross validation method is utilized to divide the training and testing datasets in order infer the regulatory relationships between all target genes and regulatory factors. For each classification problem, different classification methods are selected. If the number of positive samples is zero, the sample is classified as negative, which reveal that there are no regulatory relationships between the regulator factor i and the targets. Otherwise SVM, RF, NB, GBDT, LR, DT and KNN are utilized, respectively. When the regulations of all regulatory factors have been inferred, the algorithm stop; otherwise repeat the above process. The regulations of all TFs are integrated in order to obtain the overall GRN. The flowchart is depicted in Fig. For the inference of gene regulatory networks, the complex regulatory relationships among genes are identified, which could be evolved to two-class problems. Single-cell RNA-seq data and the corresponding regulatory relationships between genes are collected from the public databases. Count up the number of TF as Three real single-cell RNA-seq datasets are utilized to test our methods. The first dataset is derived from primitive endoderm cells differentiated from mouse embryonic stem cells, which includes 456 cells (Data1) . The secReceiver Operating Characteristic (ROC) curve considers true positive rate (TPR) and false positive rate (FPR), and could accurately reflect the relationship between TPR and FPR of a learner, which is a comprehensive manner to evaluate model sensitivity and specificity. TPR denotes the proportion of the inferred real regulatory relationships in all real regulations. FPR represents the proportion of the inferred false-positive regulatory relationships in all the true non-regulations. Area Under ROC Curve (AUC) is the area covered by ROC curve, which could reflect the performance of the learner more intuitively. In this part, ROC curves and AUC are utilized to evaluate our methods.In this part SVM with different kernel functions (linear kernel (SVM + linear), polynomial kernel (SVM + poly) and radial basis function (SVM + rbf)), RF, NB, GBDT, LR, DT and KNN are utilized to infer GRN with three real single-cell RNA-seq data, respectively. Leave-One-Out Cross Validation (LOOCV) is utilized to classify the unknown regulatory relationships. To better evaluate the performance of supervised learning algorithms, the famous unsupervised learning method GENIE3) is also utilized to infer the same GRNs, which has the highest performance in the DREAM3 Challenge. The ROC curves and the corresponding AUC values of ten methods are depicted in Figs.  is also Compared with the transcriptome data by traditional sequencing technologies, sing-cell RNA-seq data has its own internal characteristics. In this part, we compare the performances of SVM methods with different kernel functions in our proposed method. We also compare SVM with RF, NB, GBDT, DT, LR and KNN. threefold cross validation, fivefold cross validation and tenfold cross validation are utilized and the AUC results and ROC curves of nine methods with three datasets are depicted in Figs. In order to compare the performances of different supervised learning methods for GRN inference obviously, we rank these nine methods according to the performances of LOOCV Figs. , 3 and 4In this paper, a hybrid supervised learning method based on SVM, RF, NB, GBDT, LR, DT and KNN is utilized to solve the binary classification problem of gene regulatory network inference. In SVM, three different kernel functions  are also utilized. Three real single-cell RNA-seq datasets from mouse and human are utilized to test these supervised learning methods. Nine supervised learning methods and one unsupervised learning method are utilized. With Data1, Data2 and Data3, in terms of AUC, SVM, KNN and RF are 0.5%-14%, 2.1%-30.3% and 2.9%-22.1% higher than other nine methods, respectively. The inference results prove that in most cases supervised learning methods  have the better ROC and AUC performances than unsupervised learning method (GENIE3).We also compare the performances of SVM methods with different kernel functions, RF, NB, GBDT, LR, DT and KNN further. threefold cross validation, fivefold cross validation and tenfold cross validation are utilized. The results show that in most cases SVM, RF and KNN methods have the best performances among nine classifiers. Among SVM methods with three kernel functions, SVM methods with linear kernel and polynomial kernel have the better performance than SVM with rbf kernel, which prove that linear and polynomial functions are fitter to model single-cell RNA-seq data than rbf kernel.


Addressing this issue is challenging, because all of the above variables can dramatically influence gene expression signals and, therefore, cause a plethora of peculiar features in the transcriptomic profiles. Millions of transcriptomic profiles were obtained and deposited in public databases of which the usefulness is however strongly limited due to the inter-comparison issues; (2) Methods: Dozens of methods and software packages that can be generally classified as either flexible or predefined format harmonizers have been proposed, but none has become to the date the gold standard for unification of this type of Big Data; (3) Results: However, recent developments evidence that platform/protocol/batch bias can be efficiently reduced not only for the comparisons of limited transcriptomic datasets. Instead, instruments were proposed for transforming gene expression profiles into the universal, uniformly shaped format that can support multiple inter-comparisons for reasonable calculation costs. This forms a basement for universal indexing of all or most of all types of RNA sequencing and microarray hybridization profiles; (4) Conclusions: In this paper, we attempted to overview the landscape of modern approaches and methods in transcriptomic harmonization and focused on the practical aspects of their application.  The digital ocean of whole-transcriptome gene expression profiles has flooded since the early 2000s when the first generation of robust and reproducible mRNA microarray hybridization (MH) techniques was introduced into the routine laboratory practice ,2,3,4. TShortly after the critical mass of gene expression profiles has accumulated, the following two conceptual problems with the data analysis were recognized. First, poor technical compatibility of the expression profiles is obtained using different experimental platforms/equipment, protocols, and reagents ,19,20,21® PrimeView™ Human Gene Expression Array (GPL16043). The compromised compatibility of gene expression profiles obtained using different platforms and protocols was experimentally explored in the international projects MAQC (for MH) and SEQC (for RNA sequencing). Both MAQC ,18,19 anThe MAQC and SEQC projects investigated the correlations between the transcriptome profiles of the same biological type, yet obtained using the different experimental platforms. Although these correlations were high ,18,19,27As the reaction of the scientific community, a bunch of first-generation harmonization/normalization methods was generated in the first decade of the 21st century, aimed at the standardization of multi-platform expression profiles using specific algorithms. These methods were mostly trained on the different types of MH gene expression data and could dynamically transform gene profiles into a flexible yet inter-comparable form . The folFurther approaches were largely influenced by the coming era of routine next-generation sequencing (NGS) of mRNA (RNA sequencing or RNAseq) that has started roughly in the second decade of this century. Nowadays, RNAseq has become the gold standard and the basic tool for transcriptomic profiling ,48,49,50For the RNAseq data, a method called The Differential gene Expression in Sequencing, DESeq /DESeq2 ,60,61 baIn this review, we classified available intra- and cross-platform harmonization methods of transcriptomic profiles and compared their performance characteristics. Finally, we also included practical recommendations that may guide the reader to select optimal method depending on a specific task. (1)(a)Those using ranking of expression levels and setting the output levels according to the averaged values, such as QN , Feature(b)Those using piecewise rescaling of log-expression levels according to the mean/median values over distinct genes and samples, such as Column Sample (CS), Median-Centered (MC) , and AnaMethods based on statistical transformations :(2)(a)Those using negative binomial distribution, such as the DESeq /DESeq2 ,60,61;(b)Those using log-normal distribution with either covariance analysis , or with(c)Those using Dirichlet and gamma distributions as for the method PLatform-Independent Latent Dirichlet Allocation (PLIDA) ;(d)Those using the empirical superposition of conditional probabilistic (Bayesian) models that describe the generalized-type distribution as for the method applied for the comparison of the MH, NGS, microRNA, and DNA methylation data ,70;(e)Those using the Least Absolute Shrinkage and Selection Operator (LASSO) regression models ;Methods using regression and/or maximum likelihood models for validation of predefined statistical hypotheses:(3)(a)Those using piecewise linear interpolations in the log-expression space, such as Cross-Platform Normalization (XPN) ;(b)Those using piecewise cubic interpolations in the log-expression space, such CuBlock .Methods finding similar clusters in gene expression matrices of the datasets under normalization and then using iterative corrections to fit each cluster as close as possible to the target model:(4)(a)Those using the linear support vector machine (SVM) ML method, such as Distance-Weighted Discrimination (DWD) ,41,42;(b)Those using quantile-based regression models for data transfer from source to target datasets, such as Training Distribution Machine (TDM) .Methods utilizing machine learning (ML) to find and artificially remove dissimilarities between datasets to be normalized:Different harmonization methods are based on different algorithms aimed to suppress the platform bias and the batch effect. These algorithms may utilize different approaches to gene expression data processing and produce output data in different formats. Considering the mathematical apparatus, we proposed the following classification:Another important aspect that must be considered in this review is the format of output gene expression data generated by the harmonization techniques. Most of currently existing methods return the results in the flexible format. For the flexible normalization, the shape of the output transformed gene expression profiles is a variable that depends on all the profiles under harmonization. This has an important limitation that one cannot combine the output datasets generated after two or more acts of such harmonization. Even adding as few as just one transcriptional profile would require a new harmonization of the entire dataset. This clearly increases the calculation costs for large datasets that are being routinely updated.Taken together, these factors complicate the analysis of not only single gene expression levels, but also of higher order gene-based biomarkers such as gene signatures , moleculTo overcome these limitations, an alternative concept was formulated comprising conversion of a whole set of profiles under harmonization into a pre-defined output shape, e.g., into a shape of a preferred gene interrogating experimental platform. In such a paradigm, the harmonized output should look as if it would be obtained using a predefined gene expression platform. The examples of predefined-shape harmonization methods include Frozen Robust Microarray Analysis (fRMA) ,80,81,82More recently, we proposed a new family of uniformly shaped cross-platform harmonizers termed Shambhala ,33. HarmFurthermore, such harmonization may use different mathematical transforms as the engine to reshape the transcriptional profiles. The first version of Shambhala used the piecewise linear method XPN ,31 for pHarmonization of transcriptional profiles is a complex process that can distort functionally relevant features such as clustering and neighborhood on a dendrogram and fold-change of gene expression with relation to control samples. We listed in (1)(a)Correlation analysis for the gene expression profiles before and after harmonization ,31,33,34(b)Comparison of between- and within-class distances before and after harmonization ;First, different statistical criteria may be used to estimate the following endpoints:(2)(a)Logistic regression , used in(b)SVM , used in(c)Nearest shrunken centroids Prediction Analysis for Microarrays (PAM) , used inAlternatively, one may classify the samples according to gene expression data after normalization, involving various machine learning (ML) methods:The following quantitative metrics and methods may be applied to estimate the effect of harmonization:As a typical material for such normalization quality benchmarks, in many studies, the investigators used standardized reference samples, whose gene expression was interrogated with different equipment using different experimental protocols. Probably, the most important series of such cross-comparisons was performed within the Microarray Quality Control (MAQC) ,18,19 anThe MAQC and SEQC projects were focused on profiling the specific model human mRNA sample types. One was the commercial Stratagene universal human reference RNA mixture for all but brain tissues; another one was the commercial Ambion human brain reference mRNA, and the two remaining types were the mixtures of the Stratagene/Ambion samples in the ratios of 3:1 and 1:3, respectively. The quality assessment is based on the expectation that a perfect harmonization must support the similarity of gene expression profiles according to the biological nature of the sample rather than depending on the equipment and reagents used to interrogate gene expression. Thus, early approaches used visual inspection of the principal component analysis (PCA) plots and/or cluster dendrograms to assess the cross-platform harmonization benchmarks ,32,33,34SWM is WM metric for clustering according to classes corresponding to biological nature and PWM, according to the experimental platform used, may be used as a facile yet robust estimate of the harmonization quality. A higher R corresponds to a better quality, and vice versa [We recently proposed a new metric for the algorithmic cluster analysis of dendrograms ,95 callece versa .biomediDuring the last two decades, quantile normalization (QN)  and diffHowever, for the cross-platform harmonization, dozens of methods were developed for both MH and NGS types of gene expression data, but none of them was so far recognized as the gold standard. In fact, many, if not most, aspects of intra- and cross-platform normalization, such as incomparability of profiles obtained using different platforms, numerous methods for cross-platform normalization, and performance benchmarks for them, were studied in the first decade of the XXI century, in the co-called MH era. The advent of NGS, however, did not make this problem unimportant, at least because there is still a problem regarding how to harmonize old MH and new NGS data. Most of cross-platform normalization methods return the output data in the flexible format, which requires recalculation of all previously processed profiles when adding new data to the analysis. This may dramatically increase calculation time and costs which can grow exponentially with the increase of the sample size. Furthermore, some methods which show the best performance in cross-platform normalization tests , such asThus, the need for the predefined, uniformly-shaped output for data harmonization was formulated about a decade ago ,80,81,82With such a concept in mind, we obtained the best results with the cubic data transformation algorithm adopted from the CuBlock method  and builWe summarized our experience of using various harmonization methods for gene expression profiles in For the intra-platform harmonization of the MH data, QN  may seemIn case of cross-platform harmonization of two datasets with a comparable number of gene expression profiles, the best choice could be the XPN method ,31. WhenFinally, in the case of merging the MH and NGS expression datasets, or when merging of data from various platforms is needed, and the uniformly shaped (suitable for further intercomparisons) output format is required, then Shambhala-1  or ShahmTo our knowledge, Shambhala methods were the first gene expression harmonizers with a uniformly shaped output, which were applied to merge the RNAseq and MH profiles ,33. ThusHowever, one should keep in mind that Shambhala-1/2 approaches are algorithmically complex and, therefore, computational resource-demanding. Thus, parallel execution of the program code may be advantageous ,89.biom