Additionally, we found a putative reuterin biosynthetic gene cluster from a Streptococcus strain to be enriched but not exclusively found in healthy samples; metabolomic data from the same samples showed masses with fragmentation patterns consistent with (poly)acrolein, which is known to spontaneously form from the products of the reuterin pathway and has been previously shown to inhibit pathogenic Streptococcus mutans strains. Thus, we show how BiG-MAP can be used to generate new hypotheses on potential drivers of microbiome-associated phenotypes and prioritize the experimental characterization of relevant gene clusters that may mediate them.Microbial gene clusters encoding the biosynthesis of primary and secondary metabolites play key roles in shaping microbial ecosystems and driving microbiome-associated phenotypes. Although effective approaches exist to evaluate the metabolic potential of such bacteria through identification of these metabolic gene clusters in their genomes, no automated pipelines exist to profile the abundance and expression levels of such gene clusters in microbiome samples to generate hypotheses about their functional roles, and to find associations with phenotypes of interest. Here, we describe BiG-MAP, a bioinformatic tool to profile abundance and expression levels of gene clusters across metagenomic and metatranscriptomic data and evaluate their differential abundance and expression under different conditions. To illustrate its usefulness, we analyzed 96 metagenomic samples from healthy and caries-associated human oral microbiome samples and identified 252 gene clusters, including unreported ones, that were significantly more abundant in either phenotype. Among them, we found the IMPORTANCE Microbes play an increasingly recognized role in determining host-associated phenotypes by producing small molecules that interact with other microorganisms or host cells. The production of these molecules is often encoded in syntenic genomic regions, also known as gene clusters. With the increasing numbers of (multi)omics data sets that can help in understanding complex ecosystems at a much deeper level, there is a need to create tools that can automate the process of analyzing these gene clusters across omics data sets. This report presents a new software tool called BiG-MAP, which allows assessing gene cluster abundance and expression in microbiome samples using metagenomic and metatranscriptomic data. Here, we describe the tool and its functionalities, as well as its validation using a mock community. Finally, using an oral microbiome data set, we show how it can be used to generate hypotheses regarding the functional roles of gene clusters in mediating host phenotypes. BiG-MAP uses MinHash-based redundancy filtering to avoid ambiguous mapping, groups BGCs into gene cluster families (GCFs) with BiG-SCAPE , and useBiG-MAP maps shotgun sequencing reads onto gene clusters that have been predicted by either antiSMASH  or gutSMBiG-MAP is composed of four different modules : (i) BiG10.1128/mSystems.00937-21.2FIG S1https://doi.org/10.1038/s41564-017-0089-z) (PRJNA389280). The significant differential abundances of 11 gene clusters across Crohn’s disease (CD) samples and healthy (non-IBD) samples using Kruskal Wallis are shown in the heatmap. Phenotypes (CD or non-IBD) are shown in a bar on top of the heatmap. The more abundant a gene cluster is, the more yellow it is shown in the heatmap. Next to the heatmap, the bar chart represents the abundance log2 fold change values, which indicate the overall difference in abundance of an MGC between the phenotypes across all samples. On the far right, the dots represent the coverage values computed by BiG-MAP to show what proportion of the whole gene cluster has been covered by reads, in blue for the CD samples and in orange for the healthy samples. Download FIG S1, TIF file, 1.1 MB.Illustration of BiG-MAP.analyse module output when using metagenomics data. In this case, for testing and developing purposes, the CGR genomes were used as input for gutSMASH. The resulting predictions were used as input together with the metagenomics samples from Schirmer et al.  using the Kruskal-Wallis test are displayed in the heatmap. The higher the expression of a gene cluster is, the more yellow it is colored in the heatmap. Similar to Fig. S1, the logCopyright © 2021 Pascal Andreu et al.2021Pascal Andreu et al.https://creativecommons.org/licenses/by/4.0/Creative Commons Attribution 4.0 International license.This content is distributed under the terms of the The BiG-MAP.family module performs a redundancy analysis on the gene cluster collection to remove almost-identical sequences, in order to reduce the computing time and avoid problems with ambiguous read mapping. To achieve this, the protein sequences encoded in each of the gene clusters are used as input for MASH , a MinHaIn order to evaluate the overall performance of BiG-MAP and, in particular, all the default parameters chosen as defaults, such as the Bowtie alignment mode and the MASH similarity score cutoff, we designed a mock microbial community for metagenome simulation. From the Culturable Genome Reference (CGR) genome collection , we randFor orientation on the requirements and resources needed to run BiG-MAP, the time each BiG-MAP module took to run using the default settings was assessed when using as input the 3,793 gutSMASH-predicted MGCs from the 1,520 CGR collection genomes and the 78 metagenomes of the data set of Schirmer et al. . These aStreptococcus mutans have been linked to tooth decay .To illustrate the use of BiG-MAP in practice, we used it to analyze a data set from the human oral microbiota. The oral cavity is a natural habitat for many bacteria that reside in or on the gingival sulcus, tongue, teeth, and cheeks, among other surfaces. These bacteria take part in important processes such as initial digestion of food, but they are also associated with several oral diseases such as caries  and peri29–th decay . For thith decay  using Bi3H4O mass differences. With Mass Spectrometry Search Tool (MASST) searches  data together with the metabolite feature abundance table using Pathway Activity Level Scoring (PALS) , which usearches  across asearches . The forsearches . Reuteri cluster . Of note cluster . There a cluster . Further cluster . Typical cluster ; it is mbacteria , 44.10.1128/mSystems.00937-21.1TEXT S1Text S1, DOCX file, 0.02 MB.This document contains additional technical methodological details regarding the computational metabolomics analyses performed. Download Copyright © 2021 Pascal Andreu et al.2021Pascal Andreu et al.https://creativecommons.org/licenses/by/4.0/Creative Commons Attribution 4.0 International license.This content is distributed under the terms of the pdu operon in the metagenomics samples, in order to identify candidate MGCs that might be involved in acrolein formation. To this end, we ran gutSMASH on the 1,440 genomes from the Human Microbiome Oral Database  available in April 2020. Interestingly, gutSMASH identified a pdu-like operon in the genome of Streptococcus sp. strain F0442 that also includes a cobalamin (vitamin B12) biosynthetic region and is architecturally similar  to the Lactobacillus reuteri one . From this run, we found 246 gene clusters differentially abundant between groups  and 173 gene clusters when considering only the core region encoding the key enzymes used to detect the MGC. Among the significantly differentially abundant gene clusters observed when mapping reads to the core region, we found the pdu operon. While healthy samples on average have 5.62 RPKM counts/sample mapping to this gene cluster, diseased ones have 3.38 . Of note, when using a smaller data set of only 24 healthy and 23 caries-related samples , the Streptococcus sp. F0442 pdu operon did not show a statistically significant differential abundance. We also evaluated the coverage of the read mapping of the core genomic region within the expanded metagenomic data sets and found that among healthy samples, not all contained this gene cluster. For instance, of 34 healthy samples in the extended data set, we found 13 that appeared not to have the Streptococcus sp. F0442 pdu operon (coverage below 0.5), while the rest had fairly high coverage scores, with a mean coverage value of 0.77 , implying the presence of this operon or a close homologue of it   were mapped onto the predicted gene clusters and further processed using BiG-MAP.map and BiG-MAP.analyse subsequently . The muc operon from Streptococcus mutans 14D appeared differentially abundant between healthy and disease samples when using the fitZIG model , together with five other gene clusters. The muc operon from this strain shows high similarity to the one characterized by Tang et al. (PRJNA478018), no significant difference was seen. Also, the mean read coverages of the MGC core in both groups were low, 0.274 in healthy and 0.161 in caries-associated samples, which implies a relatively low abundance of the muc operon in many samples and/or partial coverage of the muc gene cluster locus with reads mapping only to flanking regions  . In ordeg et al.  and Liu g et al.  was run g et al. , which pg et al.  , predicted by antiSMASH, is significantly enriched in disease samples when applying the fitZIG model. Despite the significance, the difference in abundances between both groups is minimal, as the similar coloring depicts and the log2 fold change bar chart evinces. The plot shows another gene cluster, also coding for the biosynthesis of a nonribosomal peptide (NRP), found significantly enriched in disease samples. The heatmap has been produced by the BiG-MAP.analyse module using the 47 oral microbiome metagenomes. Download FIG S3, TIF file, 0.8 MB.The Copyright © 2021 Pascal Andreu et al.2021Pascal Andreu et al.https://creativecommons.org/licenses/by/4.0/Creative Commons Attribution 4.0 International license.This content is distributed under the terms of the pdu-like operon and muc) depended on the size of the data set used. Both examples illustrate how users would do well to study expression or abundance values of individual samples and should not draw conclusions too quickly based on the results of a single statistical test, for which the power also depends on the number of samples available.Overall, combining different omics data is a very useful approach to understand which microbes are doing what and poses a promising avenue to better understand complex biological processes. Here, we presented BiG-MAP, a command-line tool that it is able to profile the abundance and expression of a collection of gene clusters across metagenomic and metatranscriptomic data from any kind of biome, including human, plant, animal, marine, and soil microbiomes. Each of the steps in the BiG-MAP pipeline is robust, as demonstrated using a mock community. Indeed, BiG-MAP can discover interesting and relevant potential associations between genomic regions and phenotypes, which can guide experimental efforts to test MGC function. It is worth noting the usefulness of the gene cluster mapping coverage values, since they allow the user to discern between the real presence of predicted gene clusters of interest and spurious read mapping; these should always be considered together with the abundance/expression values for better interpretations of the results. Also, the associations that can be found using BiG-MAP strongly depend on the sample size and sequencing depth of the metagenome or metatranscriptome datasets. For instance, in the examples described in our study, we found that finding significant differences in the abundances of both gene clusters (the per se (besides being an average of multiple taxa and cells that may act heterogeneously). If users are interested in transcriptional regulation of MGCs instead of differences in overall transcript abundance, they may therefore want to correct for this. In future versions of BiG-MAP, we hope to make it possible for users to normalize their data by absolute abundances per sample and to allow optional normalization of metatranscriptome data relative to parent taxon abundance levels inferred from metagenome data from paired samples.In addition, there are several other caveats. It is important to note that even a statistically significant difference may not be biologically important: a consistent difference with a small fold change might have limited physiological relevance. It is also important to note that differences in relative abundance may be misleading if absolute abundances of the total microbiota are different between phenotypes. Also, differences in metatranscriptomic read abundances of an MGC may be caused by differences in their (relative) metagenomic abundances (caused by differences in abundance of underlying taxa). Therefore, differences in transcript abundances may not reflect differences in gene expression levels https://mash.readthedocs.io/en/latest/) and pick up the analysis again from that step onwards.Depending on the research question to be answered, the user may also consider different coverage thresholds depending on the scenario. For instance, lowering the coverage values might be needed to detect pathways from lowly abundant bacteria. Another limitation of BiG-MAP is that the family module is based on the MASH tool. In certain extreme cases, the number of predicted MGCs may exceed the maximum number of sequences that MASH (“sketch” and “dist” functions) is able to compare, leading to an error. In this scenario, the family module can be run in batches or the code can be slightly modified to manually run the MASH analysis and MASH “paste” function . To initially develop, test, and validate this tool, we used an inflammatory bowel disease (IBD) cohort that contains metagenomic and metatranscriptomic data from 78 individuals, 21 suffering from ulcerative colitis (UC), 46 individuals with Crohn’s disease (CD), and 11 healthy subjects  algorithm. The predicted gene clusters are then subjected to a redundancy filtering step based on their mutual sequence similarity. For that, the protein sequences of the gene clusters are extracted and used as input for MASH  for five housekeeping-coding proteins: DNA gyrase A (PF00521), DNA gyrase B (PF00204), recombinase A (PF00154), DNA-directed RNA polymerase A (PF01000), and DNA-directed RNA polymerase B (PF00562). The selection of these Pfam domains was based on the findings by Rocha et al.  that theThis module relies on Bowtie2  version (versionncb) to the length of each cluster (cl) as indicated in the formula below.Another functionality that was added in this module was to compute the read coverage of each gene cluster using the coordinates in the sorted BAM files. To do so, the sorted alignment files are converted to bedgraphs using BEDtools   GenBank output files that flag the key coding genes that are needed for the synthesis of a given molecule. Once the core genes are identified, their alignment information is retrieved using SAMtools. Next, in the same manner as RPKM values are computed for the whole gene clusters, reads aligned to the core region are extracted, counted, and corrected to finally get the RPKM counts. To perform the coverage calculation, the locations of the core genes are extracted from the bedgraph to evaluate the coverage.2 fold change that ultimately helps fitting the model to a log-normal distribution; thus, when the abundance/expression values are expected to follow normal distribution, the ZIG model is more appropriate to use. Alternatively, the Kruskal-Wallis test can be run on the normalized RPKM counts, which allows assessing whether the distribution of ranks for one group significantly differs from the distribution of ranks for the other group. Additionally, false-discovery rate (FDR) correction is applied to correct for multiple hypothesis testing. Finally, heatmaps are produced to visualize the results using the Seaborn Python package (https://github.com/mwaskom/seaborn).In order to account for sparse high-throughput sequencing, RPKM values are normalized using cumulative sum scaling (CSS) from the R Bioconductor package MetagenomeSeq . BiG-MAPPRJNA482748). Thus, the gutSMASH-predicted MGCs from each genome were used as ground truth . Next, paired-end reads were generated with a mean read length of 100 bp from the 101 CGR bacterial genomes using Grinder v0.5.3 . The resulting molecular family information together with the metabolite feature tables and sample information (metadata) were loaded into PALS  (https://masst.ucsd.edu) .To find possible leads on metabolic perturbances between healthy and caries-related samples, the processed mass spectra (MGF format) and metabolomics feature tables from Aleti et al.  were dowrg/app/)  to identcsd.edu) , the Checsd.edu) , leadingpdu operon and its presence in the oral microbiome, 1,440 oral bacteria genomes were downloaded from the HOMD collection . Next, these genomes were used as input for gutSMASH (version 0.8). The comparison between the two pdu operons from Lactobacillus reuteri (AP007281) and Streptococcus sp. F0442 (GCA_000314795.2) was done using MultiGeneBlast  with the following doi: 10.5281/zenodo.5054950. The metabolomics data used for reanalysis are available from GNPS-MassIVE accession number MSV000081832.The supporting information for this article can be found in the supplemental material of this article and in the Zenodo repository (


In recent years both research areas of next-generation sequencing and artificial intelligence have grown remarkably. Their intersection simultaneously gave rise to a panacea of different algorithms and applications. This article delineates tailored machine learning and systems biology approaches and combinations thereof that tackle the various challenges that arise in the face of big data. Moreover, it provides an overview of the numerous applications of artificial intelligence aiding the analysis and interpretation of next-generation sequencing data.The rapid improvement of next-generation sequencing (NGS) technologies and their application in large-scale cohorts in cancer research led to common challenges of big data. It opened a new research area incorporating systems biology and machine learning. As large-scale NGS data accumulated, sophisticated data analysis methods became indispensable. In addition, NGS data have been integrated with systems biology to build better predictive models to determine the characteristics of tumors and tumor subtypes. Therefore, various machine learning algorithms were introduced to identify underlying biological mechanisms. In this work, we review novel technologies developed for NGS data analysis, and we describe how these computational methodologies integrate systems biology and omics data. Subsequently, we discuss how deep neural networks outperform other approaches, the potential of graph neural networks (GNN) in systems biology, and the limitations in NGS biomedical research. To reflect on the various challenges and corresponding computational solutions, we will discuss the following three topics: (i) molecular characteristics, (ii) tumor heterogeneity, and (iii) drug discovery. We conclude that machine learning and network-based approaches can add valuable insights and build highly accurate models. However, a well-informed choice of learning algorithm and biological network information is crucial for the success of each specific research question. The development and widespread use of high-throughput technologies founded the era of big data in biology and medicine. In particular, it led to an accumulation of large-scale data sets that opened a vast amount of possible applications for data-driven methodologies. In cancer, these applications range from fundamental research to clinical applications: molecular characteristics of tumors, tumor heterogeneity, drug discovery and potential treatments strategy. Therefore, data-driven bioinformatics research areas have tailored data mining technologies such as systems biology, machine learning, and deep learning, elaborated in this review paper see . For exaTherefore, a large number of studies integrate NGS data with machine learning and propose a novel data-driven methodology in systems biology . In partIn this review, we will outline the paradigms of NGS-based cancer research during the last decades. After that, we will discuss state-of-the-art systems biology, machine learning, and DNN-based approaches applied in various cancer studies. At first, we will summarize various challenges raised by the analysis of NGS datasets and cancer genomics, and we will discuss different applications of systems biology in cancer research. Additionally, we will introduce network-oriented machine learning methodologies that have been applied to NGS cancer data. Finally, we will discuss very recent DNN-based approaches in cancer researches.During the past decade, the field of cancer genomics expanded enormously, triggered by the accumulation of massive amounts of various omics data originating from different cancer tissue types stored in large public databases such as TCGA . TCGA isThe application of machine learning on omics data has become a state-of-the-art approach to classify tumor subtypes originating from various tissues. With the aid of these large-scale NGS datasets, molecular characterization and subtyping have been done in both intra-cancer types and across the different cancer types. The TCGA’s early pan-cancer study focuses on molecular subtyping across the border of originating tissues. They were able to classify tumors with new molecular characteristics observed through NGS technology. For example, bladder cancer can be divided into two different subtypes, and one of them is more closely related to squamous cell carcinoma in the lung and head and neck cancer types . MoreoveAs described in the previous section, the combination of high-throughput sequencing technology and bioinformatics analyses extended our knowledge of cancer’s molecular background. This data-driven analysis supported the use of these approaches for molecular diagnosis and genomic medicine ,27. SubsThere might be various reasons for those issues, such as tumor microenvironment and tumor heterogeneity ,30. EverNGS data have proven to be particularly useful to analyze and understand the mechanisms and effects of drugs. Thus, NGS applications have accelerated target drug discovery and development. In particular, the molecular characteristics of cancer were robustly translated into a potential treatment target such as cell-cycle pathway or PI3K pathway ,43. FurtThe following sections will address how bioinformatics algorithms and machine learning approaches have been used in the field to address the aforementioned issues.Genes and their functions have been classified into gene sets based on experimental data. Our understandings of cancer concentrated into cancer hallmarks that define the characteristics of a tumor . This coIn this section, we will discuss how two related research fields, namely, systems biology and machine learning, can be integrated with three different approaches see , namely,The detection of potential biomarkers indicative of specific cancer types or subtypes is a frequent goal of NGS data analysis in cancer research. For instance, a variety of bioinformatics tools and machine learning models aim at identify lists of genes that are significantly altered on a genomic, transcriptomic, or epigenomic level in cancer cells. Typically, statistical and machine learning methods are employed to find an optimal set of biomarkers, such as single nucleotide polymorphisms (SNPs), mutations, or differentially expressed genes crucial in cancer progression. Traditionally, resource-intensive in vitro analysis was required to discover or validate those markers. Therefore, systems biology offers in silico solutions to validate such findings using biological pathways or gene ontology information b 58]. S. S58]. SWhile the known fraction of existing biological networks barely scratches the surface of the whole system of mechanisms occurring in each organism, machine learning models can improve on known network structures and can guide potential new findings ,74. ThisA network-based machine learning model directly integrates the insights of biological networks within the algorithm d to ultiIn summary, using network information opens new possibilities for interpretation. However, as mentioned earlier, several challenges remain, such as the coverage issue. Current databases for biological networks do not cover the entire set of genes, transcripts, and interactions. Therefore, the use of networks can lead to loss of information for gene or transcript orphans. The following section will focus on network-based machine learning models and their application in cancer genomics. We will put network-based machine learning into the perspective of the three main areas of application, namely, molecular characterization, tumor heterogeneity analysis, and cancer drug discovery.As introduced previously, the integration of machine learning with the insights of biological networks d ultimatVarious network-based algorithms are used in genomics and focus on quantifying the impact of genomic alteration. By employing prior knowledge in biological network algorithms, performance compared to non-network models can be improved. A prominent example is HotNet. The algorithm uses a thermodynamics model on a biological network and identifies driver genes, or prognostic genes, in pan-cancer data . AnotherFurthermore, in transcriptome research, network information is used to measure pathway activity and its application in cancer subtyping. For instance, when comparing the data of two or more conditions such as cancer types, GSEA as introduced in The tumor heterogeneity can originate from two directions, clonal heterogeneity and tumor impurity. Clonal heterogeneity covers genomic alterations within the tumor . While dIn contrast, tumor purity is dependent on the tumor microenvironment, including immune-cell infiltration and stromal cells . In tumoIn drug target studies, network biology is integrated into pharmacology . For insMoreover, drug-target studies are often extended to drug-repurposing studies. In cancer research, drug-repurposing studies aim to find novel interactions between non-cancer drugs and molecular features in cancer. Drug-repurposing (or repositioning) studies apply computational approaches and pathway-based models and aim at discovering potential new cancer drugs with a higher probability than de novo drug design ,106. SpeDNN models develop rapidly and become more sophisticated. They have been frequently used in all areas of biomedical research. Initially, its development was facilitated by large-scale imaging and video data. While most data sets in the biomedical field would not typically be considered big data, the rapid data accumulation enabled by NGS made it suitable for the application of DNN models requiring a large amount of training data . For inshttps://www.ncbi.nlm.nih.gov/geo/, accessed on 20 May 2021) can be leveraged to tackle critical aspects in cancer research. They store a large-scale of biological data produced under various experimental setups  data as proof of concept of the applicability of DNN for data imputation in a multi-omics integration study [Many studies in biology and medicine used NGS and produced large amounts of data during the past few decades, moving the field to the big data era. Nevertheless, researchers still face a lack of data in particular when investigating rare diseases or disease states. Researchers have developed a manifold of potential solutions to overcome this lack of data challenges, such as imputation, augmentation, and transfer learning b. Data ion study . Becauseon study ,121. Theon study . GANs caon study . In conton study . These aon study ,126. Thuon study  model toon study . Affiniton study .DNNs have been applied in multiple areas of cancer research. For instance, a DNN model trained on TCGA cancer data can aid molecular characterization by identifying cancer driver genes. At the very early stage, Yuan et al. build DeepGene, a cancer-type classifier. They implemented data sparsity reduction methods and trained the DNN model with somatic point mutations . Lyu et Another DNN model was trained to infer the origin of tissue from single-nucleotide variant (SNV) information of metastatic tumor. The authors built a model by using the TCGA/ICGC data and analyzed SNV patterns and corresponding pathways to predict the origin of cancer. They discovered that metastatic tumors retained their original cancer’s signature mutation pattern. In this context, their DNN model obtained even better accuracy than a random forest model  and, eveAs described in In addition to NGS datasets, large-scale anticancer drug assays enabled the training train of DNNs. Moreover, non-cancer drug response assay datasets can also be incorporated with cancer genomic data. In cancer research, a multidisciplinary approach was widely applied for repurposing non-oncology drugs to cancer treatment. This drug repurposing is faster than de novo drug discovery. Furthermore, combination therapy with a non-oncology drug can be beneficial to overcome the heterogeneous properties of tumors . The deeThe authors of CDRscan did an integrative analysis of cell-line-based assay datasets and other drug and genomics datasets. It shows that DNN models can enhance the computational model for improved drug sensitivity predictions . AdditioIn general, the advantage of using a biological network is that it can produce more comprehensive and interpretable results from high-dimensional omics data. Furthermore, in an integrative multi-omics data analysis, network-based integration can improve interpretability over traditional approaches. Instead of pre-/post-integration of a network, recently developed graph neural networks use biological networks as the base structure for the learning network itself. For instance, various pathways or interactome information can be integrated as a learning structure of a DNN and can be aggregated as heterogeneous information. In a GNN study, a convolution process can be done on the provided network structure of data. Therefore, the convolution on a biological network made it possible for the GNN to focus on the relationship among neighbor genes. In the graph convolution layer, the convolution process integrates information of neighbor genes and learns topological information d. ConseqIn the context of the inference problem of gene expression, the main question is whether the gene expression level can be explained by aggregating the neighboring genes. A single gene inference study by Dutil et al. showed that the GNN model outperformed other DNN models . MoreoveThe previous sections reviewed a variety of DNN-based approaches that present a good performance on numerous applications. However, it is hardly a panacea for all research questions. In the following, we will discuss potential limitations of the DNN models. In general, DNN models with NGS data have two significant issues: (i) data requirements and (ii) interpretability. Usually, deep learning needs a large proportion of training data for reasonable performance which is more difficult to achieve in biomedical omics data compared to, for instance, image data. Today, there are not many NGS datasets that are well-curated and -annotated for deep learning. This can be an answer to the question of why most DNN studies are in cancer research ,149. Morhttps://lincsproject.org/, accessed on 20 May 2021) [As described before, biological networks are employed in various computational analyses for cancer research. The studies applying DNNs demonstrated many different approaches to use prior knowledge for systematic analyses. Before discussing GNN application, the validity of biological networks in a DNN model needs to be shown. The LINCS program analyzed data of ’The Connectivity Map (CMap) project’ to understand the regulatory mechanism in gene expression by inferring the whole gene expression profiles from a small set of genes (ay 2021) ,152. Thiay 2021) .Following this study, Beltin et al. extensively investigated various biological networks in the same context of the inference of gene expression level. They set up a simplified representation of gene expression status and tried to solve a binary classification task. To show the relevance of a biological network, they compared various gene expression levels inferred from a different set of genes, neighboring genes in PPI, random genes, and all genes. However, in the study incorporating TCGA and GTEx datasets, the random network model outperformed the model build on a known biological network, such as StringDB . While nThe advance of NGS and subsequent massive production of NGS data facilitated various NGS analysis pipelines. The interplay between NGS and bioinformatics widened the spectrum of applicability of NGS and enabled the extraction and uncovering of detailed insights into the molecular mechanisms of cancer development and treatment. Accordingly, the tailored combination with machine learning and systems biology broadens our knowledge from genetics to medicine, particularly in cancer research. This review shows the development of NGS and corresponding computational methods in cancer research over the past few decades. We mainly focus on the following three levels, machine learning, machine learning with systems biology, and more recent approaches in deep learning. Among the vast amount of research trends in NGS applications, we focused on NGS data interpretations and how this developed methodology improved our understanding of cancer. In cancer genomics, sequencing data analysis extended from finding single mutated genes to mining cancer driver gene sets and their effect on biological pathways. With pathway information, genomics research was able to extend to synthetic lethality and drug combination. Similar to genomics, the cancer transcriptomics field moved from searching for differentially expressed genes to pathway activity measurement and thereby contributed to the development of a diagnosis panel. In systems biology, transcriptomics is widely used for de novo network reconstruction and finding novel pathways. Moreover, recently, deep learning algorithms are applied and were able to outperform existing models on various tasks, showing the power of a new data-driven methodology. This shift was observable in different areas of cancer research, biomarker identification, molecular characterization of cancer, tumor heterogeneity, and target-drug repurposing. However, although recently sophisticated applications of deep learning showed improved accuracy, it does not reflect a general advancement. Depending on the type of NGS data, the experimental design, and the question to be answered, a proper approach and specific deep learning algorithms need to be considered. Deep learning is not a panacea. In general, to employ machine learning and systems biology methodology for a specific type of NGS data, a certain experimental design, a particular research question, the technology, and network data have to be chosen carefully.In the last few decades, NGS enabled an improvement of diagnostics in medicine from genetic diseases to cancer. Many recent machine learning-based studies report their advancements in disease classification and prognosis analysis. However, because machine learning models are often treated as black boxes and lack transparency, the lack of explainability leads to poor acceptance among clinicians. Especially for DNN-based models, it is often hard to understand why it shows better accuracy than other methods. To solve this ’Black box’ issue, the novel field of “explainable AI” aims at improving visualization, explanation, and interpretation of machine learning models such as deep learning. Explainable AI aims to fill the interpretability gap between translational medicine and machine learning. Furthermore, it can be beneficial to improve on machine learning methods itself by identifying their weakness. For instance, interactive machine learning or human-in-the-loop AI, where machine learning models are updated based on a human expert, can be a solution to both an interpretability issue and an improving machine learning model ,156. In


Predicting the drug response of a patient is important for precision oncology. In recent studies, multi-omics data have been used to improve the prediction accuracy of drug response. Although multi-omics data are good resources for drug response prediction, the large dimension of data tends to hinder performance improvement. In this study, we aimed to develop a new method, which can effectively reduce the large dimension of data, based on the supervised deep learning model for predicting drug response.We proposed a novel method called Supervised Feature Extraction Learning using Triplet loss (Super.FELT) for drug response prediction. Super.FELT consists of three stages, namely, feature selection, feature encoding using a supervised method, and binary classification of drug response (sensitive or resistant). We used multi-omics data including mutation, copy number aberration, and gene expression, and these were obtained from cell lines , patient-derived tumor xenografts (PDX), and The Cancer Genome Atlas (TCGA). GDSC was used for training and cross-validation tests, and CCLE, CTRP, PDX, and TCGA were used for external validation. We performed ablation studies for the three stages and verified that the use of multi-omics data guarantees better performance of drug response prediction. Our results verified that Super.FELT outperformed the other methods at external validation on PDX and TCGA and was good at cross-validation on GDSC and external validation on CCLE and CTRP. In addition, through our experiments, we confirmed that using multi-omics data is useful for external non-cell line data.https://github.com/DMCB-GIST/Super.FELT.By separating the three stages, Super.FELT achieved better performance than the other methods. Through our results, we found that it is important to train encoders and a classifier independently, especially for external test on PDX and TCGA. Moreover, although gene expression is the most powerful data on cell line data, multi-omics promises better performance for external validation on non-cell line data than gene expression data. Source codes of Super.FELT are available at The online version contains supplementary material available at 10.1186/s12859-021-04146-z. Prediction of drug response for each patient is highly important in precision oncology. However, it is quite challenging as the drug response for each patient could vary owing to genetic differences . AlthougRecently, large-scale omics data have been made publicly accessible, including Genomics of Drug Sensitivity in Cancer (GDSC) , Cancer The translatability of prediction models to actual patients is challenging in drug response studies , 10. To Recently, many computational methods, such as support vector machines (SVM) , autoencDing et al.  uses AE In this study, we focused on how accurately an encoder can extract important features for classifying samples as drug response resistant  or sensitive. We proposed a novel method, Supervised Feature Extraction Learning using Triplet loss (Super.FELT). In Super.FELT, there are three stages, namely, feature selection, feature encoding, and classification. In the feature selection stage, Super.FELT uses variance threshold based on the elbow method to extract significant features of omics-data. In the feature encoding stage, each dataset is assigned to different encoders, and each encoder is trained using a supervised method, wherein an objective function is triplet loss function. Because the encoder is based on the supervised method, feature selection for reducing the large dimension of omics data would improve the efficiency of the encoder. Therefore, Super.FELT could extract important features from each omics data better than previous models. To assess the translatability of our proposed model, we trained Super.FELT using GDSC dataset and tested on CCLE, CTRP, PDX, and TCGA datasets. Our results showed that Super.FELT is superior to the other drug response prediction methods on external datasets.https://www.ebi.ac.uk/arrayexpress/experiments/E-MTAB-3610/files/raw/. CNA and mutation data were downloaded from ftp://ftp.sanger.ac.uk/pub/project/cance rrxgene/releases/release-7.0/Gene_level_CN.xlsx and ftp://ftp.sanger.ac.uk/pub/project/cancerrxgene/releases/release-7.0/WES_variants.xlsx, respectively. The drug response data based on ln(IC50) values were obtained from Table S5 (C) of Iorio et al. [https://www.cancerrxgene.org/gdsc1000/GDSC1000_WebResources///Data/suppData/TableS5C.xlsx).In this study, we used in vitro  and in vivo (TCGA) datasets, including multi-omics data  and drug responses Table . For GDSo et al.   , and CNAhttps://static-content.springer.com/esm/art%3A10.1038%2Fnm.3954/MediaObjects/41591_2015_BFnm3954_MOESM10_ESM.xlsx), where “CR” and “PR” are “Sensitive”, and “SD” and “PD” are “Resistant”. Omics data of TCGA were downloaded from Firehose Broad GDAC (http://gdac.broadinstitute.org/runs/stddata__2016_01_28/). For drug response of TCGA data, we used Table S2 of Ding et al. [For PDX and TCGA datasets, which are also used for external validation, we performed tests for drugs having at least one resistant and sensitive samples and at least four samples in total among existing drugs in GDSC. As a result, 6 and 13 drugs were tested for PDX and TCGA, respectively, where the average numbers of samples and resistant and sensitive samples are 31, 27, and 4 for PDX and 27, 13, and 14 for TCGA . We used triplet loss function, which helps the embedding vector to have similar values for samples with the same label. As the encoders were trained using the supervised mode, it should reduce the risk of overfitting. To decrease the risk, Super.FELT firstly reduced the dimension of omics data through feature selection using a variance threshold based on the elbow method. This was based on the assumption that genes with low variance might contain less important information. This approach had been used in several studies to handle omics data [Gene expression, mutation, and CNA data are denoted as ics data –39. SecoIn our study, feature selection using variance threshold for a certain omics data r(X) is the ReLU function, Next, each omics dataset was assigned to an encoding function. Here, we defined the encoding function for the omics data as response . Althougresponse , 15, 25.f(x) and d, respectively, the goal of triplet loss function is to ensure that the distance between anchor, positive, and negative is as follows.A is an anchor matrix, P is a positive matrix, and N is a negative matrix; K is the number of pairs.For encoding omics data, SET uses triplet loss function  as a cosHere, we had binary labels: resistant and sensitive. Therefore, when we select a sample with resistant label as the anchor, samples with resistant label become positive samples and those with sensitive label become negative samples and vice versa. For utilizing triplet loss function, we should define a pairing function, s(x) is a sigmoid function, The encoded data were already well organized to predict drug response owing to SET. As complex models pretend to  cause overfitting, the classifier should not be a complex model. Therefore, in our model, the classifier had a single layer with a sigmoid activation function, and dropout, weight decay, and early stopping were used. A classifier can be described as follows.To assess the performance of Super.FELT, we performed a series of experiments. In the experimental design, we introduced an overall experimental design of Super.FELT along with the other drug response prediction methods for comparison. We then assessed the performance of our model using four external datasets. The subsection ‘Cross-validation’ shows the results of We compared with MOLI  because We compared with MOLI using feature selection because we cannot say that independent training is better than joint training when comparing MOLI with Super.FELT. From the test, we could investigate which training was better and evaluate the effect of feature selection. We named this case 2 as MOLI after Feature selection (MOLIF).We compared with a model using feature selection with variation threshold and an autoencoder followed by the neural network classifier; this was the same as Super.FELT, except that a latent space of the autoencoder was used for input of the classifier instead of SET. We could compare the simple autoencoder with SET. We named this case 3 as AE.We compared with the model using feature selection and the neural network classifier; it was the same as Super.FELT without SET. Using this model, we examined how effective SET was. We named this case 4 as Artificial Neural Network after Feature selection (ANNF).We compared with AutoBorutaRF , which iWe compared with SVM after feature selection on gene expression data, which is similar to Huang et al. . The difWe compared with Super.FELT using only gene expression without mutation and CNA data. Using this test, we could compare multi-omics approach by using gene expression alone. We named this case 7 as Super.FELT E.We compared with Super.FELT using mutation and CNA data without gene expression. By testing this case 8, we could verify how influential mutation and CNA are on this task without gene expression. We named case 8 as Super.FELT M&C.The GDSC dataset used in our experiment , Doxorubicin (TCGA), Paclitaxel (TCGA), and Cetuximab (PDX), Super.FELT M&C showed the best performance. When comparing Fig. Because of its properties, Super.FELT, which used feature selection, encoder using triplet loss, independent training, and multi omics approach, outperformed the other methods with the average AUC value of 0.622 in the section ‘On PDX and TCGA’. In case of drugs for which Super.FELT was not the best, AUC values of Super.FELT were similar to those of the best model, except the case of Etoposide (TCGA) Fig. B.Recent studies have proposed models with adversarial networks by training both in vitro and in vivo datasets and obtained increased performance compared with that of models trained using only in vitro datasets . This apIn this study, although Super.FELT was applied for the drug response prediction, it can be further applied for other biomedical tasks using multi-omics datasets. Recently, disease progress prediction, such as survival and recurrence, and cancer subtype classification,  has been performed using multi-omics datasets –55. In tIn our study, we found that high translatability could be achieved between the same cell line data, but it is difficult to achieve reasonable translatability from cell line data to non-cell line data. To achieve high translatability, we used feature selection using variance thresholds based on the elbow method and triplet loss function, which are often used for improving the encoder in image classification models , 57. OurAdditional file 1. Supplementary Tables.


While single‐omics analyses on human atherosclerotic plaque have been very useful to map stage‐ or disease‐related differences in expression, they only partly capture the array of changes in this tissue and suffer from scale‐intrinsic limitations. In order to better identify processes associated with intraplaque hemorrhage and plaque instability, we therefore combined multiple omics into an integrated model.In this study, we compared protein and gene makeup of low‐ versus high‐risk atherosclerotic lesion segments from carotid endarterectomy patients, as judged from the absence or presence of intraplaque hemorrhage, respectively. Transcriptomic, proteomic, and peptidomic data of this plaque cohort were aggregated and analyzed by DIABLO, an integrative multivariate classification and feature selection method.−PDGFRα+ fibroblast‐like cell content (p = 2.4E‐05) and Arg1+ macrophage content (p = 2.2E‐04) and was driven by serum response factor (SRF), possibly in a megakaryoblastic leukemia‐1/2 (MKL1/2) dependent manner. Gene set overrepresentation analysis on the selected key features of this model pointed to a clear cardiovascular disease signature, with overrepresentation of extracellular matrix synthesis and organization, focal adhesion, and cholesterol metabolism terms, suggestive of the model's relevance for the plaque vulnerability. Finally, we were able to corroborate the predictive power of the selected features in several independent mRNA and proteomic plaque cohorts.We identified a protein‐gene associated multiomics model able to segregate stable, nonhemorrhaged from vulnerable, hemorrhaged lesions at high predictive performance (AUC >0.95). The dominant component of this model correlated with αSMAIn conclusion, our integrative omics study has identified an intraplaque hemorrhage‐associated cardiovascular signature that provides excellent stratification of low‐ from high‐risk carotid artery plaques in several independent cohorts. Further study revealed suppression of an SRF‐regulated disease network, controlling lesion stability, in vulnerable plaque, which can serve as a scaffold for the design of targeted intervention in plaque destabilization. This multiomics analysis identified a cardiovascular signature in carotid atherosclerotic lesions, which provides excellent stratification of low‐/high‐risk carotid plaques.This study highlights the advantages of multiomics analysis in terms of model robustness, biological significance, and clinical translatability.The prediction model pointed to an SRF‐regulated disease network providing valuable new insights that expedite the design of targeted intervention in plaque rupture. Laser‐capture microscopyHowever, for indepth dissection of critical processes in plaque, mere transcriptional profiling may fall short incorporating net biosynthesis and protein activity, whereas proteomics profiling alone will bias toward most abundant, rather than most relevant proteins, implying that both scales have serious shortcomings.Therefore, in the current study we set out to integrate the protein, peptide, and mRNA makeup of carotid artery lesions from the Maastricht Human Plaque Study (MaasHPS)22.1Atherosclerotic plaque samples obtained during carotid endarterectomy from 24 symptomatic patients were collected. The endarterectomy specimens were cut into parallel, transverse segments of 5‐mm thickness. Each alternating segment was snap‐frozen in liquid nitrogen and stored at −80°C, their flanking segments fixed for 24 h in formalin, decalcified for 4 h before processing and embedding in paraffin for histological evaluation.n = 24 at the start for each). However, one IPH and four non‐IPH samples did not pass the QC test and were excluded from analysis (see the following sections). Simultaneously with the microarray performance for the omics experiments, but prior to bioinformatic analysis, tissue was sectioned further for additional staining and computer‐aided quantitative measurement of plaque IPH. Based on this indepth reinspection, the three experienced pathologists  agreed to remove one ambiguous “non‐IPH” sample, showing small but surface‐detached luminal fibrin clot, as this could represent either a surgery artefact or bona fide IPH. In three allegedly “non‐IPH” samples, quantitative morphometry detected minor IPH , which was overlooked in the preclassification. These three were recategorized as IPH after inspection by the pathologists. Due to the recategorization of samples from the non‐IPH to the IPH group, three pairs of the total 26 samples of the IPH group were from three patients, respectively. To explore the potential confounder effects of this adjustment, we performed hierarchical clustering of the samples based on the plaque traits (measured as described in Section 2.2). This analysis convincingly showed that there is no patient‐specific heterogeneity among the samples, as well as that plaque phenotype is dominant over sample origin (data not shown). The final cohort for this study therefore included transcriptomics, proteomics, and peptidomics for 16 non‐IPH and 26 IPH plaques  stained plaque tissue was macroscopically preclassified for plaque stage before omics experiments. Segments were stratified into non‐IPH and IPH groups according to the absence or presence of IPH. For each CEA specimen and patient, two samples (one non‐IPH and one IPH) were collected for omics experiments , CD3+ T‐cell content , CD68+ macrophage content , iNOS+ (M1) macrophage content (% of CD68+ macrophage area), Arg1+ (M2) macrophage content (% of CD68+ macrophage area), collagen content , αSMA+ smooth muscle cell (SMC) content , αSMA–PDGFRα+ fibroblast‐like cell content , and calcification . These histological features are denoted as plaque traits, their distribution and statistical differences between paired non‐IPH and IPH plaques are shown in Figure All H&E slides were photographed at a 12.5× magnification and examined digitally using Leica Q500MC software. Total tissue, media, cap, necrotic core, hemorrhage, and luminal thrombus area were measured on H&E. Plaque size was calculated by subtracting the medial and luminal thrombus area from that of the total carotid tissue. Necrotic core and hemorrhage area were quantified relative to the plaque size. Cap thickness was measured at three to 15 positions throughout the entire cap region, at regular intervals. In addition, we measured the relative area of the following histological features: CD312O2 for 15 min, followed by heat‐induced antigen retrieval in a microwave for 10 min in a low pH target retrieval solution . After cooling down for 20 min, sections were incubated overnight at 4°C with polyclonal rabbit anti‐human MFAP4 antibody . After washing in TBS, sections were incubated for 1 h at room temperature in Brightvision poly‐HRP anti‐rabbit IgG . After washing in TBS, staining was developed using the chromogen 3,3′‐diaminobenzide  for 5 min, and sections were counterstained with Mayer's hematoxylin , dehydrated, mounted, and cover slipped. Sections were digitized using the Pannoramic 1000 scanner (3DHistech Ltd.) at 20× magnification. Morphometric analysis was done using the free software package QuPath (v0.2.3) to measure percentage area positivity/total area.For the MFAP4 immunohistochemistry for validation purpose, 4‐μm thick sections were cut from 35 plaques  left available from the total 42 plaques. Deparaffinization and rehydration was performed using xylene and graded ethanol. Endogenous peroxidase was blocked with 0.3% H2.3Snap‐frozen omics segments were pulverized and 5–20 mg of material was subjected to transcriptomic, proteomic, and peptidomic analysis.RNA was isolated by guanidium thiocyanate extraction, followed by further purification using the Nucleospin RNA II kit (Macherey‐Nagel GmbH & Co. KG). RNA concentration was measured using a Nanodrop ND‐1000 spectrophotometer . RNA quality and integrity were determined on the Agilent 2100 Bioanalyzer . Pure RNA samples that had an RNA Integrity Number (RIN) greater than 6.0 and a sample purity A260/280 ratio greater than 1.8 were taken for transcriptomic analysis (45 out of 48 samples). One analysis for a non‐IPH plaque had failed, therefore in total four unqualified non‐IPH samples were excluded from the microarray experiment.g to remove insoluble debris. DTT content of the homogenate was raised to 100 mM and protein reduction was completed through incubation at 70°C for an hour. Alkylation was completed by adding 0.5 M iodoacetamide and followed by a 30‐min incubation at room temperature. The excess alkylating reagent was quenched by addition of 12 μl of 1 M DTT. The protein content of the homogenate was recovered on a poros R1 column using a Vision Chromatography Station (Applied Biosystems). Proteins were eluted with 70% isopropanol/30% formic acid solvent mixture and dried down in a Speedvac.Proteomic analysis was carried out by utilizing the 8‐plex iTRAQ reagent for relative quantification. Reference sample used for relative quantification was created by pooling the redundant specimens derived from the equal portions of the samples for omics experiments. Pulverized tissue samples (5–20 mg) were homogenized in a Covaris E100 ultrasonic homogenizer using 400‐μl lysis buffer of 6 M guanidium hydrochloride, 1% Triton‐114, 50 mM triethylammonium bicarbonate (TEAB), 50 mM dithiothreitol (DTT), and protease inhibitor tablets . The homogenate was centrifuged for 15 min at 4000 × 2.4Biotinylated cRNA was prepared using the Illumina TotalPrep RNA Amplification Kit  according to the manufacturer's specifications starting with 100 ng total RNA. Per sample 750 ng of cRNA was used for hybridization. Hybridization and washing were performed according to the Illumina standard assay procedure. Scanning was performed on the Illumina BeadStation 500 , while image analysis and extraction of raw expression data were done using Illumina Beadstudio v3 Gene Expression software with default settings . Transcripts were measured by Illumina HumanRef‐8 v2.0 expression BeadChip.m/z 114, 115, 116, 118, 119, 121 reporter fragments in the MS/MS scans. Reference samples were labeled with the 113 and 117 reagents. iTRAQ labeled reactions were quenched by the addition of 1 M ammonium bicarbonate.Proteins were resuspended in 2 M freshly prepared urea, 1 M TEAB, 1% n‐octyl‐glucoside buffer and digested with trypsin added at 1:20 w/w ratio to the sample, for 4 h at 37°C. Digestion was stopped by increasing the temperature to 95°C for 5 min. Digested samples were labeled by the 8‐plex iTRAQ reagents following the manufacturer's protocols (Applied Biosystems) using a sample digest quantity that represents approximately 40‐μg protein content. Samples were labeled with the reagents yielding the Eight samples  constituting an iTRAQ mix were combined, desalted, and fractionated by strong cation exchange (SCX) chromatography using an Agilent 1200 instrument. Eight SCX fractions were injected for HPLC, and resolved over a 90‐min gradient of 5% solvent B (10% H2O/90% ACN/0.1% TFA) to 38% B (solvent A: 95% H2O/5% ACN/0.1% TFA). The elution volume was collected onto a plate for mass spectrometric analysis by matrix‐assisted laser desorption ionization (MALDI) as 10‐s intervals, using 10 mg/ml of alpha‐cyano‐4‐hydroxycinnamic acid in 50%–50% acetonitrile–water as the matrix. Each HPLC run was represented as a 500‐spot array on the MALDI plate. These plates were analyzed on an AB4800 mass spectrometer .m/z 113 and 117 peaks. Protein ratios were determined as the medians of all peptide ratios matching to the same protein. Peptide sequences were identified from MS/MS fragmentation spectra using the Mascot search engine  and the SwissProt database. The whole proteomic/peptidomic analysis was carried out following the same protocols as in a previous plasma proteomics study for cardiovascular disease.Peptide quantification was carried out by calculating the average ion intensity ratios relative to the The corresponding microarray profile of one IPH sample with failed MS measurement  was removed. The final MaasHPS cohort therefore included transcriptomics for 16 non‐IPH and 26 IPH plaques, with successful proteomics and peptidomics profiling from the same samples.2.5A total of 22,184 human transcripts and variants as defined by the NCBI Reference Sequence (RefSeq) were analyzed. Transcriptomic data were analyzed using the R Bioconductor lumi package (v2.38.0). First, we performed a variance stabilizing transformation, which is incorporated in the lumi package. Then, the robust spline normalization (RSN) algorithm in the lumi package was applied to normalize the data. As low‐variance genes and noise expression will not only reduce the effectiveness of subsequent clustering and machine learning but also slow down computations, we selected the top 10,000 most variable genes from a total pool of 17,759 unique detectable genes, for further analysis.k‐nearest neighbors (k‐NN) imputation (k = 7) in an unsupervised manner, as this proved superior to other methods in a pilot analysis.Measured values of the abundance of 1330 proteins and 4736 peptides were normalized using a procedure based on Vandesompele et al.Finally, 10,000 genes, 943 proteins, and 2637 peptides were detectable in the 42 samples (16 non‐IPH vs. 26 IPH) and were used for further analysis. Genes, proteins, and peptides were mapped based on the HUGO Gene Nomenclature Committee (HGNC) symbols.2.6http://mixomics.org/, v6.8.5),For single‐omics analysis of the transcriptomic, proteomic, and peptidomic data, we deployed sparse partial least squares‐discriminant analysis ,2.7SPLS‐DA requires extensive optimization of both the number of components and the number of features for constructing each component. In brief, the number of testing components was set from one to six, as in most cases, the prediction performance deteriorates rapidly when the number of the components increase above five. Then, the number of features to be tested for constructing the sPLS‐DA component was configured. For single sPLS‐DA on transcriptomics, proteomics, and peptidomics, a range of features from three to 300  were set for each component. For integrative sPLS‐DA, to achieve the comparable number of selected features, the range of testing features here for each component and each omics was set from 10 to 100 .Specifically, for integrative sPLS‐DA, a design matrix representing how close the datasets should be connected to each other needs to be configured. According to Singh et al.’s suggestion,Optimal parameters can be derived in one step from the model with the best classification in cross‐validation. A series of single and integrative sPLS‐DA prediction models were obtained from each combination of these parameter settings, and their performances were tested under stratified five‐fold cross‐validation with 1000 random repeats. Considering the unbalance of the sample phenotypes, to fairly reflect the classification performance of training models, in this tuning phase we used balanced error rate (BER), which takes the average of the errors on each class to evaluate the classification error. The optimal parameters can be concluded from Figure 2.8Inspired by the method used in a previous multiomics study,To simplify the downstream analysis, we assembled gene, peptide, and protein identifiers of selected features into a final feature set, unifying the identifiers into the corresponding gene symbol and excluding duplications. The final selections used for overrepresentation analysis, transcriptional regulatory analysis, and validation entailed 291 and 283 gene features, for single and integrative sPLS‐DA, respectively. The full lists of the selected features are presented in Table 2.92.9.1We used the function lmFit provided in the limma R package (v3.42.2) for differential expression analysis on preprocessed transcriptomics between IPH and non‐IPH groups. Positive log2 fold change (Log2FC) values indicate genes have higher expression in IPH group, and vice versa. Results of differential expression analysis for the 17,759 unique detectable genes can be found in Table 2.9.2p‐values were calculated as a cutoff to avoid presenting the false discovery of significant terms. The full list of GSOA results for both single and integrative sPLS‐DA can be found in Table We performed GSOA using the R packages clusterProfiler2.9.3iRegulon2.10n = 29; 13 early and 16 advanced plaques) and GSE43292n = 64; 32 plaque‐free artery segments and 32 atheromatous plaques).The predictive power of genes from the multiomics signature was validated in the following independent human carotid endarterectomy cohorts on microarray gene expression: GSE28829n = 283) and single sPLS‐DA components 1–2  were tested by logistic regression (solver = “lbfgs”), support vector machine  and decision tree (default settings) with stratified five‐folds cross‐validation and 1000 random repeats provided in the scikit‐learn machine learning package (v0.24.1) in Python. As the integrative sPLS‐DA component 1  was dominant in our model, we also compared the performance of the MULTI 1 signature with that of MULTI 1–4 signature from the complete four‐component model. Classification performance using full data (FULL) was evaluated as a baseline. Predictive performance was evaluated by two measures: AUC and accuracy.The prediction performance of the gene lists obtained from integrative sPLS‐DA components 1–4  and proteomics  from the BiKE cohort. Within the BiKE, two subcohorts were defined. For the first, we compared non‐atherosclerotic tissues  versus atherosclerotic arteries , based on their morphological and histological characteristics (HIST). For the second, carotid plaque tissues were classified as asymptomatic  versus symptomatic  based on the patient's clinical presentation (CLIN). For both subcohorts, we tested the classification performance of the identified multiomics signature MULTI 1 on transcriptomics and proteomics, respectively, using logistic regression with the same setting mentioned above for GSE28829 and GSE43292. Gene set enrichment analysis (GSEA) was performed on the signature MULTI 1 against the genes and proteins from BiKE transcriptomics and proteomics, with the genes/proteins ranked by log2 fold change based on the comparisons of HIST and CLIN.BiKE cohort patient inclusion for CEA surgery and enrolment in the biobank, clinical data, sample processing and large‐scale datasets have been described in details previously.2.12p‐values. Classification performances are presented as the mean ± standard deviation (SD). Distribution of plaque traits and IHC‐based MFAP4 quantification for paired plaques are presented as box and whisker plots with the first quartile, median, third quartile, and the largest or smallest values within 1.5 times the interquartile range above the third quartile or below the first quartile, respectively. Statistical significance between groups was evaluated using two‐tailed Wilcoxon rank‐sum test  or Student's t‐test . For the MaasHPS data, paired statistical testing was performed. Shapiro–Wilk test was used for the normality test. Statistical significance between the classification performance was analyzed by Student's t‐test. Significance level is denoted by *p‐value < .05, **p‐value < .01, ***p‐value < .001, ****p‐value < .0001. Statistical analyses were performed in R (v3.6.3).Component–component and component–trait associations were measured by Pearson's correlation coefficients with 3In this study, we deeply phenotyped low‐ versus high‐risk human carotid artery atherosclerotic lesion segments, as defined by the absence or presence of IPH, building an integrated gene/protein classification model for IPH. For this, we interrogated a carotid endarterectomy cohort (MaasHPS) by transcriptomic (microarray), proteomic, and peptidomic analysis (both LC‐MALDI‐MS/MS). Flanking sections were taken for detailed histological examination of cellular and acellular plaque composition and progression stage3.1First, we built single sPLS‐DA models based on the transcriptomic, proteomic, and peptidomic datasets, separately. The optimal parameters for model building can be found in Table Next, we mapped the distribution of the selected feature sets Figure , based o3.2In order to establish a more comprehensive and robust model of IPH, we sought to analyze plaque transcriptomic, proteomic, and peptidomic data in an integrated manner by integrative sPLS‐DA using the optimized parameters in Table p = 1.8E‐08) and hemorrhaged plaque area , as expected, as well as with αSMA–PDGFRα+ fibroblast‐like cells , Arg1+ healing macrophage phenotype , and collagen content of non‐IPH plaque , concordant with the GSOA findings , but not in ETS‐domain protein 4 (ELF4) regulated SRF‐dependent genes  and GSE43292n = 64; mRNA; intact arterial tissue vs. atheroma). Although the cohort setup differs from our cohort, carotid advanced plaque (GSE28829) and atheroma (GSE43292), both largely represent unstable plaque phenotypes (thin cap fibroatheroma and beyond). The main difference is in the control tissue with even lower risk than our nonhemorrhaged stable plaque: that is, early‐stage lesion (low risk) and nonlesioned arterial tissue, in the two validation cohorts, respectively, versus nonhemorrhaged stable plaque in our study, but we argued that our signature may still be valid for these broader disease trajectories.Finally, we assessed whether the IPH signature was able to discriminate low‐ from high‐risk plaques in two independent mRNA cohorts in the public domain: GSE28829Indeed, in both cohorts, the signature genes from integrative sPLS‐DA component 1 (MULTI 1) were differentially expressed between control versus diseased artery Figure , with gen = 127) and proteomics datasets (n = 36) from CEA tissue. The MULTI 1 feature set performed well on both the transcriptomics and proteomics in segregating advanced plaque from healthy carotid tissue This study illustrates the benefits of truly integrative omics analysis in terms of accuracy, visibility, and model interpretability. Our multiomics prediction model achieved similar, if not higher, accuracy in stage classification as a single‐omics model. The prediction power of the selected features was successfully validated on several independent cohorts comparing healthy artery or early (stable) plaque versus advanced (unstable) plaque or atheroma. Even, the selected features performed reasonably well in stratifying plaques from symptomatic versus asymptomatic patients at protein level (BiKE proteomics), although plaques in both groups were having an advanced (unstable) phenotype. However, the true merit of integrative analysis is in the robustness of the model, its biological significance, and the direct clinical translatability of network‐contained proteins. Significances of overrepresented terms in the multiomics signature outperformed those of the single‐omics signature, due to the strong mutual interaction of signature members per component. Interestingly, while the dominant component of the single‐ and multiomics models showed considerable overlap, the auxiliary second to fourth components of the single‐ and multiomics models differed substantially, with those of the latter being much more coherent and biologically meaningful. The dominant component in the multiomics model was overrepresented in collagen metabolism and macrophage inflammation terms, reflecting peri‐rupture modeling processes. The auxiliary components alluded to additional CVD relevant terms, such as lipid and lipoprotein metabolism, neutrophil responses and, again, tissue metabolism, mirroring early responses to hemorrhage and/or rupture. Additionally, we found that the BLVRB, which had been previously proposed by Matic et al.Literature search for the MULTI 1 elements indicated an overall relationship  to vascular smooth muscle function and inflammation in cardiovascular diseases Table . InteresMechanistically, SRF transcriptional responses depend critically on its coactivator. The major coactivator classes are T‐cell factors  and the local Medical Ethical Committee (protocol number 16‐4‐181). This study conforms to the Declaration of Helsinki, all participants have given informed written consent prior to the inclusion.All patient materials were collected in the Maastricht Pathology Tissue Collection in line with the Dutch Code for Proper Secondary Use of Human Tissue (Han Jin conceived the study, processed the data, performed the computational analyses, and drafted and revised the manuscript with input from all co‐authors. Cornelis J.J.M. Sikkink and Barend M.E. Mees performed the CEA surgery. Wouter Eijgelaar, Marco Manca, and Mat J.A.P. Daemen collected the plaque samples. Peter Juhasz performed proteomics and peptidomics experimental analysis. Immunostaining was performed by Judith C. Sluimer, Kim van Kuijk, Olivia Waring, Gregorio E. Fazzi, Marion J.J. Gijbels, and other experienced colleagues. Marco Manca helped in data analysis and multiomics data alignment. Joël M.H. Karel, Evgueni Smirnov, Martina Kutmon, and Chris T.A. Evelo gave input in the data analysis strategies. Ljubica Matic and Ulf Hedin performed analysis on the BiKE cohort dataset. Judith C. Sluimer, Pieter Goossens, Ljubica Matic, Marco Manca, and Olivia Waring offered critical revision. Erik A.L. Biessen conceived and supervised the study, provided funding and critical feedback at all stages.Supporting InformationClick here for additional data file.Supporting InformationClick here for additional data file.Supporting InformationClick here for additional data file.Supporting InformationClick here for additional data file.Supporting InformationClick here for additional data file.Supporting InformationClick here for additional data file.Supporting InformationClick here for additional data file.Supporting InformationClick here for additional data file.Supporting InformationClick here for additional data file.Supporting InformationClick here for additional data file.Supporting InformationClick here for additional data file.Supporting InformationClick here for additional data file.Supporting InformationClick here for additional data file.Supporting InformationClick here for additional data file.


Single-cell RNA-sequencing (scRNA-seq) provides more granular biological information than bulk RNA-sequencing; bulk RNA sequencing remains popular due to lower costs which allows processing more biological replicates and design more powerful studies. As scRNA-seq costs have decreased, collecting data from more than one biological replicate has become more feasible, but careful modeling of different layers of biological variation remains challenging for many users. Here, we propose a statistical model for scRNA-seq gene counts, describe a simple method for estimating model parameters and show that failing to account for additional biological variation in scRNA-seq studies can inflate false discovery rates (FDRs) of statistical tests.First, in a simulation study, we show that when the gene expression distribution of a population of cells varies between subjects, a naïve approach to differential expression analysis will inflate the FDR. We then compare multiple differential expression testing methods on scRNA-seq datasets from human samples and from animal models. These analyses suggest that a naïve approach to differential expression testing could lead to many false discoveries; in contrast, an approach based on pseudobulk counts has better FDR control.aggregateBioVar, is freely available on Bioconductor (https://www.bioconductor.org/packages/release/bioc/html/aggregateBioVar.html) to accommodate compatibility with upstream and downstream methods in scRNA-seq data analysis pipelines.A software package, Bioinformatics online.Raw gene-by-cell count matrices for pig scRNA-seq data are available as GEO accession GSE150211.  Single-cell RNA-sequencing (scRNA-seq) enables analysis of the effects of different conditions or perturbations on specific cell types or cellular states. Multiple methods and bioinformatic tools exist for initial scRNA-seq data processing, including normalization, dimensionality reduction, visualization, cell type identification, lineage relationships and differential gene expression (DGE) analysis (marker detection methods); many methods have been developed to perform this analysis  mixed models, where models for gene expression are adjusted for sample-specific or batch effects. This study found that generally pseudobulk methods and mixed models had better statistical characteristics than marker detection methods, in terms of detecting differentially expressed genes with well-controlled false discovery rates (FDRs), and pseudobulk methods had fast computation times. In another study, mixed models were found to be superior alternatives to both pseudobulk and marker detection methods  subject-specific factors (e.g. disease and intervention), (ii) variation between subjects, (iii) variation between cells within subjects and (iv) technical variation introduced by sampling RNA molecules, library preparation and sequencing. Second, we make a formal argument for the validity of a DS test with subjects as the units of analysis and discuss our development of a Bioconductor package that can be incorporated into scRNA-seq analysis workflows. Third, we examine properties of DS testing in practice, comparing cells versus subjects as units of analysis in a simulation study and using available scRNA-seq data from humans and pigs. Finally, we discuss potential shortcomings and future work.In bulk RNA-seq studies, gene counts are often assumed to follow a negative binomial distribution  study design, in which many cells from a cell type of interest could be isolated and profiled using bulk RNA-seq. The observed counts for the PCT study are analogous to the aggregated counts for one cell type in a scRNA-seq study. Furthermore, guidelines for library complexity in bulk RNA-seq studies apply to data with heterogeneity between cell types, so these recommendations should be sufficient for both PCT and scRNA-seq studies, in which data have been stratified by cell type.Theorem 1 provides a straightforward approach to estimating regression coefficients aggregateBioVar (available on Bioconductor) to facilitate broad adoption of pseudobulk-based DE testing; aggregateBioVar includes a detailed vignette, has low code complexity and minimal dependencies and is highly interoperable with existing RNA-seq analysis software using Bioconductor core data structures  that are differentially expressed. The recall, also known as the true positive rate (TPR), is the fraction of differentially expressed genes that are detected. According to this criterion, the subject method had the best performance, and the degree to which subject outperformed the other methods improved with larger values of the signal-to-noise ratio parameter subject method had a larger area under the curve than the other six methods in all simulation settings, with larger differences for higher signal-to-noise ratios. When only 1% of genes were differentially expressed, the mixed method had a larger area under the curve than the other five methods.P-value will be chosen to detect genes. P-value cutoff of 0.05. NPV is the fraction of undetected genes that were not differentially expressed.In practice, often only one cutoff value for the adjusted subject method had the highest PPV, and the NB method had the lowest PPV in all nine simulation settings. Among the other five methods, when the number of differentially expressed genes was small  curves, TPRs and false positive rates (FPRs) can be found in subject method had the shortest average computation times, typically <1 min. Four of the cell-level methods had somewhat longer average computation times, with MAST running for 7 min, wilcox and Monocle running for 9 min and NB running for 18 min. Two of the methods had much longer computation times with DESeq2 running for 186 min and mixed running for 334 min.The computations for each method were performed on the high-performance computing cluster at the University of Iowa. The cluster contains hundreds of computation nodes with varying numbers of processor cores and memory, but all jobs were submitted to the same job queue, ensuring that the relative computation times for these jobs were comparable. In addition to simulated data, we analysed an animal model dataset containing large and small airway epithelia from CF and non-CF pigs  > 1.3), whereas the other six methods detected a much larger number of genes. The number of genes detected by wilcox, NB, MAST, DESeq2, Monocle and mixed were 6928, 7943, 7368, 4512, 5982 and 821, respectively. Among the three genes detected by subject, the genes CFTR and CD36 were detected by all methods, whereas only subject, wilcox, MAST and Monocle detected APOB. Importantly, although these results specifically target differences in small airway secretory cells and are not directly comparable with other transcriptome studies, previous bulk RNA-seq  and the subject-level method (subject), with mixed sharing modest concordance with both clusters.P-values. The subject and mixed methods are composed of genes that have high inter-group (CF versus non-CF) and low intra-group (between subject) variability, whereas the wilcox, NB, MAST, DESeq2 and Monocle methods tend to be sensitive to a highly variable gene expression pattern from the third CF pig.P-values computed by each method, we compared them to the unadjusted P-values obtained from a permutation test. First, the CF and non-CF labels were permuted between subjects. For this study, there were 35 distinct permutations of CF and non-CF labels between the 7 pigs. Then, for each method, we defined the permutation test statistic to be the unadjusted P-value generated by the method. To obtain permutation P-values, we measured the proportion of permutation test statistics less than or equal to the observed test statistic, which is the permutation test statistic under the observed labels. For each method, we compared the permutation P-values to the P-values directly computed by each method, which we define as the method P-values.In order to determine the reliability of the unadjusted P-values and method P-values. Because the permutation test is calibrated so that the permuted data represent sampling under the null distribution of no gene expression difference between CF and non-CF, agreement between the distributions of the permutation P-values and method P-values indicate appropriate calibration of type I error control for each method. Overall, the subject and mixed methods had the highest concordance between permutation and method P-values. Specifically, the CDFs are in high agreement for the subject method in the range of P-values from 0 to 0.2, whereas the mixed method has a slight inflation of small P-values in the same range compared to the permutation test. All of the other methods compute P-values that are much smaller than those computed by the permutation tests. These results suggest that only the subject method will exhibit appropriate type I error rate control. The lists of genes detected by the other six methods likely contain many false discoveries.subject method better controlled the FPR of DS analysis when the expected rate of true positives is small; here, using the same animal model, we compare large and small airway ciliated cells which are expected to vary largely.Our analysis of CF and non-CF pigs showed that the subject, wilcox, NB, MAST, DESeq2, Monocle and mixed, respectively. The volcano plots for subject and mixed show a stronger association between effect size (absolute log2-transformed fold change) and statistical significance .P-values for each method. If we omit DESeq2, which seems to be an outlier, the other six methods form two distinct clusters, with cluster 1 composed of wilcox, NB, MAST and Monocle, and cluster 2 composed of subject and mixed. The intra-cluster correlations are between 0.9 and 1, whereas the inter-cluster correlations are between 0.51 and 0.62. This figure suggests that the methods that account for between subject differences in gene expression (subject and mixed) will detect different sets of genes than the methods that treat cells as the units of analysis.P-values. All seven methods identify two distinct groups of genes: those with higher average expression in large airways and those with higher average expression in small airways. The subject and mixed methods show the highest ratios of inter-group to intra-group variation in gene expression, whereas the other five methods have substantial intra-group variation. This suggests that methods that fail to account for between subject differences in gene expression are more sensitive to biological variation between subjects, leading to more false discoveries.wilcox, NB, MAST, DESeq2 and Monocle, in the simulations and animal model analysis, we only show the results for subject, wilcox and mixed. We performed marker detection analysis of cells obtained from a study of five human skin punch biopsies , the authors found that the basal cell population contained specialized subtypes  than subject (TPR = 0.34). On the other hand, subject had the smallest FPR (0.03) compared to wilcox and mixed  and had a higher PPV (0.38 compared to 0.10 and 0.23). Overall, mixed seems to have the best performance, with a good tradeoff between false positive and TPRs.subject, wilcox and mixed for DS analysis of the scRNA-seq from healthy and IPF subjects within AT2 and AM cells using bulk RNA-seq of purified AT2 and AM cell type fractions as a gold standard, similar to the method used in Section 3.5.We evaluated the performance of our tested approaches for human multi-subject DS analysis in health and disease. subject, wilcox and mixed to test for differences in expression between healthy and IPF subjects within the AT2 and AM cell populations. Overall, the volcano plots for subject and mixed look similar with a higher number of genes upregulated in the IPF group, while the wilcox method exhibits a much different shape with more genes highly downregulated in the IPF group. P-values less than 0.05 and at least a 2-fold difference in gene expression between healthy and IPF are considered true positives and all others are considered true negatives. subject and mixed have the same area under the ROC curve (0.82) while the wilcox method has slightly smaller area (0.78). Further, subject has the highest AUPR (0.21) followed by mixed (0.14) and wilcox (0.08). For the AM cells , followed by mixed  and wilcox . The subject method has the strongest type I error rate control and highest PPVs, wilcox has the highest TPRs and mixed has intermediate performance with better TPRs than subject yet lower FPRs than wilcox  suggested a modeling framework for scRNA-seq data from multiple biological sources, (ii) showed how failing to account for biological variation could inflate the FDR of DS analysis and (iii) provided a formal justification for the validity of ‘pseudobulking’ to allow DS analysis to be performed on scRNA-seq data using software designed for DS analysis of bulk RNA-seq data  and the non-CF group  would be different, but the pseudobulk method may not detect a difference, because it is only able to detect differences in mean expression. A richer model might assume cell-level expression is drawn from a non-parametric family of distributions in the second stage of the proposed model rather than a gamma family.mixed method in Section 3. The study by Zimmerman et al. provides an argument for using mixed models over pseudobulk methods because pseudobulk methods discovered fewer differentially expressed genes. In our simulation study, we also found that the pseudobulk method was conservative, but in some settings, mixed models had inflated FDR. A more powerful statistical test that yields well-controlled FDR could be constructed by considering techniques that estimate all parameters of the hierarchical model. More conventional statistical techniques for hierarchical models, such as maximum likelihood or Bayesian maximum a posteriori estimation, could produce less noisy parameter estimates and hence, lead to a more powerful DS test  and GSE143706 (scRNA-seq). The scRNA-seq data for the analysis of human lung tissue were obtained from GEO accession GSE122960, and the bulk RNA-seq of purified AT2 and AM fractions were shared by the authors immediately upon request.btab337_supplementary_dataClick here for additional data file.