ACACB, ADAMTS6, NCAM1, and CADPS in Task 1, and CD1D, PLA2G16, DACH1, and SOWAHA in Task 2. According to The Cancer Genome Atlas dataset, these genes are all prognostic factors for pancreatic cancer. Thus, the feasibility of using our deep learning-based method for the selection of genes associated with pancreatic cancer development and prognosis was confirmed.The selection of genes that are important for obtaining gene expression data is challenging. Here, we developed a deep learning-based feature selection method suitable for gene selection. Our novel deep learning model includes an additional feature-selection layer. After model training, the units in this layer with high weights correspond to the genes that worked effectively in the processing of the networks. Cancer tissue samples and adjacent normal pancreatic tissue samples were collected from 13 patients with pancreatic ductal adenocarcinoma during surgery and subsequently frozen. After processing, gene expression data were extracted from the specimens using RNA sequencing. Task 1 for the model training was to discriminate between cancerous and normal pancreatic tissue in six patients. Task 2 was to discriminate between patients with pancreatic cancer (n = 13) who survived for more than one year after surgery. The most frequently selected genes were  A great deal of research is now being conducted to apply the superior inference performance of deep learning to the field of medicine8. In deep learning, a large amount of data is required for training the model in order to extract the features that are useful for inference. In particular, in the field of image recognition, it is common to use convolutional layers that incorporate convolutional operations to capture local patterns in an image and emphasize specific features of the image. By layering multiple convolutional layers, complex higher-order features can be extracted, and image features for classification and inference with high accuracy can be extracted automatically. Therefore, deep learning has a very high proficiency for medical image processing, and various applications have been reported.Deep learning techniques have produced impressive results in many fields, including image classification, image generation, automated driving, and even board gameplayHowever, unlike images, the gene expression data used in this study are table-type numerical data that represent the expression state of each gene and are characterized by the fact that many features (genes) are interrelated. When analyzing such table-type numerical data using deep learning, it is necessary to consider all possible interactions between the input features. This is very different to processes like convolutional operations that only consider the interactions taking place around the pixel of interest, such as those in images. This means that we must consider the global interaction of all features using dense layers when analyzing such table-type numerical data. By combining these dense layers multiple times, we expect to be able to extract higher-order features that are effective for inferences.11. For typical machine learning models such as logistic regression, random forests, and gradient boosting, we can interpret the basis for their inferences to some extent. On the other hand, it is generally difficult to reason out the inference results of neural networks. This is because neural networks repeatedly perform many layers of sum-of-product computation and nonlinear transformation on the input data in order to output predictions. In image processing, techniques such as Grad-CAM12 can be used to visualize the parts of the image the model has focused on to make inferences, thereby helping us to interpret the nature of deep learning. When using a neural network with multiple fully connected dense layers in table-type numerical data, millions of operations, or even tens of millions of operations, can be performed. Although understanding the exact flow of conversions that turns input data to output prediction is practically hard in a such process, the input features that affected the inference result by using model-independent methods such as Permutation Feature Importance13 and SHAP14 can be evaluated. However, the applicability of these methods to data with a low number of samples, more than 10,000 features, and complex interactions between the features is limited.In recent years, there has been a great deal of research on the interpretability of machine learning models15. One health area where personalized medicine may be of significant benefit is cancer treatment. Pancreatic cancer is a malignant tumor with a poor prognosis. Surgery and multidisciplinary treatments, such as anticancer drug therapy and radiotherapy, are important for ensuring favorable pancreatic cancer outcomes. However, the optimal treatment method may differ for each patient. To practically realize personalized medicine, genes significantly associated with a cancer subtype need to be identified to stratify patients based on gene expression data19. However, the identification of important genes from complex gene data sets comprising tens of thousands of expression statuses is a considerable challenge. Various methods based on machine learning have been proposed for the feature selection of gene expression data, but there is no consensus on a standard method to be used23.Current medical practices and healthcare industry obligations require the involvement of personalized medicine to be able to provide more appropriate treatmentFeature selection based on machine learning can be categorized into the filter, wrapper, and embedding methods. The filter method, also known as feature ranking, is a method that quantifies and evaluates each feature according to a certain index. While it is computationally inexpensive, it is difficult to take into account the interrelationships among the multiple features and to evaluate them numerically. The wrapper method takes into account the interrelationships among the features by dynamically changing the feature subset to be evaluated, but the computational cost tends to be high because the machine learning model is trained each time the feature subset is evaluated. The embedding method is one where feature selection is done simultaneously with the training of the machine learning model, and it allows for the elucidation of the interrelationships among the features at reduced computational cost. However, depending on the machine learning model used, some problems can occur, such as over-fitting and difficulty in interpreting the results. Thus, each method has its own merits and demerits.Particularly in the case of gene expression data, where the number of samples is small and the number of features (genes) is very large, it is common for the results to be completely different for each method. In addition, since the behaviors and functions of all genes are not yet known, it is useful to evaluate the results comprehensively from various perspectives using a wide variety of feature selection methods in order to make a truly appropriate gene selection. Therefore, the main purpose of this study was to propose a methodology of feature selection based on deep learning, which includes a feature selection layer that can evaluate the importance of features in a deep learning model, and to show the feasibility of a new methodology for gene selection that has not been used before.Patient backgrounds are summarized in Table https://www.cancer.gov/about-nci/organization/ccg/research/structural-genomics/tcga).In this study, we experimented with the selection of important genes over two separate tasks. The first task was to discriminate between pancreatic cancer and normal pancreatic tissue (Task 1). The second task was to discriminate patients with pancreatic cancer who survived for more than one year after surgery (Task 2). Task 1 used gene expression data of pancreatic cancer and normal tissues sampled from six patients. Task 2 used gene expression data obtained from 13 pancreatic cancer tissues . The selected genes from Tasks 1 and 2 were validated using the Cancer Genome Atlas dataset , the third-ranked genes were different . In Task 2, CD1D and PLA2G16 were selected within the top three for both CD1D was selected first for PLA2G16 was ranked second for SOWAHA was chosen in the third place for DACH1 in the second place for In each task, the genes selected with https://www.cancer.gov/about-nci/organization/ccg/research/structural-genomics/tcga)) data for all genes in the top 10 ranking for each task demonstrated the significance of 6 out of the 10 genes in Task 1 . Task 1 involved the selection of genes for effective binary classification between pancreatic cancer tissue and normal pancreatic tissue. Task 2 involved a binary classification between better prognosis and worse prognosis. Because Task 2 directly searched for prognostic determinants, it was unsurprising that the survival analysis in TCGA dataset showed more differences in Task 2 than in Task 1.ACACB encodes acetyl-CoA carboxylase (ACC) 24. ACC CD1D (the top-ranked gene in Task 2) encodes CD1d, a transmembrane glycoprotein structurally related to major histocompatibility complex (MHC) proteins that activate natural killer T cells (NKT cells)25. The high ranking of CD1D may reflect the level of immunity it possesses against tumors. Furthermore, the other ranked genes are also likely to be strongly involved in the progression or control of pancreatic cancer, although further studies are needed to elucidate the details.k features in total, the number of all combinations of features is k is sufficiently large, it is practically impossible to evaluate all feature subsets in a full search, and a feature selection algorithm is used to select Feature selection is often used as a preprocessing step for machine learning and usually improves model performance and learning speed. If there are Our deep learning model that uses the feature selection layer can evaluate ultra-high-dimensional features, and the importance of features can be learned by considering the complex interactions among the multiple features using dense layers. By inserting the feature selection layer immediately after the input layer, the input vector, which is multiplied by the weights of the feature selection layer and performs a nonlinear transformation, is regarded as the input to the subsequent network. The weights of the feature-selection layer are successively updated to increase the discrimination performance of the subsequent network through learning. Hence, the weights of the feature selection layer are considered to represent the importance of the input features, and features with higher weights are considered to be more effective in determining the discrimination capability of the later networks than features with lower weights, which do not contribute significantly to the performance of the network.This study has several limitations. Because the number of samples used was very small , we could not prepare validation data when training the network, and we could not use external test data for evaluation. To improve the reliability of the output results, multiple trials were conducted, and further validation using TCGA database was performed. In addition, since the purpose of this study was to evaluate the feasibility of a new approach to discovering unknown genes related to cancer, we did not compare our method with other gene selection methods.In conclusion, the use of our deep learning-based gene selection method for the selection of genes associated with pancreatic cancer development and prognosis was feasible. This method can be applied to gene expression data obtained for all diseases and may contribute to cancer stratification and to improving personalized medicine.The protocol was approved by the Institutional Review Board of Chiba Cancer Center (No. 28-15) and all patients and healthy volunteers provided their written informed consent. The study was carried out in accordance with the World Medical Association’s Declaration of Helsinki. In this study, 13 patients diagnosed with pancreatic ductal adenocarcinoma (PDAC) between January 2013 and December 2017 were included. All patients underwent surgery, and both specimens of cancer and frozen samples of nearby normal pancreatic tissue were collected. No preoperative chemotherapy was administered to any patient. Patients were observed for at least one year after surgery.Total RNA was extracted from frozen tissue blocks containing Deep learning can automatically extract the features needed for inference directly from raw data. However, the extracted feature information is distributed among the weights of the multilayered network, and it is usually difficult to obtain information about the basis of the inference in a form that humans can understand. We hypothesized that the addition of a feature selection layer in the deep learning model would solve this problem. The feature selection layer multiplies the input by weight in a one-to-one relationship and outputs vectors with the same shape as the input. Thus, the values of the input features are increased or decreased by the weights and passed to the next network.A conceptual diagram of the feature-selection layer is shown in Fig. Figure The gene expression data used in this study were based on the expression levels of 11,273 genes measured in 13 patients with pancreatic cancer. The gene expression data were performed the min-max normalization in order to present all values between 0 and 1. A feature selection layer with the same number of units as the genes was placed immediately after the input layer with the 11,273 units. The classification network was then placed behind it. Each layer in a neural network can be regarded as a kind of feature extractor. In other words, it can be thought of as a process that creates new features by applying complex nonlinear transformations from all the outputs of the previous layer. In particular, in the case of dense layers, the neurons in each layers consider the output of neurons in the previous layer as a new feature, and combine all of these features to produce a new feature. In general, neural networks are trained in such a way that the weights for features that contribute to reducing the loss function and the weights for unnecessary features are increased or decreased. Due to the min-max normalization for data and the non-negative regularization for feature selection layer, training proceeds in such a way that weights corresponding to features that are not necessary for minimizing the loss function are reduced. After model training, the weight of each unit in the feature selection layer is considered to represent the importance of the corresponding features.N positions after the network was trained K times. The ranking was sorted in descending order by weight corresponding to each feature in each training trial.Our method can be categorized as an embedding type of feature selection method. However, neural networks also include random factors, such as the initial values of weights and the order in which the training data are provided. In addition, when a neural network learns using high-dimension data from a low number of samples, the network may not be appropriately trained, and the weights of the feature selection layer may not be reliable. Thus, in this study, the importance of the genes was evaluated in terms of the number of times the genes were ranked in the top http://r2.amc.nl) to separate the gene expression data of TCGA data into two groups and select the best Kaplan–Meier curves based on a log-rank test.The differences between patients’ backgrounds were assessed using Fisher’s exact test and the Mann–Whitney U test. OS was defined as the period between surgery and final observation (in days), whereas DFS was defined as the time from surgery to tumor recurrence or death. A P-value of less than 0.05 was considered significant. These statistical analyses were conducted using JMP Pro version 13.2.0 . Survival analyses were performed using the R2: Kaplan Meier Scanner (


We also briefly review feature selection methods, multi-Omics data sets, and resources/tools that constitute critical components for carrying out the integration.Through the developments of Omics technologies and dissemination of large-scale datasets, such as those from The Cancer Genome Atlas, Alzheimer’s Disease Neuroimaging Initiative, and Genotype-Tissue Expression, it is becoming increasingly possible to study complex biological processes and disease mechanisms more holistically. However, to obtain a comprehensive view of these complex systems, it is crucial to integrate data across various Omics modalities, and also leverage external knowledge available in biological databases. This review aims to provide an overview of multi-Omics data integration methods with different statistical approaches, focusing on  Moreover, curated databases for Omics interactions , and molecular pathways  www.genome.jp/kegg/, Reactome reactome.org/, and functional protein association networks (STRING) string-db.org/) are also available to incorporate known biological information in the Omics analysis. Environmental/clinical features are external sources of influence that can play a key role in the development of complex diseases  diseases . Incorpodiseases  is partisupervised and unsupervised are often used to describe different approaches to data integration. Supervised methods train a model using labeled training data with known outcome variables . In contrast, unsupervised data integration consists of a class of methods that make inferences and find patterns in input data sets without labeled outcome variables . Unsupervised multi-Omics approaches typically aim to classify  and discover biomarkers/modules (such as prioritize genes associated with a disease). There might be multiple outcome variables  which are mostly considered one-by-one in the available methods (instead of using multivariate models). Note that multiple Omics data often contain missing values, an issue particularly common for individuals with measurements by selected Omics modalities. Imputation is a typical solution to infer the missing values, see  or in a specific field . On thefinition  and featfinition  is proviThe term Omics refers to the collective characterization and quantification of biomolecules that are involved in the structure, function, and dynamics of organisms and biological processes. Categorizing the multi-Omics data integration methods is not a trivial task. There is a huge list of diverse methodologies with different objectives. One way to systematically categorize these methods is to consider their underlying statistical strategies, their biological objective, and the way they handle and treat multiple Omics datatypes. For instance, some methods  concatenate the multi-Omics data from different molecular layers to a single matrix as the input data see . WhereasRegression/Association-based Methods, 2) Clustering-based Methods, and 3) Network-based Methods. In each category, we group the methods based on their statistical approaches (see We categorize the integration methods into the following three comprehensive categories: 1) ches see . Each ofches see -4. “DatEmulti-Omics unsupervised regression-based methods into three distinct categories based on their statistical approaches, including sequential analysis, CCA- and CIA-based, and factor analysis-based methods  based on their relationship with features in other Omics layers . Multivariate analysis  is another popular approach for multi-Omics data integration due to its flexibility in accepting multiple matrices as input data. The kernel-based method gives an excellent opportunity to work with lower space similarity kernels  instead of the original (raw) Omics data. We grouped CNAmet  to calculate a global score for each gene. These scores help identify whether a gene is hypomethylated (and upregulated) or hypermethylated (and downregulated). The main hypothesis is that amplified copy number and hypomethylation result in gene upregulation. iPAC (in-trans Process Associated and cis-Correlated) , then filters the remaining genes based on in-cis correlation (>0.6), and finally checks the in-trans functionality for the remaining ones that make the final set of gens. MEMo  (via a permutation test.CNAmet  is a biorelated)  is an unModules)  is a modIllustrative Case-studies: CNAmet is applied to a cohort of glioblastoma multiforme (GBM) patients from TCGA to find synergistically regulated genes by DNA methylation CNV. It could identify this synergistic effect on well-known oncogenes, including MDM2, EGFR, and PDGFRA. CNAmet also showed that GBM patients with hypomethylated (upregulated) EGFR had a better prognosis than patients with amplified EGFR. iPAC is applied to a cohort of breast carcinoma patients. It identified a list of significant genes, including ERBB2, MAP3K7, MDM4, FGFR1, CCND1, and FADD, which are well-known cancer-associated genes. It also included some less appreciated genes such as ATAD2, TPD52, and PPM1D, which were reported as cancer genes in previous independent studies  family of proteins and metal-binding ability. studies . iPAC coCCA-based methods    is a samCIA-based methods  are computed by maximizing the covariance between the data tables: methods  is anoth). MCIA     is the enalysis)  is a spaIllustrative Case-studies: Sparse MCCA is applied on a diffuse large B-cell lymphoma dataset to assess the relationships (such as co-amplification and codeletion) between copy number changes in genome regions on separate chromosomes    is an unIllustrative Case-studies: MOFA is applied to a cohort of patients with chronic lymphocytic leukemia (CLL) to integrate mRNA expression, DNA methylation, somatic mutation, and drug response. It identified two important  markers, including the IGHV gene (immunoglobulin heavy-chain variable) and trisomy of chromosome 12. However, MOFA could find a more comprehensive and complex sub-structure for IGHV and connect it with multiple Omics, including changes in mRNA expression , DNA methylation , and with drugs  that target kinases in the B-cell receptor pathway  as one of the critical drivers which was previously underappreciated in the context of CLL. The results (factors) of MOFA are then used in a Cox-PH regression model and could predict the time to the next treatment with a reasonably high prediction accuracy (C-index∼75%). In the second and the third applications, MOFA is used to analyze Ustekinumab (UST) drug-response  kernel-based, 2) (non-negative) matrix factorization-based-based, 3) Bayesian, and 4) multivariate and other clustering methods  (DSSF (Deep Subspace Similarity Fusion) (AFN (Affinity Network Fusion) (NEMO (NEighborhood based Multi-Omics clustering) (INF (Integrative Network Fusion) ( Fusion)  is a pop Fusion)  to fuse  Fusion)  employs  Fusion)  is also stering)  is intro Fusion)  is anoth Fusion)  (Random  Fusion)  (Linear Illustrative Case-studies: SNF is applied to different multi-omics data integration studies. In the 1st application (2018) (n (2018) , it is aStandard factorization methods commonly use singular value decomposition (SVD), such as PCA. However, for some data types, such as genotypes, the original matrices are non-negative. SVD-based factorizations contain negative entries, making it difficult to interpret their results in some applications. In contrast, NMF (Nonnegative Matrix Factorization)  restrictiCluster (iNMF (integrative Non-negative Matrix Factorization) (via minimizing the following loss function (using partitioned factorization structure):iCluster  simultaniCluster  and a (s -means . It firsization)  is a mulius norm . The autIllustrative Case-studies: iCluster is applied in different studies, mainly for cancer subtyping. It is recently applied to a cohort of ovarian carcinoma patients  to identify prognostic biomarkers (UBB (ubiquitin B) and IL18BP (interleukin 18 binding protein) genes as the most prognostic biomarkers. The results suggested that lower expression of these two genes may result in higher methylation and lower CNV. Therefore, evaluating the expression of these two genes can help in early tumor diagnosis. In another study, iCluster is applied to multi-Omics data of adult soft tissue sarcomas  cases with high expression of MYLK, MYH11, ACTG2, miR-143, miR-145, lower inferred activity of the apoptosis pathway, and higher hormone receptor (ER/PR) levels. It inferred PI3K/AKT pathway activity. The authors also concluded that copy number changes were the most informatics Omics in characterizing these sarcomas (except SS).omarkers . The ressarcomas . iClustepriori assumption about what the true relationship between the data might be, which is expressed as a probability distribution. This probability is then updated as new observations become available (which is captured by a posterior distribution). This approach enables the use of prior information informing the clusters . The DPM model is one of the most widely used Bayesian nonparametric methods in the multi-Omics (multitype) clustering-based data integration. For a tutorial on DPM models, refer to  model ) subjectrefer to .LRAcluster (Low-Rank-Approximation) (MDI (Multiple Dataset Integration)  (PSDF (Patient-Specific Data Fusion) (BCC (Bayesian Consensus Clustering) (imation)  is a lowrefer to  for moregration) , TMD (Trscovery) , PSDF (P Fusion) , and BCCstering)  are fourstering) .Illustrative Case-studies: LRAcluster is applied in a study of hepatocellular carcinoma (HCC) as the major subtype of liver cancer  significantly expressed between the primary tumor compare and portal vein tumor thrombosis. Therefore, an individualized differential analysis for sequencing data was proposed to automate the process of finding these individualized genes.r cancer  to charaCOCA (Cluster-Of-Cluster Assignment) (PINS (Perturbation clustering for data INtegration and disease Subtyping) (ignment)  integratbtyping)  is a disIllustrative Case-studies: COCA has recently been applied to a cohort of Ugandan cervical carcinoma patients  in these patients. Therefore, the authors hypothesized that clade A9-infected samples might have a more active HPV infection. In another application on glioblastoma cancer patients  and 15 gene mutations, including PCDH1, CYP27B1, LPIN3, GPR32, BCL6, OR4Q3, MAGI3, SKIV2L, PCSK5, AKAP12, UBE3B, MAP4, TP53BP1, F5, RHOBTB1.patients . They copatients , COCA counsupervised multi-Omics network methods into four distinct categories based on their statistical approaches, including 1) matrix factorization-based, 2) Bayesian, 3) network propagation-based, and 4) correlation-based and other networks  that exhibit similar profiles and tend to be functionally related/co-regulated (such as gene modules) and to identify the functional relationships between different biological features . Network-based approaches do not rely solely on statistical models, but also leverage information about functional relationships and interactions available in biological knowledge databases, when integrating multi-Omics data. A network is a graphical representation (including nodes and edges) of the relationships between discrete entities. In computational network biology, nodes usually represent different features , and edges represent the relationship between pairs of nodes. When two nodes are sharing an edge, they are called neighbors, adjacent, or directly connected. The adjacency matrix of a network is then an NBS (Network-Based Stratification) (https://string-db.org/), Pathway Commons (https://www.pathwaycommons.org/), and HumanNet (https://www.inetbio.org/humannet/download.php), and constructs the patient-by-gene matrix (FUSENET (DFMF (Medusa (DFMF (Data Fusion by Matrix Factorization) (MAE (Multi-view factorization AutoEncoder) . Note that MAE can also be categorized as a supervised (deep) neural networks method. RAIMC (RBP-AS Target Prediction Based on Inductive Matrix Completion) (ication)  is a sam FUSENET  and DFMFET (DFMF  are flex (Medusa  is a modization)  to constEncoder)  is a completion)  is basedpletion)  for a copletion)  and Disopletion) .Illustrative Case-studies: NBS is applied for patient-subtype identification and discriminating the somatic mutation profiles in uterine, ovarian, and lung cancer studies obtained from TCGA. The survival result based on the identified subtypes showed that ovarian cancer patients with the most aggressive tumor had a mean survival of 32 months compared to others (∼80 months). The fibroblast growth factor (FGF) signaling pathway was enriched for this sub-network of patients with the worst survival in concordance with previous studies indicating the FGF signaling pathway as a driver of tumor progression resistant to anti-VEGF therapy  that have been referred to as BRCAness in previous studies ( therapy . The nex studies .via Markov property (PARADIGM (PAthway Recognition Algorithm using Data Integration on Genomic Models) (CONEXIC (Bayesian networks (BNs) are a combination of (directed acyclic) graph/network theory and probability models. Suppose property  of BNs: πX(b) . The mai πX(b) . PARADIG Models)  can also(CONEXIC  is anothIllustrative Case-studies: CONEXIC is applied to gene-CNV paired data from melanoma patients (CCNB2 (cell-cycle regulator) was selected from a large, amplified region. Finally, an automated literature-mining method called LitVAn  was used to find overrepresented terms in published studies. It resulted in a few well-known activated features in melanoma  and a novel process called “RAB” (Rabs regulate vesicular trafficking).patients  to identNetwork Propagation (NP)  (RWR-M (RWR-MH (TieDIE (Tied Diffusion through Interacting Events) (SNF (Unsupervised Multi-omics Data Integration Methods.on (NP)  is a stotart—RWR  is a proigraphs)  is one o) (RWR-M  and RWR- (RWR-MH ]. TieDIE Events)  accepts ts) , whereas another distinct subtype of patients with overexpressed CTSD and less responsive to TMZ [which is consistent with an in vitro study     is a genmon-Tree  is a bio sampler . It thenNetwork)  is a samNetwork)  (ANNs), Network)  model toIllustrative Case-studies: Lemon-Tree has been applied to TCGA glioblastoma expression and copy-number data  where all involved in glioblastoma driver pathways, including proliferation, apoptosis, and angiogenesis pathways. Besides the well-known genes, Lemon-Tree could also identify a few novel markers that have rarely or never studied glioblastoma. For instance, INSR was involved in several modules. It stimulates cell proliferation and is aberrantly expressed in cancer cells (INSR in glioblastoma may enhance proliferation. PAOX (polyamine oxidase) is another novel marker that might have tumor suppressor activity via amine oxidase activity and their primary involvement in cancer growth inhibition and progression  multi-step and sequential analysis (MS-SA), (B) data-ensemble (DatE), and (C) model-ensemble (ModE) (see This paper reviews key methodologies to perform odE) see .The majority of multi-Omics integration methods were applied to cancer data and mainly focused on genome and transcriptome integration. Therefore, the community needs to devote more efforts to make more publicly available data sources with more diverse Omics profiles (such as metabolome) and environmental/health factors. Many of the reviewed methods use custom pipelines where combinations of multiple methods are employed to answer the underlying biological question. Therefore, many of these methods are highly dependent on the input Omics data and prior information, making it difficult to compare these methods.There are a few benchmarking studies for some of the methods we reviewed here (mostly clustering methods). For instance, comparison and benchmarking of unsupervised multi-Omics clustering algorithms  have been performed using both real data (multiple cancer from TCGA  and a daFuture efforts should be directed toward 1) integrating more various types of data , 2) integrating into a universal pipeline, and 3) integrating the a priori biological knowledge into the system. For instance, in most cases, we have access to quantitative trait information, which can help to improve the feature’s weight assignment and prioritization and increase the accuracy of the prediction/classification tasks. One of the key challenges in integrating large-scale and heterogeneous Omics data is the small sample size and, therefore, most of the methods are data-hungry. One informative way around this issue is to leverage this extra a priori biological information into the method. Although it is beyond the scope of this review, many of the reviewed methods can, in principle, leverage this extra information. Unsupervised deep learning (DL) methods can be a good solution for considering the biological structure among the -Omics data, such as the hierarchical path from DNA to RNA and further to protein. Therefore, more effort should be devoted to utilizing DL for multi-Omics data integration problems with limited  sample sizes. Moreover, as noted in the Introduction, there might be multiple outcome variables  which are mostly considered one-by-one in the available methods. Multivariate modeling  of multi-Omics profiles may provide a more realistic picture than looking at a single outcome, and therefore provides a more powerful test of significance. Lastly, most of the reviewed methods are applied on two or three different Omics modalities, however, in principle/theory, it is possible to extend these methods for more than 2 modalities, although the technical issues become more involved.


High-throughput sequencing technology provides an efficient method for evaluating microbial ecology. Different bioinformatics pipelines can be used to convert 16S ribosomal RNA gene amplicon sequencing data into an operational taxonomic unit (OTU) table that is used to analyze microbial communities. It is important to assess the robustness of these pipelines, each with specific algorithms and/or parameters, and their influence on the outcome of statistical tests. Articles with publicly available datasets on the oral microbiome were searched for, and five datasets were retrieved. These were from studies on changes in microbiota related to smoking, oral cancer, caries, diabetes, or periodontitis. Next, the data was processed with four pipelines based on VSEARCH, USEARCH, mothur, and UNOISE3. OTU tables were rarefied, and differences in α-diversity and β-diversity were tested for different groups in a dataset. Finally, these results were checked for consistency among these example pipelines. Of articles that deposited data, only 57% made all sequencing and metadata available. When processing the datasets, issues were encountered, caused by read characteristics and differences between tools and their defaults in combination with a lack of detail in the methodology of the articles. In general, the four mainstream pipelines provided similar results, but importantly, P-values sometimes differed between pipelines beyond the significance threshold. Our results indicated that for published articles, the description of bioinformatics methods and data deposition should be improved, and regarding reproducibility, that analysis of multiple subsamples is required when using rarefying as library-size normalization method. The development of massively parallel sequencing technologies made rapid sequencing of hundreds of samples at unprecedented depth possible . This enTo date, multiple approaches have been developed to process 16S rRNA gene amplicon sequencing data . The mosHowever, algorithms and/or parameters in different processing pipelines often differ. So far, there is no single gold-standard pipeline to produce an OTU table (or higher-resolution count table), which means that both different tools and different parameters for the same step are being used in different pipelines.Many existing processing steps have been evaluated, such as the influence of chimera checking methods , denoisicf. used to conclude about scientific importance  compared among the pipelines, using portance .Articles on the oral microbiome were searched for, and their respective datasets were retrieved. To limit the influence of the 16S rRNA region, this study only searched for datasets using the V4 16S rRNA region, published during the past 5 years (Illumina MiSeq sequencing). Both sequencing and metadata had to be publicly available. Initially, articles with deposited datasets were searched for using the NCBI website as this hosts both PubMed and the Sequence Read Archive (SRA). PubMed search results were linked to SRA using LinkOut (not possible anymore in the new PubMed). However, many articles that deposited data in the SRA, with article title and DOI, were lost in this process due to incomplete linking between these databases. Therefore, studies were searched for using Google Scholar with the following query : intitle:oral 16S “V4 region” OR “V4 variable region” OR “V4 hypervariable region” “accession OR SRA.” The final papers were screened on reported P-values for comparisons: at least one test on the microbiome data had to report a P-value between 0.0001 and 0.05. Finally, sequencing data and metadata were downloaded from the NCBI.https://www.mothur.org/wiki/MiSeq_SOP, d.d. 2019-01-24). We only changed the value of maxlength in screen.seqs from 275 to 258 as the V4 region has a small length variation and as to use the same value in all four pipelines. In the VSEARCH, USEARCH, and UNOISE3 pipelines, the reads were merged and quality-filtered per sample and then combined into one file. In the (32-bit) USEARCH/UNOISE3 pipelines, (64-bit) VSEARCH was used to dereplicate these quality-filtered sequences. Since the read lengths in the different studies differed , during merging a maximum of 10% mismatches in the overlap region was used.Four different processing pipelines were built to produce OTUs tables: a mothur pipeline [version 1.41.3], a VSEARCH [version 2.11.0-linux-x86_64], a USEARCH [version 11.0.667_i86linux32], and a UNOISE3 [version 11.0.667_i86linux32] pipeline. Statistical analyses were conducted with R  and theRandom subsampling was used to normalize unequal sample depth (library size). The subsampling depth for each dataset was determined such that most samples remained in the analysis, while adhering to minimum of around 2,000 reads/sample. In addition, as sample depths depend slightly on the pipeline, the subsampling depth was chosen such that the OTU tables from the different pipelines contained the same samples. To assess the reproducibility of statistical tests, 1,000 random subsamples of the same OTU table were analyzed.Different publicly available datasets on the oral niche were searched for and processed with the VSEARCH, USEARCH, mothur, and UNOISE3 pipelines. During processing, different issues were encountered with specific datasets and pipelines that had to be addressed first. Next, the influence of the pipelines on diversity comparisons and reproducibility of results were evaluated.Dataset Search in Materials and Methods). Out of 53 articles that included an accession number to, for example, NCBI’s SRA or the European Nucleotide Archive, 45 studies actually deposited the raw data, while only 30 included the metadata in the database or in the article. Finally, 11 studies remained that used the V4 16S rRNA region and were related to the oral niche . Based on screening with the P-value criterion, five oral microbiome studies were selected from these 11 studies. This criterion was used to restrict our analyses, since results would unlikely differ for more extreme P-values.The literature search returned 60 articles of which, upon inspection, many did not satisfy our criteria  within the imposed time limit of 120 h. Finally, the deposited data of dataset 5 consisted of already merged read pairs.Therefore, the processing of these datasets was slightly altered to address these issues. In dataset 2, the tail of the reverse reads contained approximately 100 bp low-quality bases (Q <= 2) preventing the read pairs to be merged by VSEARCH. However, USEARCH merged these reads, because the used version automatically trims these low-quality tails (Q <= 2) before merging . Therefore, we pre-filtered dataset 2 using Trimmomatic v.0.38  with “TAFrom dataset 4, many sequences were removed after the mothur screen.seqs command on the aligned sequences, in which the sequences are required to span at least the V4 region (from 1968 to 11550) in the alignment. Manual inspection showed that many sequences ended one position early and that the first base call after the V4 (806R) reverse primer was absent. Therefore, for dataset 4 only, the value of the end parameter in this screen.seqs command was changed from 11,550 to 11,549 to avoid losing 57% of the sequence data.In addition, both datasets 3 and 4 contained many singletons. This caused the OTU clustering to fail in mothur. Therefore, singletons were removed from datasets 3 and 4 in the mothur pipeline . For dataset 3, it was also possible to generate an OTU table with cluster.split . To evaluate the difference between these two OTU tables , they were compared. Spearman’s correlation of the Shannon diversities , Procrustes Analysis , and the Mantel test  showed that the OTU tables were very similar. Since dataset 4 could not be processed with cluster.split within the wall-time limit of 120 h, no comparison could be made and the dataset with singletons removed was used.After the modifications described above, all five datasets were processed with all four pipelines. The total numbers of raw, merged, quality-filtered reads, and reads mapped to the OTU table are summarized in For each of the datasets, OTU tables were generated by the different pipelines. The fraction of quality-filtered mapped reads represented in the OTU table was similar  were compared for the different pipelines . A heatmap  users slightly depended on the pipeline . However, the much higher P-value reported by the authors was related to the weighted UniFrac distance metric. Indeed, when using the OTU table provided by authors, all PERMANOVA (Bray-Curtis) results became very similar . As an example, we take dataset 1, which was processed by the authors using USEARCH . In our This study also evaluated the reproducibility, defined here as “re-analysis with exactly the same pipeline and same dataset supports an identical conclusion.” To this end, each OTU table was subsampled 1,000 times, and statistical tests were done as above, for each of the 1,000 tables, thus providing 1,000 P-values  differed from other three . In the first comparison, the median depths of the groups differed a lot ; in the latter, they were much closer . Thus, when subsampling, results can show more variation since random subsamples vary more when subsample depth is low compared to the sample depth and/or when there is a bias in sample depth between groups. However, this argument does not hold for, for example, dataset 5: D2P vs HP and D1P vs HP. Here, all three sample groups have very similar medians. However, variability can also be caused by biological differences as well as sample size differences . As to exclude biological and other differences between samples, dataset 1 was subsampled at a lower depth to illustrate the increased variability using the same data . For example, in dataset 4, about 35% of the sequences was taxonomically classified as chloroplast . However, in the corresponding article (QIIME 1 was used), we did not explicitly find that these sequences were removed, although that seemed to be the case (Due to (implicit) differences between tools used for the pipelines, we sometimes had to adapt a pipeline to the data at hand  or distance metrics , since these are different downstream choices. When evaluating results from published studies, we should remember that different α-diversity indices or β-diversity metrics can lead to different conclusions. However, here, the focus was on whether different conclusions would result from different amplicon processing pipelines.Although QIIME 1 also was often used, it has not been supported since 2018, and we did not include it as to keep comparisons concise. In addition, based on previous studies, the default QIIME 1 pipeline has higher error rates due to chimeras and higher amount of spurious OTUs comparing with others . We, thesame dataset (study) resulted in different P-values. Although we cannot conclude that a lower threshold should be used, we should keep in mind that P-values just below 0.05 may not be very robust or reproducible, and a lower threshold also comes at a cost  or the European Nucleotide Archive (EBI) as indicated in BB designed the research and assisted XK in the study design, data collection, and bioinformatics analysis. XK, DD, and BB wrote the draft. DD, WC, and BB reviewed the manuscript. All authors contributed in the preparation and finalization of the manuscript. All authors contributed to the article and approved the submitted version.XK is supported by the China Scholarship Council (CSC), grant number 201808440303.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.


The pathogen exposure history of an individual is recorded in their T-cell repertoire and can be accessed through the study of T-cell receptors (TCRs) if the tools to identify them were available. For each T-cell, the TCR loci undergoes genetic rearrangement that creates a unique DNA sequence. In theory these unique sequences can be used as biomarkers for tracking T-cell responses and cataloging immunological history. We developed the immune Cell Analysis Tool (iCAT), an R software package that analyzes TCR sequencing data from exposed (positive) and unexposed (negative) samples to identify TCR sequences statistically associated with positive samples. The presence and absence of associated sequences in samples trains a classifier to diagnose pathogen-specific exposure. We demonstrate the high accuracy of iCAT by testing on three TCR sequencing datasets. First, iCAT successfully diagnosed smallpox vaccinated versus naïve samples in an independent cohort of mice with 95% accuracy. Second, iCAT displayed 100% accuracy classifying naïve and monkeypox vaccinated mice. Finally, we demonstrate the use of iCAT on human samples before and after exposure to SARS-CoV-2, the virus behind the COVID-19 global pandemic. We were able to correctly classify the exposed samples with perfect accuracy. These experimental results show that iCAT capitalizes on the power of TCR sequencing to simplify infection diagnostics. iCAT provides the option of a graphical, user-friendly interface on top of usual R interface allowing it to reach a wider audience. During T-cell development, the loci that encode TCRsα andβ-chains are rearranged by recombination of the variable (TCRV), diversity (TCRD), and joining (TCRJ) gene segments, encoding the complementary determining region 3 (CDR3). These genetic rearrangement events result in a high degree of diversity in the CDR3 regions of individual TCR loci.T- and B-cell responses are responsible for long-lasting immune memory responses to infectious agents, such as bacteria and viruses. Expansion of pathogen specific T-cells provide us with a robust resource for understanding whether an individual has been infected with a pathogen. The T-cell receptor (TCR), located on the surface of T-cells, is responsible for recognizing pathogen-specific peptides, leading to immune response and development of protective immune memory.7 unique TCRβ chains can be identified from the ∼ 1012 circulating T-cells present in a healthy human adult. A healthy human adult can have 1018 mathematically possible TCR recombinations resulting from the genetic rearrangement., The potential diversity of the repertoire coupled with the limited number of T-cells present in individuals makes identifying identical TCR sequences among multiple individuals exceptionally challenging. In addition, a specific TCR response to particular antigen can be extremely rare, which can pose an even greater challenge to identifying signals of T cell memory. However, by analyzing the large and diverse TCR repertoire using high-throughput TCRβ sequencing, it is possible to identify pathogen specific TCRs shared among different individuals exposed to the same infectious agent.During an infection or vaccination, T-cells that carry receptors recognizing pathogen associated peptides become activated and undergo rapid clonal expansion. The clonally expanded T-cells carry the same unique TCR rearrangement and a portion remain in circulation long after the pathogen has been cleared to provide long-lived immunological memory. The persistence of memory T-cells in circulation make the genetically rearranged TCR loci a stable biomarker documenting an individual’s immunological history. To utilize the diverse TCR repertoire as a potential biomarker for specific pathogen exposure, pathogen-specific TCR sequences common to different individuals exposed to the same pathogen need to be identified. This poses significant challenges given the diversity and magnitude of the TCR repertoire. On average, ∼ 10 Additionally, recent publications described an analytical approach for computationally identifying common/public TCR sequences.,, However, analyses of the TCR repertoire for diagnostic purposes have remained largely resource- and time-intensive efforts.Recently, high-throughput next-generation sequencing (NGS) techniques were employed to analyze the diverse immune cell repertoire. we have developed an R package with a user-friendly interface, the immune Cell Analysis Tool (iCAT), to identify TCR sequences statistically associated with pathogen exposure and to distinguish infected from non-infected samples. By providing an interactive interface throughiCAT, sample exposure can be assessed and predicted conveniently without requiring command-line skills.iCAT has an ability to classify target-associated receptor sequences (TARSs) and diagnose exposure with a high accuracy.Utilizing the diagnostic methodologies described in Wolf, et al. (2018),iCAT, an R package utilizing high-throughput TCR sequencing data to analyze TCR sequences and to diagnose infection in a user-friendly format  in the form of a web-application utilizing R-Shiny and a command-line R interface for batch processing of large-scale data. The simplest method to installiCAT on a system is directly from GitHub usingdevtools andinstall_github:install.packages("devtools")devtools::install_github("BioHPC/iCAT")We developed format .iCAT prshiny,iCAT also usesshinyjs, data.table, ggplot2, DT, hash, and magrittr. Required packages will be installed through theinstall_github step. Alternatively, users can clone or download the repository from GitHub and rundevtools::install("iCAT/").In addition toiCAT accepts tab-delimited files with the size limit of 10 gigabytes per file with multiple options to define TCR clonotypes within samples. TheiCAT shiny app has three tabs, separating major functionalities: Training, Library, and Prediction. Under the “Training” tab, clicking ‘Train Model’ will start the pipeline to statistically identify a subset of TARSs that will act as feature selections for training the diagnostic classifier, diagnosing samples as either negative or positive. Upon training,iCAT’s main tab provides a table summary of the data, a figure shows the distribution of TARSs between the positive and negative samples, and a classification matrix predicting the exposure status of samples used in the training data (A user can upload multiple TCR sequence repertoires from negative (control) and positive  cohorts.ng data . All figA separate tab, “Library”, is unlocked upon training and shows a table where each row describes a TARS and its presence in the positive and negative samples. All tables and figures are supplemented with a custom button for easy download .The third tab of iCAT, “Prediction”, also unlocks after training and allows the user to upload one or more independent TCR-sequencing samples for classification .Training” step. A user should provide multiple negative training samples  using theBrowse button. Then, repeat the step for positive training samples . The user should select the type of training feature. iCAT provides three options: (1)CDR3 Amino Acid Sequence , (2)TCRV-CDR3-TCRJ , (3)Nucleic Acid (DNA) . SelectingTCRV-CDR3-TCRJ is recommended as a balance between sensitivity and specificity and this option has been used for all the use cases in this paper. In addition, users can customize the range acceptable of copies per clonotype, and the minimum threshold of public sequences, which determines the minimum samples a TCR sequence must be observed in to be considered for analysis.Samples, such as from blood or lymph tissue, are collected and genetic material is purified. TCR sequences present in the sample are selectively amplified and then sequenced . The firTraining” tab isMax p-value (default: 0.1), which determines the minimal degree of statistical significance that iCAT will accept as being potentially ”associated” with the positive group. The statistical methodology of iCAT is based on identifying a subset of TARSs that informs classification. TCR sequences significantly associated with positive samples as opposed to negative samples are identified by performing a one-tailed Fisher’s exact test. iCAT determines the optimal p-value cutoff to generate the TARS library based on the idea of coverage ratio. To determine an optimal p-value threshold for identifying vaccine-associated TCRβ sequences, we applied a heuristic test that selected the optimal p-value threshold based on the ”coverage” provided by the library for both vaccinated (Cv) and naïve samples (Cn). “Coverage” is defined as the summation of the number of samples containing each TARS divided by the number of samples. In the equations below,xi denotes the number of vaccinated samples a single TCRβ is identified andnv represents the number of positive samples in the training data.yi also denotes for naïve samples andnn represents the number of naïve samples.Cv toCn is determined for each p-value. The p-value with the largestCv:Cn ratio and offers significant coverage to distinguish vaccinated (v) from naïve (n) samples was chosen.One important option of this “ Such a normal distribution has been adopted to calculate the distance of a sample to the mean. In detail, the normal distributions for the naïve and vaccinated populations in our training data were calculated based on a function of the difference between a single sample value (x) and the mean of a set of data (μ) over the standard deviation of that set of data (σ) from the below equation.Training” tab will show some exploratory tables and a figure regarding the training data and the model built, which can all be downloaded to the user’s machine easily. In addition, the “Library” and “Prediction” tabs will unlock.For the classification of vaccinated and naïve samples, iCAT calculates the percentage of TARS (% TARS) present in a sample. The % TARS for each sample in the training data is compared against the % TARS normal distribution for each group to predict if each sample is ”positive” or ”negative”, determined by which group a sample is more closely associated with.Library” tab displays a table consisting of the TARS, determined to be statistically associated with exposure to the target/agent/pathogen  for classification as “Positive” or “Negative” and determining the accuracy of the diagnostic assay. Use theBrowse button to upload such independent samples for prediction. Multiple samples can be uploaded simultaneously. ClickPredict Independent Sample will analyze the dataset. A table will appear after analysis is complete. The table displays sample names along with the prediction “Positive” (red) or “Negative” (blue), and displays the %TARS that is the percent of individual sequences from the sample that are included in the TARS library. The prediction results can be downloaded as a table.To allow the diagnostic classifier to test independent cohorts of samples, a third tab in iCAT, “ng data . The “PiCAT requires R 3.4.0 or upper and can be run on any operating system with common specifications .https://doi.org/10.17632/cf92gt44zf.1. The third data set is human TCR samples exposed to the novel SARS-CoV- 2 virus which is the cause of the ongoing COVID-19 global pandemic. The sample size is small (two cohorts), but those two cohorts’ TCR repertoires were obtained for other projects one and two years prior to infection. Therefore, this negative and positive SARS-CoV-2 human TCR sequencing data set from same cohorts can be a great example to show the great potential of iCAT as a diagnostic assay. This data is available athttps://doi.org/10.5281/zenodo.3835956.To evaluate the efficacy of iCAT, we used three TCR sequencing data sets that are publicly available. The first viral data set consists of 148 training and 20 independent mouse samples. The training set has 32 mouse pre-treatment naïve group and 116 vaccinated samples, each cohort inoculated intranasally with the ACAM2000 smallpox vaccine. The second viral data set consists of 133 (27 negative and 96 positive) training and 15 (5 negative and 20 positive) monkeypox virus. The two mouse datasets are publicly available from When applied to the training data as a baseline check, iCAT correctly classified 32 of 32 naïve samples as ”unexposed” and 58 of 58 vaccinated samples as ”exposed” (100% accuracy). We utilized TCR-sequencing files from 10 mice pre- and post-smallpox vaccination that were not involved in the training of the diagnostic classifier to act as independent cohorts to test the diagnostic accuracy of the iCAT generated classifier. The classification results are displayed in the “Prediction” tab and can be downloaded as a .txt file. From a total of 20 samples, 90% of pre-vaccination samples (9 of 10) were correctly classified as “negative” and 100% of samples post-smallpox vaccination (10 of 10) were classified as ”positive”. Training time was 2.36 minutes and classification time was 30.6 seconds. Overall, this data displays that the iCAT platform computationally identifies target-associated public TCRs, utilized to train a diagnostic classifier capable of distinguishing between exposed and unexposed samples with a high degree of accuracy.32 pre-exposure (naïve) samples were analyzed, which included 2,049,383 unique TCR sequences (clonotypes). We setup iCAT options to analyze by either the CDR3 amino acid sequence or the V-gene and J-gene names. 714,522 amino acid sequence-gene name combinations were found in the naïve samples. 58 samples taken 2- and 8-weeks post-vaccination for smallpox were analyzed, which included 1,581,619 clonotypes and 573,612 unique amino acid sequence-gene name combinations. After training, iCAT accurately generated the same virus-associated TCR library (314 TCR sequences) identified in Wolf, et al., 2018.We tested iCAT using another TCR-sequencing mouse dataset which included 27 naïve samples and 48 samples 2- and 8-weeks post infection with monkeypox. We chose to analyze based on CDR3 amino acid sequence in addition to V-gene and J-gene names. The p-value cutoff was set to 0.1 and the minimum number of public sequences was set to 1. Those parameters produced the best separation experimentally. Naïve samples included 1,772,085 clonotypes and 630,381 unique amino acid sequence-gene name combinations. Exposed samples included 1,070,615 clonotypes and 382,906 unique amino acid sequence-gene name combinations. iCAT correctly classified this training data with 100% accuracy. When tested on an independent monkey pox data set – set up by excluding 5 samples from the naïve group and 10 samples from the exposed group pre-training – iCAT correctly classified the 5 naïve samples as negative and the 10 exposed samples as positive. Thus, we demonstrated a 100% classification accuracy using iCAT on this monkeypox dataset. Training time was 47.81 seconds and classification time was 6.83 seconds. Data included 4 naïve samples from 2018 and 2019, and 4 samples collected 15- and 30-days post-infection. We chose the iCAT option to analyze by CDR3 amino acid sequences only. The p-value cutoff was set to 0.1 and the minimum number of public sequences was set to 1. The naïve data included 2,935,893 clonotypes and 1,120,606 unique CDR3 amino acid sequences. The exposed data included 1,987,608 clonotypes and 541,111 unique amino acid sequences. iCAT achieved a perfect classification accuracy on the training data, correctly assigning the 4 naïve and 4 exposed samples. Further, iCAT correctly classified 4 independent exposed samples as positive. Training time was 1.50 minutes and classification time was 33.12 seconds. This demonstrates the wide utility of iCAT and the methodology it implements.We further tested iCAT on TCR-sequencing data from two human individuals exposed to the novel SARS-CoV-2 virus.In this article, we have presented iCAT, a powerful software tool for determining pathogen exposure through TCR sequencing data. It has significant clinical applications in disease diagnosis, surveillance, as well as for determining potential vaccine efficacy. Once data interpretation is fully automated, the TCR sequencing analysis and other types of NGS will likely become a standard tool for diagnosis and management of disease. Our current datasets are from pre- and post- exposure to viruses, and serve as a proof of principle that TCR sequencing analysis can be utilized to identify individuals exposed to infectious agents or vaccines with great accuracy, speed, and accessibility. We demonstrated the use of iCAT for accurately detecting exposure to the SARS-CoV-2, the virus behind COVID-19. Although this use case was based on a few number of samples, it shows the immense potential of our software the utilization of TCRs as a biomarker. This type of analysis may be used to distinguish between two different but highly related infections, such as Zika virus and Dengue, which is one of the global concerns considering Zika virus’s association with fetal complications in infected pregnant women, and current laboratory testing cannot distinguish between the two. Parallel endeavors in our group show promising results in identifying virus-associated TCR sequences uniquely associated with a prior Zika versus Dengue virus infection in mice using iCAT. Further, the iCAT platform may prove useful for diagnosing individuals in the early stages of autoimmunity, by identifying auto-reactive TCRs before symptoms and significant tissue damage occurs. Earlier diagnosis may allow for preventative measures, better treatment, and better outcomes. Broadly, our approach can be used to diagnose autoimmune disease and possibly immune responses to cancer before or after immunotherapy.https://doi.org/10.17632/cf92gt44zf.1.Mendeley Data: Identifying and Tracking Low Frequency Virus-Specific TCR Clonotypes Using High-Throughput Sequencing,This project contains the raw sequencing data from HLA-A2 transgenic mice before and after infection with either the ACAM2000 smallpox virus or highly releated monkeypox virus.https://doi.org/10.5281/zenodo.3835956.Zenodo: Longitudinal high-throughput TCR repertoire profiling reveals the dynamics of T cell memory formation after mild COVID-19 infection,This project contains the third human TCR samples exposed to the novel SARS-CoV-2 virus is available from zenodo in mixcr format.Data are available under the terms of the Creative Commons Attribution 4.0 International license (CC-BY 4.0).https://github.com/BioHPC/iCAT.Source code available from:http://doi.org/10.5281/zenodo.4436485.Archived sourced code as at time of publication:License: MIT Please add a sentence(s) on how you calculated uncertainty.vC:nC ratio and offers significant coverage to distinguish vaccinated (v) from naïve (n) samples was chosen."Please check the following sentence: "The p-value with the largestIs the following sentence correct: "If the value is bigger, we can conclude that the sample is more associated with the training group."?Thank you so much for the improved manuscript. I have minor comments:Are the conclusions about the tool and its performance adequately supported by the findings presented in the article?YesIs the rationale for developing the new software tool clearly explained?PartlyIs the description of the software tool technically sound?PartlyAre sufficient details of the code, methods and analysis (if applicable) provided to allow replication of the software development and its use by others?PartlyIs sufficient information provided to allow interpretation of the expected output datasets and any results generated using the tool?PartlyReviewer Expertise:Genomic Epidemiology, Microbiome Epidemiology, MultiomicsI confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. No further comments.Are the conclusions about the tool and its performance adequately supported by the findings presented in the article?YesIs the rationale for developing the new software tool clearly explained?YesIs the description of the software tool technically sound?PartlyAre sufficient details of the code, methods and analysis (if applicable) provided to allow replication of the software development and its use by others?NoIs sufficient information provided to allow interpretation of the expected output datasets and any results generated using the tool?YesReviewer Expertise:Immunology, T cells, TCR, single-cell receptor sequencing,We confirm that we have read this submission and believe that we have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. This article is a valuable resource for T-cell receptor (TCR) sequencing data-based infection diagnostics. The authors provided a R software package named iCAT which provides a step-by-step analysis of TCR sequencing data to predict if individual subjects are infected or not. First, iCAT takes positive and negative pathogen-specific TCR datasets to estimate parameters involved in in model. Second, iCAT provides the Library tab which summarizes the target associated receptors sequences with p-values. Third, iCAT provides prediction tab which takes test TCR datasets and make diagnostic decision. I enjoyed reading this article and github tutorial.Major comments: It seems like that the performance of this tool does depend on the number of samples.  Please provide discussion or guideline on the number of negative and positive samples relating to the performance. It seems like that this is the first tool available for TCR data-based diagnostic tool. Please clarify if this is the first tool which uses TCR sequencing data for the purpose of diagnostic. If this is not the first tool, then comparison results with other tools should be included.Minor comments: First, I have a question for clarification purpose. Before TCR sequencing data is uploaded, does TCR sequencing data need any preprocessing step to make input compatible with iCAT? Install.packages(“devtools”) does not work with R4.0.3 on Mac OS X. The error I got is as follows: Downloading GitHub repo BioHPC/iCAT@HEAD Error in utils::download.file’. What is other? Please include information of computational time for training and prediction steps for each case. Although feature from an option of ‘TCRV-CDR3-TCRJ’ is recommended, please include the performance for other features for each test case which could help readers understand and compare features. What does TCRBV stand for? What does VATS stand for? A correction constant is not shown in equation (2). Whole genome sequences can distinguish DENGU from ZIKE with 100% accuracy. For the sentence ‘current laboratory testing cannot distinguish between the two’, are you excluding whole genome sequencing data? Is it possible to make change range of acceptable into something like 1:99 or 1~99 instead of 1 99 to indicate range? Does the max p-value change automatically according to the training data provided? Or should it be changed manually based on the selected one by Fisher’s t-test? Please revise typo in Figure 1(B) Please revise typo with ‘the minimum samples’ Please revise the following sentences: - The ratio of Cv to Cn is determined for each p-value. The p-value with the largest Cv:Cn ratio and offers significant coverage to distinguish (v)accinated from (n)aïve samples was chosen. - iCAT was configured to analyze by CDR3 amino acid sequence only If the value is bigger, we can conclude that the sample is more associated to the the training group. Is this a correct sentence? From the training data, iCAT correctly classified 32 of 32 naïve samples as ”unexposed” and 58 of 58 vaccinated samples as ”exposed” (100% accuracy). Is this a correct sentence?Are the conclusions about the tool and its performance adequately supported by the findings presented in the article?YesIs the rationale for developing the new software tool clearly explained?PartlyIs the description of the software tool technically sound?PartlyAre sufficient details of the code, methods and analysis (if applicable) provided to allow replication of the software development and its use by others?PartlyIs sufficient information provided to allow interpretation of the expected output datasets and any results generated using the tool?PartlyReviewer Expertise:Genomic epidemiology, MultiomicsI confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above. Major comments: 1. It seems like that the performance of this tool does depend on the number of samples.  Please provide discussion or guideline on the number of negative and positive samples relating to the performance. Response: We really appreciate all the great and thoughtful comments from the reviewer. The sample size is of course very important in determining the power and accuracy of the diagnostic.  We have generated positive and negative datasets of various sample sizes to determine the impact of sample numbers on accuracy.  The conclusion was that the number of samples required to generate an accurate diagnosis is determined by the nature of the samples. For example, a dataset from Zika infected mice produced a small but conserved virus-specific TCR repertoire due to the small number of immunogenic epitopes  ; this allowed for the generation of a powerful diagnostic assay using relatively small numbers of samples.  Comparatively, using a dataset from mice before and after smallpox vaccination resulted in a large pool of virus-specific TCRs due to a large number of potential epitopes; the less focused response required a greater number of samples in each group to generate sufficient coverage and diagnostic accuracy . 2. It seems like that this is the first tool available for TCR data-based diagnostic tool. Please clarify if this is the first tool which uses TCR sequencing data for the purpose of diagnostic. If this is not the first tool, then comparison results with other tools should be included. Response: To our limited knowledge, this is the first publicly available tool for diagnostic classification using TCR sequencing data.Minor comments: 1. First, I have a question for clarification purpose. Before TCR sequencing data is uploaded, does TCR sequencing data need any preprocessing step to make input compatible with iCAT? Response: iCAT takes in TCR data in tab-delimited clonotype abundance tables. These tables are the result of mapping of reads from Variable, Diversity, and Joining segments then the assembly of clonotypes. iCAT does not perform these steps so they may be considered pre-processing. Clonotype tables are provided through the immunoSEQ assay from Adaptive Technologies, which is the most commonly used immunosequencing pipeline in recently published studies. While iCAT input-parsing was built around immunoSEQ’s format, it also has a flexible interface that allows handling of different formats with different user-selected column names. 2. Install.packages(“devtools”) does not work with R4.0.3 on Mac OS X. The error I got is as follows: This issue is resolved and updated on iCAT. 3. Each input file is limited to 10Gigabyte in size. Where does this limitation come from? Response: This is an arbitrary limitation that we coded into the tool’s interface. It is not a required limitation for any of the underlying functions in iCAT. The reason we put this size limit in place is because one of our future goals with iCAT is to host it on a website where users can upload their data for analysis, and thus data size can become a limiting factor. 4. I downloaded one of test cases which contains many files. It was not easy to figure out which files to be uploaded. It would be nice that this R package provides example datasets along with the package so that they can be played in R directly. Response: Example negative, positive, and independent datasets are included in the R package under the path “inst/extdata/”. 5. There is an option named ‘Other’ for ‘analyze clonotypes by (column names in parantheses)’. What is other? Response: This option exists to allow users to input a custom column name to analyze by. TCR data can vary in formatting (e.g. some files might describe the amino acid column as “aminoAcid” while others can describe it as “aa”) so this option can help users load different data formats into iCAT. 6. Please include information of computational time for training and prediction steps for each case. Response: Use case 1: Training time: 2.36 minutes Classification time: 30.6 seconds Use case 2: Training time: 47.81 seconds Classification time: 6.83 seconds Use case 3: Training time: 1.50 minutes Classification time: 33.12 seconds The manuscript was updated to reflect this information. 7. Although feature from an option of ‘TCRV-CDR3-TCRJ’ is recommended, please include the performance for other features for each test case which could help readers understand and compare features. Response: In general, we found that using the J-gene and V-gene info improves accuracy in our testing, but other features are available in case researchers to believe they might get better results with them. Using the amino acid sequence for the monkeypox data,  1/5 negative samples were identified correctly, while 6/10 positive samples were identified correctly. Using the DNA sequence, 5/5 negative samples were identified correctly and 3/10 positive samples were identified correctly. 8. What does TCRBV stand for? Response: It stands for T cell receptor beta chain variable region. 9. What does VATS stand for? Response: It stood for vaccine-associated target sequences. This is now changed to target-associated receptor sequences (TARSs) for consistency in the manuscript. 10. A correction constant is not shown in equation (2). Response: That is correct We updated the sentence for the equation in the manuscript. 11. Whole genome sequences can distinguish DENGU from ZIKE with 100% accuracy. For the sentence ‘current laboratory testing cannot distinguish between the two’, are you excluding whole genome sequencing data? Response: We are limiting that comment to typical laboratory diagnostics which typically does not include whole-genome sequencing. 12. Is it possible to make change range of acceptable into something like 1:99 or 1~99 instead of 1 99 to indicate range? Response: We reflected this comment and iCAT now allows inputting either a space or a colon to delimit range values. 13. Does the max p-value change automatically according to the training data provided? Or should it be changed manually based on the selected one by Fisher’s t-test? Response: The max p-value can be changed manually.  The purpose is to allow the user to modify the stringency of the diagnostic. This could be done after generating the diagnostic library and the user determines they would prefer more (or less) stringent parameters. 14. Please revise typo in Figure 1(B) Response: Fixed typo in the new manuscript version. 15. Please revise typo with ‘the minimum samples’ 16. Please revise the following sentences: - The ratio of Cv to Cn is determined for each p-value. The p-value with the largest Cv:Cn ratio and offers significant coverage to distinguish (v)accinated from (n)aïve samples was chosen. - iCAT was configured to analyze by CDR3 amino acid sequence only Response: Sentences were revised in the new manuscript version to make them more clear. 17. If the value is bigger, we can conclude that the sample is more associated to the the training group. Is this a correct sentence? Response: The sentence was revised in the new manuscript version. 18. From the training data, iCAT correctly classified 32 of 32 naïve samples as ”unexposed” and 58 of 58 vaccinated samples as ”exposed” (100% accuracy). Is this a correct sentence? Response: The sentence was revised in the new manuscript version. Diagnostic methods based on the detection of disease-specific or disease-associated T-cell receptor (TCR) sequences are novel approaches whose development is still in its infancy. This paper introduces an intuitive, easy-to-use, software package for the analysis of TCR deep sequencing data with the purpose of grouping data into ‘naïve/healthy’ or ‘diseased/exposed’ groups. The statistical methods used is rather rudimentary, but serve the purpose. It demonstrates remarkably high accuracy in three use cases, including a small dataset of COVID-19. However, the choice of parameter settings and how data is split into training and test datasets, is not well explained. Thus, one is left with a feeling that some over-fitting and over-tuning may have occurred. Nevertheless the shortcomings, overall this software package is first of its kind and will be welcomed by the growing community of scientists working with immune receptors and disease prediction.Major comments:Introduction: Aside from the immensity of TCR diversity, both theoretical and within each individual, by far the largest challenge to identifying signals of T-cell memory to a particular antigen, is the rarity of specific T cells to any given antigen. In circulation during the homeostatic memory phase, this number can be as low as 1 in 100 000 for CD4+ T cells, and may be up to 1% in the CD8 compartment.Operation: Does iCAT support both TCRB and TCRA data, separately, or in the same dataset? Or does it only support TCRB data?Setting (2) during training: at which level of TRBV and TRBJ allelic similarity is the tool tuned? In other words, would human TRBV7-2*01 and TRBV7-2*02 be considered the same, or different? How about human TRBV4-2 and TRBV4-3 that are very similar, especially at the 3’-end where the last 40 amino acids are identical. Many HTS short reads would not be able to distinguish between these two gene segments. Use Case 1: The authors wrote that they analysed this dataset either by the CDR# amino acid sequence only, or with the V- and J-gene identity in addition. Did these two options behave differently?In the use cases, it is not common to report the accuracy on training dataset. it is not clear how the data was split into the training and testing dataset. More common practice is to do cross validation, and report the averaged precision, sensitivity, maybe F-score, especially for unbalanced samples. For Use case 2 for example, how were the 5 naïve and 10 exposed samples in the test group chosen, randomly? Would one get the same 100% accuracy if another random set of 5 naïve and 10 exposed samples was chosen?In the training tab, the default Max p-value is 0.1. Usually in disease associated studies, a more stringent cut-off p value is recommended. Considering the main purpose of iCAT is for diagnosis, the specificity is not as crucial, 0.1 might be a good choice. Is there any analysis supporting the choice, e.g. in the use cases or on other public available data which antigen specific sequences is known?In the prediction tab, it would be better to provide a measurement of uncertainty in addition to %TARS, Britanova et al. (2014), and DeWitt et al. (2018) suggested that MHC, age and sex also effect the immune receptor repertoire. Incorporating such meta information should be useful, even if in cases where the sample size is too limited for these factors to be modelled in the training, they are better to be randomized in training and testing datasets.Some studies such as, Schneider-Hohendorf et al. (2018)How about computing time? How long did it take to do the training, and classification of for instance the use case 2 data?There are substantial public resources including some antigen-specific database and repertoire database. It might be useful to allow loading some commonly used datasets as controls in case the user has limited sample size.Minor comments:Figure legend 1: TCR sequencing of blood samples can be done with, but not limited to, genomic DNA. Many groups also use cDNA.Why does the tip of the iCAT’s tail has an antibody? If this were a TCR analysis tool, would it not be more appropriate with a cartoon drawing of TCR?What is VATS in last paragraph in Page 5, is it the same as TARS?In the library tab, pattern for significant TARS would make more sense, e.g. are they all similar or there were numbers of clusters based on the similarity?Are the conclusions about the tool and its performance adequately supported by the findings presented in the article?YesIs the rationale for developing the new software tool clearly explained?YesIs the description of the software tool technically sound?PartlyAre sufficient details of the code, methods and analysis (if applicable) provided to allow replication of the software development and its use by others?NoIs sufficient information provided to allow interpretation of the expected output datasets and any results generated using the tool?YesReviewer Expertise:Immunology, T cells, TCR, single-cell receptor sequencing,We confirm that we have read this submission and believe that we have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however we have significant reservations, as outlined above. Major comments: 1. Introduction: Aside from the immensity of TCR diversity, both theoretical and within each individual, by far the largest challenge to identifying signals of T-cell memory to a particular antigen, is the rarity of specific T cells to any given antigen. In circulation during the homeostatic memory phase, this number can be as low as 1 in 100 000 for CD4+ T cells, and may be up to 1% in the CD8 compartment. Response: We really appreciate all the great and thoughtful comments from the reviewer. As the reviewer commented, the rarity of specific T cells to a specific antigen is also a challenge in addition to the diversity and magnitude of the TCR repertoire. We updated the Introduction from the comment cited a reference for it. 2. Operation: Does iCAT support both TCRB and TCRA data, separately, or in the same dataset? Or does it only support TCRB data? Response: iCAT has been developed and tested to analyze TCR beta chain of the T-cell receptor in the alpha/beta T cells. The column call for the V and J are vGeneName and jGeneName and do not distinguish between alpha and beta. So, as long as the remaining formatting is the same, there is no reason we couldn’t do either TCRA or TCRB.  However, it is not designed to do so in tandem. 3. Setting (2) during training: at which level of TRBV and TRBJ allelic similarity is the tool tuned? In other words, would human TRBV7-2*01 and TRBV7-2*02 be considered the same, or different? How about human TRBV4-2 and TRBV4-3 that are very similar, especially at the 3’-end where the last 40 amino acids are identical. Many HTS short reads would not be able to distinguish between these two gene segments. Response: We do not look that deeply into the different V and J chains. TRBV7-2*01 and TRBV7-2*02 are in the column vMaxResolved or jMaxResolved, which we do not use for the V and J calling. We use the vGeneName and jGeneName, which for this example, both would be called as TRBV7-2. But TRBV4-2 and TRBV4-3 will be called differently as those would be distinguished in the vGeneName column.  You can see this in the data table of the generated library, the selected V and J regions do not include the *01 or *02 details. This is also true when you look in the TSV file. 4. Use Case 1: The authors wrote that they analysed this dataset either by the CDR# amino acid sequence only, or with the V- and J-gene identity in addition. Did these two options behave differently? Response: Using the V- and J-gene identity in addition to the CDR3 amino acid sequence results in significantly higher prediction accuracy in our testing, likely due to identifying more unique markers. Specifically, it results in fewer false positives. For example, in use case 1, using the amino acid sequence only resulted in correctly identifying only 1 / 10 of the pre-vaccination samples, while still correctly identifying 10/10 of the post-vaccination samples. 5. In the use cases, it is not common to report the accuracy on training dataset. it is not clear how the data was split into the training and testing dataset. More common practice is to do cross validation, and report the averaged precision, sensitivity, maybe F-score, especially for unbalanced samples. For Use case 2 for example, how were the 5 naïve and 10 exposed samples in the test group chosen, randomly? Would one get the same 100% accuracy if another random set of 5 naïve and 10 exposed samples was chosen? Response: We agree it is not common to report the accuracy on the training data and it is not as useful as the accuracy on the independent testing data in evaluating a classification tool. However, we chose to include it to demonstrate this feature of the iCAT interface. This feature may be useful as a baseline-check after training in the usual scenarios where the ground truth for the testing data is not available. The independent samples in use case 2 were chosen randomly before training and excluded from the training set. We also repeated the test three times after receiving this question using a script to randomly select 15 samples for testing. The accuracy was at a 100% for all the new random tests. 6. In the training tab, the default Max p-value is 0.1. Usually in disease associated studies, a more stringent cut-off p value is recommended. Considering the main purpose of iCAT is for diagnosis, the specificity is not as crucial, 0.1 might be a good choice. Is there any analysis supporting the choice, e.g. in the use cases or on other public available data which antigen specific sequences is known? Response: The max P-value is set at default to 0.1, it can be changed to any value.  We found through our own testing groups that it was a reasonable cut-off  as p-values above this tended to generate libraries that were not stringent enough. However, the user can manually alter the p-value to more stringent conditions.  It is worth noting that while the max p-value of the assay may be defaulted at 0.1, iCAT includes a feature for determining the optimal p-value threshold for library generation, which is likely to be well below the max allowed p-value. 7. In the prediction tab, it would be better to provide a measurement of uncertainty in addition to %TARS Response:  This is a great comment and we totally agree that a measurement of uncertainty in addition to %TARS might be useful for iCAT users. We considered an uncertainty metric based on the difference between the probability density in the pre-exposure distribution and the probability density in the post-exposure distribution. This new feature has been implemented and now available on GitHub iCAT and we updated the README screenshot of the prediction tab. 9. Some studies such as, Schneider-Hohendorf et al. (2018)1, Britanova et al. (2014)2, and DeWitt et al. (2018)3 suggested that MHC, age and sex also effect the immune receptor repertoire. Incorporating such meta information should be useful, even if in cases where the sample size is too limited for these factors to be modelled in the training, they are better to be randomized in training and testing datasets. Response: We agree this meta information can provide insightful context for different use cases. This is one of our goals for future versions of iCAT (iCAT 2.0). For example, we are especially excited to study the effect of clustering data based on HLA subtypes. In our opinion, this would be better explored in a separate paper. *manuscript modification*10. How about computing time? How long did it take to do the training, and classification of for instance the use case 2 data?Training time: 47.81 secondsClassification time: 6.83 seconds We updated the manuscript to reflect this information. Response: Following are the results from use case 2. 11. There are substantial public resources including some antigen-specific database and repertoire database. It might be useful to allow loading some commonly used datasets as controls in case the user has limited sample size. Response: Providing commonly used datasets as default controls would be a convenient feature for some use cases. We considered including such datasets with the iCAT software package, but decided against it in the current version due to size consideration. We aimed to create a versatile and easy to use software tool. Using the instructions in this paper and on GitHub, we believe loading public datasets as needed can be performed conveniently by the average iCAT user. We will consider this feature in iCAT 2.0.Minor comments: 1. Figure legend 1: TCR sequencing of blood samples can be done with, but not limited to, genomic DNA. Many groups also use cDNA. Response: Fixed in the new manuscript version. 2. Why does the tip of the iCAT’s tail has an antibody? If this were a TCR analysis tool, would it not be more appropriate with a cartoon drawing of TCR? Response: While it is true that a TCR tail would be more descriptive of our tool, it is our opinion that an antibody makes a more aesthetic cat tail than a TCR. Its shape is also more widely recognizable and can give a general idea to non-specialists about what iCAT deals with at first glance. 3. What is VATS in last paragraph in Page 5, is it the same as TARS? Response: Yes. Fixed in the new manuscript version. 4. In the library tab, pattern for significant TARS would make more sense, e.g. are they all similar or there were numbers of clusters based on the similarity? Response: The current version of iCAT does not perform clustering of the target-associated receptor sequences based on similarity, but this can be developed and incorporated in future versions. We agree that it would be a useful addition for exploratory analyses. iCAT currently focuses on using the sequences for prediction, but provides a starting point for further analyses by allowing the download of those significant sequences and some statistical information about their frequency in the training data.