Investigation of the genetic basis of traits or clinical outcomes heavily relies on identifying relevant variables in molecular data. However, characteristics such as high dimensionality and complex correlation structures of these data hinder the development of related methods, resulting in the inclusion of false positives and negatives. We developed a variable importance measure method, termed the ECAR scores, that evaluates the importance of variables in the dataset. Based on this score, ranking and selection of variables can be achieved simultaneously. Unlike most current approaches, the ECAR scores aim to rank the influential variables as high as possible while maintaining the grouping property, instead of selecting the ones that are merely predictive. The ECAR scores’ performance is tested and compared to other methods on simulated, semi-synthetic, and real datasets. Results showed that the ECAR scores improve the CAR scores in terms of accuracy of variable selection and high-rank variables’ predictive power. It also outperforms other classic methods such as lasso and stability selection when there is a high degree of correlation among influential variables. As an application, we used the ECAR scores to analyze genes associated with forced expiratory volume in the first second in patients with lung cancer and reported six associated genes. More importantly, complex correlation patterns5 and considerable interactions6 are present between the variables . Additionally, the number of causal or relevant variables to clinical outcomes may be small9.As a result of novel biotechnology such as next-generation sequencing (NGS) technologies, genomic and clinical research have benefited dramatically from the steep increase in both quantities and quality of molecular data. Identifying important genomic factors correlated with phenotypes or clinical outcomes will help scientists investigate the genetic basis of traits or diseases and make targeted interventions possible. An example is the identification of cancer driver genes that are crucial for diagnosing and treating the disease. On the other hand, the data complexity challenges the analysis methods. In these data, the number of variables is often much larger than the number of individuals. For example, expression of > 20,000 mRNA transcripts can be measured using microarrays10, thus introducing many false negatives. Another class of methods are penalized regression models , which aim to select a small set of predictors that are associated with a trait. Despite their good performance in prediction, they face challenges in association studies. For example, the lasso tends to select only one variable from a group of highly correlated genomic factors, and it cannot select more variables than the sample size. The elastic net addresses these two problems, but its result, like other well-known approaches , can be numerically unstable when applying cross-validation to estimate the parameters. Some researchers also propose to rank and select variables based on their assigned scores. An example of this is the variable importance computed by random forest14, which has been applied in genetics15, gene expression16, methylation17, proteomics18, and metabolomics studies19. Another example is stability selection20, which is based on linear models and has flavors of both lasso and random forests. Comparing with the lasso, it is more suitable for variable selection, but the price to pay is the reduced power to identify the true signals. CAR scores21 and CARS scores22 also fall into this group, they are easy to calculate, but might not be flexible enough when the noise in data is too small or too large.Many computational tools have been developed to help the selection of genomic factors relevant to the quantitative traits or clinical outcomes. The most commonly used approach performs independent hypothesis testing on each variable, and keep those whose 21 and improves over the CAR scores in terms of selection accuracy by adjusting the parameter according to different datasets’ characteristics. Specifically, the ECAR scores aim to rank the truly influential variables as high as possible. To determine the final selected set, we apply the adaptive false discovery rate density approach23. We compared the ECAR scores’ performance to lasso, stability selection, ridge, CAR scores, and Sure Independence Screening24 (SIS) on three classes of datasets: simulated datasets with a fixed correlation structure, semi-synthetic datasets generated from mRNA expression data, and real datasets from T3/barley database. In our study, we showed that ECAR improves CAR and rivals popular methods like the lasso in terms of the variable selection accuracy and the predictive power of high-rank variables.Due to these limitations of current approaches, we developed the Elastic Correlation Adjusted Regression (ECAR) score for simultaneously variable selection in high dimensional biological data. This method is an extension of the CAR scoresSuppose we have random variables  on 500 simulated datasets consisting of 200 observations and 600 features. The correlation matrix of the features is block diagonal with the compound symmetry structure which was used in the previous researchWe adjusted Since it seems too advantageous to SIS that the influential genes are all from the first block, we also tried to select the genes from the whole set randomly. Figure https://tcga-data.nci.nih.gov/tcga/) using TCGA-Assembler27 (v1.0.3). We removed the genes with zero expression in more than 20% of the samples. For each expression data, 1000 genes with the largest variance were selected, log-transformed and normalized to form the simulation dataset. The sample sizes of simulation datasets are 512, 369, and 497, respectively. The comparison method is the same as that in the previous section. Figure We obtained mRNA expression datasets from 3 cancer projects  from The Cancer Genome Atlas (TCGA) portal . Barley samples have spike length information in the trial of experiment NSGC Spring Core Panel in 2012, and lodging degree and leaf width information in the trail of experiment UMN NSGC GWAS in 2015. Genotyping experiment BarleyNB_9K includes 2417 samples and 6913 markers. We cleaned this data by removing 63 markers with a minor allele frequency (MAF) less than 1% and 429 markers which are missing for more than 3% of data. In the data, 1, 0, − 1 represent AA, AB, BB respectively. The missing values were replaced by − 1. After removing samples with NA trait value and markers that are duplicated or have zero variance, the spike length dataset contains information for 1947 samples and 6583 markers; the lodging degree dataset has 712 samples and 6236 markers; the leaf width dataset has 738 samples and 6239 markers.We applied ECAR, CAR, lasso, stability selection, SIS, ridge on these three datasets. Since we do not know the truly influential SNPs, we compared the mean square error (MSE) on the test set indexed by the number of SNPs in the model instead. For the spike length, lodging degree, and leaf width dataset, the Tables We analyzed forced expired volume in 1 s (FEV1) in 230 patients from the TCGA LUAD cohort . In the data preprocessing step, 3781 genes which have zero value for more than 20% of samples were removed. This results in a dataset of 230 samples and 16,750 genes. After the logarithmic transformation on both the gene expressions and FEV1, we performed ECAR on the data.CHRM3, CTCFL, KCNE2, MLANA, MSMP, TTLL2, many of which have been reported to be associated with lung function or cancer. For example, CHRM3 encodes the muscarinic acetylcholine receptor M3, which is a well-characterized drug target for which many approved drugs exist, including for the treatment of asthma and obstructive lung disease28. BORIS transcripts were expressed in lung carcinoma cell lines at high to moderate levels29. KCNE2 and TTLL2 might be associated with pulmonary function31. MSMP hindered the effect of anti-VEGF therapy32, and it could promote xenograft PC3 growth and reduce the survival of PC3 metastatic mice model33. The genes selected by SIS and CAR are listed in Table The 34. We tried to estimate We developed the ECAR scores, which can simultaneously measure the importance of all the variables in regression models. We showed that by diligently searching the parameter which maximizes PR-AUC, ECAR can improve traditional methods like SIS, ridge, CAR in terms of the feature selection accuracy, while maintaining strong predictive power in high-rank features. ECAR is also highly competitive to popular variable selection methods like lasso, notably when influential factors are correlated. Moreover, it enjoys the grouping property that strongly correlated variables will tend to be selected together. Another advantage of ECAR is that its parameter is insensitive to the sampling setting: even the coefficients’ distribution used is different from the real one, the results would not be significantly different. The flexibility of ECAR not only enables it to perform well under settings that are unsuitable for CAR, but also ensure it has approximately equal performance when CAR works satisfactorily. Some researchers applied the CAR scores to SNP selection in the GAW17 dataset and found CAR performed much better than other classic methods21, which enables substantial computational savings when the sample size is much smaller than the number of features. Another issue is that it is not a completely automatic method, which means we have to estimate some parameters like One issue with ECAR is that it is computationally intensive when estimating the parameter To evaluate the performance of the ECAR scores, we used three kinds of datasets. The first kind of datasets has 200 samples and 600 features which were split into two equal-sized groups. The datasets were generated from a multivariate normal distribution with mean zero. The correlations between any two features in the same group are 0.25, and features in different groups are independent. The outcomes were generated by a Gaussian linear model in which influential features are randomly selected. The second datasets we used were semi-synthetic datasets. We called it semi-synthetic datasets because the features are from real mRNA expression data obtained from 3 cancer projects  from TCGA. The outcomes were generated in the same way as described above. We compared the variable selection accuracy on the datasets mentioned above. We also used three barley datasets downloaded from the T3/barley database whose sample and feature size ranged between 712 to 1947 and 6236 to 6583, to evaluate the prediction performance of ECAR. We applied ECAR to the LUAD dataset and analyzed genes that influence forced expired volume in 1 s (FEV1). The information of datasets is shown in Supplementary Table 35 is an example. The number of influential features 36 implemented in R package corpcor. After generating the simulated datasets, the PR-AUC  can be computed at each value of 23 (using function fdrtool from R package fdrtool).Our method takes genomic features and the outcome as input and returns scores that represent the importance of features. In ECAR scores mentclass2pt{minimThe performance of ECAR was evaluated in terms of feature selection and prediction accuracy. ECAR assigns a score to each feature in the dataset, and these features are later ranked and selected by the model in descending order according to the absolute value of their scores. For the evaluation of ECAR’s performance on simulated datasets whose true influential features are known, we used PR-AUC, which is the area under the precision-recall curve created by plotting the precision against the recall at various threshold settings. The precision is the number of true positive factors divided by the number of selected factors, while recall is the fraction of true positive factors that are retrieved. We also plotted the number of true positive factors against the number of total selected factors. For the real data, we looked at the prediction performance of the selected SNP features. The whole data were randomly split into a training set of 2/3 of total samples and a test set of the remaining 1/3. We reported the test performance evaluated by MSE (mean square error) averaged across ten different random divisions of training and test sets. For all the methods, we showed the mean of MSE for the number of selected features To test the stability of 20. Another approach used for feature ranking is CAR scores, which are calculated based on the correlations between the de-correlated variables and the response variable. The calculation of CAR scores is relatively simple, which is ECAR was compared with CAR, SIS, Ridge, Lasso, stability selection and random selection  in terms of feature selection accuracy and generalization performance of high-rank features. The rank of the variables is based on their scores calculated from different methods. Sure Independence Screening (SIS) is a univariate correlation ranking method that ranks features’ importance according to their marginal correlation with the response variable. It is helpful in ultra-high dimension settings for screening irrelevant variables; however, it could lead to a high rate of false positives regarding importance ranking problems in moderately high dimension settings. In our study, the scores are the absolute value of Pearson Correlation Coefficient R (v.3.6.0) was used for the development of ECAR; pROC (v.1.15.0) was used for computing PR-AUC; corpcor (v.1.6.9) was used for calculating the α power of the correlation matrix. glmnet (v.3.0-2) was used for comparison with ECAR. mvtnorm (v.1.0-10) was used for generating random numbers for the multivariate normal distribution.Supplementary Information.


GSE139825 is lung tissue microarray, and GSE83456, GSE19491 and GSE50834 are blood microarray. The differential genes of GSE139825 and GSE83456 were 68 and 226, and intersection genes were 11. Gene ontology (GO) analyses of 11 intersection genes revealed that the changes were mostly enriched in regulation of leucocyte cell‐cell adhesion and regulation of T‐cell activation. Kyoto Encyclopedia of Genes and Genomes (KEGG) enrichment analysis of DEGs revealed that the host response in TB strongly involves cytokine‐cytokine receptor interactions and folate biosynthesis. In order to further narrow the range of biomarkers, we used protein‐protein interaction to establish a hub gene network of two data sets and a network of 11 candidate genes. Eventually, IRF1 was selected as a biomarker. As validation, IRF1 levels were shown to be up‐regulated in patients with TB relative to healthy controls in data sets GSE19491 and GSE50834. Additionally, IRF1 levels were measured in the new patient samples using ELISA. IRF1 was seen to be significantly up‐regulated in patients with TB compared with healthy controls with an AUC of 0.801. These results collectively indicate that IRF1 could serve as a new biomarker for the diagnosis of pulmonary tuberculosis.Pulmonary tuberculosis (PTB) is a major global public health problem. The purpose of this study was to find biomarkers that can be used to diagnose tuberculosis. We used four NCBI GEO data sets to conduct analysis. Among the four data sets,  Its main clinical features are prolonged low fever, expectoration and haemoptysis. Mycobacterium tuberculosis can infect tissues and organs all over the body, including lungs, intestines, lymph nodes, joints, spine and genitourinary system.Tuberculosis is an infectious disease caused by Mycobacterium tuberculosis, which is at risk of developing tuberculosis.Mycobacterium tuberculosis is a kind of intracellular parasitic bacteria. Participation in immunity against Mycobacterium tuberculosis is mainly cellular immunity, including macrophages, T cells and NK cells.In recent years, tuberculosis has gradually become a public health problem that threatens human health. According to the tuberculosis report of the World Health Organization, about 10 million people in the world were infected with TB and 3 million died in 2018. It is also estimated that about 1/4 of the world's people are infected with Mycobacterium tuberculosis to escape the immune killing of the body.Mycobacterium tuberculosis infection has essential theoretical significance for clinical diagnosis and the study of novel tuberculosis vaccine and immunotherapy. The immune response in lung tissue can comprehensively reflect the lung immune response to Mycobacterium tuberculosis, so we can screen related molecular markers as sensitive indicators of tuberculosis infection. In this study, bioinformatics methods were used to compare and analyse the original genetic data of patients with pulmonary tuberculosis and healthy people, hoping to find the genes that may play an important role in the pathogenesis of tuberculosis, reveal the molecular immune mechanism of tuberculosis and discover the potential biomarkers of TB.However, MTB can inhibit oxidative stress, apoptosis and autophagy, inhibit the synthesis of histocompatibility complex molecules and thus affect antigen presentation.22.1GSE139825,GSE83456, including 35 pulmonary tuberculosis (PTB) samples and 35 healthy control (HC) samples. The exclusion criteria were as follows: patients with immunodeficiency, such as HIV infection, congenital immunodeficiency; patients with hepatitis B; and patients with autoimmune diseases. All of the above conditions can affect pulmonary immunity, so they were excluded. The first verification data were extracted from GSE19491, including 21 PTB samples from London, 12 HC samples from London, 20 PTB samples from South Africa and 31 latent TB samples from South Africa. The second verification data were extracted from GSE50834, including 23 HIV‐infected samples and 21 HIV/TB co‐infected samples. All sample data were downloaded for further analysis. All sample data were from public databases, so informed consent and ethical approval were not required.The data of human lung tissue samples and blood samples infected with TB were extracted from the GEO database.2.2t test method was used to calculate P‐value of genes, and the adjusted P‐value was calculated by Benjamini and Hochberg's method. The following selection criteria were used to screen out differentially expressed genes: ∣log FC∣> 1 between two sample groups; and adjusted P‐value <.05. Using Venn diagram, the DEGs of GSE139825 and GSE83456 were intersected.The original expression matrix had been normalized by the uploader, so there was no need for normalized operation. The differentially expressed genes (DEG) were screened out by limma package.2.3https://metascape.org/gp/index.html/main#/step1) The Metascape We used the Search Tool for the Retrieval of Interacting Genes (STRING) online database . The experiment was carried out according to the manufacturer's instructions.2.6GSE19491 and GSE50834, were used to evaluate the expression level of potential biomarker. We used idmap R package to find out potential biomarker expression level in two gene matrixes. The ggplot2 R package was used to draw box plot to show the different expression level in different groups. In GSE19491, there were three groups: HC groups, PTB groups and Latent TB groups. In GSE50834, there were two groups: HIV‐infected groups and HIV/TB co‐infected groups.Two data sets, 2.7Gene Set Enrichment Analysis (GSEA) is a computational method that determines whether an a priori defined set of genes shows statistically significant, concordant differences between two biological states . We used the GSEA software to conduct the GSEA. P‐value <0.05 was statistically significant.2.8P < .05 was considered to be statistically significant.All statistical analysis was carried out by R (version: 3.6.2) software. Student t test was used to test the difference between the two groups. The receiver operating characteristic (ROC) curve was used to evaluate the reliability of candidate indicators as diagnostic biomarkers. If the area under curve (AUC) was greater than 0.8, it means that the reliability of the candidate biomarkers was higher. All P values were bilateral, and 33.1P value <.05 and ∣logFC∣ >1 were defined as selection criteria. The intersection DEGs between GSE139825 and GSE83456 were as follows: IL7R, TNFAIP6, KLHDC8B, IRF1, HELZ2, ADM, CALHM6, GCH1, CD274, CCR7 and PSTPIP2. The volcano plot and Venn diagram were plotted based on the analysis results of the gene expression matrix  to analyse 11 intersection DEGs and obtained the interaction data of 11 intersection DEGs Figure . Then, 2s Figure . We tooke Figure .3.4The IRF1 was selected as a candidate biomarker for experimental verification. The experimental results showed that IRF1 was up‐regulated in tuberculosis group, which was consistent with the results of our bioinformatics analysis Figure . The ROC3.5GSE19491 was as follows: IRF1 was up‐regulated in TB group from HC group ; Writing‐original draft (lead); Writing‐review & editing (lead). Yanzheng Song: Data curation ; Funding acquisition (lead). Lin Wang: Methodology ; Software . Zilu Wen: Data curation ; Formal analysis . Yijun Zhu: Data curation ; Methodology . Qiliang Chen: Data curation ; Funding acquisition ; Investigation .


Here, using a panel of 14 chemicals with different risks of human drug-induced liver injury (DILI) and two dosing concentrations, we evaluated an HTTr platform comprised of collagen sandwich primary rat hepatocyte culture and the TempO-Seq surrogate S1500+ (ST) assay. First, the HTTr platform was found to exhibit high reproducibility between technical and biological replicates (r greater than 0.85). Connectivity mapping analysis further demonstrated a high level of inter-platform reproducibility between TempO-Seq data and Affymetrix GeneChip data from the Open TG-GATES project. Second, the TempO-Seq ST assay was shown to be a robust surrogate to the whole transcriptome (WT) assay in capturing chemical-induced changes in gene expression, as evident from correlation analysis, PCA and unsupervised hierarchical clustering. Gene set enrichment analysis (GSEA) using the Hallmark gene set collection also demonstrated consistency in enrichment scores between ST and WT assays. Lastly, unsupervised hierarchical clustering of hallmark enrichment scores broadly divided the samples into hepatotoxic, intermediate, and non-hepatotoxic groups. Xenobiotic metabolism, bile acid metabolism, apoptosis, p53 pathway, and coagulation were found to be the key hallmarks driving the clustering. Taken together, our results established the reproducibility and performance of collagen sandwich culture in combination with TempO-Seq S1500+ assay, and demonstrated the utility of GSEA using the hallmark gene set collection to identify potential hepatotoxicants for further validation.Cell-based  The U.Ss (AOPs) . To broas (AOPs) . To incrs (AOPs) . Combines (AOPs) , the surs (AOPs)  has demos (AOPs) . TempO-Ss (AOPs) .Gene set enrichment analysis (GSEA) is a powerful method for interpreting genome-wide expression profiles by grouping individual genes into gene sets that represent a specific biological state or process sharing the same chromosomal location or regulatory targets . GSEA isAlthough the S1500+ gene set and the hallmark gene set collection were independently developed, both aim to reduce redundancy without compromising on the coverage of biological space. The former achieves this at the gene level and the latter at the gene set level. We reasoned that GSEA of S1500+ genes using the hallmark gene set collection would inform the effects of a chemical on the liver and provide a common pattern of enrichment that represents hepatotoxicity. However, GSEA was originally designed to run on the entire transcriptome (Subramanian 2005), and studies comparing GSEA of the whole transcriptome to that of a subset of genes representing the whole transcriptome are limited. To examine if the S1500+ gene set could produce comparable results as the whole transcriptome in GSEA, we selected sandwich cultured rat hepatocytes as a resource-sparing and scalable model for hepatotoxicity screening, measured gene expression using the TempO-Seq ST HTTr assay, and performed GSEA with the hallmark gene set collection. We chose the collagen sandwich culture configuration amongst a plethora of 2D/3D cultures for balanced operational simplicity/robustness and maintenance of hepatocyte functions for up to a week , and limSince establishing the performance of a high-throughput platform is essential before incorporating the data for chemical risk assessment, the goals of this study were to (1) investigate the intra- and inter-batch reproducibility of the HTTr platform, as well as concordance with external data, (2), compare the TempO-Seq surrogate S1500+ transcriptome (ST) assay with the whole transcriptome (WT) assay in detecting chemical-induced changes at the gene and pathway levels, and (3) identify a general pattern of hallmark gene set enrichment that represents hepatotoxicity. Our results established the performance and reproducibility of the HTTr platform and demonstrated the potential of combining S1500+ high-throughput transcriptomics with hallmark gene set enrichment analysis for hepatotoxicity assessment.p < 0.05) ranging from zero (indomethacin and chloramphenicol at the respective low concentrations) to more than sixteen thousand  after 24 h of treatment.Fourteen chemicals representing different drug classes were selected from Open TG-GATEs for HTTr by TempO-Seq assays . Chemicain situ collagenase perfusion method described previously  based on a modified eviously . Animals5 cells/mL), the latter was placed in a reagent reservoir and transferred using a multi-channel pipette fitted with wide-bore tips. The reagent reservoir was gently rocked between each withdrawal to prevent hepatocytes from settling. After cell seeding, the plate was left undisturbed for 5 min before transferring to a cell culture incubator at 37 °C and 5% CO2. The cells were allowed to attach to the collagen-coated surface for 3 h, then the medium was removed and the wells were washed with PBS once to remove any unattached cells or cell debris. Next, 100 μL of collagen solution (0.1 mg/mL in culture medium) was added for forming the top collagen layer. The collagen solution was prepared by first neutralizing 280 µL of stock collagen (3 mg/mL) with 210 µL 1X PBS, 35 µL 10X PBS and 35 µL 0.1 M NaOH, then 0.5 mL of the neutralized collagen was mixed with 7 mL of chilled culture medium. After overnight incubation (16 h), a collagen overlay was formed and the spent medium was replaced with 100 µL of fresh culture medium. Cells were incubated for an additional day before chemical treatment. Care was taken during the removal and addition of culture medium to avoid disturbing the collagen overlay. On the day of treatment, the spent medium was replaced with 100 µL of fresh medium containing chemicals. All treatments were performed in culture medium containing 0.5 % DMSO for 24 h. Each treatment condition was tested in triplicates to examine technical reproducibility. To avoid edge effects due to different rate of evaporation, only the inner 60 wells were used while the remaining 36 perimeter wells were filled with 200 µL of RNAse-free water. Two identical sets of chemically-treated hepatocytes were prepared, one for cell viability assay and the other for TempO-Seq assay. The entire experiment was performed twice using hepatocytes isolated from different individuals in order to examine reproducibility between biological replicates.To coat collagen on a 96-well plate (Thermo Scientific™ Nunc™), we diluted bovine type I collagen  with equal volume of 0.01 M HCl and dispensed 80 µL of the diluted collagen solution to each well. After 3 h of incubation at 37 °C, the collagen solution was removed and the wells were washed with 100 µL of PBS twice. Collagen-coated plates were either immediately seeded with cells or stored at 4 °C in PBS for at most 3 days. On the day of cell seeding, 50 µL of culture medium was added to the wells followed by 50 μL of hepatocyte suspension .To prepare samples for TempO-Seq, we removed the spent medium after 24 h of chemical treatment and washed the wells with 100 µL of PBS twice. Then the cells were lysed by adding 20 µL of 1X TempO-Seq lysis buffer (BioSpyder) in PBS. The plates were incubated at r.t. for 10 min and sealed with Nunc™ Sealing Tapes prior to storage at −80 °C until shipment. Samples were shipped frozen on dry ice to BioSpyder Technologies, Inc. , where libraries and targeted sequencing of the S1500+ surrogate gene set  and the whole transcriptome  were performed. Briefly, 2 μL of each sample lysate was hybridized with detector oligos from the TempO-Seq rat ST or WT assay, with 1 h hybridization period for the ST assay and overnight for the WT assay. After ligation and nuclease digestion, the ligated products were added into an amplification mix with sample-specific PCR primer pairs, which include the standard Illumina adaptor and a sample-specific barcode sequence. Amplicons were pooled and purified using a PCR clean-up kit . Sequencing was performed on an Illumina HiSeq 2500 system with both ST and WT libraries included in the same sequencing run. Reads were de-multiplexed using Illumina’s bcl2fastq software version 2.20 into FASTQ files for each sample. Reads were aligned to reference genome (rn6) using STAR aligner version 2.5.3a allowing 2 mismatches. The Tempo-SeqR package  was used to generate read counts data. FASTQ files and read counts data have been deposited in the National Center for Biotechnology Information Gene Expression Omnibus (GEO) (GSE152128).All probes in the TempO-Seq assays were included for calculating the total number of read counts, the percentage of probes with zero read count, and the correlation of read counts between technical or biological replicates. Normalization and pre-filtering were not performed for these calculations and the raw read counts data were used. For correlation of read counts, a Pearson correlation coefficient was calculated for each replicate in comparison to the other two replicates form the same batch of samples , resulting in three correlation coefficients for each treatment. In addition, Pearson correlation coefficients were calculated between replicates from different batches of samples  to give the correlation coefficients between biological replicates. For visualization of batch effect by PCA plot, probes with less than an average of ten counts across samples from both batches were excluded from analysis and read counts were normalized by DESeq and varianceStabilizingTransformation (vst) functions of the DESeq2 package .p value of<0.05 were considered as differentially expressed. For comparison of DEGs between biological replicates, differential gene expression analysis was independently performed for each batch of samples. For comparisons of DEGs between ST and WT assays and gene set enrichment analysis (GSEA), read counts of gene isoforms were combined by addition to produce one read count per gene, and then differential gene expression analysis was performed with batch correction.Probes with less than an average of ten counts across samples from both batches were removed from the count matrix. The resulting count matrix was used as input in the DESeq2 R package to identify differentially expressed genes relative to vehicle control. Probes with adjusted GSEA was performed using the fgsea R package  and the https://www.ebi.ac.uk/arrayexpress/experiments/E-MTAB-797/). Raw CEL files for each treatment, generated using the Affymetrix GeneChip Rat Genome 230 2.0, were normalized using robust multiarray average (RMA) . The difference between the log2 normalized intensity values for 18,894 transcripts for each treatment were compared with matched controls to calculate the average log2 fold-change (L2FC) profiles across 12,962 genes. Probes in each L2FC profile were mapped to genes using the Affymetrix Rat Genome 230 2.0 Array annotation obtained from Ensembl Release 102  of samples ran by ST assay ranged from 0.6 to 2.0 (mean = 1.2 ± 0.3) and from 1.0 to 2.2 (mean = 1.6 ± 0.3) for batch 1 and 2, respectively a. For saPearson correlation of read counts was performed to assess reproducibility between technical replicates within the same batch of samples and between biological replicates from different batches of samples. Regardless of the batch or the type of assay, correlation coefficients of read counts between technical replicates were above 0.90 for the various treatments c. CorrelNext, differential gene expression analysis was independently performed for the two batches of samples. A number of observations were made by comparing the number of DEGs between different assays, batches and treatments . First, Reproducibility of differential gene expression was evaluated by determining the fraction and correlation of DEGs that overlapped between the two batches of samples . It was found that treatment with high concentrations of drugs typically produced a more reproducible transcriptomic signature than treatment with low concentration of drugs. For example, the fraction of overlapping DEGs between the two batches of acetaminophen-treated samples ran by ST assay increased from 4% at 400 µM to 75% at 10 mM a. The frThe second objective of this work is to evaluate inter-platform transcriptomic reproducibility by comparing TempO-Seq ST and WT data generated in this study with Affymetrix rat GeneChip 230 2.0 data produced by the Open TG-GATEs project . The genThe performance results for connectivity mapping using ST and WT signatures of different lengths and different algorithms are shown in There was a complex relationship between the length of signatures, connectivity-scoring algorithms and transcriptome assays (ST vs WT), but we observed some interesting trends. Increasing the signature length did not result in substantial improvement in performance for any of connectivity-score algorithms. The Fr1 performance of ST signatures for xc increased up to nq ≤ 300 and then decreased whereas the performance for sji decreased monotonically for nq > 100. On the other hand, increasing the WT signature length produces modest improvements in performance for sji (nq ≤ 200), xcs (nq ≤ 400) and gtes (nq ≤ 400). The Fr5 performance of ST signatures varied less with nq as it did for Fr1. For the WT signatures, Fr5 signatures showed consistent performance for all algorithms except gtes. Finally, the Fr10 performance of ST signatures generally decreased for nq ≥ 200. On the other hand, the Fr10 performance of WT signatures did not change much for xc, xcp and xcs with nq, it decreased for sji and it increased for gtes.The best connectivity mapping approaches for the ST and WT signatures were identified for the Fr1, Fr5 and Fr10 performance metrics. The ST signatures produced the highest Fr1 performance of 0.5 using xc (with shortest nq = 200 and shortest nr = 2000) and xcp (with shortest nq = 200 and nr = 4000). The highest Fr1 performance of 0.46 for WT signatures was produced by xcs (with shortest nq = 300 and shortest nr = 4000). The highest Fr5 performance value for ST signatures of 0.75 was produced by xcp (nq = 200 and nr = 5000) and for WT signatures as value of 0.75 was produced by sji, xcs and xc (for different signature and profile lengths). Lastly, the highest Fr10 performance value for ST signatures of 0.89 was produced by xcp and xc (nq = 100 and nr = 5000) and for WT signatures as value of 0.86 was produced by xcs and xc (for nq = 400 and nr = 5000).In The similarity between TempO-Seq ST and TG-GATES transcriptomics profiles was also driven by common mechanisms of action for the chemicals . To illuOverall, the connectivity mapping analysis of these 14 chemicals using TempO-Seq ST and WT data and Affymetrix rat GeneChip data from Open TG-GATES on rat primary hepatocytes show a high-level of inter-platform reproducibility.The third objective of this study is to evaluate the robustness of S1500+ gene set as a surrogate for the whole transcriptome. A positive correlation of L2FC values was observed between genes common to both assays, with correlation coefficients being above 0.95 for all treatments . By genePCA plots were generated using the L2FC values of the union of DEGs to visualize and compare the gene expressions between the two assays b–e. RemaNext, gene signatures captured by the ST and WT assays were compared using GSEA with the hallmark gene set collection. The number of genes within each hallmark gene set after restricting the genes present in the ST and WT assay are shown in It was also observed that ST assay generally resulted in a lower number of significantly enriched hallmark gene sets (FDR < 0.05) compared to WT assay . NonetheHierarchical clustering divided treatments according to similarities in the enrichment of hallmark gene sets. NES of samples calculated by both ST and WT assays were combined into a single matrix for the analysis. The clustering correctly paired samples by chemical treatment, that is, the nearest neighbor to a treatment ran by ST is the same treatment ran by WT , was obsBesides producing treatment-specific pairs, hierarchical clustering using NES of hallmark gene set divided the treatments into three major clusters  modeling of transcriptomic data has been developed to estimate apical BMD or point-of-departure (POD) values  values . Transcr) values  in order) values .In this study, we described an HTTr platform comprised of a collagen sandwich cell culture model and the TempO-Seq transcriptomic assay. Using a comprehensive study design involving fourteen chemicals with different DILI classifications and two treatment concentrations, we systematically evaluated technical (well-to-well), biological (batch-to-batch), and cross-platform (TempO-Seq-to-Affymetrix) reproducibility of this HTTr platform. Our results show that, while the read depth and fraction of probes with zero read count varied between samples a, b, a sIt is noteworthy that a strong correlation between replicates does not necessarily reflect agreement, as the large number of targeted genes, particularly in the WT assay, makes it difficult for linear relationships to be skewed by strong disagreement between a small number of genes . NonetheOur results also showed that the reproducibility of DEGs between biological replicates positively correlates with the magnitude of transcriptional effect induced by the treatment, which depends on the potency of the chemical and the concentration . A more On the other hand, testing concentrations above the highest non-cytotoxic concentration should be avoided as it may not yield information relevant to the mode of action of a chemical . Accordiin vitro rat primary hepatocyte subset of the Open TG-GATES data, which was most closely aligned with our study design, including 131 chemicals a set of 1177 transcriptomic profiles. Using five different connectivity-scoring algorithms to quantify similarities between all TempO-Seq and Affymetrix transcriptomic data, we found 68% of the chemicals were perfectly matched between the two platforms and 100% of the chemicals were in the top 10 hits. Connectivity mapping from TempO-Seq ST and WT and the TG-GATES data showed a high level of concordance  for accurate results. According to the GSEA User Guide version 3.0, the recommended minimum number of genes in a gene set is 15. Seven hallmark gene sets had<15 genes in the S1500+ panel, namely, Notch signaling, Hedgehog signaling, pancreas beta cells, apical surface, WNT beta-catenin signaling, KRAS signaling DN, and angiogenesis . Also, tc : A*STAR via IFCS ; via EMULSION IAF (R-185-000-350-305); MOE ARC (R-185-000-342-112); SMART CAMP; and Mechanobiology Institute of Singapore (R-714-106-004-135) funding to Hanry Yu.Fan Lee: Investigation, Formal analysis, Writing - original draft. Imran Shah: Funding acquisition, Supervision, Conceptualization, Formal analysis, Writing - review & editing. Yun Ting Soong: Investigation, Writing - review & editing. Jiangwa Xing: Conceptualization. Inn Chuan Ng: Resources. Farah Tasnim: Conceptualization, Funding acquisition, Project administration. Hanry Yu: Supervision, Funding acquisition, Writing - review & editing.Hanry Yu declares he holds significant equity in Invitrocue, Osteopore, Histoindex, Vasinfuse and Ants Innovate not related to the reported work. The authors declare they have no actual or potential competing financial interests.


TISMO also includes 1518 in vivo RNA-seq samples from 68 syngeneic mouse tumor models across 19 cancer types, of which 832 were from immune checkpoint blockade (ICB) studies. We manually annotated the sample metadata, such as cell line, mouse strain, transplantation site, treatment, and response status, and uniformly processed and quality-controlled the RNA-seq data. Besides data download, TISMO provides interactive web interfaces to investigate whether specific gene expression, pathway enrichment, or immune infiltration level is associated with differential immunotherapy response. TISMO is available at http://tismo.cistrome.org.Syngeneic mouse models are tumors derived from murine cancer cells engrafted on genetically identical mouse strains. They are widely used tools for studying tumor immunity and immunotherapy response in the context of a fully functional murine immune system. Large volumes of syngeneic mouse tumor expression profiles under different immunotherapy treatments have been generated, although a lack of systematic collection and analysis makes data reuse challenging. We present Tumor Immune Syngeneic MOuse (TISMO), a database with an extensive collection of syngeneic mouse model profiles with interactive visualization features. TISMO contains 605  Instead, syngeneic tumors transplanted into immunocompetent mice are readily available and provide reproducible results for cancer immunology research. Syngeneic mouse models have been widely used in cancer immunology studies, and a large volume of tumor expression profiles under various immunotherapy treatments have been generated . Among these databases, only NCI OMF contains syngeneic tumor model studies, although it solely provides meta-information of these studies without expression profiles. Similar to NCI OMF in providing study-level meta-information, GXD  is a comprehensive database with over two thousand uniformly processed and quality-controlled RNA-seq samples of syngeneic mouse cancer cell lines and tumor models. These datasets were uniformly processed from raw sequencing reads using a standardized workflow. In addition, immune cell infiltration and pathway enrichment levels have been inferred and phenotypic metadata have been manually annotated. TISMO provides interactive web interfaces for users to compare and visualize gene expression, pathway enrichment, and immune infiltration level across syngeneic mouse models, treatments, and response groups. The continued maintenance of TISMO will be of great utility to the cancer immunology and immuno-oncology research community.Herein, we present Tumor Immune Syngeneic MOuse (TISMO), a large-scale publicly accessible resource of syngeneic mouse models. TISMO (We developed a parser to query datasets deposited in the Gene Expression Omnibus (GEO)  between https://liulab-dfci.github.io/RIMA). RIMA is an automated Snakemake pipeline developed by our group to streamline the processing of RNA-seq data, including but not limited to read alignment, quality control, expression qualification, batch effect removal, and immune cell infiltration inference. FASTQ files containing the raw reads were downloaded or transferred. Read alignments were performed with STAR . ith STAR   , we firsin vitro samples and in vivo samples, respectively.Samples collected from different studies were processed with different protocols and platforms, and subject to technical bias. Even though TISMO conducts all sample comparisons within each cohort, we normalized the data for consistency across cohorts. Specifically, for each cohort we pre-computed false discovery rates (FDR) using DESeq2  for eachhttps://github.com/zexian/TISMO_data).To make reliable and robust immune cell infiltration estimations, we utilized Immunedeconv , an R pahttp://tismo.cistrome.org without any registration or login restriction. It is implemented with the R-Shiny framework (R version 3.6.3) on an Apache2 HTTP server, and is compatible with smartphones and tablets. The website consists of seven functional components: ‘Home’, ‘Data Browser’, ‘Gene’, ‘Pathway’, ‘Infiltrates’, ‘Data Download’, and ‘Documentation’. ‘Home’ includes a tutorial video with step-by-step instructions on using the database and website. Users could browse or search the curated metadata using the ‘Data Browser’ module to locate relevant syngeneic models. ‘Gene’, ‘Pathway’ and ‘Infiltrates’ modules enable users to select and compare gene expression, pathway enrichment, and immune infiltration level between treatments, response groups, and models. Users could explore and derive gene expression programs or immune infiltrates consistently associated with ICB treatment and response in these modules. In the pathway module, users could also upload and evaluate self-defined gene sets. In the ‘Data Download’ module, users could download phenotypic metadata, quantified gene expression, and immune cell infiltration for all samples in the database. The documentation page summarizes the data processing steps and the number of samples in different models and treatment conditions.To enable users to systematically explore the curated datasets, we developed a user-friendly web interface to host the TISMO database. The TISMO website is freely available at in vitro RNA-seq samples from 49 syngeneic cancer cell lines across 23 cancer types. TISMO also contains 1518 in vivo RNA-seq samples from 68 syngeneic mouse tumor models across 19 cancer types , allowing users to evaluate them across models, treatments, and response groups.Cytokines, including IFN and TNF, play essential roles in adaptive immunity in the TME . Due to N = 832), allowing users to explore whether specific gene expression programs or immune infiltrations are robustly associated with ICB response or resistance. Using TISMO, users could select a specific gene, pathway, or immune cell infiltration, evaluate whether its level changes upon ICB treatment, and compare it between ICB responders and non-responders . In. InN = 8s Figure . The difs Figure , and thes Figure ) and immin vivo samples, and the manually annotated metadata for all samples.TISMO’s website hosts a data browser module to help users locate relevant mouse models from our collection. Users could query sample Study ID , sample metadata, treatment condition, response status and the number of replicates in each design. The data browser module aids researchers in selecting the most relevant syngeneic mouse models to supply evidence for hypothesis generation or validation. On the TISMO website, data cohorts and treatment types could be selected for statistical comparison and visualization. The interactive filtering and visualization features enable users to systematically compare different models, cell lines, treatments, and response groups. The generated figures could be downloaded in jpg, pdf, and table formats. The download page provides access to the expression matrices of all 2123 RNA-seq profiles, immune cell infiltration estimations of the http://tismo.cistrome.org) is the first comprehensive database for users to investigate and visualize gene expression, pathway enrichment, and immune cell infiltration levels in syngeneic mouse models across different ICB treatment and response groups. Expression profiles hosted on TISMO were uniformly processed from the raw sequencing reads. The immune infiltration levels were inferred by six state-of-the-art deconvolution algorithms, and the metadata were manually annotated from publications. TISMO provides web interfaces to help users explore the syngeneic mouse model data interactively. In summary, TISMO is a comprehensive database of syngeneic mouse models that will help users select relevant syngeneic mouse tumor models, provide data to generate and test hypotheses, and reveal novel mechanisms of ICB response and resistance.Syngeneic mouse models are essential in immunotherapy research as they enable the study of cancer cells in the context of immunocompetent hosts. Large amounts of syngeneic mouse model profiles have been generated, but these data are scattered, making data reuse challenging. There is still no comprehensive, intuitive, and convenient database with user-friendly, interactive web features for researchers to explore syngeneic mouse model data. TISMO (http://tismo.cistrome.org to all users without restrictions. Annotated mouse syngeneic metadata are available on the website. All expression data and immune infiltration estimation can be downloaded from the data download page.TISMO is available at gkab804_Supplemental_FilesClick here for additional data file.


Recent technological advances have resulted in an unprecedented increase in publiclyavailable biomedical data, yet the reuse of the data is often precluded by experimentalbias and a lack of annotation depth and consistency. Missing annotations makes itimpossible for researchers to find datasets specific to their needs.Here, we investigate RNA-sequencing metadata prediction based on gene expressionvalues. We present a deep-learning–based domain adaptation algorithm for the automaticannotation of RNA-sequencing metadata. We show, in multiple experiments, that our modelis better at integrating heterogeneous training data compared with existing linearregression–based approaches, resulting in improved tissue type classification. By usinga model architecture similar to Siamese networks, the algorithm can learn biases fromdatasets with few samples.Using our novel domain adaptation approach, we achieved metadata annotation accuraciesup to 15.7% better than a previously published method. Using the best model, we providea list of >10,000 novel tissue and sex label annotations for 8,495 unique SRAsamples. Our approach has the potential to revive idle datasets by automated annotationmaking them more searchable. Next-generation RNA-sequencing (RNA-seq) has been a pillar of biomedical research for manyyears ,2.It alTo allow for efficient data reuse, publicly available data have to be harmonized and wellannotated with standardized metadata and subsequently be made accessible (and searchable); this prEfforts to predict missing or sparse metadata in public RNA-seq resources have shownpromising results. For instance, recently published studies used text mining approaches toretrieve missing annotation from associated abstracts or free text annotations in the datasources . Others Artificial neural networks (ANNs) in their various forms and functions consistentlyoutperform classical ML approaches in a large variety of biological tasks, includingclassification, data generation, and segmentation . Given lHere, we present a DA approach capable of leveraging a number of dataset biases, boostinggeneralizability of phenotype prediction. We developed the model using 3 data sources  of different size and with a different degree of bias. To validate ourapproach we compare it to a previously suggested linear model (LIN) , as wellTo train and test models we gathered data from 3 different sources , each with a different level of heterogeneity . We measGTEx (v6) comprises 9,662 samples from 554 healthy donors across 31 tissues. GTExstrives to build a highly homogeneous dataset with strict guidelines on donor selection,biopsy, and sequencing methodology [From the SRA, a total of 2,034 studies containing a total of 49,657 samples weredownloaded from recount2. Every SRA study was potentially processed at a different siteby a different technician following different standards. In addition, the underlyingbiological condition of the samples is often unclear. We assume each study to have aunique dataset bias, which makes the SRA a highly heterogeneous data source. Inaddition, data annotation is not standardized, resulting in sparse metadata with lowfidelity.RNA-seq data for TCGA were downloaded, consisting of 11,284 samples spanning 26tissues. While there are 740 samples of healthy donors across 20 tissues, >90% of thesamples are tumor biopsies from different tissues and different stages of tumorprogression. TCGA accepts sequence data from different locations using differentsequencing technologies. Despite the high level of standardization and reliability ofmetadata information, heterogeneity is also inherent to the TCGA dataset due to thebiological context  albeit not as pronounced as in the SRA.In this study, we focus on bulk mRNA-seq data because they are by far the most frequentRNA type in each of the 3 data sources used. The following approaches were used to removedata from single-cell and small RNA-seq studies from further analysis: First, weidentified small RNA-seq data on the basis of the total fraction of small RNA counts andprotein coding RNAs. Specifically, we considered a subset of the Gencode gene types . Every sample that had its maximum totalcount fraction not allocated to either protein_coding or processed_pseudogene was removedfrom further analysis . Second,We considered 3 different phenotypes for expression-based prediction. Explicitly, wepredicted the tissue of origin of a biopsy , thepatient's sex, and sample source (denoting whether the sample was from a patient biopsy ora laboratory-grown cell line) Fig. .https://gtexportal.org/  Tissue and sex annotation for GTEx were extracted from the official sample annotationtable as provided by GTEx . Homo sapiens cellline, which resulted in 3,240 annotated SRA samples. Sample source was determined usingthe sqlite file provided by MetaSRA  normalization wasapplied to normalize for gene length  and library size. Wenext reduced the number of input features (genes), aiming to keep features that containinformation and removing potentially uninformative features. First, all non-protein-codinggenes were removed, reducing the gene set by 65.5% to 19,950 genes. For sexclassification, only protein-coding genes on the X and Y chromosome (n = 913) wereselected. For retaining only genes that show significant dispersion across tissues, wecomputed the Gini coefficient [g between 0.5 and 1were retained, resulting in a feature space of dimension d = 6,974. Forsex classification, genes with 0.4 < g < 0.7 were used(d = 190). Sample source classification included genes with 0.3<g < 0.8  . First standard logfficient ,33, 34 fTissue: To ensure that dataset biases are not shared between training and test sets,SRA data were always split on the study level. For tissue of origin prediction, the 2largest SRA studies per class were put in the training set. This ensured maximal biasvariability in the remaining test data, ensuring a realistic test score. Of the 178 SRAstudies containing tissue annotated samples, 30 studies were selected for the trainingset  and 148 studies for the test set  .Sex: We noticed SRA samples identified as female by MetaSRA to have a significantamount of reads mapped to chrY . All samSample Source: A confidence cut-off of ≥0.7 was applied (provided by MetaSRA), reducingthe total amount of annotated samples for SRA from 23,651 to 17,343. MetaSRA provided 6different types of sample source. The 2 largest classes, TISSUE and CELL LINE, wereselected. In this study we renamed the MetaSRA label TISSUE to biopsy to avoid confusionwith the phenotype tissue . For each of the 2 selectedcategories we sorted all available studies by number of samples, placed the first thirdof studies into the training , the second third into the test, and the last third into the SRA validation set  . A list After determining the best model for each phenotype, we retrained the models forautomated metadata annotation. The same datasets as defined above were used for the sexmetadata annotation. Tissue: We followed the same pipeline as described above. Samplesfrom a tissue class other than the original 16 classes were pooled together into a“catch-all" class, resulting in 17 classes. In total, 44 SRA studies were selected forthe training set  and 203 studies for the test set . SampleSource: Contrary to before, for metadata annotation we used all available classes in theSRA data source. All classes that are not of type biopsy were grouped into a single“catch-all" class, while the same cut-off as before was applied. The training set  is made up of 974 SRA studies and the test set  of 492 studies.MLPs use fully connected neural network layers to learn non-linear features from a rawinput space  and consMany DA models correct bias between 2 domains, a source and a target domain. Inbiological research, however, one is often confronted with many small datasets, eachpotentially with its unique dataset bias. Therefore, we specifically designed our DA modelto be able to learn from very few data by using a Siamese network architecture . The Siaa) sampled from the bias domain,and a positive (p) and a negative sample (n) from thesource domain. The anchor and the positive sample have equal class labels, whereas thenegative sample is from a randomly selected different class. The triplet loss function[d arethe distances between the constricted embedding space of the SM and the bias mappinginto that space of the BM on samples i and j. Forimproved training time and robustness, our model is trained on semi-hard triplets [Our DA architecture is based on the Siamese network architecture. A Siamese networkusually shares the weights between 2 equal networks. Here, however, we do not use weightsharing. Weight sharing and other types of architecture did not prove to be applicableto this problem (see Methods section Other Models). It consists of 3 modules: A sourcemapper (SM) and bias mapper (BM), which correspond to the Siamese part of the model, aswell as a classification layer (CL). These modules give rise to 3 differentconfigurations, i.e., 2 training cycles and a prediction configuration see . The tasfunction was usedtriplets  \documenm. The distances are defined as Euclideandistances in embedding space: with a margin parameter σ is the sigmoid activation function for the embedding vector. Triplets are minedonline, meaning that they are newly generated for each batch . The SM-We used the metadata prediction performance of the LIN model described in Ellis et al. as a poicalculating the coverage matrix for the training samples based on the regionsreported in Ellis et al.  by usingbuilding the model by running “filter_regions" and “build_predictor" (R packagephenopredict version 0.99.0) with the same parameters used in Ellis et al. testing the model on the test samples with “extract_data," “predict_pheno,"“test_predictor" (R package phenopredict version 0.99.0)Notably, our experiments differ from the original work  solely bsmall. If the SRA train and testsets are combined for training, we write Slarge. For instance, an experimentusing an MLP, trained on a mix of GTEx and SRA and evaluating on SRA data, would be namedMLP G+Ssmall-S.Each experiment was named after the model, the training, and the test data used. Thepossible models are LIN (linear model ), MLP per class (biopsy vs cell line). At each step 1 additional SRAstudy per class was subsampled, ending with 6 SRA biases and 350 samples in the trainingset per class. As a control experiment, we chose the largest SRA study available for eachclass to create a training set with a single bias per class. Starting with 50 samples perclass in 6 iterations, we subsampled an additional 50 samples, ending with 350 samples,thereby assessing the effect on performance that can be attributed to the dataset size.Subsampling and random selection of SRA studies were repeated 10 times with differentseeds, and each configuration was trained on 10 different seeds, yielding an estimate ofuncertainty.We have identified mislabeled samples for the sex phenotype (see Methods). The followingexperiment was designed to test the ANN-based model’s susceptibility to overfitting onmislabeled training data. An MLP model was trained on GTEx data on 4 tissue classes . A range of fractions of the brain samples were randomlyassigned to skin tissue . The model wasthen trained on GTEx samples of the 4 classes, including the mislabeled brain samples. Wetested the model’s overfitting capabilities by letting it predict the label of themislabelled brain samples. If the model overfits, these samples should be predicted to befrom skin tissue. The same experiment was conducted for the sex phenotype by mislabelingmale samples as female.N is the number of samples,y the true label, and C is the number of classes,jM is the number of samples for class j,and ijy and We report micro and macro accuracy, which are equivalent to mean sample accuracy (msa)and mean class accuracy (mca), respectively. Sample accuracy is a measure of absoluteperformance on the test data. It reports the fraction of correctly classified samples overall classes: U test (scipy.stats.mannwhitneyu v 1.3.1).Accuracy distributions were tested for significance using the non-parametric Mann-WhitneyWhile developing our DA model, we did a thorough literature search and implemented andtested multiple architectures and strategies. Here, we give a brief overview of the modelsthat we found not suitable for the problem of bias-invariant RNA-seq metadata annotation.The first strategy that was tested was interpolation between source and target domain bytraining feature extractors on an increasing ratio of target to source domain data . The secThis study aims to find the best model for RNA-seq metadata annotation based on geneexpression. Three different data sources were selected for which phenotype data wereavailable Fig. . Each ofThree different ML models were compared Fig. .First, Models were trained on either GTEx or a mix of GTEx and SRA data and tested on TCGA andSRA data. Uncertainties for MLP and DA models were estimated from 10 training runs withdifferent random seeds Fig. .We first tested the performance of the LIN, MLP, and DA algorithms to predict the tissueof origin on GTEx , TCGA , and SRA datasets. A subset of 16 tissue labels was chosen that is common to all 3 data sources matched the base model’s msa score but performed slightly worse using themca metric. Similarly, the LIN Ssmall-S model matched the msa of LIN G-S butshowed an increased performance for mca . Notably, by only usingthe small SRA training dataset, we lose the advantage of the large sample size of GTEx.Based on this, we hypothesized that by combining SRA and GTEx in the training data, wemay be able to leverage both sample size and diversity.Metadata prediction on SRA was the most challenging and interesting task owing to thepotentially large number of different biases in the data source. We retrained and testedLIN G-S on our datasets and achieved msa of 0.893 and mca of 0.765 for the 16 tissuesFig. . Of notesmall-S model increased its msa to 0.908 and mca to 0.785, whichin turn is 1% lower than the LIN Ssmall-S model. The 2 best performing modelswere MLP G+Ssmall-S and DA G+Ssmall-S, outperforming LIN G-S onmsa by 2.5% and mca 5.5% . No significant difference in the meanperformance was detected between these 2 models . Crucially, however, DAG+Ssmall-S exhibited the lowest standard deviation (std = 0.003 for msa andstd = 0.009 for mca) of all models tested . LIN Slarge-T improved accuracy by 6.6% for msaand 8.6% for mca to 0.784 and 0.724, respectively. In comparison, MLPSlarge-T increased model performance by 11.4% to 0.832 (by 11.7% to 0.755)for msa (mca) with respect to LIN G-T. Combining GTEx and SRA training data reduced LING+Slarge performance to msa 0.725 and mca 0.651. The best accuracy wasachieved by our MLP G+Slarge  and DAG+Slarge  models. The DA model had thus a 15.7% and9.1% performance increase for msa compared to LIN G-T and LIN Slarge-T,respectively. In addition to being the top performer, DA G+Slarge-T also wasthe most robust model for this task, having the lowest variation in its results (std =0.004 for msa and std = 0.006 for mca)  .small, which allows us to assess the influenceof the amount of bias injection on model performance. Whereas the addition of more SRAdata to the training dataset had little influence on LIN models (except for a slightincrease of ∼0.2% for G-Slarge-T), both MLP and DA model accuracies improvedsignificantly (by between 5% and 9%) upon addition of additional SRA data (We repeated the prediction for TCGA with the models trained for SRA tissue prediction(previous section), i.e., on SSRA data .small didslightly better than MLP Slarge-G (msa and mca of 0.943). MLPSlarge-T achieved msa and mca 0.971, outperforming LIN Slarge-T (msaand mca of 0.882). MLP Ssmall-S achieved msa 0.95 and mca 0.941, outperformingLIN Ssmall-S with msa 0.89 and mca of 0.884  cell lines, whereas GTEx and TCGA data areexclusively from biopsies. Starting from the hypothesis that fundamental differences doshow on an expression level, we set out to train LIN and MLP models on SRA data to predictthe sample source of SRA, GTEx, and TCGA. Of note, while we were able to approximatelyreproduce the original results for LIN Sorted in ). LIN Sl884 Fig. .large-T. This confirms our hypothesis that the homogeneity of theGTEx data did not allow the MLP G-T model to generalize to TCGA data, while the additionof SRA training data in MLP G+Slarge-T resulted in a model with significantlyimproved generalization. To further investigate this result, we took a closer look at theper class accuracy for the TCGA tissue prediction . We first tested the trivial case MLP G-G by splitting GTExinto training and test sets, achieving sample and class accuracy of 0.995 (data notshown).For sex classification, only genes on the X and Y chromosome were used as input features. Our best model, based on the data annotationprovided by MetaSRA, was MLP G+Slarge-T with msa 0.964 and mca 0.962 . This result is significantly better than what waspreviously reported , MLP Ssmall-S (msa: 0.994 and mca: 0.994), and MLPG+Ssmall-S (msa: 0.993 and mca: 0.992). Results are shown in All linear models for the prediction of sex for SRA data achieved high accuracy (msa:0.98 and mca: 0.98 for LIN G-S and LIN G+Sa: 0.863 ). The MLAccording to MetaSRA all our training and testing data for sex prediction on SRA stemfrom patient biopsies. However, ≥2 of the largest misclassified SRA studies in the testset are clearly cultured cell lines. For example, SRP056612 is a study on the effect ofthe MERS coronavirus on cultured kidney and lung cells  and SRP0Our experiments on phenotype classification seem to indicate that increased training datadiversity might enhance classification performance. To learn more about the relationshipbetween the amount of training data and model performance, MLP G-S was trained on anincreasingly large subset of the GTEx training data for tissue classification. We observeda limited effect on model performance with increased training dataset size. The msareaches its peak with one-third of the available training data, while the mca saturates atapproximately half of the available training data .small-S for samplesource classification was trained on an increasing number of biases in the training set.As a control experiment, an MLP was trained with the same amount of data but drawn from asingle-bias source. We observed a positive correlation between msa and the number ofbiases in the training set . In the specific case of sex classification, the MLP G+S was used to predict the truecorrected label for the SEX samples that were removed from training due to lowsex-chromosome counts (see Methods). For 82% of the 132 filtered samples, the MLP modelpredicted the opposite of the presumably wrong MetaSRA labels. However, our MLP model wasable to confirm the MetaSRA label for 24 samples. These samples had a mean chrY count sumof 2.4 . Manual confirmation revealed a high modelaccuracy. For example, SRR1164833, SRR1164787, and SRR1164842 are samples from a prostatecancer study labeled as male by MetaSRA. Our MLP model correctly classified these samplesdespite the fact that their chrY total sum count was between 0.4 and 1.4. On the otherhand, SRR16076 54/56/61/62/64/65/70/71 are annotated as female by MetaSRA and the MLP buthad a chrY total sum count of 2–5.3. We see the correct classification of these borderlinecases as further evidence that no overfitting is taking place.A list of all SRA samples for which the MetaSRA labels and the predicted labelsmismatched is available in the small-S) achieved msa 0.947 and mca0.93 on a test set (data not shown) and MLP Slarge was subsequently used toidentify all of our yet unannotated SRA samples of source type biopsy. At a probabilitycut-off of 0.5 we identified 1,072 new SRA samples as originating from a biopsy.We have used our best models to predict high-quality metadata for published SRA sampleslacking information on tissue, sex, or sample source. Prediction of sex is straightforwardbecause our models were trained on all possible biological categories. For tissue andsample source, however, our models were trained on a subset of all potential classes inthe unlabeled data. If, for example, we try to label a sample of a tissue type unknown bythe model, the model will force 1 of the learned classes onto that sample. To deal withthis in the best possible way for sample source classification, we modified theclassification task into one vs all. Specifically, we first trained a new MLP model toidentify the sample source biopsy vs all other sample sources available in the SRA data asdefined by MetaSRA. This model . Trainingand test datasets were subsequently combined to train DA G+Slarge forannotation prediction of unlabeled SRA samples. We predicted the tissue of origin for allSRA samples of source type biopsy for which no entry on MetaSRA was available .Second, we extended the tissue classification task to 17 classes by adding a “catch-all"class. To this end, we extended the training data to all GTEx  and SRA  data with tissue labels and assigned the placeholder class for every sample thatdid not belong to the original set of 16 tissues. That way, we ensure that the learnedmodel will not force known classes on every tissue type. With this approach, the DAG+Slarge. Third, 8,495 SRA biopsy samples with missing sex information were predicted using MLPG+SWe developed a novel deep-learning–based DA approach for automated bias-invariant metadataannotation. To our knowledge this is the first time that DA has been applied to thisproblem. We were able to outperform the current best model  on tissuThe main strength of our method is its ability to incorporate dataset bias from datasetswith only a few samples by applying a Siamese network-like architecture. The model learns toignore bias by repeated exposure to (a few) samples in (many) different contexts, i.e., astriplets. In addition, it does not rely on feature selection but uses normalized gene counttables and lets the network learn which features carry important information.small → MLPSlarge vs MLP Ssmall → MLP G-Ssmall). Second, for SRAtissue classification, we showed that there is a limit of accuracy that can be achievedirrespective of the size of the training set. Our experiment showed that peak accuracy isalready reached by using 50% of the available data. Last, for sample source classification,we directly compared the relationship between the number of biases in the training data, thenumber of samples, and the model performance. We found a positive correlation between thediversity of the training data and the accuracy achieved by that model.Different types of experiments showed the importance of training models on a multi-biasdataset. First, we showed for every phenotype classification that models that had SRAsamples included in the training data performed better than models trained only on GTExdata. For tissue classification, we further showed that the effect of adding SRA samples tothe training data outweighs adding 3.2 times as much GTEx data . At least for the TCGA tissue classification, it seems that a minimum of roughly8 classes is needed for the DA model to be able to unfold its full potential consistently.Our experiments indicate that the difference between DA and MLP performance will keepincreasing, in favor of the DA model, the more classes we add . Adding Whereas currently the scope of our predictive models has been limited by the availabilityof data , theapproach is ready to incorporate more data, biases, classes, and more phenotypes, and thereis reason to believe that this will confer increased performance of ANN-based models, inparticular DA models. At the same time, automated annotation ensures that the vast amount ofdata currently lying idle in online repositories and institutional data centers can indeedbe leveraged. We believe that this synergy is capable of producing a large and comprehensivebody of annotated biological data that will boost knowledge discovery for biomedicalresearch.Project name: Bias invariant RNA-seq metadata annotationhttps://github.com/imsb-uke/rna_augmentProject home page: Operating system: Platform independentProgramming language: PythonOther requirements: TensorFlow 2.1License: MITThe input data as described in Supplementary Figure S1:Visualizing dataset bias. GTEx is a single-study datasource, while SRA is a multi-study data source. (A) t-SNE plot of gene expression values ofGTEx and (B) SRA samples, belonging to 5 different tissues. The GTEx data are morecoherently clustered compared to the SRA data. The individual studies in the SRA data appearto form less homogeneous clusters, indicating a larger within-variance in the datasource.Supplementary Figure S2:t-SNE on fraction of total gene count per gene type.The fraction of the total log TPM normalized counts per gene type was calculated for alltypes that can be associated with messenger RNA or small RNA. t-SNE was applied on theresulting vectors of fraction per gene type. Samples with their maximum fraction in a genetype belonging to a small RNA category were labeled orange, else blue. The scatter plotshows that samples labeled as small RNA-seq all cluster together, suggesting a validapproach.Supplementary Figure S3:Tissue label overlap between GTEx, TCGA, and SRA. GTExv6 provides samples for 31 tissues and TCGA for 26. MetaSRA provided labels for 26 of the 31GTEx tissues. This figure depicts the 40 tissues that form the union between the 3 datasources, a black square indicating that a tissue is present in the respective dataset.Seventeen tissues are shared between GTEx, TCGA, and SRA, 16 of which were used for tissueprediction.Supplementary Figure S4:Misclassification in MetaSRA. Histogram of the totalsum of normalized counts mapped to the chrY for GTEx, TCGA, and SRA. Male and female clearlyoverlap in SRA, indicating mislabeling by MetaSRA.Supplementary Figure S5:Graphical representation of architectures for ANN-basedmodels. (A) MLP models for tissue, sex, and sample source. (B) The (1) SM-CL MLP, (2) SM-BMSiamese Network, and (3) BM-CL prediction models for tissue and (C) sex. Each rectanglerepresents a layer in the neural network and is colored according to the type of layer thathas been used. BM: bias mapper; CL: classification layer; d: input dimension; n: number ofnodes; p: dropout probability; SM: source mapper. B2 and C2 show the SM to have frozenweights.Supplementary Figure S6:Supplementary. Samples are indicated according to theirclasses  and their bias . The model is ready for prediction after 2 training steps: (A) Asource mapper is trained on single bias data together with a classification layer. (B) Abias mapper is created as a duplicate of the source mapper; the weights of the source mapperare fixed. Triplets are passed through the source mapper and bias mapper configuration tolearn a bias mapping. (C) The bias mapper, equipped with a classification layer, can be usedto predict data from previously unseen datasets.Supplementary Figure S7:Per class accuracy for TCGA tissue classification. Meansample accuracy for each tissue and all ANN-based models is shown. The error bar shows thestandard deviation across 10 random seeds. The plot demonstrates the varied tissueclassification performance of different tissues. For instance, it seems to be difficult toidentify adrenal gland or pancreas with any of the models. In particular, the badclassification performance of MLP G-T for bone marrow, ovary, and uterus is especiallynoticeable, along with the observation that performance can be salvaged by addition of(biased) SRA data to the training dataset. This highlights the strength of ANN-based modelsin capturing bias from training data.Supplementary Figure S8:Sex phenotype results. (A) SRA and (B) TCGA test data.DA: domain adaptation; G: GTEx; LIN: linear model; MLP: multilayer perceptron; S: SRA; T:TCGA. ANN-based models yielded consistently worse results than the baseline model, untilnewly annotated data were incorporated into the training set.Supplementary Figure S9:Dependence of prediction performance on increasingtraining dataset sizes for MLP G-S. MLP models were trained on subsets of the GTEx data forSRA tissue classification on 10 seeds and averaged. At each step, the subset was increasedby 250 samples. Box plots from 20 iterations for the msa and mca are shown in blue andgreen, respectively. Mean sample accuracy reaches its peak with only 25% of the trainingdata, while 50% of the data is sufficient for the mean class accuracy to saturate.Supplementary Figure S10:Test of overfitting. An MLP model was trained on GTExdata. An increasing fraction of 1 class was assigned a wrong class label . The model was trained on the partially mislabeled data and the mislabeled data werepredicted by the model after training. We quantify the model’s susceptibility to overfittingby letting it correct the mislabeled training data. The MLP model was able to correct allmislabeled data up to a mislabeling fraction of 20%. We conclude that the ANN models arevery robust in dealing with mislabeled data.Supplementary Figure S11:True-positive rate for test data predicted withannotation models. (A) Sample source, (B) sex, and (C) tissue classification.Supplementary Figure S12:Relationship between number of classes and DAperformance in DA G+S-T. The 16 tissues were sorted by sample size in GTEx, and at each step1 tissue was added to the classification problem, starting with the largest 2. MLP and DAwere trained as described above for 10 seeds each and tested on TCGA data. The mean sampleaccuracy for each seed (top panel) or mean class accuracy (bottom panel) is shown. Each dotshows the difference in accuracy (DA-MLP) at each step for each seed. Seaborn regplot wasused to fit a regression line. While, on average, MLP performs better for lower number ofclasses, the performance gain by the DA model with respect to MLP increases with the numberof classes.Supplementary Table S1: Mapping from GTEx tissue names to MetaSRA tissuenames.SupplementaryTable S2: Summary of the datasets used for each phenotype afterpre-processing.Supplementary Table S3: Number of samples per class for phenotypeclassification experiments.Supplementary Table S4: Hyperparameters considered during model tuning andtheir initial range.Supplementary Table S5: Summary of the hyperparameters used for each model.SupplementaryTable S6: Sample and class accuracy given are the mean over n = 10seedsANN: artificial neural network; BM: bias mapper; CL: classification layer; DA: domainadaptation; GTEx: Genotype-Tissue Expression Project; GUI: graphical user interface; ML:machine learning; MLP: multi-layer perceptron; mca: mean class accuracy; msa: mean sampleaccuracy; RNA-seq: RNA-sequencing; SM: source mapper; SRA: Sequence Read Archive; TCGA: TheCancer Genome Atlas; TPM: transcripts per million; t-SNE: t-distributed stochastic neighborembedding.This work was supported by the Deutsche Forschungsgemeinschaft (DFG) grants CRU 306 P-C,296 P8, and CRC 1286 Z2 for S.H., H.W., and S.B., respectively.The authors declare that they have no competing interests.Authors' ContributionsSB and HW designed the project. HW designed and implemented the models and conducted theanalyses. SH implemented the LIN model and conducted the corresponding analyses. SB, HW, andKK wrote the manuscript.giab064_GIGA-D-20-00367_Original_SubmissionClick here for additional data file.giab064_GIGA-D-20-00367_Revision_1Click here for additional data file.giab064_GIGA-D-20-00367_Revision_2Click here for additional data file.giab064_GIGA-D-20-00367_Revision_3Click here for additional data file.giab064_Response_to_Reviewer_Comments_Original_SubmissionClick here for additional data file.giab064_Response_to_Reviewer_Comments_Revision_1Click here for additional data file.giab064_Response_to_Reviewer_Comments_Revision_2Click here for additional data file.giab064_Reviewer_1_Report_Original_SubmissionColin Dewey -- 3/15/2021 ReviewedClick here for additional data file.giab064_Reviewer_1_Report_Revision_1Colin Dewey -- 7/8/2021 ReviewedClick here for additional data file.giab064_Reviewer_2_Report_Original_SubmissionMaren BÃ%ttner -- 4/23/2021 ReviewedClick here for additional data file.giab064_Reviewer_2_Report_Revision_1Maren BÃ%ttner -- 6/20/2021 ReviewedClick here for additional data file.giab064_Supplemental_FilesClick here for additional data file.