We introduce the Transcriptome State Perturbation Generator (TSPG) as a novel deep-learning method to identify changes in genomic expression that occur between tissue states using generative adversarial networks. TSPG learns the transcriptome perturbations from RNA-sequencing data required to shift from a source to a target class. We apply TSPG as an effective method of detecting biologically relevant alternate expression patterns between normal and tumor human tissue samples. We demonstrate that the application of TSPG to expression data obtained from a biopsy sample of a patient's kidney cancer can identify patient-specific differentially expressed genes between their individual tumor sample and a target class of healthy kidney gene expression. By utilizing TSPG in a precision medicine application in which the patient sample is not replicated  application•We apply TSPG to The Cancer Genome Atlas data to perturb gene expression states•n = 1) gene expression tumor alterationsTSPG was used to learn patient-specific . RNA-sequencing data derived from a single patient's tumor were analyzed using this tool to determine gene expression aberrations specific to that patient's tumor. As medicine shifts from cohort-based population studies to individual-based precision treatments, our example demonstrates that deep learning is a powerful ally in the quest to understand how complex biological systems have shifted for a single patient. Cells express genes in unique combinations that enable distinct functions. Using machine learning, we present an algorithm that takes a source gene expression snapshot and learns how to change it to mimic a target snapshot. We applied the Transcriptome State Perturbation Generator algorithm to learn which genes have changed in a single patient's tumor relative to a normal tissue sample. By knowing which gene expression changes are required to leave a normal state in a single person, it is possible to design therapeutic strategies tailored for that patient. Nearly 4 years prior to the completion of the Human Genome Project,5,,Seeking to provide context to these genomic interactions, there have been several initiatives in recent years to characterize the genomes of different types of human cancers. The National Institutes of Health pioneered the creation of The Cancer Genome Atlas (TCGA) in 2005 in an effort to provide the molecular and physical map of cancer aberrations to improve our ability to diagnose, treat, and prevent cancer.Precision medicine promises to change the course of cancer care by allowing doctors to select treatments that are most likely to help patients based on a genetic understanding of their disease.One of the most challenging barriers in analyzing genomic data is the remarkably low signal-to-noise ratio (SNR). Tumors each harbor a combination of cancer-causing, or driver, mutations and largely irrelevant passenger mutations that do not contribute to the oncogenic potential of the cells. Distinguishing one from the other is critically important, but it is nearly impossible to accomplish when looking at a single patient sample in isolation. A precision medicine approach to cancer genomics is therefore plagued with the fact that a single patient sample is not replicated  provides a useful insight into the gene expression state of a biological sample by measuring tens of thousands of unique RNA transcripts.17,,27Deep learning,x, classified as a horse by a neural network, can be subtly perturbed into While deep-learning models are being successfully applied in a variety of fields, surprising vulnerabilities exist within trained models. Adversarial samples  can be designed to purposefully cause the deep-learning model to incorrectly classify the sample.Within the field of molecular biology a limited amount of work involving generative models has been reported. Ghahramani et al.In this report, we contribute to this emerging field by adapting the AdvGAN framework to model the transition between source and target tissue transcriptome states. Furthermore, we demonstrate how a single patient sample can be perturbed to resemble the expression state of a larger cohort by leveraging TCGA data as the training set for the model. By using TCGA data to train our network, we show that a model can be trained to recognize biologically relevant patterns in RNA-seq data and can effectively perturb a patient's tumor RNA-seq sample to resemble the expression patterns of normal tissue. Those perturbations can then be analyzed to determine significant transcriptional changes that occur during tumor progression, thus solving the f Accuracy, represents the target model accuracy when the input is perturbed from any input source  to the labeled target class. Thus the accuracy represents the ability of the generator to “trick” the target model into classifying a sample as a specified class.We examined the raw adversarial generation capabilities using 50 Hallmark gene sets defined in the Broad Institute Molecular Signatures Database (MSigDB).The high classification accuracy suggested that the generator was able to exploit the decision boundary between any two classes in the target model. While this was in itself interesting, we were led to three new questions concerning the nature of these perturbations: (1) Do the perturbed samples follow the distribution of the target class? (2) Are these perturbations nonsensically changing expression values to satisfy the criterion of tricking a target model? (3) Do the most highly perturbed genes encode biological function of the target condition?To test whether the adversarial samples exhibited high-dimensional structure similar to that of the original target samples, we used the dimensionality reduction and visualization tool t-SNE (t-distributed stochastic neighbor embedding).x is a heatmap of gene expression values from an original sample from the GTEx dataset of class type Brain-Hippocampus. x was passed through the generator to produce P, which was then added to x to create adversarial sample P ranges between t by a pretrained model (the target model). Perturbation values P describing the exact gene expression changes remained, on a per-gene basis, which occurred in order to transform Brain-Hippocampus tissue to Nerve-Tibial tissue. Additionally we show The right side of up in this case, indicating an activation of the gene during the transition toward Nerve-Tibial.To clarify all applications of TSPG when perturbing toward varying conditions, we adopted a naming convention of Source→Target—Direction. For example, Whole-Blood→Nerve-Tibial—UP would be used to reference a perturbation from a Whole-Blood sample toward the class of Nerve-Tibial in the positive direction. The gene would be turned The classification accuracy, t-SNE visualizations, and perturbation heatmap results suggested that the perturbation of the Hallmark gene expression patterns successfully shifted the state of source tissue to target tissue. To test whether the perturbed genes were relevant to the source or target tissue, we randomly selected a sample from each of the 53 tissues available in the GTEx dataset and passed it through the generator that was trained with Heart-Left Ventricle as the target class. We then took the 20 most positively perturbed genes (Random→Heart-Left-Ventricle—UP) and the 20 most negatively perturbed genes (Random→Heart-Left-Ventricle—DOWN) for each of the 53 state transitions. Interestingly, a total of only 85 unique genes were identified in the top 20 across all 53 GTEx experiments out of the total 4,386 genes being positively perturbed. In other words, a distinct group of 85 genes were consistently perturbed in the positive direction moving toward the target tissue. This suggests that the same genes tend to be perturbed toward the target tissue, irrespective of the source tissue. On the other hand, the 20 most negatively perturbed genes across all target tissues combined to form 642 unique genes, suggesting that a greater variety of genes must be “turned off” depending on the phenotype of the source tissue.We then tested whether the 20 most positively perturbed genes out of the pool of 4,386 genes were enriched  into one of three renal tumor subtypes: clear cell renal carcinoma (KIRC), papillary renal cell carcinoma (KIRP), and chromophobe renal cell carcinoma (KICH) as shown in While these findings confirmed the efficacy of TSPG, we wanted to make this tool relevant for individual patients. The goal became to apply this deep-learning model to perturb a single sample of patient tumor RNA-seq data toward the normal phenotype of corresponding tissue, thereby determining patient-specific genomic aberrations from a single data sample. A patient (designated BP) was diagnosed with RCC in 2014 when a large mass was discovered in his left kidney . A monthBP's RNA-seq data was processed using the same method as for the TCGA and GTEx samples. The two processed vectors of BP's tumor RNA-seq sample and healthy tissue were then appended to a larger gene expression matrix (GEM) that included TCGA samples for KIRC, KIRP, and KICH, as well as the samples for healthy “solid tissue normal” TCGA and GTEx samples. The result was a normalized, comprehensive GEM consisting of three unique kidney cancer subtypes, healthy tissue data derived from two different sources, and patient-specific tumor/normal samples. An overview of the samples included in this comprehensive GEM is listed in To ensure that this comprehensive GEM had been properly preprocessed so that BP's samples clustered with their histologically defined tissue subtypes, we again used t-SNE visualization to confirm that the unique vectors of expression data grouped with their respective tissue groups. GTEx was grouped with all other TCGA normal samples and given the common label of “Normal,” and was then compared against the other tumor subtypes. Following validation of the alignment of the comprehensive GEM, we used TSPG to perturb the expression values of BP's tumor sample toward a normal target class. Defining our perturbations in this direction allowed us to determine tumor-specific transcriptional aberrations from a single sample . It is important to note that the experimental design for these patient-specific perturbations is reversed from the original kidney experiments reflected in Two particular genes of note, MET and ERRFI1, are of known relevance to RCC.Functional enrichment was then performed on these two gene lists to search for biological functions relevant to the most highly perturbed genes in each direction /. Interestingly, although the general ratio of overlapping genes to total unique genes was similar between tumor-upregulated and tumor-downregulated genes, TSPG consistently isolated a wider variation between patients in different tumor-upregulated genes than it did tumor-downregulated genes, resulting in a larger set of total genes for the tumor-upregulated direction. The exception to this case is TCGA sample TCGA-BQ-7051, which can be explained by the fact that this is the only case in which TSPG generated more perturbations in the tumor-upregulated direction than in the tumor-downregulated direction  analysis on the TCGA KIRP dataset compared with the GTEx normal kidney data and TCGA normal correlating tissue. DESeq2 was used perform the DGE analysis, which uses the negative binomial probability distribution to determine a reference by which to compare differential expression between classes.We then compared the genes that TSPG had most significantly perturbed from all five TCGA patients, as well as BP's, KIRP expression vectors with those DGE identified to be statistically significant to determine the overlap between the two different algorithm’ sets of results as depicted by The high correlation between the established method of DGE and the novel deep-learning model of TSPG in the tumor-downregulated direction further supports the efficacy of TSPG as a proposed alternative method of determining differential expression when given only one sample. Furthermore, it implies that the genes which TSPG independently identifies as being downregulated in tumors relative to normal tissue (tumor-downregulated) for each individual patient's tumor are in large part the same genes that are downregulated for the statistical majority of KIRP tumors as identified by DGE. In stark contrast, there is significantly less agreement between the genes that TSPG perturbed in the tumor-upregulated direction for each patient and those genes identified by DGE in the correlating direction using the binomial reference distribution. When considered in conjunction with the results from comparing the TSPG results from one patient to another, this further supports the conclusion that the genes perturbed in the tumor-upregulated direction are highly unique to each patient's tumor. While the genes perturbed in the tumor-downregulated direction largely match statistical averages of other KIRP tumors, the genes perturbed in the tumor-upregulated direction exhibit inherent heterogeneity both when compared directly with other tumors and when compared with statistical averages of all related tumors.We adapted and added criteria to the AdvGAN framework, which originated from within the computer vision community, to produce meaningful biological state transitions. While we used rization  to limition  showed perturbations greater than one standard deviation from the mean degree of perturbation. While this is a notably lower threshold than was considered for the majority of this study, it is worth noting that while the perturbations were less extreme, TSPG consistently identified a gene of known relevance to p1RCC to be overexpressed in all tumors evaluated. This further validation is supportive of the fact that MET plays a critical role in the progression of p1RCC and that TSPG is capable of highlighting even subtle transcriptional changes.The seminal paper by Linehan et al. on the genomics of papillary RCCERRFI1, a negative regulator of EGFR, was also observed in 11.1% of papillary RCCs.ERRFI1 gene. This would indicate that expression of the ERRFI1 gene is significantly lower in papillary RCC when compared with healthy kidney tissue, consistent with a deletion or silencing of the gene. Although these are merely observational correlations, it is notable that some of the most critical genetic alterations of papillary RCC identified by Linehan et al.Deletion of the chromosomal region including tumor suppressor Functional enrichment results for BP's 418 significantly perturbed tumor-downregulated genes show consistent enrichment for many functions that are relevant to kidney function and development. Enrichment results for functions such as urination, circulating renin, kidney coexpression, and a diseased state of nephrocalcinosis, as shown in To understand the transcriptional changes that occurred during progression of BP's p1RCC, we can analyze the 444 genes that TSPG identified as being significantly perturbed in the tumor-upregulated direction. Interestingly, consistent significant enrichment for keratinization function was observed see . KeratinWith this initial validation of the potential clinical relevance of the genes that TSPG can identify, future studies should aim to determine regulatory pathways that affect these genes, ultimately aiming to identify therapeutics that specifically target those pathways. Furthermore, future aims involve collaborating with wet lab biologists to perform immunohistochemistry and western blotting on BP's preserved tumor tissue sample to confirm that the identified candidate genes are in fact being overexpressed in the cells. This study provides a computational tool for analyzing gene expression data that can predict biologically significant genetic aberrations. Further research should focus on determining biological and clinical validity of these predictions to establish a bench-to-bedside workflow. This is the ultimate step in bringing TSPG to a clinically applicable setting where precision medicine can be applied based on this deeper understanding of transcriptional activity of cancer cells.F. Alex Feltus is the lead contact for this study.There are no physical materials associated with this study.ahttps://github.com/ctargon/TSPGThe source code for TSPG along with all documentation is available at the following Github repository: bFigshare.https://doi.org/10.6084/m9.figshare.5330593The normalized gene expression data (FPKM) from TCGA and GTEx data are available on chttps://github.com/mrbende/RNAprepThe exact parameters used in the workflow to process patient-derived FASTQ data to be compared with public sources can be found at the following Github repository: dFigshare.https://doi.org/10.6084/m9.figshare.5330539Maximum likelihood gene expression counts from TCGA and GTEx data are available on 2 transformation and quantile normalization on the GEM in Python using scikit-learn.We used two GEMs containing RNA-seq expression levels for human tissue and tumor samples: (1) The GTEx dataset ;G, a discriminator D, and a target model f. The generator G takes an n-dimensional sample x, where n is the number of genes, and generates a perturbation D outputs a prediction of whether the generated sample came from the training data or the generator  activation, and the output layer uses the hyperbolic tangent (tanh). The tanh function allows the generator to produce a perturbation ranging from 2 fold change. The discriminator consists of three dense layers of sizes 512, 256, and 128. Each dense layer is followed by a batch normalization layer and a leaky ReLU activation function with alpha set to 0.2. The target model used in the experiments contains three dense layers of size 1,024, 512, and 128, each layer using the ReLU activation function.The architectures used by a.All three networks were trained using the Adam optimizerThe TSPG workflow consists of three steps:The target model is trained on the training data, which in our case is the combined GTEx and TCGA GEM or a subset thereof. This is the model which assigns a class label to a sample of transcription data.The generator is trained (using the same training data as the target model) to generate perturbations toward a given target class. The perturbed samples should be classified as the target class by the target model and should be indistinguishable from the training data according to the discriminator.The generator is used to perturb each sample in the test data to “appear like” the target class. The perturbed test samples are considered valid if they are classified as the target class by the target model. These perturbations can then be visualized or used for other downstream analyses such as functional enrichment.To validate TSPG, we trained the target model and the generator on the combined GTEx and TCGA data, then we perturbed 100 randomly selected samples from these training data along with the samples from 10 randomly selected classes including the target class. t-SNE plots were used to visualize both original and perturbed samples. Additionally, we used the mean vector of a target class to visually inspect how similar an adversarial sample is to the target class.b. The following files were pulled for analysis:1.kich-rsem-fpkm-tcga.txt2.kich-rsem-fpkm-tcga-t.txt3.kirc-rsem-fpkm-tcga.txt4.kirc-rsem-fpkm-tcga-t.txt5.kirp-rsem-fpkm-tcga.txt6.kirp-rsem-fpkm-tcga-t.txt7.kidney-rsem-fpkm-gtex.txtThe process for unifying TCGA tumor data with GTEx tissue data was documented by Wang et al.,By pulling only relevant kidney data, the size of the dataset was minimized while still providing several subtypes of kidney tissues for analysis to avoid overfitting the TSPG model. These datasets were then merged along gene IDs.c.Illumina RNA-seq was performed on both a biopsy sample of BP's p1RCC and healthy kidney tissue, resulting in two single-insert paired-end RNA-seq datasets in FASTQ format. The normal sample was sequenced with 150M requested reads and the p1RCC sample was sequenced with 300M reads. These FASTQ samples were processed according to protocols used for processing this GTEx and TCGA data.https://toppgene.cchmc.org)To check for biological relevance, we used the ToppFun tool . This training set contained 158 normal samples, 60 KICH samples, 475 KIRC samples, and 236 KIRP samples. The generator was trained to perturb the input to appear normal so that we could observe the transition from the patient's tumor sample to normal. After the target model and the generator were trained, we generated perturbations for both the patient-normal sample and the patient-tumor sample. A naive comparison was also performed by taking the difference between these two samples, which served as a baseline against which to compare the actual results of TSPG. By making these perturbations based on the 158 available normal samples rather than just the one patient-normal sample, TSPG was able to circumvent the high uncertainty associated with an d. DGE analysis was performed using the “kirp-rsem-count-tcga-t.txt” dataset as the source group, while “kirp-rsem-count-tcga.txt” and “kidney-rsem-count-gtex.txt” were used together as the target group. This was intended to replicate TSPG's function of measuring transitions between an aberrant source group toward a normal target group. t-SNE visualization was again used to confirm proper clustering of source and target groups. To analyze the results of DGE, we isolated statistical outliers that represented the most highly differentially expressed genes (both up- and downregulated) (2 fold change.DGE analysis was performed using the Bioconductor package DESeq2.


Cancer stem cells play crucial roles in the development of colon cancer (COAD). This study tried to explore new markers for predicting the prognosis of colon cancer based on stem cell-related genes. In our study, 424 COAD samples from TCGA were divided into three subtypes based on 412 stem cell-related genes; there were significant differences in prognosis, clinical characteristics, and immune scores between these subtypes. 694 genes were screened between subgroups. Subsequently a six-gene signature  was established. This model had strong robustness and stable predictive performance in cohorts of different platforms. Taken together, the six-gene signature constructed in this study could be used as a novel prognostic marker for COAD patients. Colon cancer is one of the most common malignant tumors in the world, with approximately 1.2 million new cases and 600,000 deaths each year . TherefoIn recent years, research has shown that there is a small group of special cells inside tumors. These cells have strong self-renewal and tumorigenic abilities and stem cell-like characteristics. Such cells are called cancer stem cells (CSCs). CSCs may be involved in the development, recurrence, metastasis, and chemotherapy resistance of tumors and promote tumor progression , 5. CSCsetc., and they are usually used to identify and isolate CSCs (CUL4B) upregulates the expression of the CSC marker CD44 to maintain colon cancer stemness and drive malignant progression to affect prognosis  in both training and validation cohorts, and it was independent of clinical characteristics. Therefore, it was recommended to use this classifier as a molecular diagnostic test to assess the prognostic risk of patients with colon cancer.(The human CSC-related pathways were downloaded from the Molecular Signature Database v7.0 (MSigDB), and 456 genes related to CSCs were sorted from the 30 lipid metabolism pathways in The Cancer Genome Atlas (TCGA) Genomic Data Commons (GDC) application performing interface (API) was used to download the latest expression data and clinical follow-up information of patients with colon adenocarcinoma (COAD). This cohort contained RNA sequencing data and clinical follow-up information of 424 samples S2_TableThe GSE39582 chip expression data in MINiML format were downloaded from the National Center for Biotechnology Information (NCBI). GSE39582 contained 536 samples with clinical characteristics S3_TableThe following steps were performed to preprocess the RNA sequencing data from the TCGA samples. 1) Remove the samples without clinical information or with progression free survival (PFS) <30 days. 2) Remove the data of the normal tissue samples. 3) Remove any gene whose fragments per kilobase million (FPKM) is 0 in half the samples. 4) Retain the CSC-related gene expression profiles.The following steps were conducted to preprocess the GSE39582 and GSE17536 cohorts. 1) Remove the data of the normal tissue samples. 2) Convert the disease-free survival (DFS) data by year or month into by day. 3) Remove the samples with DFS <30 days. 4) Use the “Bioconductor” package in R to map the chip probe to the human gene SYMBOL. 5) Retain the CSC-related gene expression profiles.We selected primary cancer samples, all of which were from before the first treatment, and we excluded samples whose follow-up time was <30 days. In addition, we randomly divided the TCGA cohort into 2 groups, one as the training set and the other as the internal validation set. Two Gene Expression Omnibus (GEO) cohorts were used as independent external verification sets. Since we wanted to verify the predictive performance of the model in different platforms, we did not specially process the GEO expression profile datasets.The statistical information of the preprocessed cohorts is shown in First, the expression profile data of the CSC-related genes were extracted from the TCGA database. Genes with expression >0 in more than half the samples were retained. A total of 412 genes were included in subsequent analysis. COAD samples were clustered by non-negative matrix factorization (NMF) with the standard of “brunet” and 50 iterations. The number of clusters, k, was 2 to 10. The average contour width of the common member matrix was determined through “NMF” in the R package, and the minimum number of member of each subclass was 10. The optimal number of clusters was determined based on cophenetic, dispersion, silhouette, and so on.DESeq2 was used to calculate the differentially expressed genes (DEGs) between the subtypes. With a false discovery rate (FDR) <0.05 and |log2FC| >1 as the threshold, functional enrichment analysis of the DEGs was conducted using the R software package “WebGestalt” to explore the pathways and functions involved with these DEGs.First, 75% of the 424 TCGA samples after preprocessing were randomly selected as the training cohort. To avoid random allocation bias affecting the stability of subsequent modeling, 100 repeated samplings with replacements for all samples were performed in advance to ensure that the distribution of the randomly selected samples was consistent with all samples in terms of age, clinical stage, and tumor-node-metastasis (TNM) stage. For all DEGs and survival data, univariate Cox proportional hazards regression analysis was performed. We then used the R package survival “coxph” function, with log rank p <0.01 as the threshold. It was necessary to further reduce the gene range and construct a prognostic model while maintaining a high accuracy rate, and the least absolute shrinkage and selection operator (Lasso) method was used as a compression estimate. It constructed a penalty function to obtain a more refined model so that it compressed some coefficients, and at the same time it set some coefficients to 0. Therefore, the advantage of subset shrinkage was preserved. It was a kind of processing biased estimation with complex collinearity data, which could realize variable selection at the same time as parameter estimation and better solve the multicollinearity problem in regression analysis.In the clinical information of the TCGA data, the related hazard ratios (HRs), 95% confidence intervals (CIs) of the HRs, and p-values were analyzed using univariate and multivariate Cox regression. Age, sex, pathologic T, N, and M stage, and tumor stage information was systematically analyzed to evaluate the clinical independence and predictive performance of our model.To observe the relationships between the risk scores of the different samples and biological functions, the gene expression profiles corresponding to these samples were selected for single-sample gene set enrichment analysis (ssGSEA) using the R software package “GSVA,” and the score of each sample for different functions was calculated to obtain the correspondence of each function. The ssGSEA score of each sample was used to further calculate the correlations between functions and risk scores.Sixty cases, with matched non-tumorous and tumorous tissue samples from Shengjing Hospital of China Medical University, were enrolled in this study. Total RNA was extracted using TRIzol reagent  following the manufacturer’s instructions. Complementary DNA (cDNA) was synthesized from high quality total RNA using PrimeScript™ RT Master Mix . Real-time qPCR was performed to validate gene expression using Power SYBR™ Green PCR Master Mix  on the 7900HT Fast Real-Time PCR System . Relative expression was calculated based on 2-ΔΔCt method.According to the cophenetic, dispersion, and silhouette indicators, the optimal number of clusters was determined to be three  among the three subtypes were compared. Age, M stage showed significant differences in the three groups  could significantly classify the TCGA training cohort into high- and low-risk groups (p <0.05), while DYDC2 and MS4A15 could not significantly do this. The formula of the final six-gene signature was as follows: RiskScore6 = 0.4448*expDYDC2 + 0.21*expMS4A15 + 0.2098*expMAGEA1 + 0.5068*expWNT7A + 0.1363*expAPOD + 0.1545*expSERPINE1.First, 75% of the 424 TCGA samples after preprocessing were randomly selected as the training cohort to construct the model. The sample information of the final training cohort is shown in DYDC2, MS4A15, MAGEA1, WNT7A, APOD, and SERPINE1) was associated with high risk, so these genes were considered risk factors. Receiver operating characteristic (ROC) analysis of the prognostic classification of the risk score was performed using the R software package “timeROC.” The prognostic classification efficiency at 1-year, 3-years, and 5-years was analyzed  was associated with high risk, so these genes were considered risk factors. ROC analysis of the prognostic classification of the risk score was performed using the R software package “timeROC.” The prognostic classification efficiency at 1-year, 3-years, and 5-years was analyzed .Survival analysis showed that clinical stage, T stage, N stage, M stage, lymphatic invasion, and venous invasion in the TCGA training cohort samples were significantly correlated with PFS in COAD  and an 11-gene signature (Kim)—for comparison with our six-gene model. To make the models comparable, the same method was used to calculate the risk scores of each COAD sample in the TCGA cohort to evaluate the ROC of each model, and divide the sample into Risk-H and Risk-L groups according to the median risk score to calculate the prognostic differences between the two groups of samples. The ROC and KM curves of the two models are shown in Based on the bioinformatics analysis results, the expression of six genes were verified in 60 colon cancer tissues and paired normal tissues. The results in To make this article easier to understand, we drew a flowchart . MS4A family genes are abnormally expressed in a variety of solid tumors  is a member of the melanoma-associated antigens family A (MAGE-A) and is closely related to the prognoses of various malignant tumors, such as esophageal squamous cell carcinoma ; the NCBI Gene Expression Omnibus .Publicly available datasets were analyzed in this study, these can be found in The Cancer Genome Atlas (The studies involving human participants were reviewed and approved by the Research Ethics Committee of China Medical University and all patients signed informed consent forms to allow analyses to be performed on their tissue samples.QS and XW contributed conception and design of the study. YL performed the experiment and statistical analysis. All authors contributed to the article and approved the submitted version.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


In this study, we aimed to identify an immune-related signature for predicting prognosis in cutaneous melanoma (CM). Sample data from The Cancer Genome Atlas  were used to develop a prognostic signature with 23 immune-related gene pairs (23 IRGPs) for CM. Patients were divided into high- and low-risk groups using the TCGA and validation datasets GSE65904 (n = 214), GSE59455 (n = 141), and GSE22153 (n = 79). The ability of the 23-IRGP signature to predict CM was precise, with the stratified high-risk groups showing a poor prognosis, and it had a significant predictive power when used for immune microenvironment and biological analyses. We subsequently established a novel promising prognostic model in CM to determine the association between the immune microenvironment and CM patient results. This approach may be used to discover signatures in other diseases while avoiding the technical biases associated with other platforms. The incidence of cutaneous melanoma (CM) is increasing more rapidly than any other cancer, and accounts for about 132,000 new cases and 65,000 deaths worldwide annually  data retrieved from The Cancer Genome Atlas (TCGA)  and the https://portal.gdc.cancer.gov) and randomly assigned to a training dataset (n = 230) or a testing dataset (n = 230). In addition, the GSE65904 (n = 214), GSE59455 (n = 141), and GSE22153 (n = 79) datasets were downloaded from the GEO database (http://www.ncbi.nlm.nih.gov/geo/) for validation of the IRGP signature. In total, 905 samples were available for analysis. A file containing 1534 IRGs that was downloaded from the ImmPort database (https://immport.niaid.nih.gov) and the CIBERSORT analytical tool (https://cibersort.stanford.edu/) were used to determine an estimation of the abundances of 22 distinct cell types in a mixed cell population based on gene expression data. Immunohistochemical images of CM patients were downloaded from The Human Protein Atlas dataset (http://www.proteinatlas.org/). All data was available for free from website.In this retrospective study, four independent RNA-seq datasets and clinical data from diverse, high-throughput sequencing platforms were comprehensively analyzed. A CM dataset (n = 460) was downloaded from the TCGA data portal (All data were preprocessed using R software (version 3.6.2). If more than one probe was matched to the same target gene, the average value of the probe was calculated to replace the expression level of the single gene. If there were multiple samples from the same patient, the average value of each gene was used as the expression level of that gene.The TCGA CM dataset was randomly divided into a training group and a testing group by R package “caret” with the ratio of training group samples to test group samples is 0.5. The distribution of CM patients gender (p = 0.068), age (p = 0.047), clinical stage (p = 0.036), follow-up time (p < 0.0001), death rate (p < 0.0001) and the number of CM samples in the dichotomies was similar between the two groups , 18. TheThe TCGA testing dataset and three microarray data files were selected to validate the signature comprised of 23 IRGPs. Every dataset was stratified into high- and low-risk groups based on the cut-off value of the prognostic signature. Next, the log-rank test and Cox analysis were performed, and a graph of OS was created to calculate the C-index between the high- and low-risk groups in each dataset. Finally, the signature of the 23 IRGPs was compared to the 1-, 3-, and 5-year area under the ROC curves (AUCs) and the C-indices.The CIBERSORT analytical tool  was usedhttps://www.gsea-msigdb.org/gsea/datasets.jsp) (Gene ontology (GO) and Kyoto Encyclopedia of Genes and Genomes (KEGG) pathways analyses of the two risk groups were performed using the R bioconductor package “fgsea.” GO analysis and KEGG pathways files  were downloaded from the Gene Set Enrichment Analysis (GSEA) website (ets.jsp) . Gene sewww.r-project.org) and Prism 8 . Training group and testing group were randomly divided by R package “caret”. OS curves were plotted using the R packages “survival” and “survminer.” A heat map of the IRGPs, risk score map, and OS status graph were created using the R package “pheatmap.” A model of prognostic IRGPs was established using the R package “glmnet”  and R package “RCircos” and “GenVisR”. We found that the genes expressed in TCGA tumor samples were independent parts, compared to TCGA normal sample (only one sample), GTEx normal skin exposed samples or not to the sun  or an old dataset (age > 65 years), where OS was expected to be better for the younger patients , GSE59455 (n = 141), and GSE54467 (n = 79) were applied for external validation. The risk score of each patient was calculated using the same 23-IRGP prognostic signature. Then, based on the median risk score, the patients were assigned to the low- or high-risk group. Interestingly, OS was poorer in the high-risk group Table 2 https://www.cbioportal.org/) was used to determine the mutation rates of the different genes. According to the results, the probability of mutation for PD-1, PD-L1, CTLA-4, PSMB8, and PSMB9 were 5%, 1.9%, 1.6%, 5%, and 4% . A bubble graph of the 16 GO terms enriched in the high-risk group with a padj value < 0.05 is presented in TCGA-CM dataset includes primary and metastatic samples, the primary samples submitted to sequence were initially diagnosed melanoma samples; however, the metastatic samples for sequencing were always from follow-up patients instead of initially diagnosed samples. So the TCGA-CM dataset did not always adopt the initially diagnosed melanoma samples for sequencing. So we established another prognostic model in CM determine the effectiveness of this approach according to the observed survival interval (OBS), which could be defined as the time interval from TCGA sampling to patient death or last follow-up . As is sThe 23-IRGP prognostic signature was also compared with the prognostic signatures of individual IRGs. First, as the TCGA CM data had only one normal sample, the samples from the Genotype-Tissue Expression dataset  and TCGACM is a solid malignant tumor with strong immunogenicity with a rapidly increasing incidence rate worldwide. Since the approval of interferon-α for the treatment of CM in 1995, the potential of other immunotherapies have received much attention from researchers . Like wiDue to the technical bias of sequencing platforms, gene expression data must be preprocessed for standardization, which is particularly significant when establishing prognostic signatures. To achieve robust prognosis prediction without the technical bias associated with different platforms, prognostic signatures of IRGPs can be established by pairwise comparison, which does not require preprocessing for data standardization. IRGP scores are calculated based on the expression levels of IRGs in the same sample. As such, the prognostic signature is not only able to overcome the batch effects of different platforms, but also does not require the scaling and normalization of data. In CM prognostic model, based on the AUC and C-index values, the prediction capability of IRGPs is more promising when compared with prognostic checkpoint (AUC = 0.729) , prognosGiven that the results of TCGA-CM dataset is made up of primary and metastatic samples, and the metastatic samples for sequencing were always from follow-up patients instead of initially diagnosed samples, IRGP-OBS prognostic model were established to make a predict comparison with 23-IRGP. As we expected, both IRGP-OBS model and 23-IRGP model had precious prediction capability with high AUC and C-index values. In addition, many researchers of CM also regarded OS as golden standard to evaluate the model predict power , 46, andvia endothelin receptor type A  that were highly related to the invasiveness of melanoma, suggesting that a high-risk score was correlated with increased melanoma metastasis and poorer survival. These results indicated the capability of the IRGP signature for predicting tumor invasion in CM patients.Nevertheless, there were some limitations to this study that should be addressed. First, the 23-IRGP prognostic signature was based on a retrospective study using the TCGA CM dataset and validated using three microarray datasets from the GEO dataset. Thus, these results should be validated against other datasets with different sample attributes in a prospective cohort. Second, as the 23 IRGPs were used to construct a prognostic signature model, different prognostic signature models are needed for comparison. Third, further validation of the 23 IRGPs by quantitative real-time polymerase chain reaction, western blotting, and immunohistochemical analyses will be needed before this approach can be applied clinically. Fourth, due to the fact that the relevant CM data  cannot be obtained, the analysis of cross-talk between the signature and ICIs cannot be compensated systematically at present.In conclusion, our findings indicate that our prognostic signature established using 23 IRGPs is a novel, promising model for predicting the prognosis of CM, indicating an association between the immune microenvironment and CM. This approach can be used to discover signatures in other diseases without the technical bias associated with different platforms.https://www.cancer.gov/about-nci/organization/ccg/research/structural-genomics/tcga); GEO dataset (https://www.ncbi.nlm.nih.gov/gds/): GSE65904, GSE59455, and GSE22153.Publicly available datasets were analyzed in this study. This data can be found here: TCGA dataset , the National Key Research Program of China (No. 2016YFC1101004), and Zhejiang Provincial Medical and Healthy Science Foundation of China (No. 2019ZD028 and 2018KY874).The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


Multi-gene expression assays are an attractive tool in revealing complex regulatory mechanisms in living organisms. Normalization is an indispensable step of data analysis in all those studies, since it removes unwanted, non-biological variability from data. In targeted qPCR assays it is typically performed with respect to prespecified reference genes, but the lack of robust strategy of their selection is reported in literature, especially in studies concerning circulating microRNAs (miRNA). Unfortunately, this problem impedes translation of scientific discoveries on miRNA biomarkers into widely available laboratory assays. Previous studies concluded that averaged expressions of multi-miRNA combinations are more stable references than single genes. However, due to the number of such combinations the computational load is considerable and may be hindering for objective reference selection in large datasets. Existing implementations of normalization algorithms  have poor performance and may require days to compute stability values for all potential reference as the evaluation is performed sequentially.We designed NormiRazor - an integrative tool which implements those methods in a parallel manner on a graphics processing unit (GPU) using CUDA platform. We tested our approach on publicly available miRNA expression datasets. As a result, the times of executions on 8 datasets containing from 50 to 400 miRNAs (subsets of GSE68314) decreased 18.7 ±0.6 (mean ±SD), 104.7 ±4.2 and 76.5 ±2.2 times for geNorm, BestKeeper and NormFinder with respect to previous Python implementation. To allow for easy access to normalization pipeline for biomedical researchers we implemented NormiRazor as an online platform where a user could normalize their datasets based on the automatically selected references. It is available at norm.btm.umed.pl, together with instruction manual and exemplary datasets.NormiRazor allows for an easy, informed choice of reference genes for qPCR transcriptomic studies. As such it can improve comparability and repeatability of experiments and in longer perspective help translate newly discovered biomarkers into readily available assays. Despite the boom of next-generation sequencing techniques, quantitative RT-PCR (qPCR) assays remain a popular method for evaluating gene expression. Furthermore, due to recent advances in design of highly specific probes this method is becoming effective for analysis of short non-coding RNAs . AlthougHowever, the approach described above is impossible to apply when the qPCR assay targets only few differentially expressed miRNAs (or genes). The mean expression cannot be used as reference in this scenario as the miRNAs are chosen on the basis of between-group differences. Therefore, in such situations, normalization must be performed with respect to endo- or exogenous control genes or total input RNA . ExogenoLiterature review , 19, 20 In these methods potential reference genes are ranked according to their respective stability scores and the most stable one is applied as a normalization factor. Stability, in this context, is understood as the invariability of expression of genes or microRNAs between experiments that are subject to changing technical aspects and whose samples may represent different pathological conditions or clinical factors. Although this approach is commonly used , it has recently been questioned by members of our research team  as an inA new algorithm was therefore proposed that instead of analyzing only single miRNAs, also calculates stability scores for all of their 2- and 3-element combinations in geNorm (GN), NormFinder (NF or NFG when samples’ groups are provided) and BestKeeper (BK). It was proven that combination-based normalizers are statistically better than single reference genes in terms of stability . MoreoveFollowing , we redeReading and preprocessing input data.Calculation of stability indices for single genes by geNorm, BestKeeper and NormFinder.Determination of all possible 2- and 3-element combinations of input miRNAs.Calculation of stability indices for combinations by the same algorithms and creating rankings of combinations according to their stability.Aggregation of results from all normalization algorithms and choice of recommended references among combinations.Normalization of input dataset with respect to selected reference (on user request).The comprehensive combination-based method of identifying reference circulating miRNAs, described in the “N) generated for a given set is dependent on the number of elements (miRNAs) in that set and can be calculated using the formula:Step 1 consists of reading data (gene expression and samples’ group designation), checking format correctness and handling missing values. For further analysis we only consider the miRNA detected in all samples, as they are most likely to not produce missing values in the next targeted experiment. In step 2 single genes are ranked from the most to the least stable one. Additionally, to reduce computation times for larger datasets (e.g. from NGS), we limit the analysis to 250 genes with the highest stability. From the remaining genes, step 3 creates the list of all combinations of given length that serves later as a queue of tasks distributed for parallel processing on a GPU. Step 4 assesses stability of all combinations and, as the most computationally expensive one, is executed on a GPU. The exact number of combinations .Parallel implementation on CUDA-enabled GPU.Limitation of miRNAs set size by the exclusion of those that are the least probable to form stable combinations.Calculation of stability indices for combinations presented the greatest area of optimization in the form of:Mathematical optimization consisted mainly in limiting operations repeated in every iteration to only one execution. Formal algebraic description of the algorithms and some minor modifications introduced by us is presented in detail in Additional File We divided the main part of the application into three separate paths, each of them corresponding to one of proposed normalization algorithms. We then implemented them as CUDA C kernels embedded into a C++ application  to the user and then allow them to normalize their dataset with one of these reference sets. The output file is kept in the same format as for the input file to allow for easy inclusion of our normalization strategy into existing pipelines. The complete stability rankings for all combinations are also available for download.Finally, having identified the need for a readily available tool for users with limited experience in programming, we designed graphical user interface (GUI) for our algorithm that is available on our institution’s server at norm.btm.umed.pl.We decided that the main measure of efficiency and user-friendliness of our application would simply be total execution time, since it is the most discernible for the end user and ultimately affects their experience the most. As a way of providing an insight into the particular parts of operation of the application, we also measured the execution times of some distinct parts of the algorithm, including purely computational one, named “kernel” in the “For the purpose of testing the performance improvement of the proposed computing method, we employed two benchmarking platforms. The system specifications of both platforms are presented in Suppl. Table 1  of the speed-up was calculated according to [where rding to  as:5\doObtained speed-ups for 3-miRNA combinations were the highest in case of BestKeeper with execution of whole algorithm 100 times faster in CUDA than in Python and the lowest in geNorm with the program still running almost 20 times faster than before  and the results are shown in Suppl. Fig. 4 and 5 in Additional File  and the Cq values (mean-centered) after different normalization protocols were applied to the data. In panel A, no normalization was applied, thus sample means and range of values were clearly different between samples. The next two panels (B and C) present the effects of normalization to the best 3-miRNA-based reference  and normalization to the mean of miRNAs detected in all samples, respectively. Those 2 approaches equalized distributions of Cq, such that distributions in all samples became very similar. Additionally, methods B and C do not produce discernibly different results; thus, they may be equally useful. In panels D  and E , references which were proposed in [Our next goal was to check the validity of identified references when they are applied to real data. Figure posed in  by the aEffect of normalization methods on variability of miRNA expression between samples was presented in Fig. MYCN-amplified and non-amplified neuroblastoma samples after applying different normalization schemes. Figure Finally, we demonstrated applicability of references found by NormiRazor in differential expression analysis on the exemplary neuroblastoma dataset (GSE121513). The results of this biologically-oriented validation are presented in Fig. In the present study we proposed a new GPU-based computational approach to the recently published  extensivIn the case of searching for reference genes/miRNA, the execution time of an algorithm is an essential part of the user experience. Since finding the reference is only a small, although important part of gene expression experiment, upon which following actions depend, no researcher would accept waiting for the results for an extended period. The longest computational section in our application, namely calculating stabilities for 3-element references by geNorm, takes about 100 seconds with a list of 250 candidate miRNAs, which was set as a technical limit in the final web application. Thus, taking into account the queuing of tasks and aggregation of results, reference miRNAs could be found in about 15 min, which we believe to be an acceptable waiting period.Another important aspect to consider is the quality of identified references. Previous work of our team showed that averaged expression of 2 or 3 miRNA is a more stable reference than any single miRNA with the measure of stability defined as a score from BK, GN and NF . In thisMYCN amplification, was detected when normalization was performed by our algorithm and by the method proposed in the article, upon which this part of validation analysis was based [Many studies stress that normalization strongly affects interpretation of experimental results , 18, 32.as based . When ouas based  who arguhttps://genorm.cmgg.be/). Currently geNorm is implemented in commercial qbase+ software, which however is not freely available to the research community. Similarly, BestKeeper cannot be downloaded from its original source. Besides, BestKeeper analysis was originally limited to at most 10 candidate genes that with currently available computational resources is no longer a reasonable limitation. NormFinder macro for Microsoft Excel is the only one that is still available and working even with newest version of Microsoft Office suite. Although it would not be effective for extensive search in combination-based approach, we must acknowledge its stability, that is relatively uncommon in case of bioinformatics tools [The entire computing suite is available as a web application. Distributed in this form it is independent of users’ computer hardware and software and it does not require the possession of any CUDA-enabled GPU. Compared with previous solutions, it definitely has several advantages. Initial implementations of geNorm, BestKeeper and NormFinder are already about 15 years old and are not directly compatible with new software. First version of geNorm as VBA applet for Microsoft Excel is no longer available, since according to its website it is “no longer compatible with latest versions of Excel, slow, buggy, difficult to use” : Platform independentProgramming language: C++, CUDA, Python, PHPLicense: GNU GPLSource code:https://git.btm.umed.pl/zbimt/normirazorAny restrictions to use by non-academics: NoneAdditional file 1 Mathematical formulation of implemented algorithms. Mathematical formulation of implemented algorithms with our changes with respect to original versions.Additional file 2 Supplementary results. This file contains all supplementary tables and figures which are refereed in text.Additional file 3 Archive with source code. This archive contains NormiRazor source code.


Advances in single-cell RNA-seq technology have led to great opportunities for the quantitative characterization of cell types, and many clustering algorithms have been developed based on single-cell gene expression. However, we found that different data preprocessing methods show quite different effects on clustering algorithms. Moreover, there is no specific preprocessing method that is applicable to all clustering algorithms, and even for the same clustering algorithm, the best preprocessing method depends on the input data.We designed a graph-based algorithm, SC3-e, specifically for discriminating the best data preprocessing method for SC3, which is currently the most widely used clustering algorithm for single cell clustering. When tested on eight frequently used single-cell RNA-seq data sets, SC3-e always accurately selects the best data preprocessing method for SC3 and therefore greatly enhances the clustering performance of SC3.The SC3-e algorithm is practically powerful for discriminating the best data preprocessing method, and therefore largely enhances the performance of cell-type clustering of SC3. It is expected to play a crucial role in the related studies of single-cell clustering, such as the studies of human complex diseases and discoveries of new cell types. Single-cell RNA sequencing (scRNA-seq) has revolutionized traditional transcriptomic studies by extracting the transcriptome information at the resolution of a single cell; therefore, this approach is able to detect heterogeneous information that cannot be obtained by sequencing mixed cells and to reveal the genetic structure and gene expression status of a single cell –7. MoreoA number of clustering methods have been developed by using scRNA-seq data; e.g., Xu and Su designed a new method by using a shared nearest neighbor approach followed by a quasi-clique-based clustering algorithm (SNN-cliq) to cluster single-cell transcriptomes . In addiGenerally, most single-cell clustering methods use gene expression data as their input, estimated from the scRNA-seq data of individual cells. A critical step in those clustering methods is to perform data preprocessing before cell-type clustering to eliminate the effects of confounding factors and reduce the effects of noise in the sample. Gene expression data record the expression value of each gene in each cell generally by transcripts per million mapped reads (TPM), counts per million mapped reads (CPM), reads per kilobase of transcript per million mapped reads (RPKM), fragments per kilobase of transcript per million mapped reads (FPKM), read counts mapped to a gene (READS), quantile normalization (QN) or others. For a given gene expression data set, the commonly used data preprocessing methods for single-cell clustering include log transformation, z-score transformation and a newly developed approach sctransform  in the sk-means, and SNN-clip, to eight commonly used single-cell gene expression data sets. The results showed that different data preprocessing methods have quite different effects on different clustering algorithms for different types of gene expression data. Additionally, some clustering methods showed the best clustering results for certain data sets without any preprocessing. Therefore, we conclude that there is no specific preprocessing approach that is applicable to all clustering methods for any gene expression data set. Based on this conclusion, we designed the graph-based SC3-e algorithm specifically for discriminating the best data preprocessing method for SC3 algorithm. When our algorithm was tested on the eight frequently used single-cell gene expression data sets, SC3-e always accurately selects the best preprocessing method for SC3 and therefore greatly enhances the performance of SC3.In this study, we analyzed the effects of data preprocessing on clustering results in detail by applying several widely employed clustering methods, such as SC3, dynamicTreecut, pcaReduce, tSNE + k-means, SNN-clip, pcaReduce, and SC3) were applied to evaluate clustering performance under four of the most commonly used data preprocessing methods  with eight frequently used data sets ; therefore, the default value of N is set to 1 in the design of SC3-e algorithm. Based on the above evaluations, we concluded that SC3-e is not only effective but also quite stable.After running SC3-e on all the eight data sets, the results showed that the parameter see Fig. . In factk-means were run 100 times to capture the average accuracy, while SNN-clip and dynamicTreecut were run only once, as their solutions are stable. After running all the clustering methods on the eight data sets, the results showed that SC3-e performed the best for six data sets, except for the Biase data set and Deng data set, where only the dynamicTreeCut algorithm performed slightly better than SC3-e , num_clusters, we tested its effects by setting different values of num_clusters. Results showed that changing the parameter num_clusters may result in the change of the best preprocessing method. For example, on the Biase data set, the best preprocessing method was z-score transformation if the num_clusters was set to 4, while the best preprocessing method was log transformation if the num_clusters was set to 3, 5, or 6 , SNN-clip, and SC3, and four of the most commonly used preprocessing methods, log transformation, z-score transformation, no transformation, and sctransform, were applied in this study to analyze the performance of different clustering methods under different preprocessing methods. Given input gene expression matrix data in which columns represent cells and rows correspond to genes/transcripts, different transformations were calculated as follows:i-th row of the input data matrix, and i-th row of the input data matrix.Five frequently used clustering methods, dynamicTreecut, pcaReduce, tSNE followed by k-means and SC3 50 times, while SNN-clip and dynamicTreecut were run only once since their solutions are stable. Then, the clustering accuracy was calculated according to the commonly used criterion of the adjusted Rand index (ARI) [After transformations, four preprocessed matrices were generated, referred to as log data (log transformation), z-score data (z-score transformation), no data (no transformation), and sctransform data (sctransform). Then, the applied clustering methods took each of these four data matrices as the input to perform cell-type clustering. To obtain stable clustering solutions for the clustering methods, we ran pcaReduce, tSNE + ex (ARI)  as follonij represents values from the contingency table, ai is the sum of the ith row of the contingency table, and bj is the sum of the jth column of the contingency table.where To select the best preprocessing method for each given data set for the SC3 clustering algorithm, we designed the graph-based SC3-e algorithm  were performed on M, and four preprocessed matrices, M-log, M-zscore, M-no, and M-sctransform were generated. Then, the SC3 pipeline was applied to each of the preprocessed matrices to produce a consensus matrix, C, and a clustering result, T. The value of cij  in the consensus matrix, C, represents the similarity between cells i and j, and the larger the value, the more similar they are. Therefore, we finally obtained four consensus matrices, C-log, C-zscore, C-no, and C-sctransform, and four corresponding clustering results, T-log, T-zscore, T-no, and T-sctransform.Given a gene expression data set, T-log, T-zscore, T-no, and T-sctransform, based on the consensus matrices C-log, C-zscore, C-no, and C-sctransform, we first built a clustering graph for each of the four clustering results. Given a clustering result, T and its corresponding consensus matrix, C, each node in the clustering graph represents a cell in the data set, and an in-cluster edge is added between two nodes, ni and nj, if and only if the two corresponding cells, i and j, are in the same cluster and their consensus value, cij, is lower than a threshold of cm,n, where m indicates that this edge belongs to the m-th cluster, and n means that it is the n-th in-cluster edge in the cluster. At the same time, an out-cluster edge is added between two nodes, ni and nj, if and only if the two corresponding cells, i and j, are in two different clusters and their consensus value, cij, is higher than a threshold of rp, where p represents the p-th out-cluster edge in the clustering graph. After the processing of each clustering result, we finally obtained four clustering graphs G-log, G-zscore, G-no, and G-sctransform.To discriminate the best clustering result from G, for each clustering result, we contracted the nodes in the same cluster into a single node, referred to as a cluster node, and two cluster nodes, vi and vj, are connected by an edge if and only if there is at least one out-cluster edge between the two corresponding clusters in the original clustering graph. Each edge in the contracted clustering graph is weighted by the value of di,j, calculated as follows:Ei,j represents the set of out-cluster edges between clusters i and j in the original clustering graph, and |Ei,j| represents the number of edges in set Ei,j. According to the definition, a smaller di,j clearly indicates lower closeness between two clusters, i and j. Then, each node in the contracted clustering graph is weighted by two values, wi and di, where wi measures the closeness among cells in the same cluster, i; di measures the closeness between cluster i and all the other clusters; and the two values wi and di are defined as follows:k is the number of edges in cluster i.D(vi) represents the set of the neighbors of cluster node vi.After constructing a clustering graph, n represents the number of clusters, and wi and di represent the two corresponding weights of cluster node vi. According to the above definition, a lower value of the C-score means that cells in the same cluster show closer relationships, while cells in different clusters show more distant relationships. Therefore, the smallest value of the C-score demonstrates the best data preprocessing for SC3 and therefore the best cell-type clustering.To choose the best data preprocessing method for SC3 based on the contracted clustering graphs, we calculate a C-score value as follows:N times  and generate N C-scores for each transformation; then, the final stable C-score can be obtained by computing the average of the N C-scores.SC3 is an unstable algorithm that may generate different clustering results and consensus matrices when it is run multiple times and therefore produces different C-scores. To obtain a stable C-score under each transformation, we can repeat the whole process Additional file 1: Table S1. This file contains 50 pairs of ARI and C-score values generated by running SC3 50 times on each data set.Additional file 2: Table S2.This file contains the ARI and C-score values under different values of the parameter num_cluster on each data set.Additional file 3: Table S3.This file contains the ARI and C-score values under different values of the two parameters α, and β on each data set.


High-throughput sequencing (HTS) has been applied successfully for virus and viroid discovery in many agricultural crops leading to the current drive to apply this technology in routine pathogen detection. The validation of HTS-based pathogen detection is therefore paramount.Plant infections were established by graft inoculating a suite of viruses and viroids from established sources for further study. Four plants  were sampled in triplicate and total RNA was extracted using two different methods (CTAB extraction protocol and the Zymo Research Quick-RNA Plant Miniprep Kit) and sent for Illumina HTS. One replicate sample of each plant for each RNA extraction method was also sent for HTS on an Ion Torrent platform. The data were evaluated for biological and technical variation focussing on RNA extraction method, platform used and bioinformatic analysis.The study evaluated the influence of different HTS protocols on the sensitivity, specificity and repeatability of HTS as a detection tool. Both extraction methods and sequencing platforms resulted in significant differences between the data sets. Using a de novo assembly approach, complemented with read mapping, the Illumina data allowed a greater proportion of the expected pathogen scaffolds to be inferred, and an accurate virome profile was constructed. The complete virome profile was also constructed using the Ion Torrent data but analyses showed that more sequencing depth is required to be comparative to the Illumina protocol and produce consistent results. The CTAB extraction protocol lowered the proportion of viroid sequences recovered with HTS, and the Zymo Research kit resulted in more variation in the read counts obtained per pathogen sequence. The expression profiles of reference genes were also investigated to assess the suitability of these genes as internal controls to allow for the comparison between samples across different protocols.This study highlights the need to measure the level of variation that can arise from the different variables of an HTS protocol, from sample preparation to data analysis. HTS is more comprehensive than any assay previously used, but with the necessary validations and standard operating procedures, the implementation of HTS as part of routine pathogen screening practices is possible.The online version contains supplementary material available at 10.1186/s12985-021-01523-1. The prevention and management of plant diseases largely depend on accurate identification of pathogens. Rapid and specific detection assays are required that need continuous development and optimisation as technology and knowledge of the pathogens advance. The increased use of high-throughput sequencing (HTS) for the construction of virome profiles of many agricultural crops has led to the discovery of many new viruses –13. SomeThe generation of sequence data using various metagenomic and enrichment strategies and the development of the associated bioinformatic pipelines has led to the identification of many known and novel viral sequences . The conThe current consensus is that HTS data analyses and the interpretation of the results for plant viral diseases require expertise in both bioinformatics and plant virology , 15. NevEven though plant virologists without training in bioinformatics can benefit from automated pipelines with graphical user interfaces, the quality and accuracy of the output is reliant on the quality of the input. The input to an HTS assay incorporates the whole process from sample collection, wet laboratory processing and data generation including data quality control. All the quality control measures up to data QC are the same or similar to any other sensitive molecular assay and need to be incorporated as assay variables. To ensure optimum data analysis, data should be evaluated for the different quality parameters, including not only the quality scores of each base, but also the sequencing depth. All these parameters can impact on the specificity, sensitivity and repeatability of the diagnostic result. The specific application of HTS will determine the acceptable level of variation that is tolerable. Identifying the exact virus or viroid variant, for example, is not required for pathogen detection. Applications of HTS are therefore varied and include both detection and discovery. It is important to identify the application and to tailor the assay, data analysis and interpretation accordingly.The citrus industry is one of the largest fruit industries worldwide with South Africa being the second largest fresh citrus exporter . HoweverViruses and viroids that are mainly spread through vegetative propagation can be effectively controlled through the use of certification programmes to generate virus free budwood/cuttings for propagation. HTS can detect multiple pathogens within a single assay, and has the advantage that data can be re-evaluated as new viral agents and variants are added to global databases.Citrus sinensis plant material infected with a range of viruses, including positive and negative sense RNA viruses, and viroids were established and subjected to HTS to evaluate the level of biological and technical variation that can arise from the RNA extraction method, sequencing platform and bioinformatic pipeline used. This study evaluated HTS variation for a citrus virome in order to use HTS as a standalone detection assay.In this study, C. sinensis cv. ‘Madam Vinous’, sweet orange) were established from seed and each graft inoculated in February 2019 with three source plants infected with a pre-determined range of pathogens (citrus tristeza virus (CTV), citrus virus A (CiVA), citrus tatter leaf virus (CTLV), hop stunt viroid (HSVd), citrus dwarfing viroid (CDVd) and citrus exocortis viroid (CEVd)). Source plant A was infected with CTV genotypes RB, VT, T3, T30, S1 and A18, CiVA, HSVd, CDVd and CEVd, source plant B with ‘Candidatus Liberibacter africanus’ (CLaf) and source plant C only contained CTLV. These source plants are well characterised sources maintained at Citrus Research International (CRI) in Nelspruit, South Africa. Plants were maintained in a temperature-controlled greenhouse (24–28 °C) with natural light in summer, but with additional lighting provided in winter months to supply a total of 16 h light per day. The infection status of the graft inoculated plants were confirmed with RT-PCR eight months post inoculation. Briefly, total RNA was extracted from leaves using an acid phenol extraction buffer [A set of plants  was also subjected to the CTAB extraction protocol as a nontarget positive control. The Quick-RNA Plant Miniprep Kit (Zymo Research) was used to extract total RNA from 0.2 μg of leaf midribs from each sample homogenised in liquid nitrogen. The three samples from each of the four plants were sequenced on the Illumina HTS platform and one sample of each plant was sequenced on the Ion Torrent HTS platform. For the samples sequenced on both platforms, the same RNA extract was divided and shipped to the respective service providers. The quality of the RNA was assessed independently at each service provider using the Agilent 2100 BioAnalyzer to determine the RNA integrity number (RIN).A representative leaf sample was taken from each of the four experimental plants, and from this sample three random samples were taken for further processing. Total RNA was extracted from these three samples per plant using two different extraction methods Fig. . One graTwo-step RT-PCRs were performed to determine the CTV genotype status of the graft inoculated plants. Complementary DNA (cDNA) was synthesized from 1 μg of total RNA using 0.15 μg of random hexamers (Promega) and Maxima reverse transcriptase (Thermo Scientific) in a final volume of 20 μl according to manufacturer’s instructions. A 2-μl aliquot of cDNA was added to 25 μl of PCR reaction mixture containing 1 × KAPA Taq buffer A (Roche), 0.2 mM dNTP mix (Thermo Scientific), 0.4 μM forward and reverse primers (IDT)   Fig. . Paired-C. sinensis plants extracted with two RNA extraction methods, were ribo-depleted , and sequencing libraries were constructed  , Stellenbosch University)  Fig. . Single-The service providers were requested to provide 20 million read pairs per sample from Illumina sequencing and 40 million single end reads from Ion Torrent.Adapter sequences were removed from the Illumina data and data was trimmed for quality using Trimmomatic  (SLIDINGC. sinensis host genome (C.sinensis_Hzau_v2.0_genome) [Both the trimmed Illumina and Ion Torrent data were subjected to de novo assemblies using SPAdes 3.14  and CLC _genome)  and the To perform the CGW assembly both the Illumina and Ion Torrent data were imported using the appropriate tool for Illumina or Ion Torrent fastq files, respectively. Subsequently the data was de novo assembled (default parameters) and the resulting contigs were identified using Blastn.All the quality trimmed reads were mapped to reference genomes extracted from Genbank. These references included the expected viruses and viroids  as well as nontarget variants of expected viruses , and the nontarget positive control virus . The Burrows-Wheeler Alignment Tool (BWA) version 0.7.13  were utiThe CTV genotype status of each plant sample was also confirmed with read mapping using the pipeline and criteria established previously .C. sinensis reference genes were identified from literature [Arabidopsis Thaliana orthologs against the organism C. sinensis  using CGW (default parameters).The average distance between the paired-end Illumina reads was calculated by mapping the paired-end data to the The read count of each reference gene, virus and viroid sequence was normalised for comparisons by calculating the transcripts per million (TPM) count. Firstly, the read count per kilobase of sequence (RPK) was calculated for each gene, virus and viroid sequence. After which the sum of all the RPKs were divided by a million to calculate the denominator for the transcripts per million (TPM) count for reference genes and pathogens separately. Each sequence’s RPK was divided by the reference genes or pathogens specific denominator to normalise the read count for biological and technical variation between samples. The TPM count for each reference sequence was used to compare the proportion of reads that mapped to a specific reference in each sample. Statistical analyses were performed using R available from the Comprehensive R Archive Network (CRAN) . The Kruhttps://github.com/lh3/seqtk). The Illumina subsets were created in paired-end mode to have a total read count for each subset equal to the selected subset sizes (e.g. the 1000 reads subset from Illumina data contain 500 read pairs). The data subsets were mapped to the different reference genes or pathogen sequences using the abovementioned pipeline.To assess the influence of data set size on the detection of virus and viroids using read mapping, data for each sample were sub-sampled randomly 10 times for 9 different subset sizes  using seqtk  and the average ribosomal RNA (rRNA) ratio was 5.5 (± 1.3 standard deviation). The average RIN value for the RNA extracted with the Zymo Research kit was 7.9 (± 0.5 standard deviation) and the average ribosomal RNA (rRNA) ratio was 1.9  and 79,000 (Zymo Research kit extractions) scaffolds with an average N50 of 1800 and 1600 nt, respectively  and five of the Zymo Research kit extracted samples . Contigs could not be assembled from the Ion Torrent data for CTLV in one CTAB extracted sample, HSVd in three samples (one CTAB and two Zymo Research kit extractions), CDVd in one CTAB extracted sample and CEVd in two samples (one per extraction method) , CTLV in sample Z122 (Ion Torrent), CEVd in sample Z131 (Ion Torrent) and HSVd and CEVd in sample Z143 (Ion Torrent).C. sinensis reference genes to identify biological variation and technical variation associated with RNA extraction method and sequencing platform  for CTV genotype RB (KU883265), CTLV MH108976.1), HSVd (KY110716.1), CEVd (KY110721.1), and CiVA RNA1 (MT720885) and CiVA RNA2 (MT720886) , CTV (MK033511) and CiVA (MT720885), respectively. The citrus Illumina data sets were also mapped to OrMV (KY769694.1) and on average 197 reads  mapped from the CTAB/Illumina data and 60 reads  of the Zymo Research/Illumina data mapped. The negative control data was also mapped to all the pathogens and the highest read count obtained was 22 reads (Illumina) and one read (Ion Torrent) for CTV (KU883265).The Subsampling the original data into nine different sized data subsets showed the influence of sequencing depth on the potential genome/transcript coverage that can be obtained. Overall, the Illumina data reached a higher genome coverage with less data for both the pathogen and gene reference sequences Figs.  and 9.FiWith the 10,000,000 reads subset an average target pathogen genome coverage of 99.5%, 99.6%, 99.2% and 98.0% was obtained for the CTAB/Illumina, Zymo Research/Illumina, CTAB Ion/Torrent and Zymo Research/Ion Torrent protocols, respectively  trees were prepared, one negative control and three graft inoculated with CTV, CTLV, CiVA, HSVd, CDVd and CEVd. Each plant was sampled three times at the same timepoint, and RNA extracted using two different methods. Two sequencing platforms were selected to generate data from three samples from four plants (Illumina) and one sample from four plants (Ion Torrent). All data sets were subjected to a reference independent de novo assembly approach and a dependent read mapping strategy to determine the virome profile of each plant sample.HTS can be utilised for both detection and discovery and the acceptable level of variation in specificity, sensitivity and repeatability that can be tolerated will depend on the application. In the present study, the detection of virus and viroid species were evaluated for application as a routine diagnostic assay. Identifying the exact virus or viroid variant, for example, was therefore not required for pathogen detection.HTS is intrinsically specific and is therefore only limited by the accuracy of the base calls, the depth of the data and the completeness of the reference databases . A publiThe composition of the starting nucleic acids additionally impacts detection specificity. The virome profiles of the Zymo Research kit extracted samples were less consistent across samples and replicates than the profiles of the CTAB extracted samples Fig. . This vaThe lower weight input restriction of the Zymo Research RNA extraction kit resulted in greater variation in the virome profiles between the technical replicates of each biological sample Fig. , probablReproducibility is required at each step of the HTS assay, from nucleic acid extraction to data interpretation. Previous studies highlighted the link between appropriate depth of coverage and the repeatability of the assay , 40, butThe two different extraction methods yielded total RNA with different rRNA profiles. No significant difference between the RIN values for the two groups was observed, however the rRNA ratio of the Zymo Research kit extracted RNA was significantly lower than for the CTAB RNA due to a higher concentration of 5S rRNA yielded by the Zymo Research kit. This indicates a potential difference in the RNA species extracted with each method and it is possible that the CTAB extraction selected against viroid sequences due to the Lithium Chloride (LiCl) precipitation step. The SPAdes de novo assembly of the Illumina data did not produce viroid scaffolds for one technical replicate of two different CTAB extracted samples, compared to all the expected viroid scaffolds assembled in the Zymo Research kit replicates  Fig. . The expThe expected virome was confirmed with RT-PCR and included five CTV genotypes , CTLV, CiVA and three viroids . The influence of read mapping to a distant variant of the target virus/viroid was assessed by including nontarget reference sequences of CTV (genotype T68) and the Cachexia variant of HSVd. An average genome coverage for CTV genotype T68 of 73–83% was obtained for the four different extraction/platform protocols. Compared to the genome coverage of the expected CTV genotype RB of > 99%, the T68 read mappings are distinguishable as false positive mappings. A previous study showed that it is possible to obtain up to 90% coverage for nontarget genotypes in mixed genotype infections and that read mapping across more than 95% of the genome is indicative of the presence of a particular genotype . Due to The sensitivity of any detection assay is directly linked to the proportion of viral RNAs among the host cellular RNAs. Therefore, sequencing depth plays an important role in the reliability of the HTS assay. The main conclusion from the subset experiment in this study was that less Illumina data was needed to obtain complete or near complete genomes of the expected pathogens and the Ion Torrent data can perform on par with Illumina if more reads are used for the read mapping. The number of bases in each subset size for the Ion Torrent data was 1.3–1.4 times more compared to the Illumina subsets as a consequence of the longer read length. However, the average distance of 196 bases between the Illumina read pairs may have increased the efficacy of the Illumina read mapping.The finding of a previous study  that shoOrnithogalum nontarget positive control or C. sinensis negative control data that was included in the Illumina and Ion Torrent sequencing runs and no pathogens associated with Ornithogalum were assembled from the citrus RNA data. The Ornithogalum data set were mapped to the citrus pathogens and negligible read counts were obtained. A maximum of 788 citrus RNA reads from the different samples mapped to ornithogalum mosaic virus however the genome coverage never reached more than 1.5%. This indicated no significant cross-contamination between samples.No citrus pathogen sequences were de novo assembled from the The reproducibility of this study was not measured specifically, as a true test of reproducibility would require an interrogation of the extent to which consistent results could be obtained by repetition of the whole experiment at different timepoints. An attempt at reproducibility was made to include both biological and technical replicates for the Illumina sequencing protocol. The technical replicates were however not sequenced on the Ion Torrent platform due to a cost implication. The Ion torrent data cost 43% more for the same amount of Illumina data.In this biological context, reproducibility will not be completely achievable as variables such as plant age, growth stage and virus concentration linked to infection duration might influence the outcome.A comparison of single-end data from the Illumina platform and single-end data from the Ion Torrent platform was not evaluated to keep the generation of data as close to a real-case scenario as possible. The two service providers, Macrogen and CAF, provide by default, paired-end Illumina and single-end Ion Torrent data, respectively.This study is a detailed measurement of technical variation in HTS data associated with the detection of viruses and viroids in citrus. The study evaluated the efficiency of using HTS to detect two single stranded RNA viruses from different families, a negative-sense single-stranded RNA virus and three viroid species. The study evaluated the influence of RNA extraction protocol, sequencing platform and data analysis pipelines on the sensitivity, specificity and repeatability of HTS as a detection tool. Each of these parameters introduce a different bias that creates variation in the data output. Even though the different extraction methods, sequencing platform and data analysis tools resulted in variation in the present study, the end result being the virome profile of each sample could be confirmed independent of the HTS approach. The study highlights the need to be aware of the level of variation associated with each approach in strategy, from sample collection to data interpretation and how these variables may impact on the initial objective of the HTS assay. This awareness is critical to enable informed adjustments to correctly interpret the data for a reliable results. The primary recommendation that follows from this study is that, irrespective of extraction method or sequencing platform, a combination of de novo assembly and read mapping be used for a routine detection assay.Since the goal of this study was to evaluate HTS as a detection tool in quarantine or certification schemes, and not for discovery purposes, a list of known pathogens should be available in these settings for read mapping. The aim of a de novo assembly in the certification scheme context will be to identify unsuspected pathogens.The application of HTS for the detection of plant viruses is commonly described as being unbiased, however this is only true within a specific context, in that it does not require any prior knowledge of the pathogens. There are however, slight biases and variations at every step of an HTS assay, as demonstrated, but which can easily be corrected for when quantified. This study provides strong evidence that the application of HTS for routine pathogen detection is attainable if the detection pipeline is critically validated.Additional file 1Species and strain-specific primer sequences used in a two-step RT-PCR to amplify citrus viruses and viroids.Additional file 2Descriptive statistics of high-throughput sequencing data generated and the identification of contigs/scaffolds per de novo assembly.Additional file 3Descriptive statistics of the TPM values for the different gene accessions used as internal control references.Additional file 4Statistical tests performed using the Kruskal-Wallis rank sum test between the data distributions of the different extraction methods and sequencing platform data sets based on transcripts per million (TPM) counts per gene analysed. Significant differences are indicated in bold if the p value < 0.05.Additional file 5The subset sizes for which the respective accession obtained a 90% and 95% genome coverage for all sample and subset replicates with read mapping per RNA extraction method and sequencing platform protocol.