Breast and prostate cancers are typical examples of hormone-dependent cancers, showing remarkable similarities at the hormone-related signaling pathways level, and exhibiting a high tropism to bone. While the identification of genes playing a specific role in each cancer type brings invaluable insights for gene therapy research by targeting disease-specific cell functions not accounted so far, identifying a common gene signature to breast and prostate cancers could unravel new targets to tackle shared hormone-dependent disease features, like bone relapse. This would potentially allow the development of new targeted therapies directed to genes regulating both cancer types, with a consequent positive impact in cancer management and health economics.twin networks recovery (twiner) penalty, as a strategy to ensure similarly correlated gene features in two diseases to be less penalized during the feature selection procedure.We address the challenge of extracting gene signatures from transcriptomic data of prostate adenocarcinoma (PRAD) and breast invasive carcinoma (BRCA) samples, particularly estrogen positive (ER+), and androgen positive (AR+) triple-negative breast cancer (TNBC), using sparse logistic regression. The introduction of gene network information based on the distances between BRCA and PRAD correlation matrices is investigated, through the proposed Our analysis led to the identification of genes that show a similar correlation pattern in BRCA and PRAD transcriptomic data, and are selected as key players in the classification of breast and prostate samples into ER+ BRCA/AR+ TNBC/PRAD tumor and normal tissues, and also associated with survival time distributions. The results obtained are supported by the literature and are expected to unveil the similarities between the diseases, disclose common disease biomarkers, and help in the definition of new strategies for more effective therapies. Breast invasive carcinoma (BRCA) and prostate adenocarcinoma (PRAD) are the two most common invasive cancers in women and men, respectively . In bothα or β may be an option in the treatment of castration resistant metastatic prostate cancer cells [ER), progesterone receptor (PR), and human epidermal growth factor receptor type 2 (HER2) [In the PRAD case, androgen/androgen receptor (AR) signaling pathway is deeply involved in the progression of the disease, and androgen deprivation therapy (ADT) with anti-androgens remains as the main treatment in early and late stage disease . Hormoneer cells . By the 2 (HER2) , and ass2 (HER2) . AR-inhi2 (HER2) .Given that BRCA and PRAD are hormone-dependent cancers, possibly sharing layers of signaling and regulatory pathways, it is important to unravel common players that could establish a link between the hormone-dependence and the fact that both types of cancers exhibit a high tropism to bone. To achieve that, one possible solution is to, given a classification model, extract the most relevant features in the discrimination between tumor and normal tissue, out of the full set of dozens of thousands of features currently delivered by high-throughtput ‘omic technologies. Classification of tumor and normal tissue can be performed separately for BRCA and PRAD, however, given the similarities between the two diseases, a common strategy to classify patients tissue, while simultaneously identifying the genes playing a role in both diseases, would be of great value. This way, a considerable reduction in the effort in defining new therapies could be accomplished.l1 regularizer and a linear combination of l1 (lasso) and l2 (ridge) penalties, respectively [Statistical learning in high-dimensional ‘omic data poses many challenges, in particular for parameter estimation, since the models are seldom identifiable. One way to cope with this problem is to add constraints in the parameters space. For instance, imposing sparsity in the solution will enable feature selection since a large subset of the parameters will be exactly zero. Several methods have been applied in the context of ‘omic data, namely the lasso and elastic net, which impose a ectively , 11. WhiG:=, V denotes vertices (or nodes) and E the set of edges. In a gene network, vertices are genes and edges represent a weighted relation between two genes. It has been advocated that incorporating network information as a constraint in the loss function potentially increases the model predictive performance of, e.g., sparse Cox and logistic regression models, as shown when modeling the survival of ovarian cancer carcinoma patients and classifying patients into breast cancer subtypes [Model constraints can benefit from external knowledge on the biological disease processes, often given by network information. For example, groups of genes are co-expressed under certain conditions or their protein products interact with each other to carry out a biological function , which csubtypes –14). Fursubtypes , 16), orsubtypes .In this work we combine correlation-based regularizers and sparse logistic regression to solve a binary classification problem with BRCA and PRAD cancer tissues, and normal tissue from breast and prostate cancer patients as classes. Different datasets will be considered in two case studies in the search for shared gene signatures in BRCA and PRAD: I) ER+ BRCA vs. PRAD, showing similarities at the ER signaling level and shared marked bone osteotropism; and II) AR+ TNBC vs. PRAD, sharing AR-signaling dependency. With the goal of identifying a common network to each BRCA subtype and PRAD data, ER+ BRCA, AR+ TNBC and PRAD gene correlation networks will be generated using the Pearson correlation between observed (gene expression) variables as similarity measure. For a given gene in the network, the more similar its correlation pattern between the two diseases under consideration is, the less penalized it will be in the regularization term of sparse logistic regression. Selected similarly correlated genes in the two diseases, i.e., playing a role in discriminating between cancer and non-cancer, can be seen as potential biomarkers candidates in the two groups of patients.https://cancergenome.nih.gov/).The transcriptomic data on breast and prostate cancer patients used in this work were obtained from The Cancer Genome Atlas (TCGA) Data Portal  and ‘0’ for normal (79 samples) tissue.The BRCA RNA-Seq Fragments Per Kilo base per Million (FPKM) dataset was imported using the ‘brca.data’ R packageY was created, with ‘1’ corresponding to TNBC individuals , and non-TNBC (‘0’) to non-TNBC (other types of BRCA) patients, whenever at least one of the three genes is ‘positive’. The individuals’ status regarding ER, PR and HER2, needed for building Y, were obtained from the BRCA clinical data available from the TCGA, composed of 114 variables, as described in Lopes et al. (2018) [tumor and 113 normal tissue samples.The TNBC dataset was built from the BRCA dataset described above. The TNBC binary response vector . 2018) , 18. Onl , 18. OnY is binary, coded with ‘1’ for tumor samples and ‘0’ for normal tissue samples. The PRAD dataset is composed of 495 tumor and 52 normal samples.The PRAD RNA-Seq Fragments Per Kilo base per Million (FPKM) dataset was imported using the ‘prad.data’ R packageFPKM normalized ER+ BRCA, AR+ TNBC, and PRAD gene expression data were log-transformed and Z-score normalized prior to data analysis. A subset of ∼ 20000 variables in each dataset was considered for further analysis, corresponding to the protein coding genes reported from the Ensembl genome browser  and the Y={Yi}i=1,...,n, which is given by the logistic function Binary logistic regression describes the relationship between one or more independent variables and a binary outcome vector Xi,i=1,...,n, is the vector of p covariates for observation i, P(Yi=1|Xi) is the probability of success for observation i, and β= are the regression coefficients associated with the p independent variables.where The parameters of the logistic model are estimated by maximizing the log-likelihood function of the logistic model, given by yi indicates success (yi=1) or unsuccess (yi=0) for observation i. By the introduction of a regularization term, the log-likelihood function becomes where the binary variable where α=1 corresponding to lasso and α=0 to ridge, and the tuning parameter λ controlling the amount of shrinkage in the coefficients.stands for the elastic net penalty, with With the specific goal of weighting variables based on their similarities across two given diseases, we propose twiner, a structured regularizer based on the pairwise correlations between variables, independently obtained from two given datasets.A and B, j=1,…,p with the remaining ones. The proposed dissimilarity measure dj of gene j between A and B is given by the angle of the corresponding vectors, i.e., Consider two correlation matrices for diseases w= is therefore based on this distance, normalized by their maximum value: The rationale of using the angle is that two patterns will be identified as similar if they have the same proportionality between the entries across the two datasets, irrespective of the magnitude of the vectors. In the context of the present application, one gene has a similar role in BRCA and PRAD, if it is similarly correlated with the remaining genes in the two diseases. This correlation-based regularization constitutes the basis of twiner, since this dissimilarity will then be used as a penalization for the cost function. The weighting vector to be used as a penalization The penalty term in Eq. w representing the factors that control how much of the penalty λ affects each coefficient, and ∘ standing for the element wise (or Hadamard) product.with vector j, the smaller the distance between j correlation pattern. Therefore, the resulting twin networks recovery (twiner) penalty enables the identification of variables with similar (twin) correlation with the remaining variables across the two diseases, with smaller penalties being associated to genes with smaller distances between the diseases’ correlation matrices. The influence these genes have in the outcome will be assessed by regularized logistic regression based on the elastic net penalty.For a given gene The modeling strategy described above will be applied in two case studies with the goal of finding common gene signatures in i) ER+ BRCA and PRAD, and ii) AR+ TNBC and PRAD, as described next.normal breast and prostate tissue samples. The overall procedure for dataset construction is illustrated in Fig. Sparse logistic regression using the elastic net penalty (EN) was used to classify RNA-Seq data from patients into ER+ BRCA, AR+ TNBC and PRAD vs. tumor, and EN applied to classify RNA-Seq data into ER+ BRCA or PRAD (tumor) vs. normal samples, herein called BRCAPRAD model. Three quarters of randomly selected samples were assigned to training samples for model construction, whereas the remaining samples were assigned to test samples for model evaluation. The classification was performed using two models: 1) EN; and 2) sparse logistic regression using the twiner penalty (twiner). For both EN and twiner models the alpha parameter was set to α=0.9, which yields a adequate number of features to be further analysed without compromising clinical interpretability. The Pearson correlation matrices from ER+ BRCA and PRAD RNA-Seq data are matrices ΣA and ΣB, as seen above, and the response Y vector is a binary vector with ‘0’ corresponding to normal tissue and ‘1’ to tumor (ER+ BRCA or PRAD) tissue. The angular distance between j in matrices ΣA and ΣB was used for building the weight vector w as explained above. With the goal of searching for shared disease biomarkers, genes showing larger angular distances between correlation vectors in the two diseases were discarded, only keeping those (out of the previous ∼ 20000) showing an angular distance less than 75 ∘, at the same time contributing to reduce model complexity. The new dimension in the variables space was thus decreased to 16367 genes.With the goal of finding a common gene signature between ER+ BRCA and PRAD cancer types, ER+ BRCA and PRAD data were grouped into a single class, i.e., EN and twiner models were generated 100 times for randomly chosen training and test sets. The median values for the mean squared error (MSE) of classification, the area under the Precision-Recall curve (AUC) , 22 and tumor samples (‘1’) vs. TNBC and PRAD normal samples (‘0’), herein called the TNBCPRAD model.The same analysis was performed in the search for a common gene signature between AR+ TNBC and PRAD cancer types. As for the ER+ BRCA vs. PRAD case, only genes (out of the previous ∼ 20000) showing lower angular distances were considered, yielding 14598 genes for model building in the AR+ TNBC vs. PRAD case. EN and twiner models were generated to classify RNA-Seq data into AR+ TNBC or PRAD For biological interpretation of the variables selected by the methods, gene correlation networks were represented only for ER+ BRCA, AR+ TNBC, and PRAD data, using the variables exclusively selected by EN, twiner and shared by the two models.p-values. Increased risk groups’ separability by a given set of genes is expected to trigger further research on the role of these genes in the disease.Finally, as an attempt to clinically validate our approach, the variables selected by EN and twiner were tested in a survival analysis on ER+ BRCA, AR+ TNBC and PRAD tumor data, using the Cox regression model . The indtumor vs. normal, independently for ER+ BRCA, AR+ TNBC and PRAD data, herein called ER+ BRCA, AR+ TNBC and PRAD models, were also built, using the same set of variables used for the combined BRCAPRAD and TNBCPRAD models, as explained above. The goal was to identify (if any) genes selected in common by independent disease models , and the overlap with BRCAPRAD and TNBCPRAD sparse logistic models aiming at extracting common gene signatures from two diseases through the twiner penalty. Similarly to the independent BRCAPRAD nd TNBCPRAD models, training and test sets were randomly generated. The optimization of the parameters λ and α based on the MSE for the models described above was performed by 10-fold cross-validation (CV), with varying α values (1 >α> 0) tested.Individual models for predicting the class, w vector was introduced as penalty factor in the glmnet function. Differentially expressed genes across tumor and normal ER+ BRCA, AR+ TNBC and PRAD tissue were identified using the limma Bioconductor R package [The glmnet R package  implemen package , in ordetumor and normal breast and prostate samples, a first non-supervised analysis was intended to visualize samples’ grouping in a reduced dimensional space. A Principal Component Analysis (PCA) was applied to a dataset comprising gene expression data from ER+ BRCA, AR+ TNBC and PRAD tumor tissue samples, along normal tissue samples from breast and prostate patients. A clear separation between ER+ BRCA/AR+ TNBC and PRAD tumor samples is observed in the space of the first two principal components  and TNBC (AR+ TNBC) is observed in the subspace represented  and 2 by twiner models (NKAPL and PAK3), with NKAPL being selected in common by the two modelling strategies.EN and twiner were applied to the BRCAPRAD dataset, as a means to identify a common gene signature between ER+ BRCA and PRAD diseases. Summary results of the two modeling strategies applied to 100 random training and test sets can be found in Table CXCR2, GLRA4, LRRC3B, NKAPL, PAK3, RP11-729L2.2, SCN5A and TMEM236) and the remaining 9  exclusively selected by twiner  were shared with the BRCA model and 1 (NKAPL) with the PRAD model, respectively  and common (green) genes in ER+ BRCA and PRAD tumor samples, compared to that observed for BRCA and PRAD normal tissue samples.Figure With the goal of finding a common gene signature in AR+ TNBC and PRAD, EN and twiner were applied to the TNBCPRAD dataset. A less pronounced model accuracy improvement was obtained for twiner over EN, compared to the BRCAPRAD case, with a median MSE decrease in 5% for the training set Table .BGN, DNAJB1, GLRA4, GSTM3 and NKAPL), and the remaining 8  exclusively selected by twiner  and 4  with the PRAD model (Table SLC35E2 and UCN) have lower weights compared to the remaining genes exclusively selected by EN and selected in common by the two strategies (Table tumor (AR+ TNBC/PRAD) and normal tissue.Thirteen genes were selected in more than 75 out of 100 twiner models Fig. ; Table 3tumor and normal samples  and two down-regulated (BMT2 and SCARA5). Amongst the three up-regulated, CLEC11A may be directly implicated in bone tropism or bone metastasis development.In this work we proposed to identify events specifically deregulated in both ER+ BRCA and PRAD cancers. We hypothesized that these events could reflect a possible association with bone tropism and/or hormone signaling. Our analysis led to the identification of nine genes exclusively selected by twiner, five identically deregulated in ER+ BRCA and PRAD Table . AmongstCLEC11A was previously found to be part of a gene set up-regulated in cancer stem cell populations upon therapeutic insult [CLEC11A encodes a C-type lectin domain protein, osteolectin, which is an osteogenesis driver usually secreted by bone stromal cells that promotes the differentiation of mesenchymal progenitors into mature osteoblasts [CLEC11A in our model may be associated with a blastic bone metastasis phenotype, typical in PRAD and also frequent in BRCA.c insult . CLEC11Aeoblasts . TherefoIn silico analysis had previously identified POLR2H as one of the key genes involved in the occurrence of PRAD, and POLR2H protein was significantly upregulated in PRAD tissues [ tissues . HoweverZBTB24 encodes the poorly characterized zinc finger and BTB domain containing 24 protein, which belongs to the large ZBTB family of transcriptional repressors. Although its function is still unknown, it was recently shown that ZBTB24 is involved in the control of DNA methylation [Finally, hylation . It willSCARA5 (Scavenger receptor class A member 5) is a candidate tumor suppressor in several malignancies; however, its role in BRCA cell growth and metastasis is still unclear. SCARA5 was found to be down-regulated in BRCA tissues and cells and correlated with clinicopathologic characteristics [SCARA5 overexpression significantly suppressed cell proliferation, colony formation, invasion, and migration, and induced G0/G1 arrest and apoptosis. Recently, another group reported that SCARA5 expression was significantly decreased in tumors (92.2%), compared to non-cancerous tissue samples, due to the hypermethylation of the promoter [SCARA5 in PRAD, but based on our results we hypothesize a similar pattern of expression. BMT2 has not been previously implicated in BRCA or PRAD.Amongst the down-regulated genes, eristics . In thispromoter . There aHER2 amplified tumors [CTU and UCN, up-regulated; and CD300LG, KLHL4, PARK2, SCARA5 and SNCG, down-regulated. Only SCARA5 was detected in the ER+ BRCA vs. PRAD analysis, suggesting specificity for AR+ TNBC.AR (androgen receptor)-signaling is particularly important in prostate cancer, however, AR is also expressed in up to 90% of ER+ BRCA, and to a lesser degree, in d tumors . Althougd tumors . TherefoCTU1 (cytosolic tRNA thiouridilase) is involved in maintaining genome stability, since post-transcriptional modifications of transfer RNAs (tRNAs) at the wobble uridine 34 (U34) base are highly conserved and contribute to translation fidelity [fidelity . Partnerfidelity .UCN (urocortin) has been previously reported to attenuate TGF β1-induced Snail1 and Slug expressions, both in ER+ BRCA and TNBC in vitro models [β1 oncogenic signaling and ultimately EMT. Therefore, UCN up-regulation would be associated with a better prognosis, a less invasive disease. However, it was also suggested that urocortin may have a dual role in cancer, since it differentially binds to CRFR1 or CRFR2 and either activates or blocks the Bcl-2/Bax/caspase-9 axis, leading to apoptosis or survival, respectively [UCN was found to be elevated in PRAD, although no correlations with clinical features were presented in this study [o models , suggestectively . In clinis study .SCARA5, and also SNCG, PARK2, CD300LG and KLHL4, were down-regulated in both AR+ TNBC and PRAD.The above described SNCG (synuclein-gamma) seems to be a molecular indicator of metastasis in a wide range of human cancers, including BRCA and PRAD [SNCG gene expression by DNA demethylation the contributing factor to malignant progression of many solid tumors and its expression in primary carcinomas is an effective molecular indicator of distant metastasis. Silencing SNCG in a prostate cancer cell line has shown to decrease proliferation and invasion in vitro, and tumor growth in vivo, with the exception of castrated mice [SNCG interacts with AR and promotes prostate cancer cellular growth and proliferation by activating AR transcription in an androgen-dependent manner, whereas SNCG was almost undetectable in benign or androgen-independent tissues prostate lesions. SNCG in PRAD was also described to be activated by Cav-1 in the tumor microenvironment [SNCG may correspond to a more indolent disease. In breast cancer, it was shown that TNBC cell lines do not express SNCG, in accordance with our results [SNCG [The loss of epigenetic control of and PRAD . In thisted mice , suggestironment . Neverth results ; althougts [SNCG .PARK2  is involved in autosomal recessive parkinsonism. PARK2 presents a partial mitochondrial localization at the outer mitochondrial membrane and its depletion results in abnormal mitochondrial morphology. An in silico analysis has shown that PARK2 may be related to cell cycle control, suggesting a role in carcinogenic processes [rocesses .CD300LG was found to be down-regulated in AR+ TNBC tissues when compared with adjacent normal studies [CD300LG down-regulation results in immune escape of cancer cells [In accordance to our findings,  studies , but itser cells .KLHL4 (Kelch like family member 4) was expected to be down-regulated in breast cancer. A previous study has shown that this gene is down-regulated downstream of IGFBP5, silenced in response to stromal cells in ER α-positive breast cancer cells [KLHL4 down-regulation is expected to be associated with worse prognosis.Also er cells . As thisFrom the biological interpretation above, several links between the results obtained by our method and disease biology have been established, which reinforces the ability of our method to identify shared disease features in breast and prostate cancers. Moreover, these genes are able to stratify patients into high or low risk, according to overall survival, and deserve further studies to clearly determine their role in the progression of BRAC and PRAD.High-dimensional data leads to ill-posed inverse problems that cannot be tackled easily. Regularized optimization is a promising strategy to cope with undetermined problems, since it adds extra constraints to the loss function, which, if chosen carefully, can provide biological and clinical insight. We presented the twiner penalty, a correlation-based regularizer designed to enable the selection of similarly correlated genes in two diseases by sparse logistic regression, as a strategy to identify common key players in both diseases. The usefulness of the strategy proposed is shown in the context of Breast Invasive Carcinoma (BRCA) and Prostate Adenocarcinoma (PRAD), which show remarkable similarities at the hormone-related signaling pathways level. While being largely supported by the literature and clinical evidence by survival analysis, our results identified putative disease biomarkers which are expected to greatly improve our knowledge on the diseases and contribute to the definition of new target therapies.


Ca–Cr) classifying these samples as low-Ca websteritic or eclogitic. The garnet REE patterns fit an equilibrium model suggesting the diamond-forming fluid shares an affinity with high-density fluids (HDF) observed in fibrous diamonds, specifically on the join between the saline–carbonate end-members. The δ13C values for the diamonds range from − 5.27 to − 22.48‰ (V-PDB) with δ18O values for websteritic garnets ranging from + 7.6 to + 5.9‰ (V-SMOW). The combined C–O stable isotope data support a model for a hydrothermally altered and organic carbon-bearing subducted crustal source(s) for the diamond- and garnet-forming media. The nitrogen aggregation states of the diamonds require that diamondite-formation event(s) pre-dates fibrous diamond-formation and post-dates most of the gem monocrystalline diamond-formation events at Orapa. The modelled fluid compositions responsible for the precipitation of diamondites match the fluid-poor and fluid-rich (fibrous) monocrystalline diamonds, where all grow from HDFs within the saline-silicic-carbonatitic ternary system. However, while the nature of the parental fluid(s) share a common lithophile element geochemical affinity, the origin(s) of the saline, silicic, and/or carbonatitic components of these HDFs do not always share a common origin. Therefore, it is wholly conceivable that the diamondites are evidence of a distinct and temporally unconstrained tectono-thermal diamond-forming event beneath the Kaapvaal craton.The petrogenesis and relationship of diamondite to well-studied monocrystalline and fibrous diamonds are poorly understood yet would potentially reveal new aspects of how diamond-forming fluids are transported through the lithosphere and equilibrate with surrounding silicates. Of 22 silicate- and oxide-bearing diamondites investigated, most yielded garnet intergrowths (The online version of this article (10.1007/s00410-019-1608-0) contains supplementary material, which is available to authorized users. The deep carbon cycle can be investigated on billion year timescales by studying the petrological and geochemical nature of mantle diamonds and their mineral/fluid inclusions and intergrowths. There are three main diamond types; monocrystalline, fibrous, and polycrystalline, where the latter are found as either framboids (i.e. boart) or diamondite  a dominance of low-Ca eclogitic to websteritic garnets relative to garnets from monocrystalline diamonds  and source(s) for the diamondite-forming fluids at Orapa, Botswana. These data are used to contrast diamondite-formation with fibrous and non-fibrous monocrystalline diamond-formation. This study examines long-standing notion that diamondite-formation represents a habitually ignored but volumetrically significant and distinct carbon-rich metasomatic (diamond-forming) event in the sub-continental lithospheric mantle . Images were generated using an accelerating voltage of 15 kV on samples coated with iridium (~ 1 nm). Major and minor element compositions were determined by electron probe microanalysis (EPMA) using a JEOL 8200 electron microprobe housed within the Institute of Meteoritics at the University of New Mexico. Qualitative analysis was performed using both backscattered electron imaging and energy dispersive spectroscopy. Quantitative analyses were performed using wavelength dispersive spectrometers. An accelerating voltage of 15 kV and a nominal probe current of 40 nA was used during each analysis. We analysed for Si, Ti, Al, Cr, Fe, Mn, Mg, Ni, Ca, Na, and P. Si and Ca were standardized using a synthetic diopside crystal. Ti was standardized using Taylor rutile, and Fe was standardized using Taylor hematite (Taylor multi-element standard documentation. C.M. Taylor Company). Al was standardized using pyrope from Kakanui, New Zealand, and Cr was standardized using chromite from Tiebaghi Mine, New Caledonia . Samples were analysed in batches of ~ 16, with multiple analyses of NIST SRM 612  of BCR-2G, used as a measure of precision, are ≤ 5% for most elements, except for 9Be (5%), 111Cd (8%), 115In (8%), 118Sn (10%), 121Sb (7%), and 182W (6%). The higher RSD for these elements is attributed to their low elemental abundances. To ensure BCR-2G provided an accurate measure of LA-ICP-MS accuracy, the Department of Terrestrial Magnetism, Carnegie Institution of Washington, USA (DTM) chip of BCR-2G was analysed by LA-ICP-MS at the Research School of Earth Sciences, Australian National University using the LA-ICP-MS protocols described in Jenner and O’Neill  inductively coupled plasma mass spectrometry  at the Department of Terrestrial Magnetism, Carnegie Institution of Washington, USA. Data acquisition was performed with 20 s of background measurement, followed by 40 s of sample ablation using an ablation diameter of 238 µm and a repetition rate of 7 Hz. To minimize the potential for oxide production, the LA-ICP-MS system was tuned to low oxide production rates  region using a standard light source, KBr beam splitter, and MCT detector. Approximately 256 scans were performed for each IR spectrum acquired at a resolution of 4 cm−1. Background spectra were collected under the same analytical conditions before each analysis and used to calculate absorbance by dividing each sample spectrum and then taking the base-10 logarithm. The IR spectra were deconvoluted using the excel version of the DiaMap software ] (20%), platelet band position and height (± 1 cm−1), hydrogen-related band heights at 3107 and 1405 cm−1 (± 1 cm−1). While these uncertainties are very conservative, the reason for the large uncertainty on I(B´) is due to the exceptionally large platelet features that occur in some of the spectra and how variations in the baseline can significantly affect the result  configured for the mid- to near-IR at the Geophysical Laboratory, Carnegie Institution of Washington, USA. Data were collected over the mid-IR (750–7800 cm2 laser in a 30-torr atmosphere of BrF5. Oxygen gas liberated by the fluorination of heated samples was transferred on molecular sieve (5A) frozen by liquid nitrogen to the inlet of a Thermo MAT252™ stable isotope ratio mass spectrometer at the Geophysical Laboratory, Carnegie Institution of Washington, USA. Analytical data are reported as per mil in relation to VSMOW (Vienna Standard Mean Ocean Water) by calibrating against reference materials NBS 28 and UWG-2  were run between each sample to ensure there was no carry-over or a memory effect from un-combusted diamond/minerals. The elemental composition of the samples can account for complete combustion and no sample-to-sample contamination. In-house standards were analysed at regular intervals to normalize and correct these data  and to monitor the accuracy and precision of the measured isotopic ratios and elemental compositions throughout the run. These in-house standards have been calibrated against international and other certified standards. All carbon isotopic values are reported relative to Pee Dee Belemnite (PDB) with the analytical error of ± 0.1‰. To demonstrate the applicability of the method employed here for the 13C/12C determinations of diamond, the results from our technique were compared to data from a single sample of known isotopic composition that has been previously analysed in two different laboratories using two different methods , which is within statistical error of the previous studies that produced an average δ13C value for sample Dia053 of − 20.9 ± 0.5‰  for the garnets vary among different samples over a large range Mg#’s = 47–85). The websteritic garnets from diamondites show Mg#’s intermediate between the eclogitic and peridotitic garnets of the monocrystalline diamonds from Orapa , suggesting the mean values for multiple analyses of the samples represent the bulk value for the sample in question. The only peridotitic sample in the Orapa suite has a δ13C value of − 21.9‰. Although this is not uncommon for peridotitic diamondites , with an average value of + 6.91 (σ ± 0.6)‰ and the sole eclogitic garnet analysed shows a δ18O value of + 6.9‰  are evidence for a long and protracted record of carbonaceous metasomatism in the SCLM   of varying degree , to pairs of nitrogen atoms , to 4 nitrogen atoms tetrahedrally arranged about a vacancy  the 13C-depletion is the product of high-temperature carbon isotope fractionation  are very small and cannot explain a shift for a δ18O value > ± 1‰  with an average value of + 6.9 (± 0.6)‰. There are no δ18O data available for the fibrous diamonds, but the observed 18O-enrichment in the diamondite garnets is akin to what is observed for garnet inclusions in fluid-poor monocrystalline diamonds from Orapa and the neighbouring kimberlites of Damtshaa  from Orapa reveal that the diamond-forming carbon need not share a common origin Fig.  despite The cluster of kimberlites in the Orapa region sample all three main diamond types; monocrystalline, fibrous, and polycrystalline (diamondite). There are important differences for the petrogenesis of these three diamond types. The nitrogen in diamondites show more aggregated states than the fibrous diamonds but are less advanced than most fluid-poor monocrystalline diamonds. Thus, the nitrogen aggregation data offer the possibility that diamondite-forming event(s) at Orapa are younger than most fluid-poor monocrystalline diamonds but older than the fibrous diamond-forming events. The stable C–O isotope data for the Orapa diamondites support a model for a hydrothermally altered and organic carbon-bearing subducted crustal source(s) for the diamond- and garnet-forming media, which is similar to the volatile element source for some of the eclogitic fluid-poor monocrystalline samples, but distinct from the fibrous diamonds. Diamondites form as the result of rapid nucleation of polycrystalline diamond intergrown with silicates and oxide mineral phases (dominantly garnet). The fluids responsible for the precipitation of diamondites, and fluid-poor and fluid-rich (fibrous) monocrystalline diamonds share a geochemical affinity, where all grow from HDFs within the ternary saline-silicic-carbonatitic system. However, while the nature of the parental fluid(s) share a common lithophile element geochemical affinity, the origin(s) of the saline, silicic, and/or carbonatitic components of these HDFs do not always share a common origin.Diamondites preserve the record of a distinct melting and metasomatic event in the SCLM initiated by the subduction of hydrothermally altered and organic carbon-bearing subducted crustal source(s). Therefore, it is wholly conceivable that the diamondites are evidence for a different and temporally unconstrained tectono-thermal diamond-forming event beneath the Kaapvaal craton.The individual EPMA major element analysis on every inclusion/intergrowth analysed in this study (XLSX 124 kb)The individual LA-ICP-MS acquired trace element abundance datum for every inclusion/intergrowth analysed in this study (XLSX 97 kb)Below is the link to the electronic supplementary material.


Subpopulation identification, usually via some form of unsupervised clustering, is a fundamental step in the analysis of many single-cell RNA-seq data sets. This has motivated the development and application of a broad range of clustering methods, based on various underlying algorithms. Here, we provide a systematic and extensible performance evaluation of 14 clustering algorithms implemented in R, including both methods developed explicitly for scRNA-seq data and more general-purpose methods. The methods were evaluated using nine publicly available scRNA-seq data sets as well as three simulations with varying degree of cluster separability. The same feature selection approaches were used for all methods, allowing us to focus on the investigation of the performance of the clustering algorithms themselves. We evaluated the ability of recovering known subpopulations, the stability and the run time and scalability of the methods. Additionally, we investigated whether the performance could be improved by generating consensus partitions from multiple individual clustering methods. We found substantial differences in the performance, run time and stability between the methods, with SC3 and Seurat showing the most favorable results. Additionally, we found that consensus clustering typically did not improve the performance compared to the best of the combined methods, but that several of the top-performing methods already perform some type of consensus clustering.https://github.com/markrobinsonuzh/scRNAseq_clustering_comparison). In addition, an R package providing access to data and clustering results, thereby facilitating inclusion of new methods and data sets, is available from Bioconductor (https://bioconductor.org/packages/DuoClustering2018). All the code used for the evaluation is available on GitHub ( This opens up new possibilities for deconvolution of expression patterns seen in bulk samples, detection of previously unknown cell populations and deeper characterization of known ones. However, computational analyses are complicated by the high variability, low capture efficiency and high dropout rates of scRNA-seq assays11, as well as by strong batch effects that are often confounded by the experimental factor of interest12.Recent advances in single-cell RNA-seq (scRNA-seq) technologies have enabled the simultaneous measurement of expression levels of thousands of genes across hundreds to thousands of individual cells13.Given a collection of single cells, a common analysis task involves identification and characterization of subpopulations, e.g., cell types or cell states. With lower-dimensional single-cell assays such as flow cytometry, cell type detection is often done manually, by visual inspection of a series of two-dimensional scatter plots of marker pairs (“gating”) and subsequent identification of clusters of cells with specific abundance patterns. With large numbers of markers, such strategies quickly become unfeasible, and they are also likely to miss previously uncharacterized cell populations. Instead, subpopulation detection in higher-dimensional single-cell experiments such as mass cytometry (CyTOF) and scRNA-seq is often done automatically, via some form of clustering. As a consequence, a large number of clustering approaches specifically designed for or adapted to these types of assays are available in the literature15, there are to date fewer such studies available for scRNA-seq. The latter is complicated by the large number of different data generation protocols available for scRNA-seq, which in turn has a big effect on the data characteristics. Menon16 specifically evaluated three methods , illustrating their different behavior in low and high read depth data. A recent paper20 compared 12 clustering tools on scRNA-seq data sets from the 10x Genomics platform, showing that different methods generally produced clusterings with little overlap. An overview of several different types of clustering algorithms for scRNA-seq data is given by Andrews and Hemberg21.While extensive evaluations of clustering methods have been performed for flow and mass cytometry dataHere, we extend these initial studies to a broader range of data sets with different characteristics and additionally consider simulated data with different degrees of cluster separability. We evaluate 14 clustering algorithms, including both methods specifically developed for scRNA-seq data, methods developed for other types of single-cell data, and more general approaches, on a total of 12 different data sets. In order to focus on the performance of the clustering algorithms themselves, we use the same preprocessing approach  for all methods, and investigate the impact of the preprocessing separately. In addition to investigating how well the clustering methods are able to recover the true partition if the number of subpopulations is known, we evaluate whether they are able to correctly determine the number of clusters. Further, we study the stability and run time of the methods and investigate whether performance can be improved by generating a consensus partition based on results from multiple individual clustering methods, and the impact of the choice of methods to include in such an aggregation.SC3 andSeurat generally performed favorably, withSeurat being several orders of magnitude faster. In addition,Seurat typically achieved the best agreement with the true partition when the number of clusters was the same, while other methods, likeFlowSOM, achieved a better agreement with the truth if the number of clusters was higher than the true number. Finally, we show that generally, combining two methods into an ensemble did not improve the performance compared to the best of the individual methods.We observed large differences in the clustering results as well as in the run times of the different methods.https://bioconductor.org/packages/DuoClustering2018). In addition, the complete code used to run all analyses is available onhttps://github.com/markrobinsonuzh/scRNAseq_clustering_comparison. The current system uses a Makefile to run a set of R scripts for clustering, summarization and visualization of the results.Given the high level of activity in methods research for preprocessing, clustering and visualization of scRNA-seq data, it is expected that many new algorithms (or new flavors of existing ones) will be proposed. In order to facilitate re-assessment as new innovations emerge and to provide extensibility to new methods and data sets, all (filtered and unfiltered) data sets as well as all clustering results are accessible via an R/Bioconductor package, leveraging the Bioconductor ExperimentHub framework , SRP073808 (Koh24) and GSE52529-GPL16791 (Trapnell25). These data sets were chosen to represent different degrees of “difficulty” in the clustering task. In particular, theTrapnell data set was not generated with the aim of detecting subpopulations, but rather to investigate a continuous developmental trajectory. Nevertheless, it was included in our evaluation as an example of a data set where the phenotype designated as the “true” cluster labels (see below) may not represent the strongest signal present in the data.conquer, the gene-level length-scaled TPM values  and the phenotype were extracted from the MultiAssayExperiment27 object provided byconquer and used to create a SingleCellExperiment object. We also estimated transcript compatibility counts (TCCs) for each of these data sets usingkallisto29 v0.44, and used these as an alternative to the gene-level count matrix as input to the clustering algorithms.Three real scRNA-seq data sets were downloaded fromKumar data set, we grouped the cells by the genetic perturbation and the medium in which they were grown. For theTrapnell data set we used the time point (after the switch of growth medium) at which the cells were captured, and for theKoh data set we used the cell type annotated by the data collectors (obtained through FACS sorting). We note that the definition of the ground truth constitutes an intrinsic difficulty in the evaluation of clustering methods, since it is plausible that there are several different, but still biologically interpretable, ways of partitioning cells in a given data set, several of which can represent equally strong signals. Many public droplet-based data sets contain cell type labels, but these are typically inferred by clustering the cells using the scRNA-seq data, and thus any evaluation based on these labels risks being biased in favor of methods similar to the one used to derive the labels in the first place. By using ground truths that are defined independently of the scRNA-seq assay, we thus avoid artificial inflation of the signal that could result if the truth was derived from the scRNA-seq data itself.The selected cell phenotype was used to define the “true” partition of cells when evaluating the clustering methods. For theconquer, we obtained UMI counts from the Zheng data set5, generated by the 10x Genomics GemCode protocol, fromhttps://support.10xgenomics.com/single-cell-gene-expression/datasets. We downloaded counts for eight pre-sorted cell types  and combined them into three data sets, with a mix of well-separated  and similar cell types  and uniform and non-uniform cluster sizes. For the data set denotedZhengmix4eq, we combined randomly selected B-cells, CD14 monocytes, naive cytotoxic T-cells and regulatory T-cells in equal proportions . For theZhengmix4uneq data set, we combined the same four cell types, but in unequal proportions . For theZhengmix8eq data set, we combined cells from all eight populations, in approximately equal proportions (400–600 cells per population). For these data sets, we used the annotated cell type (obtained by pre-sorting of the cells) as the true cell label.In addition to the data sets fromKumar data set as input, we simulated scRNA-seq data with known group structure, using thesplatter package26 v1.2.0. We generated three data sets, each consisting of 500 cells, with varying degree of cluster separability. For theSimKumar4easy data set, we generated 4 subpopulations with relative abundances 0.1, 0.15, 0.5 and 0.25, and probabilities of differential expression set to 0.05, 0.1, 0.2 and 0.4 for the four subpopulations, respectively. TheSimKumar4hard data set consists of 4 subpopulations with relative abundances 0.2, 0.15, 0.4 and 0.25, and probabilities of differential expression 0.01, 0.05, 0.05 and 0.08. Finally, theSimKumar8hard data set consists of 8 subpopulations with relative abundances 0.13, 0.07, 0.1, 0.05, 0.4, 0.1, 0.1 and 0.05, and probabilities of differential expression equal to 0.03, 0.03, 0.03, 0.05, 0.05, 0.07, 0.08 and 0.1, respectively. The GitHub repository (https://github.com/markrobinsonuzh/scRNAseq_clustering_comparison) contains a link to acountsimQC report30, comparing the main characteristics of the simulated data sets to those of the underlyingKumar data set.Using one subpopulation of thescater package31 v1.6.3 was used to perform quality control of the data sets. Features with zero counts across all cells, as well as all cells with total count or total number of detected features more than 3 median absolute deviations (MADs) below the median across all cells , were excluded. Depending on the availability of manual annotation, we filtered out cells that were classified as doublets or debris. Thescater package was also used to normalize the count values, based on normalization factors calculated by the deconvolution method from thescran package32 v1.6.2, and to perform dimension reduction using PCA33 and t-SNE34. Either the raw feature counts or the log-transformed normalized counts were used as input to the clustering algorithms, following the recommendations by the authors (seeTheSeurat17 to estimate the variability of the features and retained only the most highly variable ones (HVG). Finally, we usedM3Drop35 to model the dropout rate of the genes as a function of the mean expression level using the Michaelis-Menten equation (M3Drop). The gene-wise Michaelis-Menten constants are computed and log-transformed, and the genes are then ranked by their p-value from a Z-test comparing the gene-wise constants to a global constant obtained by combining all the genes. After filtering, we usedscran to renormalize each data set, excluding cells with negative size factors. Supplementary Figure 2 shows the overlap between the retained genes with the different filtering methods, for each of the 12 data sets, and Supplementary Table 1 provides the number of cells retained after each type of filtering.We evaluated three methods for reducing the number of genes provided as input to the clustering methods. For each filtering method, we retained 10% of the original number of genes (with a non-zero count in at least one cell) in the respective data sets. First, we retained only the genes with the highest average expression  value across all cells (denoted Expr below). Second, we usedSeurat andSC3, and methods developed for other types of high-throughput single-cell data (FlowSOM). The collection of methods include representatives for most types of algorithms commonly used to cluster scRNA-seq data. The type of the underlying clustering algorithm for the different methods is shown inFourteen clustering methods, publicly available as R packages or scripts, were evaluated in this study .Seurat instead requires a resolution parameter, which indirectly controls the number of clusters. For each data set, we ran each method with a range of k values . We ranSeurat with a range of resolution parameter values, yielding approximately the range of k values evaluated for the other methods. A subset of the methods provide an estimate of the true number of clusters; we record this estimate for comparison with the true number of subpopulations. For each choice of k (or resolution), we ran each method five times, allowing us to investigate the intrinsic stability of the obtained partitions. Note that the data is the same for all five instances, and thus only the stochasticity of the clustering method affects our stability evaluation. All parameter values except for the number of clusters were set to reasonable values following the authors’ recommendations or the respective manuals  for comparing two partitions50 to evaluate whether the methods preferentially partitioned the cells into clusters of equal size, or whether they preferred to generate some large and some small clusters. Given proportionsp1, . . . ,Np of cells assigned to each ofN clusters, the normalized Shannon entropy is defined byWe used a normalized Shannon entropySince the true degree of equality of the cluster sizes varies between data sets, we subtracted the normalized entropy calculated from the true partition to obtain the final performance index.clue R package51 v0.3-55. Next, for each data set and each imposed number of clusters, we calculated the ARI between the partitions for each pair of methods, and used hierarchical clustering based on the median of these ARI values across all data sets to generate a dendrogram representing the similarity among the clusters obtained by different methods. To investigate how representative this dendrogram is, we also clustered the methods based on each data set separately, and calculated the fraction of such dendrograms in which each subcluster in the overall dendrogram appeared.To evaluate the similarities between the partitions obtained by different methods, we first calculated a consensus partition from the five independent runs for each method, using theclue R package, and used the ARI to evaluate the agreement with the true cell labels. We then compared the ensemble performance to the performances of the two individual methods used to construct the ensemble.Finally, we investigated whether clustering performance was improved by combining two methods into an ensemble. For each data set, and with the true number of clusters imposed, we calculated a consensus partition for each pair of methods using theThe 14 methods were tested on real data sets as well as simulations with a varying degree of complexity  and acroKumar,KumarTCC andSimKumar4easy). When filtering by expression or variability, close to all methods achieved a correct partitioning of the cells in these data sets, while theM3Drop filtering led to a poorer performance for the simulated data set. All methods failed to recover the partition of the cells by time point in theTrapnell data sets, where the ARIs were consistently below 0.5. This indicates that there are other, stronger, signals in this data set that dominate the clustering.As expected, excellent performances were achieved for the well-separated data sets with a strong difference between the groups of cells . At most just over 20% of the cells in the expression and HVG filtering and up to approximately 40% of the cells for theM3Drop filtering were excluded, making a direct comparison between the filterings difficult. Furthermore, the genes retained in theM3Drop and expression filterings showed a low degree of overlap in many of the data sets . Overall, only small differences were seen between the results for the data sets containing gene abundances and those containing transcript compatibility counts (TCCs).We note that theSC3 andSeurat often showed the best performance. These methods were also the only ones that achieved a good separation of the cell types in the droplet-basedZhengmix data sets, which have a much higher degree of sparsity and a larger number of cells than the other data sets. This is consistent with a previous study16 showing thatSeurat performed better than other types of algorithms on data with low read depth. Generally, the performance ofSeurat was also not strongly affected by the gene filtering approach (except for the simulated data sets), while other methods, likeSAFE, were more sensitive to the choice of input genes for some data sets.FlowSOM showed a poor performance for the true number of clusters . However, if the number of clusters was increased, the performance ofFlowSOM improved considerably, and if the methods instead were compared at the number of clusters that gave the optimal performance for each method,FlowSOM showed a better performance .RtsneKmeans, a general-purpose method, showed a higher average performance across the data sets and filterings than many of the clustering algorithms specifically developed for scRNA-seq data. Compared toSC3 andSeurat,RtsneKmeans showed poorer performance for theSimKumar8hard andZhengmix4uneq data sets. The subpopulations in these data sets are nested in the t-SNE space, explaining the difficulty in clustering for the K-means algorithm .While none of the methods consistently outperformed the others over the full range of the imposed numbers of clusters in all data sets,52. No strong association was found for the simulated data sets , indicating that there is low inherent bias in the clustering algorithms. For most of the real data sets, we found highly significant differences in the number of detected features between cells in different clusters. However, it is unclear whether this represents a technical effect or a biological difference between the cell populations.We also investigated whether the number of detected features per cell differed between the clusters, using a Kruskal-Wallis testRtsneKmeans for the same data set  gave slightly shorter run times than applying theSC3 clustering to all cells. The method could potentially be accelerated by using a lower proportion of cells as a training subset. A detailed overview of the run time and the dependence on the number of clusters is given in Supplementary Figure 7 and Supplementary Figure 8. Apart fromSC3 andSC3svm, the imposed number of clusters did not affect the run time.We measured the elapsed time for each run, using a single core and excluding the time to estimate the number of clusters if this was done via a separate function. Since the run times are strongly dependent on the number of features and cells in a data set, we represent them as normalized run times, by dividing with the time required forata set .Seurat Seurat was generally the fastest method, especially for the droplet-based data sets, but at the same time provided among the best partitionings of the data.Plotting the run time versus the ARI for a subset of the data sets   further Zhengmix4uneq) and plotting the run time as a function of the number of cells . The majority of the methods showed a linear increase in run time as a function of the number of cells, whileCIDR andRaceID2 scaled worse. The run time ofSC3 andSC3svm, and to some extentSAFE, showed more complex patterns since these methods reduce the number of random starts of the Kmeans algorithm drastically if the number of cells exceeds 2,000.The scalability of the methods was investigated by subsampling the largest data set , indicating that the selection of genes may be suboptimal.M3Drop led to the exclusion of a large number of cells in theZhengmix data sets, and these cells can not be used for the comparison. For the stable methodsCIDR, TSCAN, ascend andPCAHC, the type of filtering had a relatively large impact on the clustering solutions, and often filtering on the mean gene expression and the gene variability gave more similar clusters than filtering withM3Drop. The stochastic methods showed both a high variability between the individual runs for a given filtering and between runs with different filterings.A summary of the variability both within and between the different filterings is shown in Supplementary Figure 10. It is worth noting that comparing the performances between the different filtering approaches is difficult for two reasons: first, the variability of the clustering runs for a given filtering might exceed the variation between the filterings, and second, filtering withFlowSOM was more variable, and often tended to group the cells into one large cluster and a few very small ones . One consequence of this was thatFlowSOM often showed higher ARI values for a larger number of clusters, while the performance of many of the other methods decreased with increasing k . These methods tended to have more equally sized clusters for larger numbers of clusters than the true number, leading to a higher disagreement between the true classification and the clusterings .By computing the Shannon entropy for the various partitions, we obtained a measure of the equality of the sizes of the clusters . Since tFlowSOM, which required a relatively large number of clusters to reach its maximal performance.Above, we investigated the performance and stability of the methods when the true number of clusters (the number of different labels in the partitioning considered as the ground truth) was imposed. Whether this number of clusters actually provided the highest ARI value  mainly depended on the difficulty of the clustering task , and theSC3, CIDR, ascend, SAFE andTSCAN all have built-in functionality for estimating the optimal number of clusters. In most cases, the estimated number was close to the true one; however,ascend andCIDR had a tendency to underestimate the number of clusters, whileSC3 andTSCAN instead tended to overestimate the number . The tendency ofSC3 to overestimate the cluster number is consistent with a previous publication16. The agreement with the true partition at the estimated number of clusters is shown in Supplementary Figure 13.SC3 is still the best-performing method in this situation.45. EvenSC3 andSC3svm had surprisingly different clusterings; in less than a third of the data sets, these two methods showed the most similar clusterings. In addition, no apparent association between the similarity of the clusterings and the type of input or the dimension reduction or underlying type of clustering algorithm was seen (The similarity between each pair of methods was quantified by means of the ARIs for each pair of consensus clusterings (across the five runs of each method for each data set and number of clusters).as seen .SC3, CIDR, Seurat, tSNE+Kmeans), where the ensemble performance was generally on par with the best individual method45. It is still possible that an ensemble method could provide a general improvement over agiven single method, since it is unlikely that the same method will be the best performing in all conceivable data sets. In fact, among the methods we evaluated, bothSC3 andSAFE combine multiple individual methods to achieve the final clustering result. Studying individual combinations in more detail, we observed that combiningSC3 orSeurat with almost any other method led to a worse performance than obtained by these methods alone (consistent with the observation that they were among the methods giving the best performance). On the other hand, methods likeCIDR, FlowSOM andTSCAN could often be improved by combining them with another method , whileSeurat will determine the number of clusters based on a resolution parameter set by the user.In this study, we have evaluated 14 clustering methods on both real and simulated scRNA-seq data. There were large differences in the ability of the methods to recover the annotated clusters, and performance was also strongly dependent on the degree of separation between the true classes.The same preprocessing steps and fixed gene sets were used for all clustering methods. This enabled us to investigate the impact of the clustering algorithm itself, rather than entire pipelines or workflows. The selection of the filtering approach had an impact on the majority of the methods and resulted in different clustering solutions. Specifically for the more difficult data sets there was a higher dissimilarity. However, this did not necessarily affect the performances of the methods.53 or by simulating technical replicates54. Freytag20 showed thatSC3, Seurat, CIDR andTSCAN were stable under cell-wise perturbations. In our study, we evaluated the methods with respect to their sensitivity to random starts. Overall, the methods showed a high degree of stability across all data sets, except for the simulated data sets in combination with the M3Drop filtering, where the stochastic methods showed a decrease in stability. This may be due to a disagreement between the mean-dropout relationship in the simulated data and the one assumed byM3Drop, leading to a suboptimal gene selection.The stability of clustering algorithms can be evaluated by generating perturbed subsamples of the data set and redoing the clusterings. These subsamples can be created in several ways, e.g., by random subsampling with or without replacement, by adding noise to the original dataascend allows for other alternatives). Thus, no clear advice on the type of algorithm that is best suited for clustering single-cell RNA-seq data can be made based on our results. In fact, the two best-performing methods,SC3 andSeurat, rely on very different underlying clustering algorithms.The evaluated methods are based on a broad spectrum of approaches for dimensionality reduction and clustering. We note that the majority of the methods use PCA or PCoA for dimension reduction or Euclidean distances as the distance metric  data sets, the clustering results from our study and functions for summarizing the performance of different scRNA-seq clustering methods. Additionally, helper functions and descriptions for the evaluation of new methods and data sets are provided.The R/Bioconductor data packageThe package is available under the terms of the GPL (>=2) license.et al. 2018.https://doi.org/10.5281/zenodo.4165026Zenodo: Supplementary Figures and Tables for DuòThis project contains the following extended data:310564e6-a2fe-44c5-8518-698d2633e7df_supp_file_1.pdf Data are available under the terms of the CC-BY 4.0 license.https://doi.org/10.5281/zenodo.1314743Zenodo: Archived R scripts as at time of publication.This project contains the R scripts used to run the analyses presented in this article. Material from this repository are available under the terms of the MIT license.https://github.com/markrobinsonuzh/scRNAseq_clustering_comparison.GitHub: Latest version of R scripts used for analysis.This repository contains the latest version of the R scripts used to run the analyses presented in this article. Material from this repository are available under the terms of the MIT license. I am satisfied with the changes the authors have made. Furthermore, I want to commend the authors for making the data accessible through an R package to facilitate further benchmarking of single cell clustering methods.Is the work clearly and accurately presented and does it cite the current literature?YesIf applicable, is the statistical analysis and its interpretation appropriate?YesAre all the source data underlying the results available to ensure full reproducibility?YesIs the study design appropriate and is the work technically sound?PartlyAre the conclusions drawn adequately supported by the results?YesAre sufficient details of methods and analysis provided to allow replication by others?YesReviewer Expertise:NAI confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. The authors have thoroughly addressed my comments. Based on the new revised results, I have the following minor comment: - The authors note that "We found substantial differences in the performance, run time and stability between the methods, with SC3 and Seurat showing the most favorable results." Please clarify what metric(s) you are using to assess "most favorable results" ie. "most favorable results in terms of overall accuracy in cell-type identification", "in terms of run time", "in terms of stability", etc. Based on your new Supplemental Figure 9, it seems that SC3 would not be favorable in terms of runtime/scalability.  With this minor comment addressed, I find this article now suitable for indexing in F1000 and look forward to the authors' future follow-up work.Is the work clearly and accurately presented and does it cite the current literature?PartlyIf applicable, is the statistical analysis and its interpretation appropriate?PartlyAre all the source data underlying the results available to ensure full reproducibility?YesIs the study design appropriate and is the work technically sound?PartlyAre the conclusions drawn adequately supported by the results?PartlyAre sufficient details of methods and analysis provided to allow replication by others?YesReviewer Expertise:NAI confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. Thank you for your comment. SC3 and Seurat show the highest accuracy when the true number of clusters is used as the input parameter. Also, SC3 has the highest median ARI when the estimated number of clusters is imposed. Similarly, using the number of clusters that gives the maximum performance, SC3 evaluates best. These two methods have both the lowest variability within and between the different filterings. Seurat is stable with regards to random starts and at least for the datasets filtered on the average expression and the variability. Finally, whereas Seurat was generally the fastest method, we agree that SC3 is currently not favorable in terms of the runtime or scalability. Overview The authors present comprehensive benchmarking of clustering tools in R on real and simulated single-cell RNA-seq datasets. Their work includes performance, stability and run time analysis. Furthermore, they also investigate whether combining results from different methods increases performance. Major commentsThroughout the entire manuscript the authors should make it clear that only clustering tools available in R were investigated. This is important, as there are quite a number of popular python applications for clustering of single cell RNA-seq data available.Like Jean Fan, I am concerned about the appropriateness of the Trapnell et al. dataset and the Zheng et al. 10x datasets. Furthermore for the Zheng et al. dataset, I would like to know why the authors did not use all 10 pre-sorted cell populations available? Furthermore, how did the authors choose which cell populations to combine for their Zhengmix4 and Zhengmix8 datasets?  The authors show nicely that Seurat is not very strongly affected by gene filtering. Could this be a result of its clustering approach being based on the 500 most variable genes?On page 7 in the paragraph “Run Times vary widely between methods” the authors use Adjusted Rand Index instead of its already introduced abbreviation Could the size of Figure 5 be increased?Why did some methods get raw and some methods log-transformed normalized counts?Consider changing Supplementary Figure 2 to a visual representation that represents size differences between sets, like UpSetR plots.On page 10 the authors say: ”In addition, no apparent association between the similarity of the clusterings and the type of input or dimension reduction or underlying type of clustering algorithm was found.” Could the authors explain in more detail how this analysis was performed.On page 6, the authors speculate that there are stronger signals that dominate clustering in the Trapnell et al dataset that are not time points. What could these be? Have the authors investigated cell cycle? Minor commentsIs the work clearly and accurately presented and does it cite the current literature?YesIf applicable, is the statistical analysis and its interpretation appropriate?YesAre all the source data underlying the results available to ensure full reproducibility?YesIs the study design appropriate and is the work technically sound?PartlyAre the conclusions drawn adequately supported by the results?YesAre sufficient details of methods and analysis provided to allow replication by others?YesReviewer Expertise:NAI confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above. Thank you for reviewing our manuscript and for your constructive comments. Below are point-by-point responses to the individual comments. Throughout the entire manuscript the authors should make it clear that only clustering tools available in R were investigated. This is important, as there are quite a number of popular python applications for clustering of single cell RNA-seq data available.This has been clarified in the Abstract as well as in the Methods part of the text. Some of the most widely used clustering methods implemented in Python  implement the same or similar clustering methods as those evaluated in this study, and could thus be considered to be implicitly investigated. Also, the evaluation system we provide (via the code in the GitHub repository and the associated data package) is not strictly limited to methods implemented in R; other methods can be included e.g. using system calls.  Like Jean Fan, I am concerned about the appropriateness of the Trapnell et al. dataset and the Zheng et al. 10x datasets. Furthermore for the Zheng et al. dataset, I would like to know why the authors did not use all 10 pre-sorted cell populations available? Furthermore, how did the authors choose which cell populations to combine for their Zhengmix4 and Zhengmix8 datasets?We agree that the Trapnell data set was not generated with the purpose of finding cell types - however, we still find it useful to illustrate the performance of the methods in a data set where the “true clusters” (defined as the time point at which the cells where collected) do not represent the main/strongest signal in the data . We have clarified this in the “Methods-Real data sets” section of the revised paper. For the Zhengmix data sets, our aim was to generate data sets with a mix of well-separated   and similar cell types . In addition, we wanted to investigate if the number of cell populations and/or the equality of the population sizes had an impact on the performance. The included cell type combinations were selected to allow us to address these questions; however, given the richness of this data set, there are certainly many more possible combinations to explore. We have expanded the description in the “Methods-Real data sets” section a bit to highlight these goals.  The authors show nicely that Seurat is not very strongly affected by gene filtering. Could this be a result of its clustering approach being based on the 500 most variable genes?In all our investigations, we preselect the genes that are used as input for each clustering algorithm using three different variable selection methods, and internal variable selection or filtering steps are disabled. Specifically, for Seurat we perform the PCA using all the genes remaining after our filtering, and the clustering is then performed in the principal component space. Thus, the stability of Seurat should be affected in the same way as that of the other methods by the selection of variables.  On page 7 in the paragraph “Run Times vary widely between methods” the authors use Adjusted Rand Index instead of its already introduced abbreviationThanks for noticing this, we now use the abbreviation also here. Could the size of Figure 5 be increased?We have increased the size of Figure 5B. Why did some methods get raw and some methods log-transformed normalized counts?The methods are based on different distributional assumptions and underlying models, affecting the type of values that are most suitably used as input. We followed the recommendations of the authors of the respective methods, and the type of input used for each method is summarized in Figure 4. Consider changing Supplementary Figure 2 to a visual representation that represents size differences between sets, like UpSetR plots.We have replaced the Venn diagrams in Supplementary Figure 2 with UpSet plots.  On page 10 the authors say: ”In addition, no apparent association between the similarity of the clusterings and the type of input or dimension reduction or underlying type of clustering algorithm was found.” Could the authors explain in more detail how this analysis was performed.This conclusion is drawn based on Figure 4, where no association between the clustering of methods by cluster similarity and any of the method characteristics can be seen. This has been clarified in the “Results-Inconsistent degree of similarity between methods” section of the revised paper.  On page 6, the authors speculate that there are stronger signals that dominate clustering in the Trapnell et al dataset that are not time points. What could these be? Have the authors investigated cell cycle?We have not explicitly investigated the interpretation of the strongest signal in the Trapnell data set. However, Supplementary Figure 1 suggests that the annotation that we used to define the “true” clusters (the time at which the cells were collected) does not fully explain the grouping of the cells in the t-SNE visualization . As noted above, the main purpose of including this data set was to investigate the behaviour of the various methods in a data set where the clusters were less apparent. Overview Duo et al compare multiple single-cell RNA-seq clustering approaches on real and simulated single-cell RNA-seq datasets.  Major comments - Quite a number of single-cell RNA-seq datasets are available for benchmarking but only a few were explored here. While an exhaustive interrogation of all single-cell RNA-seq datasets available is beyond the scope of this paper, it would be worthwhile for the readers if the authors could comment briefly on the appropriateness of the datasets used here in terms of their cell-type diversity or other factors that may impact benchmarking. As the authors note, a method's performance is inherently tied to the degree to which the tested subpopulations are truly   transcriptionally distinct. In particular, I am concerned about the appropriateness of the Trapnell dataset, as it was originally intended for pseutotime/trajectory inference and may not even contain discrete transcriptional subpopulations. The poor performance as noted in Figure 1 for this dataset may simply arise from different methods cutting along this continuous trajectory in different ways. Similarly, for the Zheng 10x datasets, since each cell-type was sorted and sequenced separately, there is inevitably some degree of confounding of cell-type specific effects with batch effects that could make clustering much easier.  - As datasets get bigger, the scalability of each method will be an important consideration. The authors provide a preliminary look into this via the different run time of each method in Figure 2, but how this run time depends on the number of cells is unclear. Readers will be interested in whether some methods scale better than others. It is worth having an additional figure of run time as a function of number of cells (via downsampling cells and then extrapolating to larger datasets) to fully capture the scalability of each method.  - With regard to the stability between cluster runs, some methods may internally set various random seeds to ensure reproducibility. Please double check that the stability observed in Figure 3 is not simply the result of which methods uses random seeds. If a method does use an (or likely multiple) internal random seed, the seed must be changed to accurately assess stability.  Minor comments - There are quite a number of single-cell RNA-seq clustering approaches and the list keeps growing (https://github.com/seandavi/awesome-single-cell). Only a fraction is represented in this comparison. While an exhaustive comparison of all methods is beyond the scope of this paper, the authors should comment briefly on how these particular 12 clustering algorithms were chosen. - While nearly all methods assessed use dimensionality reduction as a first step, it is unclear why some were allowed to reduce to 30 dimensions while others 50. It seems that particularly as datasets get larger with presumably more cell-types captured in each datasets, we will likely want to increase the number of PCs to fully capture the variation present in the data. While the authors have left the investigation into the effects of the number of PCs to future research, they should briefly note the reason for the choice of PCs used for each method.Is the work clearly and accurately presented and does it cite the current literature?PartlyIf applicable, is the statistical analysis and its interpretation appropriate?PartlyAre all the source data underlying the results available to ensure full reproducibility?YesIs the study design appropriate and is the work technically sound?PartlyAre the conclusions drawn adequately supported by the results?PartlyAre sufficient details of methods and analysis provided to allow replication by others?YesReviewer Expertise:NAI confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above. Thank you for reviewing our manuscript and for your constructive comments. Below are point-by-point responses to the individual comments. Quite a number of single-cell RNA-seq datasets are available for benchmarking but only a few were explored here. While an exhaustive interrogation of all single-cell RNA-seq datasets available is beyond the scope of this paper, it would be worthwhile for the readers if the authors could comment briefly on the appropriateness of the datasets used here in terms of their cell-type diversity or other factors that may impact benchmarking. As the authors note, a method's performance is inherently tied to the degree to which the tested subpopulations are truly   transcriptionally distinct. In particular, I am concerned about the appropriateness of the Trapnell dataset, as it was originally intended for pseutotime/trajectory inference and may not even contain discrete transcriptional subpopulations. The poor performance as noted in Figure 1 for this dataset may simply arise from different methods cutting along this continuous trajectory in different ways. Similarly, for the Zheng 10x datasets, since each cell-type was sorted and sequenced separately, there is inevitably some degree of confounding of cell-type specific effects with batch effects that could make clustering much easier.There is indeed a large (and increasing) number of public scRNA-seq data sets available, generated with many different types of protocols. However, the main issue  is that no independent annotation of the cells is available, which implies that they are not suitable for unbiased benchmarking like we are doing here. Many public droplet-based data sets do contain “cell type labels”, but these are typically inferred by clustering the cells based on the scRNA-seq data itself, and thus any evaluation risks being biased in favor of methods similar to the one used to derive the labels in the first place. This is the main reason behind the selection of these data sets. We agree that the Trapnell data set was not generated with the purpose of finding cell types - however, we still find it useful to illustrate the performance of the methods in a data set where the “true clusters” (defined as the time point at which the cells where collected) do not represent the main/strongest signal in the data . For the Zheng data set, it’s true that there could be confounding with batch effects, and ambiguous cells may be excluded, which would also make clusters more distinct. For our Zhengmix data sets, we therefore included both very different  and more similar  cell types . We have expanded the discussion in the “Methods-Real data sets” section of the revised paper to clarify these issues.  As datasets get bigger, the scalability of each method will be an important consideration. The authors provide a preliminary look into this via the different run time of each method in Figure 2, but how this run time depends on the number of cells is unclear. Readers will be interested in whether some methods scale better than others. It is worth having an additional figure of run time as a function of number of cells (via downsampling cells and then extrapolating to larger datasets) to fully capture the scalability of each method.Thanks for pointing this out. We have included a plot illustrating the scalability, investigated by downsampling of the largest data set, in Supplementary Figure 9.  With regard to the stability between cluster runs, some methods may internally set various random seeds to ensure reproducibility. Please double check that the stability observed in Figure 3 is not simply the result of which methods uses random seeds. If a method does use an (or likely multiple) internal random seed, the seed must be changed to accurately assess stability.Two of the methods (TSCAN and monocle) set random seeds internally and do not allow these to be changed by the user. Other methods  set a random seed but let the user specify it. For these methods, we explicitly set the random seed to different values in the five runs. We have clarified this in the “Results-High stability between clustering runs” section of the revised text.  There are quite a number of single-cell RNA-seq clustering approaches and the list keeps growing (https://github.com/seandavi/awesome-single-cell). Only a fraction is represented in this comparison. While an exhaustive comparison of all methods is beyond the scope of this paper, the authors should comment briefly on how these particular 12 clustering algorithms were chosen.The methods were chosen to represent the most common types of algorithms used for clustering of scRNA-seq data. We have tried to include the most widely used methods, but also to include methods from tangential fields as well as more traditional clustering methods to serve as a baseline. We have clarified this in the text. While nearly all methods assessed use dimensionality reduction as a first step, it is unclear why some were allowed to reduce to 30 dimensions while others 50. It seems that particularly as datasets get larger with presumably more cell-types captured in each datasets, we will likely want to increase the number of PCs to fully capture the variation present in the data. While the authors have left the investigation into the effects of the number of PCs to future research, they should briefly note the reason for the choice of PCs used for each method.We extracted 50 principal components for the methods that performed an additional dimension reduction (by t-SNE), and 30 principal components for methods where the clustering was done in the principal component space. The only exception was FlowSOM; this was unintentional and has been harmonized in the revised version to use the same number of PCs as the rest of the methods.


Quantifying or labeling the sample type with high quality is a challenging task, which is a key step for understanding complex diseases. Reducing noise pollution to data and ensuring the extracted intrinsic patterns in concordance with the primary data structure are important in sample clustering and classification. Here we propose an effective data integration framework named as HCI (High-order Correlation Integration), which takes an advantage of high-order correlation matrix incorporated with pattern fusion analysis (PFA), to realize high-dimensional data feature extraction. On the one hand, the high-order Pearson's correlation coefficient can highlight the latent patterns underlying noisy input datasets and thus improve the accuracy and robustness of the algorithms currently available for sample clustering. On the other hand, the PFA can identify intrinsic sample patterns efficiently from different input matrices by optimally adjusting the signal effects. To validate the effectiveness of our new method, we firstly applied HCI on four single-cell RNA-seq datasets to distinguish the cell types, and we found that HCI is capable of identifying the prior-known cell types of single-cell samples from scRNA-seq data with higher accuracy and robustness than other methods under different conditions. Secondly, we also integrated heterogonous omics data from TCGA datasets and GEO datasets including bulk RNA-seq data, which outperformed the other methods at identifying distinct cancer subtypes. Within an additional case study, we also constructed the mRNA-miRNA regulatory network of colorectal cancer based on the feature weight estimated from HCI, where the differentially expressed mRNAs and miRNAs were significantly enriched in well-known functional sets of colorectal cancer, such as KEGG pathways and IPA disease annotations. All these results supported that HCI has extensive flexibility and applicability on sample clustering with different types and organizations of RNA-seq data. SC3 is a user-friendly clustering method that works well for smaller datasets. However, it takes too long in terms of computation time because of amount of calculating correlation matrix of cells. Besides, CIDR  now allows for omics analysis of individual cells, which can expose exciting biological processes, novel medical insights and efficient clinical applications , can integrate joint high-order correlation matrices, where the iterative use of Pearson's correlation coefficient in sample data are incorporated into our previously developed pattern fusion analysis method (PFA) , which are not informative for the cell clustering. And, the normalization for each column data is carried to maintain the feature stability of each cell or sample. Then, we can get a filtered expression matrix X.F1, the correlation of the gene expression profiles Xm·n, in which the expressions of m genes are measured for n samples and xkj denotes the expression level of gene k in sample j, the correlation of sample i and j can be calculated by the Pearson correlation coefficient (Wi − ci1T)T. It is important noted that the sensible default values di of matrix X is chosen according to di is the r largest eigenvalues of (Wi − ci1T)(Wi − ci1T)T and the number of the non-zeros eigenvalues is p. Meanwhile, the di-dimension of matrix X.where Y. The detailed adaption method can be seen in the original study ]. The RDC can be calculated as:The ratio of distance between clusters (RDC) is calculated to estimate the number Din is the average sample distance in clusters; Dout is the average sample distance between clusters.where Since the reference labels of cells or samples are already known for all published datasets, the Adjusted Rand Index (ARI) Hubert,  is appliFigure 4A. In the same way, we calculated the high-order matrices XI , where n is number of samples in data. And then we integrated all input datasets XI and high-order correlation matrices Y, we can get the differentially expressed mRNAs (or miRNAs) from heterogeneous genomic datasets according to the coefficient matrix UI*. In this work, we calculated a coefficient of variation for each element on the rows of UI*:The multi-level network is integratively constructed by using HCI schematically shown in i is the average weight of mRNA i (or miRNA i) in UI*, and δi is the standard deviation. We can define differentially expressed mRNA (or miRNA) i if ci is greater than a given threshold T, and they called DEGs (or DE-miRNAs).where μBesides, we also performed functional enrichment analysis for genes by Gene Ontology and KEGG. We also analyzed DEGs using Ingenuity Pathway Analysis (IPA), providing the association between a particular gene set and known functions, pathways, networks and associated diseases. An online database miRDB was used for miRNA target prediction and functional annotations.We defined key genes that significantly enriched in cancer dependent on KEGG, GO and IPA analysis. We found the key genes in the DEGs, which can be linked and correlated by the combined functional couplings of protein-protein interactions of STRING. MicroRNAs which can regulate key DEGs were defined as key miRNAs (degree s > 80) . Besides, in order to determine the robustness as a consistent performance under different conditions, the same analysis on four datasets were both repeated 50 times under different systematic conditions  respectively, where CV genes mean ones with largest expression variances. Similarly, the performance of HCI under different correlation matrices or conditions was better  than other methods according to the ARI and the RDC , provides the reasonable cluster number of cells. When we applied HCI to the whole cells included 430 annotated cells and 205 dropped cells, the results suggested that the K is 7. Obviously, there are potential new cell types included, and we found there were 27 annotated mesenchymal cells in the ambiguous cells. This result also showed that the other ambiguous cells can be clustered well in seven cell types separately  and Adrenocortical carcinoma (ACC), and one from the GEO   and disease (42 samples) can be clustered into two discriminative groups  (2018)P = 3.86e-05), beta-catenin destruction complex (P = 1.57e-04), colorectal cancer (P = 2.73e-46), and pathways in cancer (P = 6.82e-41). We also found that the hub genes  in this network have been linked to cancer in wide literature reports. For example, the MAPK-signaling pathways have been identified as one of the most strongly associated gene markers to colorectal cancer (CRC)  P = 3.86eThe distinct types of biological data could provide a precise explanation for understanding the complex biological processes (Ghazalpour et al., As genomic data sources is increasing in diversity and volume, HCI can fit the data structures on both one level data or multiple level data, so that, HCI could provide new avenues for the systematic explanation of various data and complex biological phenotypes at a system-wide level. Indeed, there are still a few future topics to further extend HCI method, e.g., integrating discrete data types including somatic, SNP, and CNV information.HT and TZ developed the methodology. HT executed the experiment. HT and TZ carried out the data analysis and wrote this paper. HT, TZ, and LC revised the manuscript. LC and TZ supervised the work, and LC critically reviewed the paper. All authors read and approved the final manuscript.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.