Lung cancer is one of the most commonly diagnosed cancer worldwide. As one of the liquid biopsy analytes, alternations in cell-free DNA (cfDNA) methylation could function as promising biomarkers for lung cancer detection.  In this study, differential methylation analysis was performed to identify candidate markers, and lasso regression with 10-fold cross-validation (CV) was used to establish the diagnostic marker panel. The performance of the binary classifier was evaluated using the receiver operating characteristic (ROC) curve and the precision-recall (PR) curve.  We identified 4072 differentially methylated regions (DMRs) based on cfDNA methylation data, and then a 10-DMR marker panel was established. The panel achieved an area under the ROC curve (AUROC) of 0.922 and an area under the PR curve (AUPR) of 0.899 in a cfDNA cohort containing 29 lung cancer and 74 normal samples, showing outstanding performance. Besides, the cfDNA-derived markers also performed well in primary tissue datasets, which were more robust than the tissue-derived markers.  Our study suggested that the 10-DMR marker panel attained high accuracy and robustness and may function as a novel and promising target for lung cancer detection. Lung cancer is one of the leading causes of cancer death in the world . Tissue Cell-free DNA (cfDNA), one of the liquid biopsy analytes, refers to extracellular DNA fragments mainly released from cells through apoptosis and necrosis . BesidesDNA methylation, a crucial epigenetic modification, plays an imperative role in the biological process . AberranIn this study, we explored the specific cfDNA methylation markers that distinguish patients with lung cancer from healthy individuals significantly. Four independent cohorts were employed in the analytical pipeline, including two cfDNA datasets and two tissue datasets. Differential methylation analysis was performed to screen out candidate markers in the discovery phase, and then lasso regression with 10-fold CV was performed to identify diagnostic markers in the model construction phase. Analyses showed that tissue-derived markers did not perform as well in cfDNA. However, cfDNA-derived markers achieved an AUROC greater than 0.90 in both cfDNA and tissue datasets, implying the potential benefits offered by identifying markers directly from cfDNA samples.p value < 0.01 as a cutoff. Poor performing probes with detection p value < 0.01 and those associated with SNPs or located on sex chromosomes were also filtered out. The level-3 DNA methylation data of the LUNG cohort were downloaded from the UCSC Xena platform to create tissue set 2, with sex chromosome and cross-reactive probes removed for downstream analysis. The cohort was profiled by the Illumina HumanMethylation450 platform and included 832 lung cancer and 75 normal samples. CfDNA set 1 was retrieved from GSE122126, comprised of MethylationEPIC data of 4 NSCLC and 12 normal samples. Raw data from IDAT files were read in, and the processing procedures were as described above for tissue set 1. Then, data were normalized using the preprocessFunnorm function. CfDNA set 2 was comprised of processed reduced representation bisulfite sequencing (RRBS) data from Cell-Free Epigenome Atlas (CFEA) database [Four independent datasets were used in differential methylation analysis and model construction phase, two for tissue samples and the others for cfDNA samples. Tissue 450 k data of 121 NSCLC and 12 normal samples were obtained from GSE56044 to create tissue set 1, and low-quality samples were removed using mean detection database . The RRBThe preprocessing procedures were conducted using the minfi package in R 4.0.2. The hg19 human reference genome was used for annotation. Differentially methylated regions or positions were determined using cfDNA, and tissue sets 1 and diagnostic models were finally trained on cfDNA and tissue sets 2.M-value for differential analysis of methylation levels and model construction. Differentially methylated positions (DMPs) between lung cancer and nontumor samples were determined using empirical Bayesian methods in the limma package, with ∣log2FC | >2 and adjusted p value < 0.05 considered statistically significant. To improve detection reliability, the exact workflow was applied to both datasets, and the overlaps between DMPs of two tissue sets with the same alternation patterns were considered as potential biomarkers.Beta-value of tissue sets was converted to Methylation data of potential markers in tissue set 2 was then partitioned into 10 disjoint groups randomly. Nine subsets were used as the training set, and the remaining subset was used as the validation set. Lasso regression was then first trained on the training set and assessed on the validation set using the glmnet package. The procedure was repeated 10 times, with each of the 10 subsets used as the validation set, to improve the data efficiency.M-value located in such regions separately. Parameters dist.thresh and bp.thres.clust were set to 1 and 300, referring to similarity distance threshold and maximum length between neighbor CpG sites. The procedure rendered tissue 450 k data and cfDNA RRBS data comparable. Multiple imputation was then performed to fill in missing values in RRBS data, and the diagnostic model was also built using lasso regression.The performance of tissue-derived markers was also evaluated in cfDNA samples. Considering the low coverage of RRBS data in cfDNA set 2, A-clustering algorithm was implemented to detect regions of coregulated CpG sites and merge Differential methylation analysis was performed on cfDNA set 1 using the DMRcate package. Two crucial parameters lambda and C were set to 1000 and 2, referring to Gaussian kernel bandwidth and scaling factor for bandwidth. DMRs were determined with the thresholds of HMFDR < 0.01 and treated as a source of candidate markers. Then, cfDNA set 2 was imported following the strategy below. For CpG sites located in each DMR location, the methylated and unmethylated intensity values in RRBS data were merged to measure the average methylation. DMR locations where methylation data of more than 60% samples were missing were removed for low quality. Wilcoxon signed-rank test was performed on cfDNA set 2 between lung cancer and normal samples to select potential diagnostic markers. The strategy solved the problem caused by the low coverage of RRBS data and rendered the selection more robust to some extent. Multiple imputation was implemented to fill in missing values using the mice package.Lasso regression with 10-fold CV was implemented to build cfDNA diagnostic models using the glmnet package. Parameters family and type.measure were set to “binomial” and “auc” to fit the model. Cross-validation could prevent overfitting and improve the generalization capability of the classification model. To figure out the robustness of makers identified, the cfDNA-derived diagnostic markers were validated in tissue samples. The genomic positions of cfDNA-derived markers were mapped to 450 k data from tissue set 2 using findOverlaps function in GenomicRanges packages. Then, lasso regression with 10-fold cross-validation was performed to construct the diagnostic model.ROC curve was used to evaluate the performance of marker panels derived from two different sources by plotting sensitivity against specificity at various thresholds. AUROC could reflect the probability that the rank of positive instance was higher than that of negative instance predicted by the classifier. Considering the imbalance of datasets in the model construction phase, PR curve and AUPR, an evaluation tool summarizing the trade-off between precision and recall, were used as an alternative evaluation metric. A higher AUPR value indicates better binary classifier performance, which is similar to AUROC.p value was calculated to assess the statistical significance of enrichment and the performance of markers in isolating lung cancer samples.XL-minimal hypergeometric (XL-mHG) test was also implemented to assess the discrimination ability of each marker. The method treated lung cancer and healthy group as two clusters and binarized methylation data of markers in a marker-specific and cluster-specific manner. Then, XL-mHG p values <0.1 were considered to be enriched significantly using the hypergeometric test.Functional enrichment analysis, based on the Gene Ontology (GO) knowledgebase and Kyoto Encyclopedia of Genes and Genomes (KEGG) database, was performed on the genes whose predefined promoters overlapped with DMRs identified. GO classifies functions in the aspect of molecular function, cellular component, and biological process, and KEGG provides the information of molecular interaction and reaction networks. GO terms or KEGG pathways with BH-adjusted TRAF1, RPTOR, and SPON2) were determined using data analytics in cfDNA and pyrosequencing in tissue. Whole genome bisulfite sequencing (WGBS) data of cfDNA was retrieved from the European Nucleotide Archive (ENA), including 5 lung cancer and 45 normal samples . The WGBS data was preprocessed in the same workflow for RRBS data. Then, Kolmogorov-Smirnov test (K-S test) was performed to validate that methylation levels were different between lung cancer and normal samples. Genomic DNA isolated from 3 lung cancer vs. 3 control tissue samples was bisulfite-converted using EZ DNA Methylation-Gold™ Kit. And the procedure was approved by the Ethics Committees on Human Research of Shanghai Chest Hospital (Approve ID: KS(Y)1987). Then, PCR was performed with biotinylated primers designed by PyroMark Assay Design Software 2.0 , of which 2707 (66.48%) regions were hypomethylated and 1365 (33.5%) regions were hypermethylated. The methylation patterns of the 4072 DMRs were significantly different, according to which the normal and lung cancer samples were separated into two clusters . Across M-value is more statistically valid, log2 ratios of methylated and unmethylated reads were calculated. Then, 16 candidate markers that distinguished between lung cancer and normal states were selected using the Wilcoxon test with p value < 0.001 and ∣log2FC | >1.2. Lasso regression was performed on the remaining candidate markers using the 10-fold CV technique. Finally, a 10-DMR diagnostic model was constructed in the form of a linear equation. The detailed information was listed in LYPD8, VPS13D, AMPD3, RPTOR, TRPV2, VGLL4, SPON2, CXXC5, and TRAF1. To assess the performance of the binary classifier, ROC and PR curves were created  are significantly enriched in biological processes related to energy metabolism and cell behavior, such as regulation of GTPase activity and cell-cell adhesion . KEGG paTRAF1, RPTOR, and SPON2 in cfDNA samples and normal controls  and 3134 DMGs of patients with lung cancer compared to healthy individuals using the Infinium MethyaltionEPIC array data. The biological functions of DMGs were significantly enriched in physiological and pathological processes, including TNF signaling pathway, Notch signaling pathway, and Fc gamma R-mediated phagocytosis. Then, we merged the M-value of DNA methylation data at a dynamic cutoff and introduces two parameters X and L to control the number of true positives and false positives [p value. Based on hg19, DMR_7 was annotated as TNF Receptor Associated Factor 1 (TRAF1), a member of the TRAF family. TRAF1 is crucial in activating the BRAF/MEK/ERK signaling pathway and nonsmall cell lung (NSCLC) carcinogenesis [TRAF1 in human lung cancer cells, which is consistent with our analysis outcome. Knockdown of TRAF1 could inhibit proliferation of NSCLC cells and induce cellular apoptosis, and thus, TRAF1 was a promising diagnostic and therapeutic target for NSCLC.Inspired by single-cell sequencing data clustering, the XL-mHG test was introduced to evaluate the single marker considering the similarity between classifying patients and cell types. As a rank-based, nonparametric method, XL-mHG binarizes the ositives . Consequogenesis . ResearcWe then performed lasso regression on the 36 CpG sites from tissue set 2 that was overlapping with DMRs of marker panel to assess the performance of cfDNA-derived markers in tissue samples. An AUROC of 0.973 and an AUPR of 0.990 were obtained, suggesting the robustness of 10 DMR markers. However, the cfDNA-derived panel may not be the optimal solution to the lasso regression based on the methylation levels of tissue samples, so it was different from the tissue-derived panel. Given that the majority of studies concerning the identification of cfDNA based markers start with primary tissue, we also evaluated tissue-derived marker panel in RRBS data. Although AUROC was calculated to be 0.863, AUPR was only 0.363 because of the highly imbalanced data. In fact, only one sample was predicted to be the patient in the dataset which included 29 lung cancer and 74 normal samples indeed. Therefore, our findings suggest that markers derived directly from cfDNA samples would be more robust than those derived from primary tissue.EGFR mutation in plasma [Apart from epigenetic features, tumor-related elements have various forms in blood circulation . Alteratn plasma . Proteinn plasma . Cancer n plasma .We also acknowledged several limitations in our study. First, the cfDNA set 2 is sparse, and the distribution of detected CpG sites has a low consistency across samples. Second, the sample size of the cfDNA dataset is relatively small, which requires future studies in a larger independent cfDNA database to validate our findings and improve the generalization ability of the cfDNA marker panel. Finally, RRBS and the Infinium array both have limited coverage of CpG sites considering that around 28.3 million CpG sites are present in the human genome, and thus, more accurate detection techniques with high coverage for DNA methylation are needed .In summary, our study performed a genome-wide methylation analysis on cfDNA methylation data directly. A novel 10-DMR marker panel was constructed with high sensitivity and specificity both in cfDNA and primary tissue datasets, which could be heralded as promising diagnostic markers in lung cancer.


We use Schema to infer cell types by integrating gene expression and chromatin accessibility data; demonstrate informative data visualizations that synthesize multiple modalities; perform differential gene expression analysis in the context of spatial variability; and estimate evolutionary pressure on peptide sequences.A complete understanding of biological processes requires synthesizing information across heterogeneous modalities, such as age, disease status, or gene expression. Technological advances in single-cell profiling have enabled researchers to assay multiple modalities simultaneously. We present Schema, which uses a principled  High-throughput assays can now measure diverse cellular properties, including transcriptomic –3, genomSimultaneous multimodal experiments present a new analytic challenge of synthesizing agreement and disagreement across modalities. For example, how should one interpret the data if two cells look similar transcriptionally but are different epigenetically? Moreover, given the rapid biotechnological progress that continues to enable novel measurement modalities and easier simultaneous multimodal profiling, a multimodal analysis paradigm should scale to massive single-cell datasets, be robust to noise and sparsity in the data, and be able to synthesize two or more arbitrary modalities in an interpretable way. Many existing methods, however, struggle with scalability, overfitting, or are specialized to specific multimodal tasks  important to its binding specificity. Schema is thus designed to support the continually expanding breadth of single-cell technologies while retaining the power, tunability, and interpretability required for effective exploratory analysis.across modalities in a mutually consistent way. Our key intuition is that each modality gives us information about the biological similarity among cells in the dataset, which we can mathematically interpret as a modality-specific distance metric. For example, in RNA-seq data, cells are considered biologically similar if their gene expression profiles are shared; this may be proxied as the Euclidean distance between normalized expression vectors, with shorter distances corresponding to greater similarity.Before the advent of multimodal single-cell experiments, computational analysis has focused on variation within a single modality. In contrast, analysis of simultaneous multimodal single-cell experiments  critically requires reasoning about information metric learning  modality, consisting of observations that are mapped to points in a multi-dimensional space. In the analyses presented here, we typically designate the most informative or high-confidence modality as the primary or the reference modality, with RNA-seq being a frequent choice , a common approach in statistics for inferring information from cross-covariance matrices, or autoencoders, a deep learning approach for mapping multiple datasets to a shared latent space [In our approach, the researcher starts by designating one of the modalities as the t choice . The coont space –30, is pTo see how Schema’s transformation synthesizes modalities, consider the case where the primary dataset is gene expression data. While the points close in Euclidean space are likely to be biologically similar cells with shared expression profiles, longer Euclidean distances are less informative. Schema’s constrained optimization framework is designed to preserve the information contained in short-range distances, while allowing secondary modalities to enhance the informativity of longer distances by incorporating, for example, cell-type metadata, differences in spatial density, or developmental relationships. To facilitate the representation of complex relationships between modalities, arbitrary distance metrics and kernels are supported for secondary modalities.Schema’s measure of inter-modality alignment is based on the Pearson correlation of distances, which is optimized via a quadratic programming algorithm, for which further details are provided in “We first sought to demonstrate the value of Schema by applying it to the increasingly common and broadly interesting setting in which researchers simultaneously profile the transcriptome and chromatin accessibility of single cells . FocusinWith RNA-seq as the primary  dataset and ATAC-seq as the secondary, we applied Schema to compute a transformed dataset in which pairwise RNA-seq distances among cells are better aligned with distances in the ATAC-seq peak counts data while retaining a very high correlation with primary RNA-seq distances ≥ 99%, “”. We thep = 6.7 × 10− 15), leading to an overall improvement in clustering quality , thus emphasizing the RNA-seq subspace where support from the secondary modality signal was strongest. These genes  are enriched for regulation of macromolecule metabolic process  and regulation of nitrogen compound metabolic process .To further analyze why combining modalities improves cell type clustering, we obtained Leiden cluster labels using either the RNA-seq or the ATAC-seq modalities individually. We then evaluated these cluster assignments by iterating over subsets of the data, each set covering only a pair of ground truth cell types and used the ARI score to quantify how well the cluster labels distinguished between the two cell types. While RNA-seq clusters have higher ARI scores overall, indicating a greater ability to differentiate cell types, ATAC-seq does display a relative strength in distinguishing proximal tubular (PT) cells from other cell types Fig. a. PT celity Fig. e. FurtheIn general, synthesis of multimodal data can also be done by statistical techniques like canonical correlation analysis (CCA) or deep learning architectures that represent multiple modalities in a shared latent space –33. A keTo evaluate an autoencoder-based synthesis of these modalities, we applied scVI  and totaWhile these CCA and autoencoder results were likely due to overfitting, the Schema-based synthesis constrains the ATAC-seq modality’s influence, enabling us to extract additional signal provided by ATAC-seq while preserving the rich information provided by the transcriptomic modality. We believe that this regularization offered by Schema’s constrained optimization formulation is a key advantage that will be crucial in multimodal single-cell data synthesis. We also note that Schema offers additional advantages: unlike CCA, it can incorporate more than two modalities simultaneously and, unlike totalVI, its synthesis is interpretable, revealing a more accurate characterization of PT cells.Drosophila melanogaster neurons [Another powerful use of Schema is to infuse information from other modalities into RNA-seq data while limiting the data’s distortion so that it remains amenable to a range of standard RNA-seq analyses. Since widely used visualization methods such as UMAP  do not a neurons  profiledWhile some age-related structure does exist in the original data, Schema-based transformation of the data more clearly displays a cellular trajectory consistent with biological age Fig. . ImportaRm62, CG5010 and IscU) active at the start while ribosomal genes like Rpl22 and Rpl23A were active at the end. We also confirmed that there was a significant overlap between Schema’s highest-ranked genes and those found by a standard differential expression test between timepoints . To additionally verify that Schema was infusing additional age-related structure into RNA-seq data, we performed a diffusion pseudotime analysis of the original and transformed datasets and found that the Spearman rank correlation between this pseudotime estimate and the ground truth cell age increased from 0.365 in the original data to 0.405 and 0.436 in the transformations corresponding to minimum correlation constraints of 0.999 and 0.99, respectively.Analysis of Schema’s feature selection indicated an up-weighting of genes differentially expressed at the start or end of the aging process Fig. e, with gWe note that the constrained optimization of Schema was again important to retaining biological signal during the synthesis: in comparison, an unconstrained synthesis by CCA led to a lower pseudotime correlation (0.059) than seen in the original RNA-seq dataset; the corresponding CCA-based UMAP visualization was also less clear in conveying the cellular trajectory , and thus cell types are located throughout the tissue , 38. Intand in dense regions, but not all granule cells are in dense regions and not all cells in dense regions are granule cells. We specified RNA-seq data as the primary modality and spatial location and cell-type labels as the secondary modalities. In the spatial location modality, the distance metric was defined such that two cells are similar if their spatial neighborhoods have similar density , ion transport , and electron transfer . This finding suggests potentially greater neurotransmission activity within these cells  and with two methods specifically intended for spatial transcriptomics, namely SpatialDE  and Trenthree modalities: gene expression, cell-type labels, and spatial density. We adapted CCA by correlating two modalities at a time and combining the sub-results (density variation). To adapt these, we collated their results from separate runs on granule and non-granule cells . We thus find that Schema’s results are substantially more robust across the three replicates. Compared to CCA’s unconstrained synthesis, Schema’s constrained formulation avoids overfitting to sample-specific noise, enhancing its robustness  relates to antigen binding specificities . De novoTo estimate location-specific selection pressure, we ran Schema with the CDR3 peptide sequence data as the primary modality and the binding specificity information as the secondary modality, performing separate runs for α and β chains. In the primary modality, each feature corresponds to a CDR3 sequence location and we used the Hamming distance metric between observations . The residue and location preferences estimated here can directly be used in any algorithm for computational design of epitope-specific CDR3 sequences to bias its search towards more functionally plausible candidate sequences.Next, we used Schema to investigate the selection pressure on amino acids present in the variability-prone locations identified above . We firsSchema’s ability to efficiently synthesize arbitrarily many modalities, with their relative importance at the researcher’s discretion, allows information that might otherwise be set aside  to be effectively incorporated, enhancing the robustness and accuracy of an analysis. In Applying Schema on a mouse gastrulation dataset  consistiWe have designed Schema to process large single-cell datasets efficiently, with modest memory requirements. On average, Schema processes data from a Slide-seq replicate  in 34 min, requiring less than 5 GB of RAM in the process  modality that drive the integration.Our approach enables the researcher to supervise the synthesis by choosing which modality to transform, the degree to which it can be distorted, and the desired level of agreement between modalities. While existing methods like Seurat v3  and LIGEWhen choosing a primary modality, we generally recommend selecting the most high-confidence modality or the one for which feature selection will be most informative, though it can sometimes be productive to integrate insights across multiple invocations of Schema with varying primary modality choices. In many of our demonstrations, we chose RNA-seq as the primary modality since it is often the modality where preprocessing and normalization are best understood, boosting our confidence in it; additionally, transformed RNA-seq data lends itself to a variety of downstream analyses. Once a primary modality has been designated, Schema can synthesize an arbitrary number of secondary modalities with it. In contrast, methods designed around pairwise modality comparison need ad hoc adaptations to accommodate additional modalities. Schema’s approach is advantageous not only for datasets with more than two modalities , 51 but Intuitively, our correlation-based alignment approach has parallels to kernel canonical correlation analysis (kernel CCA), a generalization of CCA where arbitrary distance metrics can be specified when correlating two datasets. While Schema offers similar flexibility for secondary modalities, it limits the primary modality to Euclidean distances. Introducing this restriction enhances scalability, interpretability, and robustness. Unlike kernel CCA, the optimization in Schema operates on matrices whose size is independent of the dataset’s size, enabling it to scale sub-linearly to massive single-cell datasets. Also, the optimal solution is a scaling transform that can be naturally interpreted as a feature-weight vector. Perhaps most importantly, Schema differs from kernel CCA in performing a constrained optimization, thus reducing the distortion of the primary dataset and ensuring that sparse and low-confidence secondary datasets do not drown out the primary signal.The constrained optimization in Schema acts as regularization, helping ensure that the computed transformation and feature selection remain biologically meaningful. By choosing a high-confidence modality as the primary modality and bounding its distortion when incorporating the secondary modalities, Schema enables information synthesis while retaining high-confidence insights. This bound on the distortion is an important parameter, directly controlling how much the secondary modalities inform the primary dataset; values approaching 1 will increasingly limit the influence of the secondary modalities. Therefore, we recommend that studies using Schema for feature selection should aggregate the results over a range of values of this parameter while analyses that utilize only a single parameter should keep it high  to preserve fidelity with the original dataset . If suffInteresting future methodological work could explore alternative formulations of the Schema objective, potentially including more complex nonlinearities than our quadratic program formulation. Schema can also be used in conjunction with data-integration methods , 17 desihttp://schema.csail.mit.edu and as the Python package schema_learn.Given the current pace of biotechnological development, we anticipate that high-throughput experiments, and their conclusions, will increasingly rely on more than one data modality, underscoring the importance of Schema and its conceptual framework. Schema is publicly available for use at Underlying our definition of the alignment of metrics is the intuitive notion that metrics are similar if the ordering of pairwise distances between the two metrics are close. A proxy for measuring this alignment is the Pearson correlation coefficient. For Schema, the goal is thus that pairwise distances in the transformed space be highly correlated with pairwise distances under each metric.quadratic programming (QP). QP is a generalization of linear programming, allowing a quadratic objective function. We learn a scaling transformation u  are highly correlated with the pairwise distances in the secondary modalities. We codify our intuition of the importance of the primary dataset by requiring that the correlation of transformed pairwise distances with the original dataset be higher than some researcher-specified threshold. The scaling transformation has the appealing property of being interpretable as a feature selection: the higher the coordinate ui, the more important that coordinate is for alignment. Thus, by selecting the top coordinates by their weights, we can access the genes most important for aligning the modalities.One of the advantages of the Pearson correlation coefficient is that it is amenable to optimization via N observations across r datasets Dj, j = 1, 2, …, r, where D1 as the primary dataset and the rest as secondary. Each dataset’s dimensionality and domain may vary. In particular, we assume D1 is k-dimensional. Each dataset Dj should also have some notion of distance between observations attached to it, which we will denote ρj, so n and m in Dj. Since our entire framework below deals in squared distances, for notational convenience we will let ρj be the squared distances between points in Dj; also, we drop the superscript in D1 and its data.Suppose we have Ω with Ω(D) generating a dataset D∗ such that the Euclidean metric ρ∗ on D∗ aligns the various metrics ρj, each informed by its respective modality. Non-Euclidean primary distance metrics ρ1 are allowed if they can be computed as a sum of k terms, one for each feature . We emphasize that none of the secondary ρj need to be Euclidean. This setup is quite general, and we now specify the form of the transformation Ω and the criteria for balancing information from the various metrics. Here, we limit Ω to a scaling transform. That is, Ω(D) = {diag(u)x : x ∈ D} for some u ∈ Rk and diag(u) is a k × k diagonal matrix with u as its diagonal entries. Then, the squared distance between points under the transformation is given byThe goal is to find a transformation w is the element-wise square of u, i.e., u acts as a feature-weighting mechanism: it chooses the features of D1 that align the datasets best . We note here that a natural extension would be allowing general linear transformations for Ω; however, in that context, the fast framework of quadratic programming would need to be substituted for the much slower framework of semidefinite programming.where ρj is to learn a metric ρ∗ that aligns well with all of them. Our measure of the alignment between ρ∗ and ρj is given by the Pearson correlation between pairwise squared distances under two metrics. Intuitively, maximizing the correlation coefficient encourages distances under ρ∗ to be large when the corresponding ρj distances are large and vice versa. This can be seen from the expressionHere, our approach to integration between the metrics ρ∗ and the distances on each of the metrics, allowing the user to specify how much each modality should be weighted. We also allow a hard constraint, whereby the correlation between the pairwise distances in the transformed data and in the primary dataset is lower-bounded. Our goal is thus to findγj and s are hyperparameters that determine the importance of the various metrics. We have also highlighted that ρ∗ is a function of w and is determined entirely by the solution to (2). In the rest of our discussion, we will abuse notation and primarily use w, rather than ρ∗, to refer to the optimal metric. The machinery of quadratic programming makes this optimization feasible.To deal with multiple modalities, we try to maximize the correlation between Q, q, G, h, A, and b, where Q is a positive semidefinite (psd) matrix and the notation ≼ means the inequality is true for each coordinate .As motivated above, quadratic programming (QP) is a framework for constrained convex optimization problems that allows a quadratic term in the objective function and linear constraints. The general form islinear in the transformation and variance is quadratic:al and bl are k-dimensional vectors that depend only on D1 and Dl; and S and T are N × k matrices that depend only on D1; and P is the set of pairs of observations, where |∙| denotes set cardinality. It is also not hard to show that To put our optimization (2) in a QP formulation, we expand the covariance and variance terms in the definition of correlation in 1) and show that the covariance is  and showquotient of the covariance and the standard deviation, and the QP framework cannot handle quotients or square roots. However, maximizing a quotient can be reframed as maximizing the numerator (the covariance), minimizing the denominator (the variance), or both.There is one more difficulty to address. The correlation is the λ is the hyperparameter for regularization of w, which we want to penalize for being too far away from the all-ones vector . One could also regularize the l2 norm of w alone , which would encourage w to be small; we have found that empirically the choices yield similar results.We now have the ingredients for the QP and can frame the optimization problem asw∗ can be used to transform unseen input data, using u∗ ∈ Rk, where This program can be solved by standard QP solvers , we can access them by varying the hyperparameters λ controls whether w satisfies q and that α and β control whether the correlation constraint s is satisfied. To satisfy these constraints, we simply grid search across feasible values of {λ, α, β}: we solve the QP for fixed values of λ, α, and β, keeping only the solutions for which the {s, q} constraints are satisfied. Of these, we choose the most optimal. The efficiency of quadratic programming means that such a grid search is feasible, which gives users the benefit of more easily interpretable and natural hyperparameters.Intuitively, we note that the choice of s (minimum correlation) and not q (maximum feature weight) be used to control Schema’s optimization. The default value of q in our implementation is set to be very high (103) so that it is not a binding constraint in most cases. We recommend not changing it and in future versions of Schema we may reformulate the QP so that q is entirely removed. To limit the distortions in the primary modality, we recommend that s be set close to 1: the default setting of s is 0.99 and we recommend values ≥ 0.9. When Schema is used for feature selection, we recommend aggregating results across an ensemble of runs over a range of s values (a wide range is recommended here) to increase the robustness of the results.We recommend that only u that we generate can be interpreted as a feature-weighting mechanism, we can identify the directions (in PCA) or factors (in NMF) most relevant to aligning the datasets. Here the user can employ arbitrary feature sets including, for instance, a union of features from two standard methods  or those generated by another single-cell analysis method, like MOFA+ 21.Standard linear decompositions, like PCA or NMF, are useful as preprocessing steps for Schema. PCA is a good choice in this regard because it decomposes along directions of high variance; NMF is slower but has the advantage that it is designed for data that is non-negative . Since the transform ϵ, the fraction of misaligned neighborhood relationships will be less than manifold interpretation that is also compelling: assuming the high-dimensional data lies on a low-dimensional manifold, small Euclidean distances are more accurate than large distances, so the local neighborhood structure is worth preserving. We can show intuitively that optimizing the correlation aims to preserve local neighborhood structure. Using correlation in the objective also affords the flexibility to broaden Corr in (2) to any function fj of the metric, i.e., Corr; this allows us to invert the direction of alignment or more heavily weigh local distances. As RNA-seq dataset sizes reach millions of cells, even calculating the O(N2) pairwise distances becomes infeasible. In this case, we sample a subset of the pairwise distances. As an estimator, sample correlation is a robust measure, allowing Schema to perform well even with relatively small subsets; in fact, we only need a sample size logarithmic in our desired confidence level to generate high-confidence results  on the same dimensionality-reduced primary and secondary datasets as supplied to Schema and computed 30 CCA factors, performing Leiden clustering using these.scvi-tools (version 0.8.1). To accommodate ATAC-seq data as an input to scVI and totalVI, we reduced the data’s dimensionality from 247,293 to 2629 by first excluding peaks with non-zero counts in fewer than 10 cells and then aggregating the count data of ATAC-seq peaks within 1 Mb genomic windows. To investigate if totalVI’s performance suffered because of the noise and sparsity in the ATAC-seq data, we evaluated it also on a synthetic, less-noisy dataset constructed by reusing and partially randomizing RNA-seq observations, our goal being to design a secondary modality that is not identical to the primary RNA-seq modality but nevertheless agrees well with it. We constructed each column of this dataset as the sum of the RNA-seq counts of 5 randomly chosen genes, with 10% of the final counts randomly set to zero. We found that totalVI did achieve stronger results when synthesizing this dataset with the RNA-seq modality (ARI of 0.088), substantially higher than what was achieved with using ATAC-seq as the secondary modality.We performed scVI and totalVI analysis using the Python package k = 300 clusters by k-means clustering of RNA-seq data; results were robust to the choice of k. Each cluster (“pseudocell”) was represented by the average ATAC-seq profile of its member cells, with these aggregated profiles forming the input to the Leiden clustering algorithm.We also evaluated a heuristic approach described in the original study : group cWe chose gene expression as the primary modality, reducing it with non-negative matrix factorization (NMF) to the top 50 components, and used temporal metadata as the secondary modality. We estimated differential pseudotime using the implementation in Scanpy  of Haghvew, w being the Schema-computed weights; cell loadings were computed similarly. This softmax approach is parameter-free and ensures that gene rankings are informed primarily by the features with the highest Schema weight.We used gene expression as the primary modality, while spatial density and cell type labels were the secondary modalities. We first computed spatial density information for each cell by learning a two-dimensional Gaussian-kernel density function on cell locations; it assigns higher scores to regions with denser cell packing Fig. c. We theTo adapt CCA for a three-way modality synthesis, we tested two approaches: (1) combining spatial density and cell-type information into a composite measure that was then correlated to gene expression, or (2) performing two separate CCA analyses  and combining them. In the first CCA-based approach, we combined spatial density and cell-type labels by learning a Gaussian kernel density function only on cells labeled as granule cells and then inferring its value for other cells. This score was then used in CCA. In the second CCA-based approach, where we integrated results from two preliminary CCA runs, the combined cell loadings were computed as the average of the normalized cell loadings from the two CCAs, with the final gene scores then computed by a matching pursuit technique , 57: theWe also needed to adapt SpatialDE and Trendsceek, both of which have unsupervised formulations, to select for genes whose expression shows spatial variation in granule cell types but not in non-granule cell types. To do so, we ran them separately on granule and non-granule cells and then ranked genes based on the difference of gene ranks between the two runs.ith element of the primary modality feature vector was the 1-letter code of the amino acid at the ith sequence position or a null value if the sequence length was shorter than i. We defined the distance between two sequences as the number of elements that disagreed. In the original space, this corresponds to the Hamming distance; in the transformed space, it is a location-weighted version of the Hamming distance. The secondary modality corresponding to binding specificity against the 44 ligands was represented as a 44-dimensional Boolean vector, with the Euclidean distance metric. Each Schema run was an ensemble of sub-runs, with varying parameter choices of minimum correlation between the original and transformed datasets and the maximum allowed feature weights. Feature weights produced in each sub-run were normalized by linearly mapping the lowest weight to 0 and the highest to 1.When estimating location-specific selection pressure Fig. c, d, we I.norm) for the location in each TCR motif and averaged it across all motifs in the database. Here, Schema and VDJdb scores have opposite orientations: for a location that demonstrates low variability, the associated Schema weight will be high while the VDJdb score will be low. Therefore, when comparing the Schema and VDJdb scores, we inverted the orientation of Schema scores by subtracting them from 1  across the selected locations in all TCR motifs in the database.To compute the selection pressure on amino acids, we focused on segments 3–7 in TCR α chains and 5–11 in TCR β chains, choosing these locations for their high sequence variability as estimated by Schema and VDJdb above. To compute Schema scores, an ensemble of sub-runs was performed, and as described above, Schema scores were normalized. VDJdb scores for an amino acid were computed as the average frequency-weighted relative entropy scores  or by epitopes (10 randomly divided subsets), we compared the baseline two-modality setup against the new three-modality setup and found that the latter produced slightly more stable results than the former, with smaller standard deviations of Schema-computed weights across the subsets of data . In general, we recommend the use of cross-validation or an independent metric to calibrate the relative weights of secondary modalities in such use-cases.To exemplify how Schema can synthesize additional modalities, we also incorporated proteomic measurements of 12 cell-surface markers. Hypothesizing that cell-surface protein levels should be unrelated to the V(D)J recombination variability, we added a low weight term to Schema’s objective function that Additional file 1:. Figures S1-S10, Text S1-S6, Table S1.Additional file 2:. Tables S2, S3.


Full-length isoform quantification from RNA-Seq is a key goal in transcriptomics analyses and has been an area of active development since the beginning. The fundamental difficulty stems from the fact that RNA transcripts are long, while RNA-Seq reads are short.Here we use simulated benchmarking data that reflects many properties of real data, including polymorphisms, intron signal and non-uniform coverage, allowing for systematic comparative analyses of isoform quantification accuracy and its impact on differential expression analysis. Genome, transcriptome and pseudo alignment-based methods are included; and a simple approach is included as a baseline control.Salmon, kallisto, RSEM, and Cufflinks exhibit the highest accuracy on idealized data, while on more realistic data they do not perform dramatically better than the simple approach. We determine the structural parameters with the greatest impact on quantification accuracy to be length and sequence compression complexity and not so much the number of isoforms. The effect of incomplete annotation on performance is also investigated. Overall, the tested methods show sufficient divergence from the truth to suggest that full-length isoform quantification and isoform level DE should still be employed selectively.The online version contains supplementary material available at 10.1186/s12859-021-04198-1. Alternative splicing and isoform switching play central roles in cell function; and disruption of the splicing mechanism is associated with many diseases and drug targets , 2. The For investigations specifically focused on splicing, one also has the option of working at the local splicing level . If, foAnother reason for estimating isoform level expression is to give better estimates of gene level expression. Indeed, it is not clear how to achieve gene level quantification from local splicing information. For various purposes, full length isoform quantification must be more informative than local splicing information when it can be achieved, and the primary reason local splicing methods are popular right now is due to the relative difficulty in working with full length.The fact that isoform quantification is a key goal for modern transcriptomic profiling is reflected in how active the community has been in developing methods and how popular those methods have been, in spite of their notoriously high false positive rates. Despite many published algorithms, in practice effective quantification of full-length isoforms from short-read RNA-Seq remains problematic and therefore has never been routine. The fundamental limitation is that individual short reads do not contain information on long-range interactions that would associate splicing events that are separated by more than the fragment length. Regardless, methods can exploit additional biological and stochastic information, like canonical splice sites, which combined with alignment information can increase accuracy –7.Although long sequence read technology is improving, compared to short read technology it continues to be lower throughput with a much higher base-wise error rate and is generally more expensive. Therefore, most RNA-Seq studies are still performed with short reads and this will likely remain the case until competing technologies mature. Short reads are typically 100–150 bases long, and usually obtained from both ends of short 200–500 base fragments. Meanwhile a significant portion of RNA transcripts are over 1000 bases and many are much longer.kallisto [Salmon [Given the difficulty in full-length isoform quantification, many RNA-Seq studies simply quantify at the gene level, which is much easier because uniquely aligning reads are rarely ambiguous at the gene level. Indeed, unless the investigator is specifically interested in splicing, gene level analysis will likely lead to the same conclusions, since all isoforms of the same gene typically have the same pathway annotations. Meaningful, unbiased benchmarking conclusions rely on independent investigations and realistic benchmarking data where the ground truth is known or well-approximated. There are in fact a few independent studies that compare the performance of transcript quantification methods using simulated data , real dakallisto  and Salm [Salmon . Angelin [Salmon  take the [Salmon . Also, tZhang et al.  use the Hayer et al.  investigbona fide alignment [Approaches for quantifying isoform expression can be divided into three main categories. The first approach uses reads mapped to the genome by an intron-aware aligner, e.g. STAR . The genlignment , 13. In lignment –22. Pseulignment , 24.Fig.Here we present a benchmarking analysis of the six most popular isoform quantification methods: kallisto, Salmon, RSEM, Cufflinks, HTSeq, and featureCounts, based on a survey of the literature Fig. . HTSeq aA hybrid approach is taken to obtaining benchmarking data, where real samples are emulated to generate simulated data where the true isoform abundances are known; this was done using a modified version of the BEERS simulator . IdealizUsually, the aim of an RNA-Seq analysis is to inform a downstream differential expression (DE) analysis. Therefore, we also evaluate the methods on this level, using both real and simulated data. However, it is much more challenging to produce realistic data with known ground truth at the DE level. Unlike isoform level quantification which is sample-specific, DE ground truth is established at the population level, and therefore involves much more complex benchmarking data. Our simulated samples reflect the complex joint distribution of expression across biological replicates, and thus it is meaningful to perform a DE analysis on them. This is described in more detail below but briefly, in lieu of knowing the ground truth in terms of which isoforms are differentially expressed, for each method we compare the DE analysis performed on the known true isoform quantifications of the simulated data to the DE analyses performed on the estimated counts determined using the particular method. The more different the two analyses are, the less accurate the quantification method must be in informing the DE analysis. This then allows us to compare the methods in terms of their accuracy of quantification. It is possible that a method underperforms another method at the level of quantification, but outperforms it in the DE analysis.For the simulated data we started with 11 real RNA-Seq samples: six liver and six hippocampus samples from the Mouse Genome Project . IsoformWe performed a comparative analysis of seven of the most commonly used full-length isoform quantification algorithms; kallisto , Salmon X-axis indicates an unexpressed transcript which was erroneously given positive expression. A point on the Y-axis indicates an expressed transcript which was erroneously given zero expression.The idealized data has no indels, SNP’s, or errors, includes no intron signal, and deviates from uniform coverage across each isoform only as much as may happen due to random sampling. Under such perfect conditions we expect that all methods will achieve their best performance. The data were aligned to the reference genome or transcriptome with STAR  and quanIn liver, we observe that kallisto, Salmon, RSEM and Cufflinks’ are fairly concordant with the truth, while NRP performs moderately well considering its simplicity. HTSeq and featureCounts show high deviation from the truth and in particular tend to undercount. That is because both methods ignore any read which is ambiguous between isoforms.Accuracy is somewhat lower in hippocampus, which has more complicated transcription with more alternate splicing, but similar conclusions hold  fold change between the inferred counts relative to the truth. This will be denoted logFC and if the absolute value is taken then |logFC|. Before taking ratios, all counts are adjusted by addition of a pseudo-count of 1 see “”.|< c for each c > 0. Specifically, a point  on the graph means x% of transcripts have |logFC|< y. The lines are monotonically non-decreasing and the closer to the x-axis, the better. When the |logFC| for a transcript is close to zero, it suggests that the method gives an accurate estimate. When the p-th percentile of the cumulative distribution is close to zero it means the top p percent of the best estimated transcripts are all very accurate. Genes where all of their isoforms have zero expression will be generally easy for the algorithms to get right, since there will usually be no reads mapping anywhere to the gene locus. To avoid those exaggerating the average performance, such genes were removed from consideration before generating these graphs. These graphs were also generated with all genes removed that had three or less expressed isoforms , sequencing errors, intron signal, and non-uniform coverage across each isoform. Non-uniform coverage mimics observed coverage in real data but does not attempt to model coverage-associated sequence features such as GC content. As such, the non-uniform coverage options of Salmon that involve GC content could not be utilized. The non-uniform coverage option of RSEM was used since it is sequence agnostic.Similar to the idealized data, points in Fig. The percentile plots of |logFC| are given in Fig. The correlation boxplots for the realistic data are given in Fig. Salmon is marginally superior here, in both absolute comparisons and correlations.versus liver, shows all methods reasonably close to the truth except HTSeq and featureCounts  is largely unaffected . This produces, for each method/tissue, a list of the most discordant isoforms. For a given structural property, such as length, the background distribution is the  distribution of the property over all isoforms (which is the same for both tissues). And the foreground distribution is the  distribution of the property over the 2000 discordant isoforms. These two distributions can be plotted together and interpreted visually—and can also be compared using the Kolmogorov–Smirnov test. The Teqila tool was used for this analysis .p values were multiple-testing corrected by Bonferroni for testing multiple methods and multiple properties. The significant properties are shown in Fig. A number of structural properties were investigated, such as number of isoforms, hexamer entropy, transcript length, transcript sequence compression complexity , exon coIt is worth looking at the distributions of some of the properties. A select few are shown here, others are in Additional file The observations about length and exon count apply equally well to all methods. However, for number of isoforms, NRP is quite different from the others see Fig. .Fig. 7FeAnnotation guided quantification is only as good as the annotation itself. And no annotation is perfect, as, in a given sample, there likely exist both annotated isoforms that are not present as well as unannotated isoforms that are present. It can quickly get complicated trying to sort out the effect of annotation issues on the performance of the algorithms. Sometimes missing annotation can be beneficial, for example if a non-expressed isoform is missing from the annotation, then it cannot erroneously be assigned reads. On the other hand, if a highly expressed isoform is missing, then the method must figure out what to do with the orphaned reads. It should be able to figure out that they should be ignored. Otherwise, they will be assigned to the wrong isoform. To investigate these two issues, we modified the annotation in two ways. First, all isoforms that were not expressed were removed. The extent to which their absence improves accuracy gives an indication of how well the method is handling unexpressed isoforms when they are present. Second, the highest expressed isoform of each gene was removed. This should cause major problems to the algorithms, so it is informative to see which methods handle this better.Figure Figure p values are unaffected. To investigate this, the two tissues were compared against each other; different enough tissues so that there is an abundance of differentially expressed genes. Six hippocampus samples and six liver samples of the realistic data were quantified, with each of the seven methods, and the resulting quantified values were used as input for DE analyses with EBSeq [p values generated from the true counts are compared to p values from the inferred counts—the assumption being that the closer a DE analysis on the inferred counts is to the corresponding DE analysis on the true counts, the more effectively the method has quantified the expression, with respect to informing the DE analysis. Kallisto and Salmon are recommended to be run with Sleuth, however since Sleuth cannot take true counts as input, the comparison would not be meaningful. Since we are comparing EBSeq (truth) to EBSeq (inferred) for all methods, it should be meaningful to compare methods to each other with this metric.Next, we use differential expression to assess quantification accuracy. If differential expression analysis is the downstream goal for the quantified values, then it does not matter if the absolute abundances differ from the truth, if the DE th EBSeq , which in most significant genes, as n varies from 1 to 30,000. Since EBSeq reports a lot of zero p values, rounded down from their limit of precision, ties were broken with the logFC. The vertical axis is the Jaccard index [n DE transcripts determined using the true counts and the top n DE transcripts determined using the inferred counts. The Jaccard index of two subsets of a set is the size of the intersection divided by the size of the union. The higher the curve, the better. Salmon and Cufflinks are performing best from this perspective, followed by RSEM. NRP and kallisto appear roughly equivalent. In Fig. q-value cutoff  is less relevant than the length or number of exons. That hexamer entropy is associated with poor quantification suggests a direction for improvement in future methods developments, much as Salmon offers GC bias correction. We did not however investigate the extent of an isoform’s homology to other genes, which could potentially have greater impact than the number of sibling isoforms. Nor did we investigate the impact of secondary structure, beyond how it relates to sequence complexity.In real data, we isolated one example of large quantification differences between methods. In this, the inclusion of a single read causes kallisto and RSEM to disagree by 137 counts to 0, and the difference resolves if that read is removed. This edge case occurred because only two reads were unambiguous to the two isoforms of a highly expressed gene. The transcript-level DE method Sleuth  uses booOverall, kallisto and Salmon as alignment-free methods require less computational time while achieving similar or better accuracy compared to other methods whereas RSEM and Cufflinks perform well among the alignment-dependent methods. However, our results indicate that all tested methods should be employed selectively, especially when long transcripts with many isoforms or transcripts with low sequence complexity are the candidates of interest for the study. NRP is a straightforward and simple approach that is relatively robust to polymorphisms, non-uniform coverage and intron signal; however, it struggles with a greater number of isoforms. In any case it performs nearly as well as more sophisticated methods, suggesting that information extraction and inference from short RNA-Seq reads is largely saturated and future, more complex models might offer only small benefits in gene isoform quantification.Previous benchmarking studies agree with our conclusions about RSEM as a top performer , 12, 16.These results indicate the differing strengths of different approaches to this problem. As such, it may be possible to leverage the different methods to achieve overall greater accuracy. For example, NRP, HTSeq and featureCounts appear to do better on one-isoform genes. So, it may make sense to treat those genes separately. In any case this must continue to be an active area of research before the technology can transform transcriptomics and realize the advantages of full-length isoform quantification.percent spliced included (PSI) values across all of the Mouse Genome Project (MGP) [n spliceforms, randomly select a gene with the same number of spliceforms from the empirical set. (2) For this empirical gene, randomly select the PSI values from one MGP sample. (3) Assign these PSI values across all samples for the gene in the simulated set. (4) To add inter-sample variability, randomly add/subtract a random number (uniform from 0 to 0.025) to the PSI values in each sample, such that PSI values for the gene/sample still sum to 1. These estimated gene expression counts and PSI values, for both the empirical set and remaining set of genes, served as input into the BEERS simulator [We used the same method for generating simulated data as described in Norton et al. . For allct (MGP)  liver anct (MGP) . Brieflyimulator . For theimulator . We alsohttp://bioinf.itmat.upenn.edu/BEERS/bp3/) based on transcriptome alignment, as well as the genome alignment based methods RSEM [The two simulated RNA-Seq datasets were aligned to both the GRCm38 build of the mouse genome and transcriptome with STAR-v2.7.6a . For allods RSEM , 36, HTSods RSEM , and feaods RSEM  was usedods RSEM . The comhttps://github.com/COMBINE-lab/wasabi) to convert Salmon to the Sleuth input format. All methods were run on the realistic simulated data and compared the five hippocampus samples to the six liver samples and on the real samples, six hippocampus versus six liver samples. For the simulated data, we also ran DESeq2 and EBSeq given the true quantified variables for comparison with the inferred quantifications.Transcript-level differential expression was assessed via three methods. DESeq2-v3.12 and EBSeq-v1.30.0 were run on the inferred quantified values from all quantification methods. In addition, the Sleuth-v0.30.0 method was run on the quantifications from Salmon and kallisto, using 50 bootstrap samples and the Wasabi package  [3′ bias) . Like kaRSEM is a gene/isoform abundance tool for RNA-Seq data which uses a generative model for the RNA-Seq read sequencing process with parameters given by the expression level for each isoform [ isoform , 38. A sTo prepare input for Cufflinks, HTSeq and featureCounts, the real and simulated data were aligned to a STAR genome index built with the GRCm38 Ensemblv75 transcript annotation file.Cufflinks suite [Cuffdiff2  is an alks suite , which eks suite . We provHTSeq [For HTSeq , htseq-cfeatureCounts [reCounts  is a reaNaïve Read Proportioning (NRP) approach as a baseline. This is essentially the method described by Mortazavi et al.[As a baseline comparison, we considered a vi et al. but with2 fold-change (fold-change after adjusting numerator and denominator by pseudocount of 1) for estimated counts relative to the known simulated true counts. For example, if x is the true count and y is the estimated count for a particular method, we calculate the quantity of As a measure of the accuracy of each method, we compute the absolute value of the logIn order to better represent the distribution of the |logFC| values for each method, we plot (for the set of expressed isoforms) the value of |logFC| corresponding to every tenth percentile starting from 0. If the method has high accuracy, we expect the graph to be close to 0. Thus, if the graph for method A is higher than method B, we conclude that tool B is more accurate.p values for each transcript property, followed by Bonferroni correction for multiple testing, to identify the properties that exhibit significant deviation from the global distribution.Moreover, we identify the genomic properties of the data that affect the accuracy of the methods. For each method, we identified the most discordant transcripts sorting by |logFC|. Using the Ensembl annotation and genome sequence for GRCm38, we created a database of transcript properties (such as number of isoforms, hexamer entropy, transcript length, compression complexity* , exon co*Transcript sequence compression complexity is a metric that captures the amount of lossless compression of the transcript sequence. The higher the sequence complexity, the lower the compression, which implies higher transcript sequence compression complexity.Additional file 1. Supplemental FiguresAdditional file 2. Supplemental Tables


Although tissue heterogeneity exacerbates irreproducibility, its prevalence in gene expression data remains unknown. Here, we systematically analyse 2 667 publicly available gene expression datasets covering 76 576 samples. Using two independent data compendia and a reproducible, open-source software pipeline, we find a prevalence of tissue heterogeneity in gene expression data that affects between 1 and 40% of the samples, depending on the tissue type. We discover both cases of severe heterogeneity, which may be caused by mistakes in annotation or sample handling, and cases of moderate heterogeneity, which are likely caused by tissue infiltration or sample contamination. Our analysis establishes tissue heterogeneity as a widespread phenomenon in publicly available gene expression datasets, which constitutes an important source of variance that should not be ignored. Consequently, we advocate the application of quality-control methods such as BioQC to detect tissue heterogeneity prior to mining or analysing gene expression data.Lack of reproducibility in gene expression studies is a serious issue being actively addressed by the biomedical research community. Besides established factors such as batch effects and incorrect sample annotations, we recently reported  Systematic mining of existing data and interrogation of new data can reveal molecular foundations of pathology and disease   and ARCHnakemake  workflowBioQC provides 155 sets of tissue-enriched genes (tissue signatures hereafter) derived from four large-scale tissue gene expression datasets  whi project . We idenGEOmetadb  pairs. As signatures can be correlated , we exclude correlated signature pairs so that they do not inflate false-discovery proportions.s annotated as tissue t is tested for enrichment with the query-signature kquery resulting in a P-value pquery. Let kref be the reference signature associated with tissue t and pref the P-value of testing s for enrichment of kref. Let pquerys as not heterogeneous, else continue. (ii) We fit a robust linear model using rlm from the R package MASS of |log10(pquery)| against |log10(pref)| for all samples annotated as t. (iii) If the slope of the linear model is ≥ 0.01, we exclude the pair of signatures from the results. If the slope is < 0.01 and the FDR-adjusted pquery < severe, if additionally the unadjusted pref > 0.05. (v) Finally, we compute the fraction of heterogeneous samples by dividing the number of samples that have at least one signature passing the above criteria by the total number of samples per tissue. Confidence intervals have been derived by bootstrapping (n=1 000) using the R package boot.The detection process consists of five steps as illustrated in Figure bookdown  and pancreas (30%) samples affected most frequently and blood samples affected least frequently (1.4%) Figure , which iIn general, heterogeneity was higher in ARCHS4 than in GEO, which can likely be attributed to the higher sensitivity of sequencing compared to microarrays. However, the overall patterns (highest heterogeneity in pancreas and skin) are comparable between the platforms, suggesting that the issue of sample heterogeneity is platform-independent. We further observe that heterogeneity is not equally distributed across studies. While most studies (84.3% in GEO and 73.8% in ARCHS4) contain no samples with detected heterogeneity, a considerable proportion  contains ‘severely heterogeneous’ samples . Using a linear model, we conclude that tissue heterogeneity is not associated with the year of the study, suggesting that this issue exists since the early days of transcriptome profiling and persists .A closer investigation of the source of tissue heterogeneity reveals additional insights Figure . First, BioQC depends on the ranking of genes within a sample, per-gene normalized expression profiles in the GEO repository could also not be evaluated. We find it especially problematic that no standardized mapping from probe id to gene symbols was available for many of the remaining samples. The low percentage of usable samples begs the question if our findings generalize to the entire sample population in GEO. However, many of the trends we observed in GEO (e.g. that skin and pancreas exhibit the highest level of heterogeneity) are also found in ARCHS4, which collects data of different samples acquired with an entirely different technological platform.Due to a lack of annotation, only a small fraction of GEO microarray datasets (12.9%) could be used for our analysis. In particular a lack of tissue annotation disqualified the majority (61%) of the datasets. Since BioQC has the additional benefit of explaining the source of variance, and even works with single samples, or when all samples are affected.We also note that the issue of tissue heterogeneity is specific to bulk gene expression data and does not affect single-cell RNA-seq studies, as contaminating cells form an independent cluster of cells which can either be ignored or incorporated in data analysis. In fact, single-cell RNA-seq offers the chance to study biological sources of tissue heterogeneity at a previously unimaginable depth. However, due to its lower cost and simpler sampling procedure, the majority of expression profiles will still be sequenced in bulk in the foreseeable future. In addition, many studies strive to use information derived from bulk-sequenced samples to inform both experimental design and analysis of single-cell studies. Hence, identifying samples affected by tissue heterogeneity will remain an important aspect of data analysis. Standard methods, such as principal component analysis (PCA) can identify heterogeneous samples as outliers. Using signature-based methods such as et al. (A limitation of the study is that we focused only on bulk gene expression datasets based on mRNA profiling using either microarray or Illumina sequencing. High-throughput gene expression data derived from other modalities, such as third-generation sequencing techniques and mass spectrometry-based proteomics, usually require many cells as input and hence may suffer from tissue heterogeneity as well. A recent study by Yoo et al. , for insBioQC to assess tissue heterogeneity and to ensure reproducibility of gene expression studies.Detecting and modelling tissue heterogeneity is of particular importance in systems medicine studies, where tissue-specific signals can mask disease-specific signals, thus preventing the successful detection of disease mechanisms, patient stratification, or drug target identification and validation. Based on the prevalence of tissue heterogeneity in gene expression data, we advocate the routine use of methods such as https://doi.org/10.5281/zenodo.4298774. The source code to reproduce the analysis can be found at https://github.com/grst/bioqc_geo. Pygenesig is available from https://github.com/grst/pygenesig. BioQC is available from R/Bioconductor and the documentation is available at https://accio.github.io/BioQC/.The reference signatures and raw results table including all accession numbers tested is available from lqab077_Supplemental_FilesClick here for additional data file.


Colorectal cancer is one of the most common types of cancer, and it can have a high mortality rate if left untreated or undiagnosed. The fact that CRC becomes symptomatic at advanced stages highlights the importance of early screening. The reference screening method for CRC is colonoscopy, an invasive, time-consuming procedure that requires sedation or anesthesia and is recommended from a certain age and above. The aim of this study was to build a machine learning classifier that can distinguish cancer from non-cancer samples. For this, circulating tumor cells were enumerated using flow cytometry. Their numbers were used as a training set for building an optimized SVM classifier that was subsequently used on a blind set. The SVM classifier’s accuracy on the blind samples was found to be 90.0%, sensitivity was 80.0%, specificity was 100.0%, precision was 100.0% and AUC was 0.98. Finally, in order to test the generalizability of our method, we also compared the performances of different classifiers developed by various machine learning models, using over-sampling datasets generated by the SMOTE algorithm. The results showed that SVM achieved the best performances according to the validation accuracy metric. Overall, our results demonstrate that CTCs enumerated by flow cytometry can provide significant information, which can be used in machine learning algorithms to successfully discriminate between healthy and colorectal cancer patients. The clinical significance of this method could be the development of a simple, fast, non-invasive cancer screening tool based on blood CTC enumeration by flow cytometry and machine learning algorithms. Cancer is extremely complex and heterogeneous. It includes various processes , which manifest as cancer’s irregular dynamics in multi-level spatio-temporal scales. In particular, at the molecular level, a large number of interacting molecules  constitute a complex network, which results in complex intracellular signaling, non-linear reaction kinetics, gene mutations and dysregulations, regulatory circuits, pathway cross-talks and others ,2,3,4. TColorectal cancer (CRC) is one of the most common types of cancer, and it can have a high mortality rate if left untreated or undiagnosed. It is estimated that 10% of all annually diagnosed cancers are CRC . Risk faDespite the significant advances in the field, colorectal cancer therapy, recurrence and metastasis continue to face difficulties and new challenges due to cancer’s inherent complexity. Therefore, detailed biological information  and the utilization of advanced mathematical methods are of great importance ,12,13,14In this direction,  developeIn this study, we built an ML classifier for discriminating colorectal cancer samples from non-cancer, healthy samples. The ML classifier was based on Support Vector Machines (SVM), which were chosen as an appropriate approach for classifying colorectal cancer and non-cancer/healthy samples. We used SVM since they are among the most commonly applied ML algorithms within the field of cancer research and more generally in computational biology ,31,32,33As a dataset, we used experimental data derived from circulating tumor cells (CTCs)  detectedThe aim of this study was, firstly, to validate our method for CTC determination, and secondly, to use these data to perform binary classification between colorectal cancer and healthy samples. The clinical significance of this method could be the development of a non-invasive cancer screening tool based on blood CTC enumeration by flow cytometry and ML.This study was not a clinical trial and did not include any interventions. The study was reviewed and approved by our institutional ethics committee. Informed consent was obtained from all patients. Blood samples from a total of 41 healthy individuals/non-cancer patients and 41 CRC patients were collected in sterilized 50-mL falcon tubes containing 7 mL 0.02 M EDTA as an anti-coagulant. Healthy individuals were identified as healthy/non-cancer by their physicians.A total of 2 mL of blood was mixed with 2 mL of fetal bovine serum in 15-mL centrifuge tubes to regain the cells’ shape. The samples were then centrifuged at 1200 rpm for 10 min at room temperature and the supernatant was discarded. A total of 100 μL of sample was transferred to round-bottom tubes for flow cytometry analysis.Antibodies used were CD45-PE/Cy7, CD31-RPE and pan CK-PE/Cy5. Samples were fixed and permeabilized using LEUCOPERM according to the manufacturer’s instructions. Briefly, first samples were stained with surface antibodies for 20 min , washed with PBS and then fixed with 100 μL Leucoperm Reagent A, washed with PBS, permeabilized with 100 μL Reagent B and stained intracellularly with 5 μg/mL pan-CK antibodies for 20 min and washed again with PBS. After the last wash, cells were re-suspended in 500 μL PBS and were ready for acquisition in a Beckman Coulter FC500 cytometer.A total of 31 healthy and 31 cancer samples were known to the investigators and were used for the training and validation of the algorithm. Twenty samples were blinded by using 5-digit codes and were used for prediction (test) analysis.Circulating tumor cells were defined as CD45-negative, CD31-negative and pan-cytokeratin-positive cell populations. Non-hematological cells were gated out using a CD45-negative selection. The endothelial cells were then removed using a CD31-negative gating selection. Tumor cells were identified by pan-CK-positive selection. Unstained samples were used as a negative control for gating. FCS Express software was used for fcs data analysis. The two-sample Kolmogorov–Smirnov test  is a nonThe Wilcoxon rank sum test is a nonparametric test for two populations . In parti, and the optimal hyperplane can be defined as:In this paper, in order to solve this binary classification problem, we apply a powerful classifier, the support vector machine (SVM). SVM aims to create a decision boundary between two classes in order to predict the labels from one or more feature vectors ,45. ThisTherefore, the objective of training an SVM model is to find the proper However, many binary classification problems do not have a simple hyperplane as a useful separating criterion. For such problems, instead of using a linear SVM classifier, we can alternatively use the kernel method. This method enables us to model higher dimensional, non-linear models, while retaining nearly all the simplicity of an SVM separating hyperplane. Specifically, the kernel method transforms the data into higher dimensional spaces to make the data separable.G is the kernel function, x and y are n dimensional inputs and f is used to map the input from n dimensional to m dimensional space. Finally, the term p is some positive integer, while RBF kernel is defined as:In general, a kernel function is defined as:Of course, the choice of kernel function, among other parameters, can greatly influence the performance  of an SVM model. One can choose between the available kernels through trials and, depending on the nature of the problem, select the best one. One way to find the optimal kernel in a statistically rigorous fashion is by using cross-validation.k-fold cross validation procedure. In particular, this method splits the data randomly into k equal  parts. Then, the algorithm runs k times, using k-1 of the parts as a training set and the remaining part as a test set. Each time the algorithm runs, a different test set is used, so that over the k runs of the algorithm, all the instances in the dataset are used as a test set. The success of the algorithm is the sum of the correct classification over each of the runs. However, even cross-validation can overestimate the prospective performance of ML methods. Therefore, we also conducted a truly blind test in order to demonstrate the prospective capabilities of our cross-validated model [Particularly, cross-validation is a procedure used to avoid under- and overfitting . It is aed model .Maximum number of splits: 1–163, Split criterion: Gini’s diversity index, Maximum deviance reduction. (b) Discriminant Analysis: Linear, Quadratic, Diagonal Linear, Diagonal Quadratic. (c) Naïve Bayes: Distribution names: Gaussian, Kernel, Kernel type: Gaussian, Box, Epanechnikov, Triangle. (d) SVM: Multiclass method: One-vs.-All, One-vs.-One, Box constraint level: 0.001–1000, Kernel scale: 0.001–1000, Kernel function: Gaussian, Linear, Quadratic, Cubic, Standardize data: true, false. (e) kNN: Number of neighbors: 1–82, Distance metric: City block, Chebyshev, Correlation, Cosine, Euclidean, Hamming, Jaccard, Mahalanobis, Minkowski (cubic), Spearman, Distance weight: Equal, Inverse, Squared inverse, Standardize data: true, false. (f) Ensemble: Method: Bag, GentleBoost, LogitBoost, AdaBoost, RUSBoost, Number of learners: 10–500, Learning rate: 0.001–1, Maximum number of splits: 1–163, Number of predictors to sample: 1–2.One drawback of this study is the relatively small dataset, which can lead to biased models that are not generalizable. Therefore, in order to further test the generalizability of our method, we also compared the performances of many classifiers, in addition to the SVM classifier, developed by methods frequently utilized in ML applications. In particular, we developed optimizable models from classification trees , discrimIn order to perform the comparison, since the dataset is small, we updated the original dataset, generating over-sampling datasets. Therefore, we tested the performance of all the above classifiers using the over-sampling datasets. In order to create the over-sampling datasets, we used a robust method named Synthetic Minority Over-sampling Technique (SMOTE) ,54. Thiserate it ,54. For erate it .The performance analysis of the model can be measured in terms of sensitivity, specificity, accuracy and area under the curve (AUC). They are all based on true positives  samples); true negatives  samples), false positives  and false negatives  .In particular, Accuracy is the percentage of correctly predicted samples, and is defined as:Sensitivity or True Positive Rate (TPR) is the percentage of samples correctly predicted as cancer samples, and is defined as:The opposite of sensitivity is called False Negative Rate (FNR) or Miss Rate and is equal to FNR = 1 − TPR.Specificity or True Negative Rate (TNR) is the percentage of samples correctly predicted as non-cancer/healthy samples, and is defined as:Precision or Positive Predictive Value (PPV) is the percentage of samples correctly predicted as cancer from all positive predictions, and is defined as:The opposite of Precision is False Discovery Rate (FDR) equal to FDR = 1 − PPV.Area under the Curve (AUC) is a measure of the model’s overall performance. AUC for binary classification  is givenThe maximum AUC is 1, which corresponds to a perfect classifier, while for a classifier that randomly assigns observations to classes, AUC = 0.5. Larger AUC values indicate better classifier performance. A rough rule of thumb is that the accuracy of tests with AUCs between 0.50 and 0.70 is low; between 0.70 and 0.90, the accuracy is moderate; and it is high for AUCs over 0.90 .AUC is the primary statistic we obtain from a Receiver Operating Characteristics (ROC) curve , which pAfter data acquisition, CTCs from the 31 cancer and 31 healthy samples were calculated using FCS Express. p-value equal to p-value equal to Before we built the classifier, we tested for differences between the cancer and non-cancer/healthy distributions and their medians. This was achieved using two non-parametric hypothesis tests, namely the two-sample Kolmogorov–Smirnov (KS) test and the Wilcoxon rank sum (WRS) test. Both tests revealed significant statistical differences in terms of distributions and medians. In particular, the KS test rejected the null hypothesis, namely that the data are from the same continuous distribution with a n MATLAB , using tbayesopt to find the best (optimized) classification SVM model. In particular, the hyperparameter search range included box constraint level: 0.001–1000, kernel_scale: 0.001–1000 and kernel_function: Gaussian, linear, quadratic, cubic. The optimized SVM model consisted of a quadratic kernel function  and box constraint level equal to 3.0685. The data were standardized.We used a 5-fold cross validation and MATLAB’s Bayesian Optimization function The results of the optimized SVM are shown in the confusion matrix . In partIn addition, in the right panel, the row summary displays the percentages of correctly and incorrectly classified observations for each true class. This panel shows that the sensitivity (TPR) is equal to 23/31 × 100% = 74.2% and the miss rate (FNR) is equal to 8/31 × 100% = 25.8%. This means that 23 samples were correctly classified as cancer samples and eight samples were falsely classified as non-cancer/healthy  out of 31 cancer samples. Similarly, the specificity (TNR) is 28/31 × 100% = 90.3%, while 3/31 × 100% = 9.7% were falsely classified as cancer samples.Finally, the bottom panel displays a summary of the percentages of correctly and incorrectly classified observations for each predicted class. Specifically, this panel shows the results concerning the precision (PPV) and False Discovery Rate (FDR) of the optimized SVM model. As it is shown, PPV is equal to 23/26 × 100% = 88.5% for the cancer samples and 28/36 × 100% = 77.8% for the non-cancer/healthy samples. The FDR is 100% − 88.5% = 11.5% for the cancer samples and 100% − 77.8% = 22.2% for the non-cancer/healthy samples, respectively.In In addition, as it is shown, the optimal point  for the classifier is found for TPR = 0.74 and FPR = 0.10, near the Y-axis. Classifiers appearing on the left-hand side of an ROC graph are rather “conservative”, namely they make positive classifications only with strong evidence, making few false positive errors .In order to further test the performance of the optimized SVM classifier, we examined its performance in a totally blind set. As mentioned, this set includes 10 cancer and 10 non-cancer/healthy samples. The results are summarized in Overall, the results demonstrate that the SVM classifier, based on CTCs enumerated by flow cytometry, can successfully discriminate between healthy and colorectal cancer patients with high values of performance measures.smote [In order to further test the generality of our results, we compared different classifiers developed by models frequently utilized in ML applications. All computations were performed in MATLAB  using thsmote . This fuIn addition, for the generation of the over-sampling datasets, both the training as well as the blind sets were taken into consideration. In particular, we generated six new datasets by varying both the amount of over-sampling (N) as well as the number of considered nearest neighbors (K). We considered N = 1 (2-fold observations), N = 3 (3-fold observations) and N = 10 (10-fold observations), and K = 5, 10, 20, 30. Therefore, since the initial set (including both the training and blind sets) consists of 41 colon cancer samples and 41 healthy samples, the resulting datasets contained: D1 = 164 samples, D2 = 164 samples, D3 = 328 samples, D4 = 328 samples, D5 = 902 samples and D6 = 902 samples.bayesopt). As it can be seen, all models achieved high validation accuracies, above 84%, while the differences are not so big, even for the linear benchmark method of logistic regression. This is expected, since we only have one feature as an input. The highest validation accuracies (%) were found for the classifiers based on SVM , for on ensemble classifiers  and for on classification trees . The results of the performances of the classifiers are shown in We also estimated the AUC for each of the optimized models and the results are presented in Taking into account the results concerning validation accuracies as well as AUC, it can be concluded that, even though all ML classifiers yielded high performances, SVM performed better according to the validation accuracy metric, while ensemble methods performed better according to the AUC metric. However, compared to AUC, accuracy is simpler and easier to interpret, while it is mostly used for evaluating supervised binary classifiers with balanced classes, taking into account both true positive as well as true negative predictions. Therefore, based on accuracy results, in this study, we chose SVM for developing efficient and robust ML classifiers.In the present study, we developed an SVM classifier for performing binary classification between colorectal cancer and non-cancer/healthy samples. The main feature used for the classification is the number of CTCs from cancer and non-cancer/healthy samples, as obtained from flow cytometry. In this study, 31 colorectal cancer and 31 non-cancer/healthy samples were used for the development of the SVM classifier. In addition, the SVM classifier was tested in a blind test set, which included 10 cancer samples and 10 non-cancer/healthy samples. Finally, in order to further test the efficiency and generalizability of the proposed method, we generated various over-sampling datasets by applying the SMOTE algorithm and used these datasets in order to develop and compare various ML classifiers.The results of this study revealed the efficiency of the developed SVM classifier both on the training set as well as on the blind set. In particular, for the training set, the performance measures of the SVM classifier were found to be: accuracy equal to 82.3%, sensitivity (TPR) equal to 74.2%, miss rate (FNR) equal to 25.8%, specificity (TNR) equal to 90.3%, precision (PPV) equal to 88.5% and AUC equal to 0.85. For the blind set, the performance measures of the SVM classifier were found to be: accuracy equal to 90.0%, sensitivity (TPR) equal to 80.0%, miss rate (FNR) equal to 20.0%, specificity (TNR) equal to 100.0%, precision (PPV) equal to 100.0% and AUC equal to 0.98.One drawback of this study was the relatively small dataset, which can result in misclassifications, while the estimators may produce unstable and biased models, which can fail to generalize efficiently. However, the analysis of over-sampling SMOTE-generated datasets revealed that ML classifiers can also be effective for much bigger (up to 10-fold) datasets. In particular, the estimation of the performance measures of the optimized classifiers showed that all classifiers exhibited very good performances, yielding values above 0.84 for validation accuracy and above 0.84 for AUC. Additionally, SVM performed better according to the validation accuracy metric, while ensemble methods performed better according to the AUC metric. Considering accuracy as a more relevant metric for this supervised binary with balanced classes study, SVM was the selected method.Therefore, as the results of this study demonstrate, the drawback of the small dataset size is surpassed by the dataset quality, namely the careful feature selection , which provides significant information for the development of effective ML classifiers. In particular, our results indicate that flow cytometry, using the gating strategy described, can be a valuable tool for CTC enumeration with high sensitivity and specificity. In addition to the accuracy of the method, other advantages are also present. Additional markers can also be studied. Immunophenotyping CTCs, that is, the determination of the expression of markers related to steaminess or metastasis, could provide useful clinical information that can aid in cancer prognosis and/or treatment decisions. Additionally, using flow cytometry and sorting, CTCs can be isolated alive and cultured for downstream applications.Overall, the results show that CTCs enumerated by flow cytometry can provide significant information, which when “fed” into ML algorithms can successfully discriminate between non-cancer/healthy and colorectal cancer patient subjects. Even though the results seem promising, more experiments have to take place in order to obtain larger datasets, while the exploitation of more sophisticated classification techniques is needed to verify and extend the results of this study. ML algorithms are not static products, and can continue to change and improve even once deployed, as new training data become available. However, these issues will be addressed in following studies. In conclusion, the results of this study are promising towards the development of a simple, fast and non-invasive screening method for cancer, using CTC enumeration by flow cytometry from blood samples and machine learning.