High-throughput sequencing enables the analysis of the composition of numerous biological systems, such as microbial communities. The identification of dependencies within these systems requires the analysis and assimilation of the underlying interaction patterns between all the variables that make up that system. However, this task poses a challenge when considering the compositional nature of the data coming from DNA-sequencing experiments because traditional interaction metrics  produce unreliable results when analyzing relative fractions instead of absolute abundances. The compositionality-associated challenges extend to the classification task, as it usually involves the characterization of the interactions between the principal descriptive variables of the datasets. The classification of new samples/patients into binary categories corresponding to dissimilar biological settings or phenotypes  could help researchers in the development of treatments/drugs.Here, we develop and exemplify a new approach, applicable to compositional data, for the classification of new samples into two groups with different biological settings. We propose a new metric to characterize and quantify the overall correlation structure deviation between these groups and a technique for dimensionality reduction to facilitate graphical representation. We conduct simulation experiments with synthetic data to assess the proposed method’s classification accuracy. Moreover, we illustrate the performance of the proposed approach using Operational Taxonomic Unit (OTU) count tables obtained through 16S rRNA gene sequencing data from two microbiota experiments. Also, compare our method’s performance with that of two state-of-the-art methods.Simulation experiments show that our method achieves a classification accuracy equal to or greater than 98% when using synthetic data. Finally, our method outperforms the other classification methods with real datasets from gene sequencing experiments. Microorganisms living inside and on humans are known as the microbiota. When integrated with their genes’ information, it is known as the microbiome. The Human Microbiome Project (HMP) was an endeavor for the characterization of the human microbiota to further understanding its impact on human health and diseases .In recent years, biological sciences have experienced substantial technological advances that have led to the rediscovery of systems biology –4. TheseWe initially conducted a literature search in different medical, biological, and engineering databases as well as academic sites prestigious journals such as BMC Bioinformatics, PLOS ONE, ScienceDirect, and IEEE Xplore using the queries “correlation structure for gene expression classifications,” “classifiers for compositional data,” and “classifiers based on correlation structures” in order to identify papers in English using procedures for sample classification based on correlation structures in the 2009–2019 time window. Figure Operational Taxonomic Unit (OTU) count tables are the usual output when processing the 16S rRNA sequences of microbiota samples . These txj is an element of {x1, x2, x3…, xn}. Because one value xj is selected as the denominator to build the log-ratios, the alr has been criticized as being subjective since the outcome depends mostly on the value of xj selected , where zi is calculated as:In  authors However, implementing the ilr transformation poses serious practical difficulties for high-dimension data as the computational complexity increases rapidly with dimensionality .After transforming the data, the next step is to separate the data into train, test, and validation sets, although in some cases only the train and test sets are considered. One of the most common problems prior to that step is the limitation of the number of data samples. Indeed, for a normal classifier to be employed using multivariate metrical techniques, the sample size required for optimum training is in order of thousands. This is known as the “curse of dimensionality” problem, and the usual way to overcome this limitation is by using a dimensionality reduction technique to collapse all the attributes (variables) into a lower-dimension space where the most dominant information of the dataset can be retrieved , 22.Feature selection methods are usually separated into three categories: filter, wrapper, and embedded. Table The abovementioned filter methods tend to be computationally efficient. Wrapper methods, on the other hand, tend to have a better performance in selecting features since they take a model hypothesis into account, meaning that a training and testing procedure is made in the feature space. However, this approach is computationally inefficient and is more problematic as the feature space grows , 29, 30.In , authorsThe final step after finding the most relevant features of the transformed data is to select a classifier. In clinical and bioinformatic research, prediction models are extensively used to derive classification rules useful to accurately predict whether a patient has or would develop a disease, whether the treatment is going to work, or even whether a disease would recur –35. TablDepending on the data, a classifier can belong to one of two groups: supervised or unsupervised . In supeDembélé and Kastner  presenteBelciug and F. Gorunescu  proposedIn , authorsIn this paper, we present a new method to classify samples into two groups with different characteristics  when data of compositional nature is available. Our method relies on a new metric to quantitatively characterize the overall correlation structure deviation when comparing the two datasets and a new dimensionality reduction approach. The proposed method is assessed and compared, based on classification accuracy, to two state-of-the-art methods using both synthetic datasets and real datasets from RNA-16s sequencing experiments.Here, we explain in detail the proposed classification method. First, in section “Data pretreatment”, we introduce the Data Pretreatment stage, and in section “Assessing correlation structure distortion”, a novel metric to be used as the metric to assess correlation structure distortion is described. Finally, in section “Dimensionality reduction technique”, we present the proposed classification rule, which is based on the previously defined metric and a proposed dimensionality-reduction approach to assess the disruption of a dataset’s correlation structure after a new sample is included.m features are assessed in nc and nv samples from control and case individuals, respectively. In the expressions above, the superindex ρ indicates the datasets are ‘raw’ or without pretreatment. From now on, g = c for control, or g = v for case).Let ℝm1 ×   be the i-th row of the matrix i = 1, 2, …, ng). The BM algorithm replaces the null counts byWhen analyzing OTU counts tables, a log-ratio transformation, such as the clr, is to be applied , 18, 19 ti, j = m−1 and si = m.  Let When using the Bayes-Laplace prior, we set k is an arbitrary constant . Let xp ∈ ℝn1 × To ensure the data’s compositionality on peration , 18, 19 Xg, and Σg ∈ ℝm × m is a diagonal matrix that contains the standard deviation  of all variables.Finally, a normalization is applied, resulting in:φ, a new metric to quantitatively assess the distortion in the correlation structure of a dataset after the incorporation of a new sample. The Pearson correlation matrix for Xg is calculated as follows  Python package was used. The number of experimental replicates was B = 100.For every pair Merge each i = {1, 2, …, 2n}, and classify For every pair Compute the average classification accuracy as:N is the number of correctly classified samples.where We used the correct classification rate (accuracy) as the assessment criterion to measure the performance of our method as follows:m decreases and the sample size n increases.Table In this section, we study the performance of the proposed method using two real-world datasets, which contain OTU count tables obtained through 16S rRNA gene sequencing data from microbiota experiments. We also compare the classification accuracy of our method with those of two state-of-the-art methods: SVM  and SVM-The first dataset is from the American Gut Project (AGP) , which iC = 1 and γ = 0.05, where C trades off the correct classification of training examples against the maximization of the decision function’s margin, and γ defines how far the influence of a single training example reaches.Datasets were preprocessed as described in section “Data pretreatment”. Further, the proposed method, as well as the SVM and SVM-RFE methods, were applied after separating the whole data set into training, testing, and validation sets using 70, 20, and 10% of the data, respectively. For the SVM-RFE method, the number of features to select was Table For the GG dataset, although the number of variables is small, the SVM-RFE and our method showed accuracy values above 90%, while the accuracy for the SVM method is below this threshold. It is worth highlighting that, for this data set, our method outperforms both the SVM and SVM-RFE methods. The latter result is thanks to the artificial dimensional reduction conducted to balance the natural weights when the number of samples is greater than the number of variables. Figure The ability to characterize populations of patients, species, or biological features, usually comprising a large number of variables in order to use the extracted characteristics to classify new samples into one of such populations’ categories is a relevant tool for biological and medical studies. When data describing these populations is compositional, further limitations and challenges arise.Here, we proposed a new method to classify samples into one of two previously known categories. The method uses a new metric developed to quantify the overall correlation structure deviation between two datasets, and a new dimensionality reduction technique. Although we illustrated the usefulness of our proposal with compositional data, its application is not limited, under any circumstances, to data of this nature. In fact, when data is not compositional, the centered log-ratio transformation and the zero-replacement algorithm must not be applied.Validation with synthetic data showed that the proposed method achieves accuracy values above 98%. Moreover, comparison of the performance of our method with that of SVM and the SVM-RFE , using two real-world datasets from 16 s RNA sequencing experiments, showed that our method outperforms the SVM method in both data sets, outperforms the SVM-RFE method in the GG data set, and is a close competitor of the SVM-RFE method in the AGP data set.ρc = 0.1 and ρv = 0.2. Moreover, our method may be extrapolated for multi-category classification, and a performance assessment may be conducted to test its classification accuracy in non-binary scenarios.Future studies may address the ability of our proposed method to perform accurately for a broader range of dimensions (number of variables and samples) and assess its performance for more scenarios of dissimilar correlation structures other than that for


A question of fundamental biological significance is to what extent the expression of a subset of genes can be used to recover the full transcriptome, with important implications for biological discovery and clinical application. To address this challenge, we propose two novel deep learning methods, PMI and GAIN-GTEx, for gene expression imputation. In order to increase the applicability of our approach, we leverage data from GTEx v8, a reference resource that has generated a comprehensive collection of transcriptomes from a diverse set of human tissues. We show that our approaches compare favorably to several standard and state-of-the-art imputation methods in terms of predictive performance and runtime in two case studies and two imputation scenarios. In comparison conducted on the protein-coding genes, PMI attains the highest performance in inductive imputation whereas GAIN-GTEx outperforms the other methods in in-place imputation. Furthermore, our results indicate strong generalization on RNA-Seq data from 3 cancer types across varying levels of missingness. Our work can facilitate a cost-effective integration of large-scale RNA biorepositories into genomic studies of disease, with high applicability across diverse tissue types. High-throughput profiling of the transcriptome has revolutionized discovery methods in the biological sciences. The resulting gene expression measurements can be used to uncover disease mechanisms  and singular value decomposition (SVD) , and lower-case bold symbols to denote column vectors  denote scalar values or functions.In this section, we introduce two deep learning approaches for gene expression imputation with broad applicability, allowing us to investigate their strengths and weaknesses in several realistic scenarios. Throughout the remainder of the paper, we use script letters to denote sets  and k quantitative covariates , respectively. Our goal is to recover the original gene expression vector x ∈ ℝn by modeling the conditional probability distribution Consider a dataset We first introduce a novel imputation method named Pseudo-Mask Imputer (PMI).Formulation. Let m ∈ {0, 1}n. Our model is a function f : ℝn × {0, 1}n × ℝk × ℕc → ℝn that imputes the missing expression values (1 − m) ⊙ x as follows:Here ⊙ denotes element-wise multiplication. The recovered vector of gene expression values is then given by Optimization. We optimize the model to maximize the imputation performance on a dynamic subset of observed, pseudo-missing components. In particular, we first generate a pseudo-maskb ∈ {0, 1}n is a vector sampled from a Bernoulli distribution B and α ∈  and β ∈  are hyperparameters that parameterize a uniform distribution U. Using the pseudo-maskpseudo-observed components pseudo-missing components where pseudo-missing components:The imputed components are then given by where pseudo-mask mechanism generates different sets of pseudo-observed components for each input example, effectively enlarging the number of training samples. Specifically, the hyperparameters α and β control the fraction of pseudo-observed and pseudo-missing components through the probability p ~ U. On one hand, a low probability p yields sparse pseudo-observed vectors p yields denser pseudo-observed vectors p is set to 1 and the pseudo-maskm.Importantly, the Architecture. We model the imputer f as a neural network. We first describe how we use word embeddings, a distinctive feature that allows learning rich, dense representations for the different tissue types and, more generally, for all the covariates q ∈ ℕc.qj be a categorical covariate  with vocabulary size vj, that is, qj ∈ {1, 2, …, vj}, where each value in the vocabulary {1, 2, …, vj} represents a different category . Let qj = k and dj be the dimensionality of the embeddings for covariate j. We obtain a vector of embeddings Formally, let vj entries, where each entry contains a learnable dj-dimensional vector of embeddings that characterize each of the possible values that qj can take. To obtain a global collection of embeddings e, we concatenate all the vectors ej for each categorical covariate j:where each c is the number of categorical covariates and ∥ represents the concatenation operator. We then use the learnable embeddings e in downstream tasks.where f as follows:In terms of the architecture, we model where MLP denotes a multilayer perceptron and in-silico , encouraging a bijective association between noise components and genes. Before passing the output where Discriminator. The discriminator takes the imputed samples D : ℝn × ℝn × ℝk × ℕc → ℝn that outputs the probabilities Formally, the discriminator is a function i-th component ŷi is the probability of gene i being observed (as opposed to being imputed by the generator) for each i ∈ {1, …, n} and the vector h ∈ ℝn corresponds to the hint mechanism described in Yoon et al. , where α ∈  and β ∈  are hyperparameters. This accounts for a high number of genes n and allows to trade off the number of mask components that are revealed to the discriminator.where Optimization. Similarly to GAN and GAIN, we optimize the generator and discriminator adversarially, interleaving gradient updates for the discriminator and generator.q and r, and the hint vector h. Since the hint vector h readily provides partial information about the ground truth m (Equation 11), we penalize D only for genes i ∈ {1, 2, …, n} such that hi = 0.5, that is, genes whose corresponding mask value is unavailable to the discriminator. We achieve this via the following loss function The discriminator aims at determining whether genes have been observed or imputed based on the imputed vector Z = 1 + (1 − b)⊤(1 − b) is a normalization term. The only difference with respect to the binary cross entropy loss function is the dot product involving (1 − b), which we employ to ignore genes whose mask has been leaked to the discriminator through h.where The generator aims at implicitly estimating where Architecture. We model the discriminator D and the generator G using neural networks. Similar to PMI, D and G leverage independent instances eG and eD of the categorical embeddings described in Equation (6). Specifically, we model the two players as follows:where MLP denotes a multilayer perceptron and In this section, we provide an overview of the dataset and describe the experimental details, including all the different case studies and imputation scenarios that we considered. We also describe the implementation details of PMI see  and GAINDataset. The GTEx dataset is a public genomic resource of genetic effects on the transcriptome across a broad collection of human tissues, enabling linking of these regulatory mechanisms to trait and disease associations  vs absolute levels  is also biologically meaningful, with important applications, e.g., differential expression analysis  that is robust to expression outliers. To this end, we normalized the expression data via the standard score, so that the standardized expression values have mean 0 and standard deviation 1 for each gene across all samples.Training, validation, and test splits. To prevent any leakage of information between the training and test sets, we enforced all samples from the same donor to be within the same set. Concretely, we first flipped the GTEx donor identifiers , we then sorted the reversed identifiers in alphabetical order, and we finally selected a suitable split point, forcing the two sets to be disjoint. After splitting the data, the training set, which we used to train the model, consisted of ~60% of the total samples. The validation set, which we used to optimize the method, consisted of ~20% of the total samples. The test set, on which we evaluated the final performance, contained the remaining ~20% of the data.We benchmarked the methods on two case studies:Case 1: Protein-coding genes. As a first case study, we selected the intersection of all the protein-coding genes among the 49 GTEx tissues, resulting in a set of 12,557 unique genes. This case study is challenging for imputation methods that are not scalable across the number of input variables.Case 2: Genes in a pathway. We selected a subset of 273 genes from the Alzheimer's disease pathway extracted from the Kyoto Encyclopedia of Genes and Genomes  ⊙ x. Importantly, for this scenario we assumed that the data is missing completely at random  algorithm is an efficient method that is commonly used for imputation  Xeon(R) Processor E5-2630 v4. RAM: 125GB), after which we halted the execution. For example, MICE and MissForest were unfeasible for each imputation scenario on the complete set of protein-coding genes. An empirical study of the scalability of both methods (see hods see  showed thods see .R2). This metric ranges from −∞ to 1 and corresponds to the ratio of explained variance to the total variance. Negative scores indicate that the model predictions are worse than those of a baseline model that predicts the mean of the data. Here, to evaluate the performance, we generated random masks with a missing rate of 50% and computed the imputation R2 per gene. We repeated the last step 3 times and reported the overall mean R2 and the average per-gene standard deviation of the R2 scores, averaged across the 3 runs. In inductive mode, blood surrogate and median imputation exhibited negative scores. Under in-place imputation on the protein-coding genes, GAIN-GTEx outperformed all the other methods (0.638 ± 0.007). Under inductive imputation on the protein-coding genes, PMI showed the best overall performance (0.707 ± 0.001) among all the methods.In terms of the evaluation metrics, we report the coefficient of determination  to ~0.78 . Kidney cortex, the tissue with the smallest sample size, had the highest variability in R2 with an interquartile range of Q3 − Q1 = 0.30.j indexes the tissue dimension. Strikingly, the tissue representations showed strong clustering of biologically-related tissues, including the central nervous system , the gastrointestinal system , and the female reproductive tissues . The clustering properties were robust across UMAP runs and could be similarly appreciated using other dimensionality reduction algorithms such as tSNE  gene expression data in acute myeloid leukemia  and GAIN-GTEx, were able to approximate the gene expression manifold from incomplete gene expression data and relevant covariates  and impute missing expression values. A characteristic feature of our architectures is the use of word embeddings, which enabled to learn distributed representations of the tissue types see . Importanon-diseased individuals. In particular, we observed that EBV transformed lymphocytes, an accessible and renewable resource for functional genomics, are a notable outlier in imputation performance. This is perhaps not surprising, consistent with studies about the transcriptional effect of EBV infection on the suitability of the cell lines as a model system for primary tissues , allowing us to test the performance of our method in a broad collection of tissues see . The bioores see .R2 values and therefore did not perform well. Median imputation, although easy to implement, had a very limited predictive power. Imputation methods based on k-Nearest Neighbours were computationally feasible and yielded solid but poorer R2 scores. In terms of state-of-the-art-methods, MICE and MissForest were computationally prohibitive given the high-dimensionality of the data and we halted the execution after running our experiments for 7 days. In particular, we performed an empirical study of the scalability of both methods . Together with the fact that the pseudo-mask mechanism dynamically enlarges the training size, this subtlety allows training considerably bigger networks without over-fitting. Finally, we observed that a simplification of GAIN-GTEx, GAIN-MSE-GTEx, performed similarly well, suggesting that the mean squared error term of the generator's loss function is driving the learning process. In In terms of the comparison between PMI and GAIN-GTEx, our experiments suggest that the latter is generally harder to optimize  to perform imputation on The Cancer Genome Atlas gene expression data in acute myeloid leukemia, lung adenocarcinoma, and breast cancer. In addition to technical artifacts , generalizing to this data is highly challenging because the expression is largely driven by features of the disease such as chromosomal abnormalities, genomic instabilities, large-scale mutations, and epigenetic changes (Baylin and Jones, Broader Impact. The study of the transcriptome is fundamental to our understanding of cellular and pathophysiological processes. High-dimensional gene expression data contain information relevant to a wide range of applications, including disease diagnosis (Huang et al., In this work, we developed two methods for gene expression imputation, which we named PMI and GAIN-GTEx. To increase the applicability of the proposed methods, we trained them on RNA-Seq data from the Genotype-Tissue Expression project, a reference resource that has generated a comprehensive collection of transcriptomes in a diverse set of tissues. A characteristic feature of our architectures is the use of word embeddings to learn distributed representations for the tissue types. Our approaches compared favorably to several standard and state-of-the-art imputation methods in terms of predictive performance and runtime, and generalized to transcriptomics data from 3 cancer types of the The Cancer Genome Atlas. PMI and GAIN-GTEx show optimal performance among the methods in inductive and in-place imputation, respectively, on the protein-coding genes. This work can facilitate the straightforward integration and cost-effective repurposing of large-scale RNA biorepositories into genomic studies of disease, with high applicability across diverse tissue types.https://gtexportal.org/. A detailed summary of the GTEx samples and donor information can be found at: https://gtexportal.org/home/tissueSummaryPage. Our code is publicly available at https://github.com/rvinas/GTEx-imputation.The datasets analyzed for this study can be found in the GTEx portal: Ethical review and approval was not required for the study on human participants in accordance with the local legislation and institutional requirements. Written informed consent for participation was not required for this study in accordance with the national legislation and the institutional requirements.RV and TA developed and trained the deep learning algorithm, generated all the results and figures. ERG provided the standardized RNA-seq data. ERG and PL supervised the study as joint senior authors. All the authors wrote and approved the manuscript.Circulation Research of the American Heart Association, as a member of the Editorial Board. The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.ERG receives an honorarium from the journal


The R package will be deposited to CRAN as part of our PINSPlus software suite.Cancer is an umbrella term that includes a range of disorders, from those that are fast-growing and lethal to indolent lesions with low or delayed potential for progression to death. The treatment options, as well as treatment success, are highly dependent on the correct subtyping of individual patients. With the advancement of high-throughput platforms, we have the opportunity to differentiate among cancer subtypes from a holistic perspective that takes into consideration phenomena at different molecular levels . This demands powerful integrative methods to leverage large multi-omics datasets for a better subtyping. Here we introduce Subtyping Multi-omics using a Randomized Transformation (SMRT), a new method for multi-omics integration and cancer subtyping. SMRT offers the following advantages over existing approaches: (i) the scalable analysis pipeline allows researchers to integrate multi-omics data and analyze hundreds of thousands of samples in minutes, (ii) the ability to integrate data types with different numbers of patients, (iii) the ability to analyze un-matched data of different types, and (iv) the ability to offer users a convenient data analysis pipeline through a web application. We also improve the efficiency of our ensemble-based, perturbation clustering to support analysis on machines with memory constraints. In an extensive analysis, we compare SMRT with eight state-of-the-art subtyping methods using 37 TCGA and two METABRIC datasets comprising a total of almost 12,000 patient samples from 28 different types of cancer. We also performed a number of simulation studies. We demonstrate that SMRT outperforms other methods in identifying subtypes with significantly different survival profiles. In addition, SMRT is extremely fast, being able to analyze hundreds of thousands of samples in minutes. The web application is available at  Since cancer is a heterogeneous disease, the correct identification of cancer subtypes is essential for accurate prognosis and improved treatment. With the advancement of high-throughput platforms, subtyping methods have shifted toward multi-omics integration in order to differentiate between subtypes from a holistic perspective that takes into consideration phenomena at different molecular levels . Vast amounts of molecular data have accumulated in public repositories, including The Cancer Genome Atlas datasets (TCGA) , GenomicCurrent approaches for multi-omics integration and cancer subtyping can be categorized into four categories based on their integration strategy. The first strategy is to concatenate different types of data into a single matrix and then partition the patients using the concatenated data. For example, users can normalize and concatenate multiple data types  into one single matrix and then apply well-known methods developed for single-omics analysis, such as ConsensusClusterPlus , to deteThe second strategy is to model the multi-omics data as a mixture of statistical models. Methods in this category include LRACluster , rMKL-LPThe third strategy is to project all data types into a joint latent space. A common technique used for this strategy is non-negative matrix factorization. Methods in this category include MvNMF , MultiNMThe fourth strategy is also called similarity-based strategy. Methods in this category include SNF , PSDF 2, PFA 2626, IS-KmHere we introduce Subtyping Multi-omics using a Randomized Transformation (SMRT), a new method for cancer subtyping and big data analysis. This method offers important advantages over existing software: (i) it allows researchers to analyze hundreds of thousands of samples in minutes, (ii) it can integrate data types with different numbers of patients, (iii) the ability to integrate and analyze un-matched data of different types, and (iv) the web application offers a convenient data analysis pipeline. We also improve the efficiency of our ensemble-based, perturbation clustering to support analysis on machines with memory constraints. Our extensive analysis on 37 TCGA and two METABRIC datasets shows that SMRT is more accurate than state-of-the-art subtyping methods in identifying subtypes with significantly different survival profiles. In addition, our simulations with big data show that SMRT is fast and many-fold more scalable than existing methods. Specifically, SMRT is able to analyze hundreds of thousands of samples in minutes.The overall workflow of SMRT is presented in X ∈ ℝn×m as the input matrix, where n is the number of samples/patients, and m is the number of genes/features. Briefly, the RSVD method starts by generating a random projection matrix P ∈ ℝm×r from a standard normal distribution where r ≪ m. It then projects X ∈ ℝn×m to the column space of P to get a matrix Z such that Z = XP. Due to the random projection, Z and X will have approximately the same dominant columns (features). Now, we can obtain the orthogonalized matrix Q of Z by using QR decomposition, where Q has the same size as Z of n × r. In the next step, the method projects X into a smaller space to get a matrix Y ∈ ℝr×m such that Y = TQ^ *X and then computes singular value decomposition (SVD) of Y as Y = UΣV* using the traditional SVD method . For data with hundreds of thousands of dimensions , this step substantially reduces the required computational power while maintaining the clustering accuracy. Let us denote D method . U and Vsampled set by multiplying it with the rotation matrix obtained from the RSVD for the sampled set. This implementation allows us to perform SVD in at most a few seconds, even for datasets with hundreds of thousands of samples and features.In practice, RSVD is faster and requires less memory than the traditional SVD. To further speed up our approach, we implement a parallel version of RSVD that can efficiently utilize multiple cores available in modern processors. Note that when the input data is large , we do not perform RSVD on the whole input. Instead, we split the data into two sets of patients: a sampled set and a propagated set. We first perform RSVD on the sampled set, and then project the original data matrix (both sampled and propagated set) to the subspace of the The output of this module is multiple matrices – one per data type. In each matrix, the rows represent patients while the columns represent the principal components (PCA). These matrices will serve as input of the next module: perturbation clustering that will be described in the next section. This will compute the perturbed connectivity matrices and determine the subtypes.Given a single data type, SMRT utilizes our previously developed perturbation clustering (PINS) , 30 to pn × n for clustering (n is the number of patients). The time and space complexity (running time and memory usage) of this method increase quadratically when the number of samples increases. Therefore, when the number of samples is large , we perform a sub-sampling process over the original data to obtain a subset of 2,000 patients/samples. Next, we transform the data into a lower-dimensional space, and use the perturbation clustering to partition these patients. After this step, each of the 2,000 patients has a subtype. Let us refer to this selected set of 2,000 patients as the sampled set. The next step is to determine the subtypes for the rest of the patients, called the propagated set. For this purpose, we use the fast k-nearest neighbor searching algorithms (FKNN) algorithm . By default, k is determined using the Elbow method on the sampled set using 5-fold cross-validation. The sampled set is divided randomly into 5 equally smaller sets. In each round, the combination of 4 sets is used as the training set, and the other is used as the validation set for the KNN algorithm with k ranges from 5 to a maximum of 50. The k that yields the lowest average classification error rate will be used as the optimal k. However, users are also free to modify the value of this parameter. versus using a fixed number of k.When the number of samples is large, the perturbation clustering becomes slow and memory-inefficient. The perturbation clustering algorithm relies on the pair-wise connectivity of size lgorithm , 35 to asampled set by multiplying it with the projection matrix obtained from the RSVD for the sampled set. After this transformation, the pair-wise distance between patients will be calculated in the new space with a much lower number of dimensions.One note of caution is that the number of dimensions of the data can be high, thus slowing the process of distance calculation and neighbor finding. Therefore, instead of calculating the distance between patients in the original space, we calculate the distance between patients in the principal component (PC) space of the sampled set. As described above, we project the original data matrix (both sampled and propagated set) to the subspace of the n ≤ 2,000), we utilize an ensemble strategy to partition the patients. The method first clusters each data type  and constructs the perturbed connectivity matrices. It then merges the connectivity matrices of all data types to a single similarity matrix that represents the similarity between patients across all data types by averaging the connectivity values for each pair of samples. Next, to cluster the similarity matrix, it uses several similarity-based algorithms, including hierarchical clustering, partitioning around medoids  that have data in all data types, and the remaining patients will be in the propagated set.When the number of samples is large , we perform a sub-sampling and classifying procedure that is similar to the algorithm described in the Section 2.3. The difference here is that multiple data types are involved. First, we randomly select 2,000 samples/patients and then apply the multi-omics algorithm described above to partition the selected samples. We refer to this selected set of 2,000 patients as the http://SMRT.tinnguyen-lab.com. The website is built using the R Shiny framework  in In this article, we analyze 37 TCGA and 2 METABRIC datasets. For TCGA datasets, we downloaded the matched mRNA, DNA methylation, and miRNA expression data from the TCGA data portal. For the METABRIC datasets, we were able to obtain matched mRNA and copy number variation data from the European Genome-Phenome Archive. We also downloaded clinical data and survival information of each patient, which will be used to assess the performance of the subtyping methods. https://github.com/danro9685/CIMLR for CIMLR, NEMO v0.1.0 at https://github.com/Shamir-Lab/NEMO for NEMO, mogsa v1.16.0 on Bioconductor for moCluster, iClusterPlus on Bioconductor v1.18.0 for iClusterBayes, LRACluster v1.18.0 at http://bioinfo.au.tsinghua.edu.cn/member/jgu/lracluster/ for LRACluster, PMA v1.2.1 on CRAN for MCCA, and IntNMF on CRAN v1.2.0 for IntNMF. When the number of dimensions exceeded 2,000, we used only the top 2,000 variables with the largest variance for iClusterBayes, IntNMF, and MCCA, because these methods cannot analyze the data on the whole-genome scale. For all methods, we used default parameters and let all methods automatically determine the optimal number of clusters. For MCCA, which is not a clustering method itself, we follow the implementation at https://github.com/Shamir-Lab/Multi-Omics-Cancer-Benchmark for cluster analysis.We compare SMRT with eight state-of-the-art subtyping algorithms: SNF , CIMLR , NEMO 2, moClustUsing each method, we partition the patients in each dataset, and then assess the survival difference of the discovered patient groups using Cox regression . Overallp = 0.0002 using the one-tailed Wilcoxon test).The running time of each method is shown in To reveal the contribution of each data type, we used SMRT to partition the patients using each of the data types independently. Next, we calculated the Cox p-values obtained from each data type and compared them with those obtained from subtyping the multi-omics data. log10 of p-values for gender and age are shown in Next, we investigated the association between discovered subtypes and clinical variables. We performed our analysis on gender, age, cancer stage, and tumor grade, which are available for at least 15 datasets. We perform the following analyses: (1) Fisher’s exact test to assess the significance of the association between gender  and the discovered subtypes; (2) ANOVA to assess the age difference between discovered subtypes; and finally (3) calculate the agreement between the discovered subtypes and known cancer stages and tumor grades using Normalized Mutual Information (NMI). The distributions of –Here we perform an in-depth analysis for the GBMLGG (Glioma). We further conduct pathway analysis using the discovered subtypes on the Consensus Pathway Analysis platform  using tha posteriori to assess the accuracy of each clustering method. The memory of our server is limited to 376 GB.In order to assess the scalability of the nine subtyping methods, we generate a number of simulated datasets with a fixed number of genes/features of 5,000 and varying numbers of samples . In each dataset generated, there are three classes of samples – each with a different set of up-regulated genes. The true class information was used In this article, we introduced SMRT, a fast yet accurate method for data integration and subtype discovery. In an extensive analysis using 39 cancer datasets, we showed that SMRT outperformed other state-of-the-art methods in discovering novel subtypes with significantly different survival profiles. We also demonstrated that the method could accurately partition hundreds of thousands of samples in minutes with low memory requirements. At the same time, the provided web application will be extremely useful for life scientists who lack computational background or resources. Although the software was developed for the purpose of cancer subtyping, researchers in other fields can use the web application and R package for unsupervised learning and data integration.http://smrt.tinnguyen-lab.com/.Publicly available datasets were analyzed in this study. This data can be found here: HN and TN conceived of and designed the approach. HN, DT, and BT implemented the method in R, performed the data analysis and computational experiments. MR, AC, and SDa helped with data preparation and some data analysis. HN, DT, SD, and TN wrote the manuscript. All authors reviewed and approved the manuscript.This work was partially supported by NIH NIGMS under grant number GM103440, and by NSF under grant numbers 2001385 and 2019609.Any opinions, findings, and conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of any of the funding agencies.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.


Pan troglodytes) are a genetically diverse species, consisting of four highly distinct subspecies. As humans’ closest living relative, they have been a key model organism in the study of human evolution, and comparisons of human and chimpanzee transcriptomes have been widely used to characterize differences in gene expression levels that could underlie the phenotypic differences between the two species. However, the subspecies from which these transcriptomic data sets have been derived is not recorded in metadata available in the public NCBI Sequence Read Archive (SRA). Furthermore, labeling of RNA sequencing (RNA-seq) samples is for the most part inconsistent across studies, and the true number of individuals from whom transcriptomic data are available is difficult to ascertain. Thus, we have evaluated genetic diversity at the subspecies and individual level in 486 public RNA-seq samples available in the SRA, spanning the vast majority of public chimpanzee transcriptomic data. Using multiple population genetics approaches, we find that nearly all samples (96.6%) have some degree of Western chimpanzee ancestry. At the individual donor level, we identify multiple samples that have been repeatedly analyzed across different studies and identify a total of 135 genetically distinct individuals within our data, a number that falls to 89 when we exclude likely first- and second-degree relatives. Altogether, our results show that current transcriptomic data from chimpanzees are capturing low levels of genetic diversity relative to what exists in wild chimpanzee populations. These findings provide important context to current comparative transcriptomics research involving chimpanzees.Chimpanzees ( SignificanceST between them, they have much larger effective population size than humans and a complex past involving multiple admixture events. Using publicly available data from 486 RNA sequencing (RNA-seq) samples—the vast majority of all bulk RNA-seq chimpanzee data available on SRA—we characterize genetic diversity in current chimpanzee transcriptomic samples and find that transcriptomics falls very short of capturing extant genetic diversity in chimpanzees, with only 89 unique individuals in our sample, the majority of them from the same subspecies. We propose ways to ameliorate this unbalance, but given the status of chimpanzees as critically endangered, this will require proactive collaboration with zoos globally.Chimpanzees are genetically much more diverse than humans. Currently categorized into four subspecies with high FPan troglodytes) are one of two extant members of the genus Pan, estimated to have diverged from the ancestral human lineage between 6 and 13 Ma  and Eastern chimpanzees (Pan troglodytes schweinfurthii), found in equatorial Africa, and the second clade is comprised of the Western chimpanzees (Pan troglodytes verus), found in upper Guinea and the recently designated Nigeria-Cameroon (Pan troglodytes ellioti) chimpanzees from the Gulf of Guinea  . Howevereat apes . They caeat apes . The firf Guinea . Each suf Guinea , althougf Guinea . Pairwisstimates . There istimates .In transcriptomic studies of chimpanzees, tissue samples are often collected post-mortem from captive individuals, following death from unrelated causes. However, it is unclear how representative of the diversity in wild chimpanzees these small captive populations are. A genetic survey looking at European zoos and research institutes showed that the majority of individuals (70%) had some Western ancestry . A similPan troglodytes. We focused on 3,221 samples sequenced by Illumina HiSeq 2000, 2500, and 4000 to reduce biases in the data caused by different sequencing technologies and instruments. After removing single-cell RNA-seq samples and further stringent filtering steps , we retained 486 BioSamples from 40 different BioProjects  broadly recapitulates the supervised results and supports our ancestry inference approach  not belonging to a replicate set also had relatedness ≥0.5. As the 468 samples used in these analyses included both known duplicate samples and, likely given the relatively small captive chimpanzee population, close relatives, which Somalier is not designed to detect, this finding was not surprising. On the whole, these analyses suggested that we are well powered to determine whether sets of replicate samples in our data are actually identical or not.Given this, we then filtered our data set to retain only one sample from each replicate set , resulting in a set of 237 samples. An UPGMA dendogram  online oWe then repeated the Somalier analysis with the set of 237 samples B. AveragWe also noticed some likely sample swaps, where known duplicate samples failed to cluster together. For instance, four samples from S23 appeared to be mislabeled amongst themselves  revealedIn our Somalier analyses, 18 pairs of samples not assigned to the same duplicate cluster had relatedness > 0.5  online, Using these values as a guide, we took advantage of known relationships in our data set to define thresholds specific to our data. We first analyzed only the 38 samples from S31, which contains eight known pairs of first-degree  relatives . All eigWe applied the same approach to all research centers or zoos with more than five samples, as well as to the entire data set  online. Our analyses also readily identified four clear full sibling pairs, visible in the center of Our analyses identified between 132 and 135 genetically unique individuals in our data set; a simple UPGMA dendogram of all 135 is presented in As our analyses above demonstrated, these individuals are not unrelated, nor do they represent a random sample of the genetic diversity of the wild chimpanzee population. Beyond the three sample pairs that appeared to be genetically identical, 72 individuals had at least one second-degree relationship with another individual in the data set, and 36 were part of at least one parent/offspring or sibling pair; a single individual was part of 4. On average, any individual had at least 2.09 second-degree relatives in the data set, and 7 had over 10; in the most extreme instance we inferred 19 second-degree relationships involving S3-c34-M-31y. We iteratively excluded individuals with the most relationships until no first or second-degree relative pairs remained in the data set; this subset contained between 86 and 89 individuals, depending on which relative we removed from certain pairs.Furthermore, of the 135 individuals, 85 (63%) were entirely of Western ancestry (>93.75% as predicted by ADMIXTURE), and 130 out of 135 (96%) have significant (>6.25%) Western chimpanzee ancestry. There were 48 hybrid individuals, 42 of which are of two different ancestries and 6 of three different ancestries. The final data set contained only one Central chimpanzee, one Nigeria-Cameroon chimpanzee, and no Eastern chimpanzees at all.Genetic diversity, tissue type, and relationships amongst the 15/14 chimpanzee iPSC lines in our data set. Ancestry estimates are as in Finally, we considered the subset of samples from iPSCs or iPSC-derived cell types . As iPSCOf the 15 different individuals in our data with iPSC data, 6 are entirely Western and the rest are two-ancestry hybrids with at least one Western parent, again highlighting the difficulty of successfully capturing existing chimpanzee genetic diversity in functional genomics resources. Additionally, our IBD analyses suggested that C40210 and C8861 are second-degree relatives, as are C40210 and S008919. Notably, all but one of the sample swaps we described above involve iPSCs or iPSC-derived samples, emphasizing the increased complexity of cell culture relative to observational data generation from frozen tissues .Comparisons with chimpanzees are essential for understanding human evolution and the genetic basis of human-specific traits. However, chimpanzees are a critically endangered species, and increasing their captive population is unethical, and indeed illegal in many countries. Samples are therefore strictly limited to those from animals currently living in captivity, and to post-mortem tissue and cell line collections. These populations were established decades ago, sometimes with little regard to subspecies differences, and have not always been managed to maximize genetic diversity. Breeding practices in captive colonies, especially prior to the establishment of genetically guided breeding programs, have produced hybrids with highly variable ancestry components and this lack of specificity makes them a poor resource .Pan troglodytes verus) in public data sets. This is particularly noteworthy as, of the four extant chimpanzee subspecies, Western chimpanzees have the smallest effective population size, and show the highest amount of genetic drift  unique individuals and a maximum of 89 unrelated individuals within our initial set of 468 chimpanzee RNA-seq samples. At least 82 samples have a first or second-degree relationship with another sample in the data set. As expected from captive-bred individuals, some of these relationships are between samples from the same source. However, we see a significant number of cryptic relationships between institutes. This means that even sampling from multiple sources or from tissue repositories like GW could include some cryptically related samples.In addition, many individuals in our data have been sampled more than once. The least concerning scenario of repeat sampling is sequencing of different tissues from the same donor—but only 29 individuals have transcriptomic data from more than one tissue type. If we consider the 58 individuals with neural RNA-seq data, 37 appear in more than one study, and 21 of those only in the context of neural samples across different studies. As half of these samples are derived from the National Chimpanzee Brain Resource at George Washington University, a post-mortem brain bank, it is likely that these studies are repeatedly sequencing the same brain sample. Similarly, of the 60 heart/muscle samples, 40 have been used more than once, 17 only for heart/muscle purposes—all 17 have been used by both S31  and S28 Unlike with human studies, however, more diverse sampling of chimpanzees is largely not possible. The only currently viable solution is deliberate sampling of captive donors of specific subspecies, and the expansion of existing iPSC collections. Unfortunately, genetic surveys of captive chimpanzees in zoos and research institutes worldwide consistently find that over 75% of captive individuals are of solely Western ancestry , and indBecause so little is known about the variability of gene expression between chimpanzee subspecies, it is difficult to quantify the degree to which the poor representation of chimpanzee genetic diversity hinders inter-species comparisons. However, multiple lines of evidence argue that poor sampling is likely to lead to incomplete inferences. Although they had modest sample sizes and featured primarily individuals of Western ancestry, the two largest studies of gene expression in chimpanzees, Other organisms offer additional evidence. In mice, where the use of a single inbred strain in a study is common, the same local genotype can lead to very different phenotypes across strains and genetic backgrounds . In humaPan troglodytes, sequenced through 17 different technologies, with the main ones being the Illumina HiSeq 2000 (483 samples), Illumina HiSeq 2500 , and HiSeq 4000 instruments (422 samples). To avoid potential biases from calling SNPs on sequencing data generated with vastly different technologies , we only retained five randomly chosen replicates in order to reduce computational load; thus our description of sample swaps and mislabeled samples is limited. For paired-ended samples, we retained only the R1 file. Two data sets (PRJNA445737 and PRJDB1766) failed processing with GATK. We additionally excluded two data sets  as the individuals in these studies were already well-represented in our data set. Our final data set includes 486 BioSamples, 339 of which were single ended, and 147 pair ended, from 40 different BioProjects  online aBrain: brain, iPSC-derived cortical spheroids, iPSC-derived neural progenitors, iPSC derived neuronsHeart: heart, iPSC-derived cardiomyocytes, muscle, myoblastBlood: whole blood, peripheral blood mononuclear cells, lymphoblastoid cell lines, endothelial tissueSkin: skin, dermal fibroblastsiPSCsKidneyLiverTestisColonIn parallel, we downloaded the complete set of 28,559,256 SNPs genotyped in 59 wild-born chimpanzees from all four subspecies  generated by de Manuel et al. through high coverage whole-genome sequencing . We usedhttps://gatk.broadinstitute.org/hc/en-us/articles/360035531192-RNAseq-short-variant-discovery-SNPs-Indels-, last accessed November 18, 2021). We additionally incorporated an initial quality control step to this process, and used Trimmomatic 0.38  pipeline /) between each pair of samples. Approximately 1,000 SNPs with a minor allele frequency as close to 0.5 as possible are needed for a high-confidence analysis; we therefore considered only the 946 SNPs in our data with MAF ≥ 0.25. When filtering replicate or duplicate samples from clusters, we chose at random for sets that contained 2 individuals, and retained the one with the highest average relatedness to the rest of individuals in the cluster otherwise.We used PLINK to compute genome-wide IBS pairwise distances between all samples. We then subtracted the IBS values from 1 to get a distance matrix. Using the Because neither set of results provided enough resolution to detect higher-degree relationships, we also used PLINK to calculate pairwise IBD metrics Z0, Z1, and Z2 to identify relative pairs within our data set as in We used SmartPCA (part of Eigensoft 7.2.1)  to perfoK = 4, hoping to identify the four chimpanzee subspecies, but the presence of duplicates and significant admixture in the RNA-seq samples (both known and unknown) confounded the algorithm at K = 4 because identical individuals sampled repeatedly clustered together as separate populations. Therefore, we used the WGS samples as reference to perform a supervised ADMIXTURE analysis. RNA-seq samples were assigned a particular subspecies ancestry if that subspecies contributed more than 6.25% to its ancestry, equivalent to an unadmixed great-grandparent.We also used ADMIXTURE 1.3.0  to quantOur initial dendogram of 486 samples included a clade of 18 samples that had consistently large pairwise IBS distances (>0.1) with all other samples in the data  online. https://gitlab.unimelb.edu.au/navyas/chimpanzee-snp-calling .All analyses described were carried out using custom bash and R scripts, and are available at Genome Biology and Evolution online.https://www.biologiaevolutiva.org/tmarques/data . GEO IDs of all analyzed data sets are available in https://melbourne.figshare.com/articles/dataset/RNA-seq_derived_chimpanzee_genotypes/14822289 .All data sets in this study are publicly available. A VCF of genotype data from wild-born chimpanzees used as reference is available at evab247_Supplementary_DataClick here for additional data file.


N = 13,897) as well as adults . The analyses showed quite similar mean cut scores for the two standard-setting procedures in two of the samples, whereas the third sample showed more pronounced differences. Importantly, these findings demonstrate that model-based approaches provide a valid and resource-efficient alternative for external validation, although they can be sensitive to the ability distribution within a sample.In order to draw pertinent conclusions about persons with low reading skills, it is essential to use validated standard-setting procedures by which they can be assigned to their appropriate level of proficiency. Since there is no standard-setting procedure without weaknesses, external validity studies are essential. Traditionally, studies have assessed validity by comparing different judgement-based standard-setting procedures. Only a few studies have used model-based approaches for validating judgement-based procedures. The present study addressed this shortcoming and compared agreement of the cut score placement between a judgement-based approach  and a model-based one . This was performed by differentiating between individuals with low reading proficiency and those with a functional level of reading proficiency in three independent samples of the German National Educational Panel Study that included students from the ninth grade ( In order to justify high-stakes decisions and draw relevant conclusions about persons with low reading skills, stringent criteria are required to validly classify individuals as being at risk. Therefore, it is essential to use validated standard-setting procedures that allow for distinguishing between groups of individuals with different reading proficiency levels and facilitate a criterion-referenced interpretation of a person’s skills at a given proficiency level. In this context, validation studies play a crucial role in validating the chosen cut scores , 2. In pThe present study addresses this shortcoming and presents a validity study that investigates validity arguments for the interpretation of cut scores applied for the purpose of standard setting in a German large-scale assessment, the National Educational Panel Study , to difWe begin with a brief overview of the two standard-setting approaches, followed by the reasons for cross-validating the two approaches, before presenting the results of our empirical study.In practice, two classes of standard-setting approaches are used for setting proficiency levels. The first class incorporates subjective expert judgments either on test items (item-centered approaches), such as the Bookmark procedure , 6 and AP(xvi = 1) is the response probability for an individual v to solve item i, πg is the relative class size, and πig is the response probability for item i in class g (class-specific response probability). In contrast, MRM combines LCA and the Rasch analysis [P(xvi = 1) is the response probability for an individual v to solve item i, πg is the relative class size, θvg is the ability of an individual v given that he or she belongs to class g, and σig is the difficulty parameter of item i in class g [The second class of standard-setting procedures includes model-based classifications in order to assign individuals to a certain proficiency level –21. Two analysis –33 by as class g –33. To e class g –36, the Given that empirical support for validity is central to the conclusions that can be drawn on the basis of the cut scores, a number of studies examined classification agreement between the cut scores resulting from different standard-setting procedures. With respect to the Bookmark procedure, existing studies have largely compared the results of the Bookmark procedure with other judgement-based standard-setting procedures. The results show that different procedures sometimes produce very different cut scores, and the Bookmark procedure tended to produce lower cut scores and, therefore, a lower percentage of individuals falling into the lowest proficiency level –15. In cThereby, cross-validation of a judgment-based standard-setting approach with a model-based approach, as carried out in the abovementioned studies, offers several advantages compared to cross-validation between two similar approaches. First, in judgment-based standard-setting procedures, the setting of cut scores is per se a subjective process, as the procedures incorporate subjective expert judgments on test items or test takers –15, 37. In summary, both approaches have specific strengths and limitations. Using a combination of judgement-based and model-based standard-setting procedures could increase validity support, as the strengths of one approach can compensate for the limitations of another approach. As empirical evidence for the applicability of both the approaches for validation purposes is rare, our study aims to compare the cut scores obtained from the Bookmark procedure and a variant of mixture Rasch modeling.National Educational Panel Study (NEPS) that provides, among others, data on domain-specific competencies for different representative samples across the life course [10.5157/NEPS:SC4:10.0.0) who attended ninth grade across different schools in Germany . About 12.62% had a non-German background, meaning that their first language was not German. All the participants were tested for reading competence in wave 2, 2011/12 [This study involved the use of data from the German e course . The preure, see ). All th 2011/12 , by trai10.5157/NEPS:SC6:10.0.0), including adults living in private households in Germany who were born between 1944 and 1986  and N = 3,145  respondents, respectively. The participants’ ages ranged from 24 to 69 years, with means of M = 47.66 years (SD = 10.91) and M = 49.37 years (SD = 11.40), respectively. The proportion of respondents with a non-German background was 5.70% and 8.11%, respectively. All the participants were individually tested by trained interviewers in their private homes. Further information on the data collection process is available on the project website (www.neps-data.de).Samples 2 and 3 comprised two representative (non-overlapping) samples of the NEPS “Starting Cohort Adults” . The reres, see . The samres, see . The samIn order to obtain a coherent measurement of reading competence in different samples over the life course, age-appropriate tests were developed based on a common theoretical framework with a focus on the functional component of reading (literacy approach ). The NEM = -0.03 . The Adult Samples 1 and 2 had means of M = -0.04  and M = -0.36 , respectively . The PLDs included specific “can-do” descriptions of what low-literate readers probably can do and probably cannot do, thus providing guidance for the Bookmark procedure , the NEPS reading framework, the Bookmark procedure, and the PLDs. The panelists were invited to share their thoughts with the other panel members and to discuss emerging questions.Implementation of the Bookmark procedure: The Bookmark procedure took place in three rounds . In rounr = 1.00). The item difficulty parameters of the reading items from the Rasch models and final cMRMs are given in Standard setting generally assumes unidimensionality for the entire population in order to define proficiency levels. This is the case, for example, with the Bookmark procedure, which assumes that the test fits a unidimensional Rasch model , 6. As dn1 = 6,949, n2 = 6,948). The two split-half samples did not differ significantly on gender, migration background, age, or cognitive abilities .For each sample, cMRMs with one to seven classes were fitted to the data. All the models were estimated using maximum likelihood with heteroscedasticity-robust standard errors . To prevThe agreement between the Bookmark procedure and cMRM in the assignment of individuals to either the low-literacy or literacy group was examined by focusing on only two classes identified by the cMRM: the class with the lowest reading proficiency represented the low-literacy group, whereas all remaining individuals were pooled into the literacy group.A 2 × 2 contingency table see  was creaMcNemar’s χ2 test  was usedCohen’s Kappa κ measures the global agreement between the Bookmark procedure and cMRM . AccordiSensitivity and specificity are the most commonly used measures of the accuracy of a diagnostic test as compared to an existing reference test (gold standard) , 69. Thelow) and within the literacy assignment (DISlit).To overcome the challenge of defining a gold standard, when there is no perfect standard-setting procedure, we calculated an alternative measure for assessing agreement between both procedures, a within-proficiency level disagreement rate. It measures the proportion of agreement within the low-literacy proficiency assignment  following respective McNemar’s χ2 tests for the student sample and Adult Sample 1, while there was no significant differences for Adult Sample 2  is shown in κ coefficient of consistency was almost perfect at .85 and .93 for the student sample and Adult Sample 2, respectively, but relatively lower, albeit still substantial, at .71 for Adult Sample 1. For all samples, the cMRMs showed high sensitivity and specificity for detecting low-literate and literate readers, respectively, ranging between 94.42% and 98.11% sensitivity and 93.70% to 98.68% specificity. Furthermore, the disagreement rate showed that for students, the Bookmark procedure and cMRM disagreed by 24.75% for the low-literacy group and by 1.68% for the literacy group. For Adult Sample 2, the Bookmark procedure and cMRM disagreed by 11.95% for the low-literacy group and by 2.31% for the literacy group. For Adult Sample 1, the disagreement rate for the low-literacy group was 40.59% and 6.51% for the literacy group  and constrained mixture Rasch modeling (cMRM) (model-based approach). The comparison was conducted with three subsamples within the German National Educational Panel Study.Overall, the results showed high consistency in the proficiency level assignment in the student sample and Adult Sample 2, whereas more pronounced differences were observed in Adult Sample 1. The assignment to the respective proficiency levels in the student sample and Adult Sample 2 differed by approximately 2% between the two standard-setting procedures. Thus, neither procedure was significantly different, suggesting that both the Bookmark procedure and cMRM provided feasible cut scores for differentiating low-literate and literate readers, and therefore both can be used for further analysis. These results are quite notable, as both procedures differ substantially in their assumptions of the underlying data structures. In the Bookmark procedure, the calculation of the item difficulty parameters is based on item response theory (IRT) assuming a normal trait distribution of all persons on the trait continuum, whereas MRM assumes that the trait distribution is a mixture of normal distributions in latent classes –33. HoweIn light of prior discussions of the importance of using different standard-setting approaches for proficiency interpretation, the degree of inconsistency in classification was further not as pronounced as might have been expected. It might be argued that the high degree of consistency reflects the development of low literacy over the life course as the result of an unfavorable socialization in literacy that leaves behind a small and unique group of low-literate readers whose reading proficiency developed very slowly or perhaps even stopped and declined , 71. ThiOur results further showed that the choice of classification indicators is an important factor to consider when interpreting validity in assigning individuals to different levels of proficiency. For example, the results showed that sensitivity and specificity of the respective cMRMs were surprisingly high. But, as outlined above, sensitivity and specificity are somewhat problematic indicators because both assume the Bookmark procedure as the gold standard and, therefore, the results should not be overestimated. Our proposed indicator, the within-proficiency disagreement rate, provides an alternative, when no perfect gold standard exist, and has the distinct advantage of showing how two standard-setting procedures disagree in assigning individuals to a particular proficiency level.With regard to Adult Sample 1, the comparison yielded some larger discrepancies in the cut scores between the two standard-setting procedures that affected about 6% of the sample and occurred especially at the low proficiency level. The results indicate that the ability distribution may be a reason for the lower agreement. Although both adult samples covered a wide range of ability distribution, Adult Sample 2 had an overall lower reading competence and participants with lower abilities were more strongly represented in the low ability regions than were participants of Adult Sample 1. Therefore, Adult Sample 1 may not have provided the necessary differentiation in the low ability regions, which may have led to an overestimation of the “true” low reading proficiency level. The implication of this finding is that the cut score placement is sensitive to the variance of the sample’s ability distribution when using a cMRM. This underlines the need to ensure that the sample has sufficient variance when using a cMRM for setting standards. It is less likely that meaningful interpretations of the proficiency levels can be made if potential ability levels are excluded from or underrepresented in the range of possible cut scores. It is further possible that the larger discrepancies in Adult Sample 1 could be a result of a generally worse model fit of both, the Rasch model and cMRM. Therefore, further studies examining the influence of model fit on standard setting would be of interest.Moreover, in line with previous results, the comparison showed that the cut scores determined by the Bookmark procedure are lower than those resulting from other standard-setting procedures , 11–15. Another main implication is that the use of two different standard-setting procedures provides a promising alternative for cross-validation. On the one hand, the cMRM does not rely on external panelists’ judgements, which makes the method not only more objective but also less resource-intensive in comparison to other judgement-based procedures. Thus, it provides a promising alternative to previous validation studies. On the other hand, given the satisfactory agreement between the two procedures, the Bookmark procedure provides the necessary content validity, which the cMRM does not, and thus allows a criterion-referenced interpretation of a person’s skills at a given proficiency level.There are several limitations to this study, which will need to be addressed in future studies. First, the focus here was not to examine the effect of the ability distribution on estimating proficiency levels within the cMRM. In further research, it may be worth examining this relationship. The second limitation of this study is that only one cut score was provided for the Bookmark procedure, and, therefore, the proficiency levels had to be pooled from class 2 upwards in the cMRM. It would certainly have enhanced the significance of the comparison if multiple cut scores had been set for the Bookmark procedure as well. Thus, in future studies, comparison with multiple cut scores should be conducted to support generalizability of the results. And finally, a challenge with this sort of analysis is how reading proficiency was modeled by the test developers. Reading competence was modelled as a continuous trait, thus, the instruments were not designed to capture distinct well-defined proficiency groups. Although our findings support the validity of the proficiency levels, it is more difficult to have a clear signal in the data to detect latent proficiency classes when the instrument aims to capture a continuous trait. Therefore, further studies would be interesting to validate both approaches on instruments explicitly designed to capture different proficiency levels.S1 Table(DOCX)Click here for additional data file.S2 Table(DOCX)Click here for additional data file.S3 Table(DOCX)Click here for additional data file.S4 Table(DOCX)Click here for additional data file.S5 Table(DOCX)Click here for additional data file.S6 Table(DOCX)Click here for additional data file.S7 Table(DOCX)Click here for additional data file.S8 Table(DOCX)Click here for additional data file.S9 Table(DOCX)Click here for additional data file.S10 Table(DOCX)Click here for additional data file.S11 Table(DOCX)Click here for additional data file.S12 Table(DOCX)Click here for additional data file.S1 FigThe density plots display the reading ability distribution for the different samples according to the respective Rasch models. The dashed vertical lines mark the cut scores for the Bookmark procedure.(TIF)Click here for additional data file.S2 FigThe figure shows the univariate and bivariate model fit information for the respective Rasch models and final cMRM class solutions among the two split-half student samples and two adult samples. The univariate model fit compared predicted and observed frequencies of responses for all reading items marginally. The bivariate model fits compared predicted and observed frequencies of responses for each pair of reading items. Given the large sample size, standardized residuals > |6| were considered as having a noticeable item misfit. Both model approaches, the Rasch model and cMRM showed a comparable model fit and therefore indicate that both approaches represent an acceptable measurement model. For univariate model fit, 0.00% to 2.90% items showed a noticeable misfit. For bivariate model fit, 1.79% to 5.02% items showed a noticeable misfit. Note: Some local dependence is induced by the testlet design in NEPS, that is, that the reading tests consisted of five texts, each with a set of reading items referring to the same stimulus, and by specific response formats that consist of item bundles referring to a common stimulus.(TIF)Click here for additional data file.S3 Fig(TIF)Click here for additional data file.S4 FigThe figure shows the distribution of the most likely class membership depending on the WLE estimates of the person parameters by the two split-half student sample. As can be seen, there is a clear association between class membership and the WLE estimates; less competent persons are primarily belong to class 1 (here: low literacy group).(TIF)Click here for additional data file.S5 FigThe figure shows the distribution of the most likely class membership depending on the WLE estimates of the person parameters by the two adult sample. As can be seen, there is a clear association between class membership and the WLE estimates; less competent persons are primarily belong to class 1 (here: low literacy group).(TIF)Click here for additional data file.


Preoperative evaluation of lymph node (LN) state is of pivotal significance for informing therapeutic decisions in gastric cancer (GC) patients. However, there are no non-invasive methods that can be used to preoperatively identify such status. We aimed at developing a genomic biosignature based model to predict the possibility of LN metastasis in GC patients.n = 300) from Gene Expression Ominus (GEO). In the exploratory stage, 300 GC patients from GSE62254 were involved and the differentially expressed RNAs (DERs) for LN-status were determined using the R software. GC samples in GSE62254 were randomly allocated into a learning set (n = 210) and a verification set (n = 90). By using the Least absolute shrinkage and selection operator (LASSO) regression approach, a set of 23-RNA signatures were established and the signature based nomogram was subsequently built for distinguishing LN condition. The diagnostic efficiency, as well as the clinical performance of this model were assessed using the decision curve analysis (DCA). Metascape was used for bioinformatic analysis of the DERs.We used the RNA profile retrieving strategy and performed RNA expression profiling in a large GC cohort  and verification sets . DCA demonstrated the clinical value of this nomogram. Functional enrichment analysis of the DERs was performed using bioinformatics methods which revealed that these DERs were involved in several lymphangiogenesis-correlated cascades.In this study, we present a genomic signature based nomogram that integrates the 23-RNA biosignature based scores and Lauren classification. This model can be utilized to estimate the probability of LN metastasis with good performance in GC. The functional analysis of the DERs reveals the prospective biogenesis of LN metastasis in GC.The online version contains supplementary material available at 10.1186/s12885-021-08203-x. Globally, gastric cancer (GC) is the 5th most prevalent cancer type and the 3rd highest cause of cancer-associated mortalities . Some sthttp://www.ncbi.nlm.nih.gov/geo/)in its processed format, using the package ‘GEOquery’ in R. The overview of the screening strategy used in this study is shown in Fig. https://www.nature.com/articles/nm.3850) on May 20th, 2020. The dataset obtained from the GEO database had been anonymized and, therefore, ethical approval was waived. The samples in GSE62254 were randomly clustered into a learning set and a verification set.Gene expression information and sample data from GSE62254 dataset in this research were retrieved from GEO (https://asia.ensembl.org/index.html) for RNAs annotation on May 20th, 2020. Samples in the GSE62254 dataset were divided into LN-negative and LN-positive arms according to the source information. The differentially expressed RNAs (DERs) were identified using the package limma [Human gene annotation files (GRCh38.p12) were obtained from the Ensembl repository (ge limma . DERs wege limma . A volcage limma .i) were screened out from the DERs. For each gene, univariate analysis was performed to investigate the association between gene expression levels and lymph node metastasis levels. A risk score was calculated for each patient using the linear combination of expression data (Expi) of selected genes that were weighted by their corresponding coefficients (βi) and intercept. Based on the above process, a risk-score formula was developed as:The least absolute shrinkage and selection operator (LASSO) regression approach which is applicable in the regression analysis of high-dimensional data was performed using the R package “glmnet” . For higi × Expi) + InterceptRisk score (RS) = The R package “OptimalCutpoints” was applied in determining the optimum cutoff point for risk score. The optimum cutoff was employed to cluster the patients into high- or low-risk classes. It was obtained when the Youden index in receiver operating characteristic (ROC) curve predicting LN metastasis reached its maximum in the learning set. Samples were clustered into high- or low-risk clusters by utilizing the optimum cutoff.Candidate predictors including age, sex, Borrmann classification, Lauren classification, tumor location and the risk score were embedded into the logistic regression analysis to design a diagnostic model for predicting LN metastasis in the learning set , 22. To http://metascape.org/gp/index.html) was used to predict the potential biological functions of the differentially expressed genes [Metascape  or R software (version3.5.3). All tests were dual-sided and n = 210, Additional file n = 90, Additional file Samples in the GSE62254 dataset were randomly clustered into a learning set  and LN-positive (262 samples) groups. 186 DERs *ExpRHOA + 0.0452*ExpIGFBP2 + 1.4984*ExpC11orf80 + (− 0.0937)*ExpZNF74 + (− 0.9888)*ExpFOXN2 + 0.6580*ExpGOLGA8A + 0.9803*ExpRSRP1 + (− 0.4094)*ExpUSP10 + 0.3896*ExpCLTB + (− 1.2924)*ExpPIK3R1 + 1.5335*ExpPABPN1 + (− 0.3669)*ExpCLCN4 + (− 1.4978)*ExpPARD6B + 0.0329*ExpTRPA1 + (− 0.0174)*ExpBAG3 + 0.4511*ExpZNF26 + 0.0381*ExpGDPD3 + 1.1286*ExpSPTBN5 + 2.3647*ExpKLHL28 + 1.0420*ExpGTPBP8 + 2.5667*ExpTXNDC11 + 0.1489*ExpTMEM163 + Intercept.p < 0.05 Additional file We also compared the expression of each of the 23 genes between LN-positive and the LN-negative groups. Most of the genes were correlated with LN metastasis (p < 0.05) are shown, in Fig. n = 177) while the rest (n = 33) were assigned to the low-risk group  while the rest (n = 30) were assigned to the low-risk group  and risk score  were confirmed as independent risk factors for LN metastasis Table . LN metaMetascape was used for cascade and process enrichment analysis of the DERs  triggered chronic gastritis and successive atrophic and metaplastic gastritis [Based on the Lauren classification, GC can be grouped into intestinal or diffuse kinds . The intastritis . The difastritis , 37. Difastritis –40. Our We constructed and verified a diagnostic, genomic biosignature based nomogram as a noninvasive strategy for preoperative estimation of LN metastasis in GC. This nomogram incorporates two items of genomic signature based risk scores and Lauren classification. Though deviation was obviously found in the verification set when probability was below 75%, the nomogram exhibited ideal coincidence to the authentic probability in the learning set. The possible reason for deviation observed in the verification set may be the predictive model has an over-fitting problem as it was built based on data from the learning set. Therefore, it did not perform as well in the verification set as it did in the learning set when predicting LN metastasis. The areas under the ROC curve for the learning and verification sets implied that the nomogram had good utility. The DCA is a simple method for evaluating the clinical performance of a prediction model. It can quantify different strategies and determine an optimal threshold range. This LN metastasis prediction model can assist surgeons to balance between the quality of life and aggressive lymphadenectomy.To provide insights into the potential biological processes, “metascape” was performed for the functional and enrichment analysis of DERs. The DERs were enriched in three signaling pathways, including PDGF signaling, Interleukin-7 signaling and in the Intrinsic pathway for apoptosis. The PDGF receptor cascade constitutes a signaling network that is essential for the growth of cells of mesenchymal parentage. Dysregulation of this pathway can lead to extracellular matrix reconstruction in a tumor-enhancement manner to promote the migration, infiltration, angiogenesis, and lymphangiogenesis , 42. ForCompared to previous nomograms –17, our In conclusion, this nomogram incorporates both genomic signature based risk score and Lauren classification to estimate LN metastasis in preoperative GC.Additional file 1. Learning set.Additional file 2. Verification set.Additional file 3. 186 DERs were reduced to 23 RNAs on the basis of 210 patients in the learning set.Additional file 4 Table S1 These 23 RNAs with their corresponding coefficients and univariate analysis result between gene expression level and lymph node metastasis level. p-values are based on t-test.Additional file 5 Figure S1 The optimum cutoff point of risk score. (a) The cutoff value (1.3806) was acquired when the ROC curve reached optimum sensitivity (94.05%) and specificity (88.00%) for predicting LN metastasis. (b) The cutoff value (1.3806) was acquired when Positive Predictive Value (PPV) reached 98%.Additional file 6. Risk score and risk status for each sample in Learning set.Additional file 7. Risk score and risk status for each sample in Verification set.Additional file 8. Figure S2 Decision curve analysis for the genomic signature based nomogram.Additional file 9. Metascape Analysis result.Additional file 10 Fig S3 Network of enriched terms: (a) Colored by the cluster-ID, in which the nodes with similar cluster ID are frequently close to each other. (b) Colored by p-value, in which the terms with more genes tend to have a more remarkable p-value.


Gene-set analysis (GSA) is a standard procedure for exploring potential biological functions of a group of genes. The development of its methodology has been an active research topic in recent decades. Many GSA methods, when newly proposed, rely on simulation studies to evaluate their performance with an implicit assumption that the multivariate expression values are normally distributed. This assumption is commonly adopted in GSAs, particularly those in the group of functional class scoring (FCS) methods. The validity of the normality assumption, however, has been disputed in several studies, yet no systematic analysis has been carried out to assess the effect of this distributional assumption. Our goal in this study is not to propose a new GSA method but to first examine if the multi-dimensional gene expression data in gene sets follow a multivariate normal (MVN) distribution. Six statistical methods in three categories of MVN tests were considered and applied to a total of 24 RNA data sets. These RNA values were collected from cancer patients as well as normal subjects, and the values were derived from microarray experiments, RNA sequencing, and single-cell RNA sequencing. Our first finding suggests that the MVN assumption is not always satisfied. This assumption does not hold true in many applications tested here. In the second part of this research, we evaluated the influence of non-normality on the statistical power of current FCS methods, both parametric and nonparametric ones. Specifically, the scenario of mixture distributions representing more than one population for the RNA values was considered. This second investigation demonstrates that the non-normality distribution of the RNA values causes a loss in the statistical power of these GSA tests, especially when subtypes exist. Among the FCS GSA tools examined here and among the scenarios studied in this research, the N-statistics outperform the others. Based on the results from these two investigations, we conclude that the assumption of MVN should be used with caution when evaluating new GSA tools, since this assumption cannot be guaranteed and violation may lead to spurious results, loss of power, and incorrect comparison between methods. If a newly proposed GSA tool is to be evaluated, we recommend the incorporation of a wide range of multivariate non-normal distributions or sampling from large databases if available. Gene-set analysis (GSA) is one of the standard procedures used in biomedical research when interest lies in the evaluation of certain biological functions, such as pathways, or of the collective effect of a set of genes on disease status, phenotype, or other response variables (overrepresented (or enriched) in a candidate gene-set/pathway; that is, if the number of DE genes in the set is beyond chance alone. When this is the goal, an over representation analysis test is carried out based on the hypergeometric distribution, chi-square test, or Fisher’s exact test , and pathway topology methods. Several studies have reviewed and compared these different GSA categories  or log-normal distribution. Despite doubts about this assumption, these distributions are frequently used to generate expression values in simulation studies when evaluating the performance of a newly proposed GSA tool, especially those GSAs categorized as FCS methods. In other words, many GSA tools inherently assume that the gene expression data are normally or log-normally distributed, or that the GSA tests are robust to the distributional assumptions. Since the normality assumption may not be true or guaranteed  in a logistic regression model  and The Cancer Genome Atlas (TCGA) websites, and the scRNA-seq data (rma in R package affy) and quantile normalization  for data processing. Next, considering that expression levels are affected by age, gender, ethnicity, cancer grade and stage, these downloaded data were stratified according to these factors, if available, and then the groups containing a larger number of samples were selected for the following analyses. For instance, in the first data set  sample Energy test  were selected deliberately. These signaling pathways were constructed based on interacting molecules involving specific biological functions, where the gene nodes in the pathway network (gene-set) would be expected to exert some correlation. Such pathways targeted by GSA therefore are the focus of our examination of the normality test. The 10 signaling pathways defined in KEGG are the p53, mTOR, Jak-STAT, PI3k-Akt, Wnt, ErbB, MAPK, RAS, TGF-β, and TNF pathways, which contain 60, 69, 32, 85, 76, 47, 115, 69, 58, and 82 gene nodes, respectively. These have been reported to associate with one or more of the cancers considered here. For example, the p53 pathway was reported to show association with breast  in the logistic regression model , where both response groups, termed as case and control with 50 subjects in each group, were generated from MVN distributions with different mean vectors (the difference is denoted as  = 0.1).To evaluate the performance of these GSA tools with respect to different distributions of the gene expression values, the following scenarios were considered. The first two are single-component models. Simulation setting A was designed under the control) and 2 (case) in setting A:Response group 1 ( = 0.5.Setting B aims for a larger heterogeneity than A and the data were instead generated from multivariate t distributions (MVT) with 3 degrees of freedom, the same mean vectors as in A and a compound symmetry covariance matrix with control group, the mean vectors of the two component MVNs are the zero mean vector case group, the two mean vectors are In this setting, we consider distributions containing more variation than the above scenario by assuming a mixture of two distributions. In setting C, the expression values were randomly generated from a mixture of two MVN distributions, each with 50% weight. Specifically, in the control) in setting C:Response group 1  group contains more than one component distribution. These situations can arise due to disease subtypes such as cancer grade, staging, and tumor tissue type  and a 2-component mixture of t distributions (setting F), respectively; whereas in settings G and H, it is a 3-component mixture distribution. In contrast to the previous scenarios where both distance between the distribution of the simulated data and The use of mixture distributions for tumor subtypes incorporates the scenario of skewed distributions which are apparently non-normally distributed  or 5 (if the number is below 30) genes from each pathway (for the array RNA and RNA-seq data) or from each chamber or subcluster (for the scRNA-seq data) and carried out the above six MVN tests. This procedure was repeated 1000 times and the rejection rate, the proportion P-values generating the P-values  subjects. Three selective results from setting A, D, and H are shown in In each of the three simulation settings, the difference in the mean of the two groups is denoted as Generally, all tests maintain a reasonable type I error rate at the nominal level 0.05 . The R code for the four MVN tests  can be downloaded from https://github.com/r05849032/Four_MVN_tests. For the parametric and nonparametric FCS GSA tools examined in this study, the software and functions are listed in All the human microarray RNA and RNA-seq gene expression data included in this study can be freely downloaded from the NCBI or TCGA website. GEO accession number or TCGA identification are listed in https://github.com/r05849032/Four_MVN_tests.G3 online.jkab365_Supplementary_FiguresClick here for additional data file.jkab365_Supplementary_TablesClick here for additional data file.jkab365_Supplementary_FileS1Click here for additional data file.jkab365_Supplementary_FileS2Click here for additional data file.