Antimicrobial resistance (AMR) poses a threat to public health. Clinical microbiology laboratories typically rely on culturing bacteria for antimicrobial-susceptibility testing (AST). As the implementation costs and technical barriers fall, whole-genome sequencing (WGS) has emerged as a ‘one-stop’ test for epidemiological and predictive AST results. Few published comparisons exist for the myriad analytical pipelines used for predicting AMR. To address this, we performed an inter-laboratory study providing sets of participating researchers with identical short-read WGS data from clinical isolates, allowing us to assess the reproducibility of the bioinformatic prediction of AMR between participants, and identify problem cases and factors that lead to discordant results. We produced ten WGS datasets of varying quality from cultured carbapenem-resistant organisms obtained from clinical samples sequenced on either an Illumina NextSeq or HiSeq instrument. Nine participating teams (‘participants’) were provided these sequence data without any other contextual information. Each participant used their choice of pipeline to determine the species, the presence of resistance-associated genes, and to predict susceptibility or resistance to amikacin, gentamicin, ciprofloxacin and cefotaxime. We found participants predicted different numbers of AMR-associated genes and different gene variants from the same clinical samples. The quality of the sequence data, choice of bioinformatic pipeline and interpretation of the results all contributed to discordance between participants. Although much of the inaccurate gene variant annotation did not affect genotypic resistance predictions, we observed low specificity when compared to phenotypic AST results, but this improved in samples with higher read depths. Had the results been used to predict AST and guide treatment, a different antibiotic would have been recommended for each isolate by at least one participant. These challenges, at the final analytical stage of using WGS to predict AMR, suggest the need for refinements when using this technology in clinical settings. Comprehensive public resistance sequence databases, full recommendations on sequence data quality and standardization in the comparisons between genotype and resistance phenotypes will all play a fundamental role in the successful implementation of AST prediction using WGS in clinical microbiology laboratories. Sequence read files for all samples used in this study have been deposited in the European Nucleotide Archive under the project accession number PRJEB34513 and the following sample accession numbers: SAMEA5789893 (sample A-1), SAMEA5789894 (sample A-2), SAMEA5789895 (sample B-1), SAMEA5789896 (sample B-2), SAMEA5789897 (sample C-1), SAMEA5789898 (sample C-2), SAMEA5789899 (sample D), SAMEA5789900 (sample E), SAMEA5789901 (sample F), SAMEA5789902 (sample G).Antimicrobial resistance (AMR) is now recognized as a worldwide public-health issue, and identifying those infections that are resistant to common antibiotics quickly and accurately is a leading priority. The improvement of molecular methods of analysing bacterial DNA, especially whole-genome sequencing (WGS), has raised the possibility of using it as a single assay that can identify the pathogen, antibiotic susceptibility and track transmission. In this study, we compared methods for predicting AMR from bacterial DNA sequences through an inter-laboratory study. This is, to the best of our knowledge, the first study of its kind to blind sets of participants to any contextual information on the samples they were analysing and they were free to choose any analytical pipeline they wanted. This led to variation among the methods used, but also variation in the results reported. Inter-laboratory studies such as these are useful as a precursor to the formal external quality-assurance schemes that come later when assays have been embedded into clinical service. We have shown that although there were discrepancies between results reported, these discrepancies could be traced back to problems such as sequence quality, database choice and user error, all of which can be addressed for WGS to fulfil its potential in clinical settings.Antimicrobial resistance (AMR) is a major, global, public-health threat, with projections of up to 10 million deaths per annum by 2050 . MoleculWhole-genome sequencing (WGS) is emerging as a routine clinical test that could be used to determine the bacterial species, undertake transmission tracking and identify multiple AMR-associated mutations and genes in a single assay . Whilst There are currently a wide array of bioinformatics tools and pipelines to predict AMR from WGS data . These hGaining laboratory accreditation is an important, often essential, step for tests in clinical microbiology, but is less advanced for clinical bioinformatics due to its comparatively recent development. Bioinformatic reproducibility studies have been performed for clinically relevant bacterial sequence typing methods . HoweverThis multi-centre study used genomic DNA sequences from clinical carbapenem-resistant organisms, specifically chosen to be of varying quality and complexity, to identify the range of methods used and contributors to discordant AMR predictions. Participants included a mixture of independent individuals and teams using non-commercial AMR prediction pipelines from research groups, hospital laboratories, public-health laboratories and clinical diagnostic companies. The observations made underpin our recommendations for future method developments.Enterobacterales and also Acinetobacter baumannii, and contained six distinct families of carbapenemase genes  were generated from seven clinical isolates . The bacteria were isolated between 2014 and 2017 from stool specimens from patients attending Great Ormond Street Hospital (GOSH), UK, or University Hospital Galway (UHG), Ireland. They represented six clinically relevant bacterial pathogens, including diverse se genes .http://www.eucast.org) and meropenem, ertapenem, cefotaxime, amikacin, gentamicin and ciprofloxacin. The isolates were confirmed as carbapenemase producers by PCR at a reference laboratory .Phenotypic AST was performed at UHG and GOSH using the European Committee on Antimicrobial Susceptibility Testing (EUCAST) disc diffusion method  using DNA Blood 350 µl kits with an additional bead beating step. For eight samples, the NEBNext Ultra II DNA library prep kit (New England Biolabs) and NextSeq (Illumina) 150 bp paired-end sequencing was used. For two samples, the Nextera DNA library prep kit (Illumina) and HiSeq 100 bp paired-end sequencing was used . The fasPotential inter-laboratory participants were invited in an individual capacity, both in person and by email, at the meeting ‘Challenges and New Concepts in Antibiotics Research’, March 2018, at Institut Pasteur, France. Fifteen individuals were also emailed directly to participate in the study. From those invited, nine sets of participants agreed to take part in the study. We will refer to these sets as ‘participants’ throughout. These participants were labelled Lab_1 to Lab_9; ‘Lab’ is used as a catch-all term for an individual or team of participants, who came from a mixture of research groups, hospital laboratories, public-health laboratories and clinical diagnostic companies. All participants agreed to take part in a personal capacity using non-commercial pipelines under the condition of anonymity of the results. Each participant was not made aware who the other invited participants were at that stage.Participants were sent ten paired fastq files (labelled AMRIL_1 to AMRIL_10) and were blinded to their contents. The samples included two exact duplicates A-1 and A-2 (renamed copies of the same fastq files). Two duplicates with different depths of coverage, B-1 and B-2 . Two samples sequenced from the same isolate, C-1 and C-2 . The remaining four samples, D, E, F and G, represented diverse bacterial species and carbapenemases.Participants were asked to report a species identification for each pair of fastq files provided, as well as the presence of all AMR-associated genes present in that sample. They were asked, using the above data, to make a categorical prediction on whether that sample would be resistant to ciprofloxacin, gentamicin, amikacin and cefotaxime. Lastly, participants were asked to provide a description of the analysis pipeline they used.Participants returned results via an Excel spreadsheet . Results were collated for all species identifications and resistant or susceptible predictions from each participant. Collated AMR-associated genes had each name manually checked between each participant to identify minor differences in nomenclature used.https://github.com/phe-bioinformatics/kmerid). Four participants also used the web-based tools wgsa (https://pathogen.watch/), blast (https://blast.ncbi.nlm.nih.gov/Blast.cgi) and KmerFinder (https://cge.cbs.dtu.dk/services/KmerFinder/). All participants identified species from raw reads, apart from three participants that used assembled reads . Lab_3 used both raw reads and assemblies to assign species ID using mash and wgsa, respectively. Six of the nine participating laboratories assembled the raw reads into a draft assembly before identifying AMR-associated genes. Only Lab_4, Lab_7 and Lab_9 used methods that required no assembly of the reads. Of those participants assembling their reads, SPAdes  and arg-annot’s reference sequence from an A. baumannii integron (NCBI accession no.: HM036079). While there is variation at the nucleotide level, both encode the same IMP-1 enzyme.All isolates included in this study were carbapenem resistant. The reporting of carbapenemase genes from WGS from all participants matched the reference PCR result in 91 % of cases (91/100) . Eight otet(A), tet(B) and tet(C)], some aminoglycoside modifying enzyme gene variants [aac(3)-IIa and aac(3)-IIc] and β-lactamases (blaACT-14 and blaACT-18). We also observed differences between the same participants analysing samples from the same original isolate. Due to the very low read depth, the genes reported in B-1 bore little resemblance to B-2 across all participant results. However, even in the samples from the same isolates with sufficient sequencing depth (C-1 and C-2), we observed differences in the genes identified in four out of nine participants. This suggests that resequencing, and even small increases in read length and quality, can produce variation in results. It is worth noting that all but one of these differences were additional genes identified in C-2, which had a higher read depth than B-2 (156 vs 37× median read depth). The additional genes in C-2 included ant(3′′)−Ia (Lab_2 and Lab_8), fosA7 (Lab_2 and Lab_8) and tet(C) (Lab_3), but the reported reference breadth of coverage of ant(3′′)−Ia and fosA7 was low  and the sequence similarity between the purported tet(C) sequence and the reference was also low (75%). We also found no systematic differences in genes present or absent between those participants that used tools that required assembly of short reads first and those that took unassembled short reads as input .We compared all AMR-associated genes identified by each participant in each sample. As previously noted, the largest discrepancies were the 55 efflux pump gene sequences that were present only in card . To understand the other factors influencing discordant reporting, we removed these genes that were only present in one database from our comparisons . A pairwGiven the differences in the AMR-associated genes identified in the samples by each participant, we also compared predictions of antibiotic resistance to phenotypic AST results and each other. Two participants (Lab_2 and Lab_4) did not submit any results for phenotypic resistance prediction, so were not included in the subsequent analysis. A pairwise comparison between genotypic prediction results reported by all participants, on all antibiotics and samples, showed an overall consensus of 79 %  and the interpretation of those results. Where single gene calling is required (e.g. presence of a carbapenemase), results are mainly affected by sequence quality. However, once multiple genes are involved, all three analytical issues become important. We found the largest contributors to discrepant results between the gene variants reported in each sample and the phenotypic resistance predictions were the sample sequence quality, read depth and the choice of reference resistance-gene database. Samples must be sequenced to a sufficient depth as well as sufficient breadth of coverage for the expected size of the genome, usually inferred by mapping to a suitable reference genome, of at least above 90 %. Based on our own experience and these results, we recommend 30× depth as a lower limit. This also tends to be a default setting for many read assembly tools, but generally most samples should have a higher depth of coverage than this for meaningful prediction. Some participants did flag that they would not normally analyse the low depth of coverage samples  and if those samples are excluded from this analysis sensitivity in comparison to phenotypic AST rises from 76 to 98 %. This is highly encouraging as it suggests that as long as the sequence data produced is of sufficient depth and quality (e.g. current Illumina error rates) genotypic prediction of resistance phenotype can be comparable to AST. However, we also note that many sets of participants provided little information on their employment of quality control and filtering steps. Our results, therefore, suggest an increased emphasis on data quality control is highly relevant to improving sensitivity. Conversely, we have observed the choice of sequencer and DNA library preparation method has a small effect on closely related gene variants, but little discernible effect on the inference of resistance phenotype.blaIMP-1 in card and arg-annot). If precise identification of gene variants is required, we would strongly recommend avoiding this, as it effectively leads to ‘double-dipping’ using the same reads. Multiple reference databases could be used, but after screening for reads that have already been assigned a hit against one of the databases. This would avoid multiple different genes reported at the same genomic loci. However, it would be better to merge the different reference databases and remove the redundant sequences before comparisons are made against the test data. Sequence identity, and to lesser extent breadth of coverage cut-offs, should be kept high when comparing test data to a reference database. Based on this study, we would recommend using a sequence identity cut-off of at least 90%, in combination with an up to date curated reference resistance-gene database. Although lowering of these thresholds does identify more candidate genes within a sample, many were false negatives; thus, not improving concordance with phenotypic AST results in this study.Some participants ran the same set of read data against different reference databases and merged the results, which led to different gene variants being reported at the same loci. In practice, different variants of the same gene may not always result in a different clinically relevant phenotype. However, we also found reference sequences in different databases for same gene variant can differ by 15 % nucleotide identity . Web-hosted tools that provide a ‘black box’ solution to assembly and identifying resistance from uploaded WGS data should be avoided if possible, because of the lack of interpretability. Tools are needed that are open source, designed for clinical purpose and can be subjected to thorough troubleshooting when erroneous results arise [A limitation of this study is that we focused on the use of short-read sequence data, which produces sequences far shorter than the length of genes being identified. However, we feel this is more reflective of the WGS data that is more routinely generated in clinical laboratories at this point in time. If these short reads need to be assembled into longer contiguous sequences, we found it essential to use an actively developed short-read assembler such as SPAdes (ts arise . To thisOther limitations of this study include our focus on acquired genes rather than point mutations or many of the other resistance mechanisms found in bacteria (e.g. target site modifications and efflux pumps). We also only required reporting on categorical resistance predictions. Furthermore, because our focus was on WGS, and although we validated AST at two independent laboratories, we did not investigate potential variability and discordance in phenotypic prediction. More work needs to be done on the prediction of minimum inhibitory concentrations (MICs) from WGS data before it can be implemented in laboratories. This will be aided by more systematic reporting of accompanying MIC data when making WGS data available.We have outlined recommendations for improving the current state of prediction of AMR from WGS data. Some of these recommendations, such as a standardized database and better dissemination of phenotype/genotype relationships, cannot be implemented immediately. However, current pipelines can be improved right now by robust quality control of starting sequence reads to make sure that the genome breadth of coverage is high (>90 %) and that there is sufficient depth of coverage (>30×). We also recommended that running the same sequence read data set against multiple databases should be avoided due to the erroneous results, and that sequence identity between the predicted and reference AMR genes should be higher than 90 % to avoid non-specific hits. We found little difference between the results of participants depending on what reference database they chose to use, between which Illumina short-read sequencer was used and whether they used assembly or assembly-free methods.In conclusion, we have identified some of the current contributors to discrepancies in predicting AMR-associated genes and phenotypes from bacterial isolate WGS data. We have provided recommendations for improving the current reporting of results. Despite its clear potential, even after accounting for poor sequence data, we found that the current public methods, in particular databases, are not adequate ‘off-the-shelf’ tools for the prediction of AMR from bacterial WGS data as a universal clinical test at this point in time.1. Doyle, RM. Sequence read files for all samples used in this study have been deposited in the European Nucleotide Archive under the project accession number PRJEB34513 (2019).Click here for additional data file.Click here for additional data file.


The patient’s immune system plays an important role in cancer pathogenesis, prognosis and susceptibility to treatment. Recent work introduced an immune related breast cancer. This subtyping is based on the expression profiles of the tumor samples. Specifically, one study showed that analyzing 658 genes can lead to a signature for subtyping tumors. Furthermore, this classification is independent of other known molecular and clinical breast cancer subtyping. Finally, that study shows that the suggested subtyping has significant prognostic implications.−4 in a test cohort) difference in survival. To obtain these improved results we develop a feature selection process that matches the high-dimensionality character of the data and the dual performance objectives, driven by survival and anchored by the literature subtyping.In this work we develop an efficient signature associated with survival in breast cancer. We begin by developing a more efficient signature for the above-mentioned breast cancer immune-based subtyping. This signature represents better performance with a set of 579 genes that obtains an improved Area Under Curve (AUC). We then determine a set of 193 genes and an associated classification rule that yield subtypes with a much stronger statistically significant (log rank p-value < 2 × 10 Molecular profiling of tumors has improved disease management and personalized cancer medicine. In particular, gene expression-based signatures distinguish five breast cancer subtypes –3—Lumina−4 in a test cohort) difference in survival. This set of genes is reported in Point B gene list in A global approach to understanding the role of the tumor’s immune context in breast cancer was taken by several recent studies –15. In  Ω = Ω−smallest_setendwhiledo argmin(|w|) smallest_w_index =  Ω = Ω−smallest_w_indexendreturn ΩA.We combine our iterative approach with a training/validation split. In each iteration, we calculate the validation accuracy. Classifiers are trained with a three-way loss function for clusters A, B and C. For the validation we evaluate Cluster B vs. Not Cluster B. We select the best performing set of genes. This process yields Point A in B.In this process we incorporate the log rank test into the training phase. We take the test cohorts with 1300 samples and split them into validation and test sets. We use the new validation set in the iterative process to evaluate the performance after each iteration. In each iteration we predict the label of all validation samples and then conduct a log rank test comparing the samples called Cluster B to those called Not Cluster B. We choose the set of features with the smallest log rank p-value. This process yields Point B in B, we trained a new classifier based on their relative expression as derived from the raw data that underlies the samples in our training set. The idea is that the relative expression levels of these genes can be computed in the context of any signature sample, without the need to normalize it back to the context of the training cohorts. For this purpose, consider the raw expression data instance E measured for every gene g and a single sample d. We define a relative expression value Q for every g ∈ ΓB and every training sample d. First we sort E to get the ranks 1 ≤ R ≤ N = |ΓB| (= 193). Now define:In practice, results derived from the normalized data are not directly useful in working with new samples. When diagnosing a single patient, no normalization context can be assumed—beyond that which is internal to the patient’s sample. To enable practical use of the genes in ΓB, in the context of sample d.In other words, we map every gene to its relative decile within the set ΓQ, to classify the samples into two prognostic subclasses: GPx = Not_Cluster_B and BPx = Cluster_B. To classify a new sample (which has not been seen in the training process), we just transform all gene expression levels there, for all g ∈ ΓB, to relative levels (rank deciles) as above and execute the trained classifier. Note that 10 is obviously a hyperparameter of this process.We now train a Naive Bayes classifier, based on the features lifelines. They include the 0.95 confidence envelope and an indication of the Q3 level of survival for each class with corresponding colors.The Kaplan Meier curves in this paper were generated using the Python library The two feature selection approaches described in the Methods section resulted in two gene sets with the best observed performance:579 genes for the classification accuracy driven approach; designated as Point A in 193 genes for the log rank performance driven approach; designated as Point B in Note that when using different training validation set splits, we may get different results. This point is addressed below. An overview of the selection process, as run on our data, is depicted in We compared our results to . As showWorking with these 579 signature genes that we found, we classified the test samples and applied log-rank statistical analysis on the results of the classification. We observe, as in , a prognReducing the number of selected genes reduces the classifier’s performance as well, as we can see in Working with these 193 signature genes that we found, we classified the half of the test samples we left for testing and applied log-rank statistical analysis on the results of the classification. As in approach 1 above, the prognostic disadvantage to the samples called Cluster B remains significant see .GPx (good prognosis) and BPx (bad prognosis). Survival analysis results in a log rank p-value of 10−3 when computed for n = 653 test samples . We checked that the association between the clusters and the lymphocyte infiltration did not break. As we can see in these of .We used the GOrilla (Gene Ontology enRIchment anaLysis and visuaLizAtion) , 22 toolBPx class.To understand more deeply our findings, we examined the BPx, in terms of known breast cancer subtypes and other patient characteristics. The results are described in S1 Fig in BPx, in terms of PAM50 subtypes, is further addressed below.First, we analyzed the composition of BPx to [CBΔBPx. Of these, we observed 268 patients in CB\BPx and 179 patients in BPx\CB. Ranking genes according to differential expressions between these two sets (using WRS), we found 21 genes that have a significantly lower expression in CB at FDR = 0.05 . GATA3 is one of three genes (the other two are TP53 and PIK3CA) that were reported to have somatic mutations at more than 10% incidence across all breast cancers [GATA3, with some studies suggesting that GATA3 functionality acts to inhibit, and other studies suggesting that it acts to promote, the development, growth, and/or spread of breast cancer. The second most significant gene with mutations observed, in terms of differences between BPx and GPx, is MAP3K1 . Both GATA3 and MAP3K1 are known to have important roles in breast cancer [GATA3 and MAP3K1. All ten patients were BPx . Moreover, nine of them were LumA and Her2+ BPx patients .Finally, in Metabric  the auth cancers . There it cancer –27. In t−4 on test data. As we noted above, using different training validation set splits may yield different results. We ran the process with ten different splits (see −100).In this work we applied feature selection techniques to improve a prognostic immune related signature previously suggested in . We obtalits see . We obseIt is important to note that the survival data used in this paper is overall survival data. In particular, our findings do not imply a signature that would be indicative of the response to any particular therapy.To develop classifiers driven by survival data, one needs to use validation data for which feature values and survival information are available. The task of optimizing a classifier using quantitative features requires, and merits, much deeper investigation, in a broader context. The methods described in  address BPx and GPx. Note that although the PAM50 partition is not independent of the class , neither prognostic subset is significantly aligned with a PAM50 subtype.It is also interesting to examine the classification obtained by the Naive Bayes classifier on the test samples. One obvious future research direction is the development of even more efficient prognostic signatures for breast cancer. A related task would be to include other types of measurement  to improve the predictive power.Can the combination of Tumor Infiltrating Lymphocyte (TIL) levels with the expression signature lead to an even more significant prognostic indication? According to  and to oThe development of expression-based signatures calls for using sparse feature selection techniques. The literature addresses sparse feature selection in a variety of machine learning contexts. Questions related to which of these methods work for survival analysis is a topic for further research. In this study we explored several heuristic approaches. Some of them can be further generalized. Moreover, the direct use of survival data to drive feature selection and the development of methods for doing so will require further research and development.In this work we applied feature selection techniques to produce a more efficient signature that follows the one proposed by Tekpli et al.  and thatThe immune related signature suggested in Tekpli et al.  is likelS1 Text(PDF)Click here for additional data file.


In recent years, an increasing number of cabled Fixed Underwater Observatories (FUOs) have been deployed, many of them equipped with digital cameras recording high-resolution digital image time series for a given period. The manual extraction of quantitative information from these data regarding resident species is necessary to link the image time series information to data from other sensors but requires computational support to overcome the bottleneck problem in manual analysis. As a priori knowledge about the objects of interest in the images is almost never available, computational methods are required that are not dependent on the posterior availability of a large training data set of annotated images. In this paper, we propose a new strategy for collecting and using training data for machine learning-based observatory image interpretation much more efficiently. The method combines the training efficiency of a special active learning procedure with the advantages of deep learning feature representations. The method is tested on two highly disparate data sets. In our experiments, we can show that the proposed method ALMI achieves on one data set a classification accuracy A > 90% with less than N = 258 data samples and A > 80% after N = 150 iterations, i.e., training samples, on the other data set outperforming the reference method regarding accuracy and training data required. The human impact on the marine ecosystem has increased in recent decades . ActivitLimited background knowledge: Often, it is not known a priori which species can occur in the data set.Expensive expert annotation: Quality controlled annotations are expensive in the context of marine imaging because expert knowledge from the domain of marine biology is needed for the annotation process.Low abundant classes: It is time-consuming to manually find a sufficient number of examples of rare species for the training set.One main problem in providing computational support for taxonomic classification by employing supervised learning classifiers is the amount of manual expert work required to collect a training set of labeled image patches of sufficient size for all object classes. To collect such a set, several challenges must be faced, some of them very special for underwater computer vision applications:sampling strategy.One approach that is particularly well suited to tackling these three challenges is active learning. The core idea of active learning is to select training samples automatically from the set of samples to be classified instead of leaving the selection of training samples to human experts. This automated selection of training samples is usually done in an iterative fashion. First, a training sample is selected automatically from the set of all samples to be classified. Next, the selected sample is labeled by an expert, and the process continues by going back to the first step. This is repeated until enough training samples have been selected and labeled. Performing the first step—selecting a training sample—requires an explicit description of a so-called In light of the three challenges listed above, we define the following criteria for an efficient sampling strategy in this context. In order to make efficient use of the data and domain expert’s time collecting a training set, the sampling strategy should…a.… avoid samples that do not show any instance of a class ,b.… prioritize samples that show an instance of a class that is not yet in the training set,c.… prioritize samples that show an instance of a class that is underrepresented in the training set, andd.… prioritize samples that can help to discriminate a class better from the other classes.dominant color features, an established color feature representation method that is often applied in image retrieval contexts. The dominant color feature for an image patch is extracted by grouping the pixels  into a number of five clusters using the modified median cut algorithm   T, let gCOATL-approach proposed in [In this experiment, the proposed method is evaluated on the dataset HG. First, the result of the principal component analysis is inspected. As described in posed in . The corposed in  is used.t, the performance is evaluated on all samples except for the t labeled samples. By doing so, the test data set consists of t. To avoid testing on a too small test dataset, only 1500 iterations are performed which leaves 315 test samples after the last iteration. In both methods COATL and ALMI, a number of T iterations. That is why a slight modification of the accuracy given in Equation . Along with the mean, t averaged over all runs of the experiment, the standard deviation of the accuracies after iteration t is defined asThe accuracy The area between the curves of ethod in  needs 27A and B differ significantly. In the Levene test, the null hypothesis states that the variances of A and B are equal. Next, a p-value is computed that describes the probability that the variances of A and B differ more or equal than the actually observed variances under the assumption of the null hypotheses. In our experiment, the Levene test results in a p-value of about A is significantly larger than the mean of B. In the one-sided Welch’s test, the null hypothesis states that the mean of A is lower or equal than the mean of B. Next, a p-value is computed that describes the probability that A and B under the assumption of the null hypothesis. In our experiment, the Levene test results in a p-value of about To test these findings about improving accuracy for statistical significance, we apply two tests in order to check whether the variance and/or the mean of the accuracy obtained after 279 iterations differs significantly depending on whether ALMI or the reference method from [posed in  to test h’s test  to test The aim of this work was to present a generic method for marine image classification that shows an improved learning performance due to the use of generic features and a reasoned choice of training samples in order to increase the efficiency of the manual annotation task performed by human experts. The proposed method ALMI is a single label-image classification method, i.e., the images of the processed dataset are required to contain maximum one object per image. However, if that is not the case, single-object images can be extracted from large scale images prior to using ALMI fully automatic. Some methods are proposed in ,15,16 whTo evaluate the extent to which the method meets this objective, its performance was compared to other related works. The evaluation focused on several aspects. First, the method was evaluated on very disparate data sets in order to assess the effectivity of the generic feature approach. Second, results from previously published evaluations of existing methods on the same data sets had to be available so results can be reproduced. Third, the evaluation was done in the same way as the previously published evaluations in order to visualize the progress. Regarding the evaluation in When all the samples of an underrepresented class are in the training set, the underrepresented class is not part of the test set anymore.When all the samples that are not in the “no object” class are in the training set, the accuracy is 0 because the test set only contains “no object” images that are not counted as true positives.it can achieve higher accuracies than previously published methods andit has a steeper learning curve than, i.e., ALMI achieves a certain level of accuracy with less training samples.In our experiments on two data sets that use the same evaluation and the same data set selected by the authors of the previously published method, ALMI shows thatThese effects are more prominent on the data set HG and are especially remarkable on the data set LV, because on this data set the previously published method outperformed the results of other known methods to such an extent that further improvement seemed difficult to achieve. Considering the large differences between the two marine image data sets, this all suggests that ALMI has the potential to apply to a wide range of marine image data sets and makes us confident that it can be useful in the biodiversity estimation in different types of marine habitats.


In terms of clinical features, the TMEscore is closely related to the T staging, Grade, and HPV. As for genomic characteristics, the genomic features of the Risk-H samples are a low expression of immune-related genes and high-frequency mutations of TP53 and CEP152. This model was validated in an external test set, in which the prognosis for Risk-H group and Risk-L group was also significantly different (log rank p = 0.017). A quantitative method of TME infiltration pattern is established, which may be a potential predictor of HNSC prognosis.The tumor microenvironment (TME) is of great clinical significance for predicting the therapeutic effect of tumors. Nonetheless, there was no systematic analysis of cellular interactions in the TME of head and neck cancer (HNSC). This study used gene expression data from 816 patients with HNSC to analyze the scores of 22 immune cells. On this basis, we have established a novel TMEscore-based prognostic risk model. The relationship between TMEscore and clinical and genomic characteristics was analyzed. The sample was divided into risk-H and risk-L groups based on the prognosis risk model of TMEscore, with significant differences in overall survival between the two groups (log rank  In most countries, HNSC patients over the age of 50 are more common. Among them, more than 90% are squamous cell carcinomas2. Because of the complexity of head and neck anatomy, it is difficult to perform surgery. Although significant progress has been made in the treatment of HNSC in recent years, the total global survival rate of HNSC is only 50%3. Surgery and radiotherapy are the standard modalities for patients with early head and neck cancer. However, when head and neck cancer is diagnosed, more than 50% of the patients are in clinical phase stage III or IV, and lose their best chance of operation. Also, for patients with recurrence after surgery, secondary surgery trauma is more dangerous. For such locally advanced patients, the prognosis is poor5.Head and neck cancer (HNSC) is one of the most common malignant tumors worldwide7. At the same time, some immune related parameters have also been reported to predict the prognosis of HNSC patients. These studies further show that different immune states have a significant effect on the prognosis of HNSC patients9. The sensitivity of therapeutic targets is different among patients due to the heterogeneity of tumors. Immunotherapy, therefore, has a certain selectivity for the patient population10. Further differentiation of the immune subtype of cancer is necessary to identify patients who may benefit from immunotherapy.Given the poor prognosis after standard treatment and the low level of targeted therapies in HNSC, immunotherapy is a promising additional approach. It is currently undergoing intensive research11. More and more studies have shown that TME plays an essential role in tumor progression and therapeutic response. Some studies have shown that the infiltration of a large number of immune cell into tumor tissue is strictly related to the prognosis of patients12. In particular, changes in the quantitative of cytotoxic T cells, helper T cells, dendritic cells (DCs), tumor-associated macrophages, mesenchymal stem cells, and associated inflammatory pathways and fibroblasts affect the prognosis of a variety of malignancies18. Indeed, it is generally believed that an individual’s immune state is too complex to be illustrated by a single immune marker. Therefore, it is urgent to integrate a large number of HNSC transcription data to construct a new immune-related prognostic factor.Different tumor microenvironment (TME) can induce various adverse and beneficial consequences. Immune cells are most likely to be affected by TME and can be activated to promote tumor growth and progression22. In She and Chen's studies, prognosis signatures based on immune genes and the corresponding immunotherapy for different immune subgroups have been proved to be feasible strategies for head and neck cancer24. Currently, there are some algorithms associated with immune scores that can be used to estimate the abundance of immune cells in TME. However, there is still a lack of research on the quantitative model of TME infiltration by these algorithms.At present, the study of TME has shown great potential in the treatment of solid tumors, such as melanoma, non-small cell lung cancer, renal cancer, and prostate cancerIn this study, we used genetic expression data and clinical information from TCGA and GEO public databases to evaluate the proportion of 22 immune cells in HNSC TME. On this basis, the TME infiltrative patterns of 816 HNSC patients were evaluated, and the phenotypes of TME were systematically correlated with the genomic and clinical pathological characteristics of HNSC, thus establishing a quantitative method for TME infiltrative pattern. The flow chart of the experiment is shown in Fig. The RNA-Seq data of 546 HNSC samples were downloaded from the TCGA database on April 14, 2019. The clinical follow-up information of these samples was also downloaded at the same time. Additionally, samples without clinical data and follow-up time of less than 30 days were removed. The final study included 491 HNSC samples as training sets. Similarly, we downloaded the HNSC gene expression dataset GSE65858 from the Gene Expression Omnibus (GEO) database, which was annotated by the Illumina HumanHT-12 V4.0 expression beadchip platform. Samples with a follow-up time of less than 30 days were removed, and 270 samples were finally included as a test set. We scaled the gene expression data of the two platforms. The sample statistics of the two sets of data are shown in Table 25. It can use RNA sequence data to infer the type proportion from large tumor samples with mixed cell types. To quantify the proportions of immune cells in HNSC samples, we used the CIBERSORT algorithm and the LM22 gene signature, which allows for sensitive and specific discrimination of 22 human immune cell phenotypes, including B cells, T cells, macrophages, etc. The standardized gene expression data was loaded into the CIBERSORT website (https://ciberfort.stanford.edu/). Set the threshold value to p < 0.05, and exclude the sample data that does not reach the threshold value. In total, 452 TCGA samples and 199 GEO samples were eligible. Scores of 22 immune cells were obtained with LM22 signature and 1,000 permutation.CIBERSORT is a tool to deconvolute the expression matrix of immune cell subtypes based on the principle of linear support vector regression26. We evaluated the optimal number of clusters between k = 2–10 and repeated 1,000 times to ensure the stability of the results27. The cluster map was drawn using the pheatmap package in R.Consensus clustering to obtain molecular subtypes was associated with TME-infiltrating cells. Consensus clustering was performed using the ConcensusClusterPlus package in R to determine subtypes of HNSC based on TME permeabilized cellsTo identify genes associated with TME cell infiltration patterns, we used a linear model to analyze differentially expressed genes (DEGs) between subgroups. The gene expression data of HNSC samples in TCGA was selected for DEGs analysis of TMEC1 and TMEC2. The DEGs were calculated by DESeq2 package in R, which accorded with FDR < 0.05, | log2FC |> 2.28. To further investigate the relationship between differential gene expression and the TME phenotype, we used the NMF method to re-cluster HNSC samples and analyze their clinical features. The NMF method selects "brunet" and performs 50 iterations. Set the number of clusters k to 2–10. The calculation is performed using the NMF package in R, and the minimum member of each subclass is set to 1029.Nonnegative matrix factorization (NMF) is an unsupervised clustering method widely used in the discovery of genomics-based tumor molecular subtypes30. The principal component analysis (PCA) was performed on gene expression data of each signature G using the psych package in R. After 100 iterations, the first principal component was extracted as a signature score.To obtain robust TME gene signatures, we first evaluated the prognostic value of each DEG. DEGs with significant effects on prognosis were selected, and random forest algorithms were then used to assess their importance. Genes with significant prognosis were included for randomForest feature selection using randomForest of R software package. We set the mtry for each split to be 1–279, ntree = 500. The mtry value with the lowest error rate is selected as the optimal mtry value of the random forest algorithm. Finally, the DEGs were sorted according to the degree of importance, and DEGs with cumulative importance > 95% were selected as candidate feature genes. Candidate feature genes were clustered using the k-means algorithm and then were defined as signature GsThe Cox multivariate regression analysis was used to establish a prognostic risk model for signature Gs. The TMEscore was calculated as follows:Among them, β is the multifactor regression coefficient for each signature G, and PC1 is the score of the first principal component of each signature G.To observe the relationship between TMEscore and clinical features, we divided HNSC samples into two groups by the median of TMEscore. The prognostic differences between high TMEscore and low TMEscore were compared. The relationship between TMEscore and clinical features, such as T staging, N staging, M staging, TNM stage, age, Grade, gender, smoking history and HPV was analyzed.To observe the relationship between TMEscore and immune-related gene expression, we collected three types of immune-related genes. (1) Immune-stimulating genes, including CXCL10, CXCL9, GZMA, GZMB, PRF1, IFNG, TBX2, TNF, CD8A. (2) Immunological checkpoint genes, including PDCD1, CTLA4, LAG3, PDCD1LG2, IDO1, CD274, HAVCR23. (3) Activation genes of the TGF/EMT pathway, including VIM, ACTA2, COL4A1, TGFBR2, ZEB1, CLDN3, SMAD9, TWIST1. The expression profiles of these genes were extracted to further analyze the differential expression of these three types of genes in high TMEscore and low TMEscore.p < 0.05.To observe differences in genomic variation between high TMEscore and low TMEscore samples, we downloaded SNP data from TCGA to remove the intron and silent mutations. Genes for differential mutations in the two samples were analyzed using fisher's exact test with a threshold of 31. For the comparison of the two groups, the normal distribution variables were analyzed by the unpaired student t-test, and the non-normal distribution variables were analyzed by the Mann–Whitney U test. For comparisons of more than two groups, the Kruskal–Wallis test and the one-way ANOVA were used as non-parametric and parametric test methods, respectively32. Correlation coefficients were computed by Spearmanand distance correlation analysis. Two-sided Fisher exact tests were used to analyze contingency tables. We used the Benjamini–Hochberg method to convert P values to FDR. The Kaplan–Meier method was used to generate survival curves for each subgroup. The log-rank test was used to determine the statistical significance of the difference, and the significance was defined as p < 0.05. All of these analyses were performed in R 3.4.3, based on default parameters unless otherwise stated.The Shapiro–Wilk normality test was used to test the normality of the variables unless otherwise statedp < 0.05, HR > 1). The score of T cells CD4 memory activated, T cells follicular helper, T cells CD8, etc. was associated with a better prognosis . The detailed results are shown in Fig. p < 0.0001), as shown in Fig. We used CIBERSORT to calculate 22 immune cell scores in the training set and analyzed the correlation between them Fig. . We usedp = 0.015).Similarly, we performed unsupervised clustering in the GSE65858 test set. The optimal number of clusters between k = 2–10 was evaluated and repeated 1,000 times. k = 2 was selected as the optimal cluster number based on the CDF value and Delta area Figure . There wp < 0.05). Finally, based on 279 gene expression profiles, we used the NMF algorithm to re-cluster the HNSC samples in TCGA. As shown in Fig. We selected 714 DEGs between TMEC1 and TMEC2 for subsequent analysis in the training set Fig. A. Among We evaluated the importance of 279 DEGs using a random forest algorithm. Set ntree = 100 according to the error rate, as shown in Figure p = 0.0031, HR = 1.52 (1.16–2.00) Fig. The TMEscore of different subtypes was calculated based on the prognostic risk model, and the detailed information is shown in Tables We evaluated the relationship between TMEscore and clinical information, such as T staging, N staging, M staging, TNM stage, Age, Grade, Gender, smoking history and HPV. The results showed significant differences in TMEscore in different T staging samples. There are significant differences in TMEscore for different Graded samples. TMEscore was significantly higher in HPV negative samples than that in HPV positive samples. This significant difference was not observed in other clinical information Figure .We analyzed the relationship between immune activation genes and TMEC, GeneC, and TMEscore. These genes have different expression patterns in different subtypes Fig. A. Among p < 0.01). . Fig. . Among tWe used the same method to perform TMEscore calculations in the test set. The median value of the TMEscore was chosen as a threshold to classify samples into the Risk-H group and the Risk-L group. In the test set, we also analyzed the relationship between TMEscore and prognosis, clinical data, and immune gene expression.p = 0.017), as detailed in Fig. We ploted survival curves for the Risk-H group and the Risk-L group. As a result, it was found that the AUC value of five-years was 0.70 . After that, the samples classified by TMEscore were correlated with clinical features. The results showed that there were significant differences in TMEscore for samples of different T staging, grades, and HPV. Comprehensive analysis showed that the TMEscore may be a potential predictor of prognosis in HNSC.We further screened and clustered prognosis-related DEGs to obtain signatures G1, signature G2, and signature G3. On this basis, we have established an algorithm for a comprehensive evaluation of TMEscore. A high TMEscore is associated with poor prognosis. There was a significant difference between the Risk-H group and the Risk-L group


Reverse-phase protein array (RPPA) technology uses panels of high-specificity antibodies to measure proteins and protein post-translational modifications in cells and tissues. The approach offers sensitive and precise quantification of large numbers of samples and has thus found applications in the analysis of clinical and pre-clinical samples. For effective integration into drug development and clinical practice, robust assays with consistent results are essential. Leveraging a collaborative RPPA model, we set out to assess the variability between three different RPPA platforms using distinct instrument set-ups and workflows. Employing multiple RPPA-based approaches operated across distinct laboratories, we characterised a range of human breast cancer cells and their protein-level responses to two clinically relevant cancer drugs. We integrated multi-platform RPPA data and used unsupervised learning to identify protein expression and phosphorylation signatures that were not dependent on RPPA platform and analysis workflow. Our findings indicate that proteomic analyses of cancer cell lines using different RPPA platforms can identify concordant profiles of response to pharmacological inhibition, including when using different antibodies to measure the same target antigens. These results highlight the robustness and the reproducibility of RPPA technology and its capacity to identify protein markers of disease or response to therapy. A crucial source of information in this context is the proteome and, notably, the activation status of dynamic cell signalling pathways through post-translational protein modifications. Indeed, genomic mutations are not always associated with activated signalling pathways and, conversely, pathway activation can occur in the absence of mutations, as exemplified by PIK3CA3 and human epidermal growth factor receptor 2  signalling4 in breast cancer.In the era of personalised medicine and targeted cancer therapies, identifying those patients that will benefit from existing and new therapies is paramount. Genetics is already being used to assist clinical decision-making in specific cases7, among which reverse-phase protein array (RPPA) is a technology of choice for its unequalled sample throughput. The RPPA technology uses panels of monospecific affinity reagents  to quantify, with high precision and sensitivity, the abundance of specific proteins and their post-translationally modified forms in biological specimens9. Protein samples derived from cells or tissues are immobilised on a solid substrate, deposited as small spots on multiple arrays, and each array is probed with a single, epitope-specific antibody. This enables simultaneous quantification of multiple proteins and post-translational modifications in hundreds of samples, a multiplex capability not available in any other current proteomic technology. The capacity to analyse large sample numbers enables analysis of multiple sample conditions, such as drug treatments, dose responses and time courses, resulting in data series that can support systems biology and drug discovery pipelines11. The high sensitivity (in the picomole–femtomole range) and good reproducibility of RPPA technology16 have motivated its application to a wide range of sample types, including cell lines, preclinical (e.g. xenograft) models and patient-derived material. Indeed, the microscale printing of very small amounts of samples is of particular benefit for analysis of limited clinical or preclinical material, and RPPA has become a powerful addition to the biomedical analytical toolbox for the investigation of disease mechanisms, diagnostics and prognostics, notably in cancer26.Several technologies exist for protein biomarker discovery and validation27. Moreover, no standard tools exist to quantify, normalise and quality control RPPA data. Several RPPA data processing methods have been developed40, but these are often designed for specific technical set-ups, requiring, for example, particular array layouts or raw data formats. To our knowledge, no extensive cross-platform validation has been reported for RPPA technology. There is, therefore, a need to assess and understand the impact of variation between RPPA data derived from different platforms across international research centres.The RPPA workflow is composed of several distinct steps, which can each be adapted to the needs of the laboratory or the study. RPPAs thus offer a highly flexible, modular proteomic technology, enabling numerous possible technical set-ups and protocols. As most laboratories using RPPA technology have developed a customised set-up, there are essentially as many workflows as there are RPPA platforms. Differences between platforms are diverse and can include the type of printer used to create the arrays, sample spotting conditions, slide substrate chemistry, primary and secondary antibodies used for immunostaining and slide scanner optics42, there is a growing need for the evaluation of the reproducibility of RPPA platform outputs. Herein, we use collaborative RPPA-based proteomics, employing distinct RPPA workflows at multiple research sites in different countries, to characterise a range of human breast cancer cell lines and their biochemical responses to two clinically relevant cancer drugs. We combine data derived from different RPPA platforms to assess inter-platform variation and use integrative analysis to identify platform-independent protein markers of response to drug treatment. Such cross-platform validation may thus have utility in multi-centre evaluation of robust protein markers of disease and therapeutic response.As RPPA workflows are being developed for use in clinical settings, for which robust assays are paramountTo assess the reproducibility of RPPA technology across multiple laboratories, we developed an international multi-platform approach that integrated RPPA data derived from three research sites across Europe . We selected for this study six breast cancer cell lines, encompassing different breast cancer molecular subtypes and presenting distinct drug sensitivities  for the Paris, Heidelberg and Edinburgh platforms, respectively. Normalised RPPA data derived from each platform were then clustered antibody-wise to identify similarities between validated antibodies  and analysed the integrated dataset using unsupervised learning. For dataset integration, antigens targeted by each antibody were classified, capturing antibody recognition of related protein isoforms or family members where applicable . Each data point was linked to the RPPA platform from which it was derived to enable downstream data analysis. We reduced the dimensionality of the integrated multi-platform RPPA dataset using principal component analysis. This unsupervised analysis of the six breast cancer cell lines revealed separation in feature space for several of the cell types, indicating cell type-specific expression of the proteins and phosphoproteins measured Fig. . MoreoveTo interrogate the similarity between antigen expression profiles for distinct antibodies used on different RPPA platforms, we devised a data-driven representation of antibody similarity, which we termed a clustered antibody antigen map. Antibodies used at each RPPA platform  clustered with expression of the same antigen determined by at least one other RPPA platform Fig. c,d. In a43; selumetinib is a selective ATP-independent allosteric inhibitor of mitogen-activated protein kinase kinase 44. To capture the signalling dynamics of the proteins under investigation, cells were treated with either drug or with DMSO (vehicle control) for 20 min and 24 h , we extended the integrative analysis to include multi-platform RPPA data of the six breast cancer cell lines treated with two clinically relevant drugs, lapatinib  and selumetinib . Lapatinib is a reversible ATP-mimetic tyrosine kinase inhibitor of epidermal growth factor receptor  and Her2rs = 0.70) , as expected  were generally well correlated , with replicate samples clustering together, indicating robust data-driven grouping of the dataset Fig. . ClusterWith RPPA technology poised for adoption into routine clinical laboratory assays, there is a need to assess reproducibility and variation between RPPA data derived from different platforms. Herein, we employed a collaborative RPPA-based proteomics approach to evaluate the consistency of results obtained from the same samples using different RPPA workflows. We combined the RPPA data derived from multiple research sites to assess inter-platform variation and found that different RPPA platforms using distinct set-ups yield remarkably consistent results, including when several different antibodies are used to recognise the same antigen (protein or phosphoprotein) of interest. Indeed, antigens detected by different antibodies used at different RPPA platforms were, in general, almost as well correlated as those used at the same RPPA platform. These observations strongly suggest that the different instrumental set-ups and analytical workflows used at these representative RPPA platforms do not preclude the generation of comparable and reproducible data when using high-quality, validated antibodies.Using different workflows, the layout of samples spotted on arrays differed among the three RPPA platforms, which prevented application of the same automated data normalisation method. Each research site therefore applied their own in-house data normalisation procedure tailored for the respective RPPA platform. All platforms normalised signal intensities for total amounts of printed protein, but the way they accounted for sample dilutions differed. The Paris platform applied non-parametric curve fitting to the serial dilutions of each sample, the Heidelberg platform fitted curves for control samples only and reported experimental samples to these curves and the Edinburgh platform computed a linear fit of the serially diluted samples. Despite these different approaches, we show that normalised data improves the distinction between concordant data (derived from antibodies recognising the same antigen class) and non-concordant data , as compared to raw data. Future efforts to develop normalisation pipelines that are compatible with multiple array layouts may further improve data reproducibility among platforms and expand opportunities for cross-platform validation of RPPA technology. Such cross-platform validation may have utility in the appraisal of robust markers of disease and therapeutic response and their application as prognostic or predictive biomarkers.7. Proteomic and phosphoproteomic datasets are well placed to complement genetics-based biomarker strategies by providing additional information on activation states of dynamic pathway signalling. However, the application of proteomic technologies presents a number of challenges, including consistent high-quality sample preparation, sensitive detection of low-abundance proteins and post-translational modifications, high sample throughput at reasonable cost and rapid turnaround time necessary for clinical application or drug discovery and development pipelines. The development of RPPA workflows attempts to address many of these challenges, yet the reproducibility of RPPA data generation and analysis across distinct RPPA platforms and research centres has not been extensively evaluated. We used an integrative RPPA approach to characterise a range of human breast cancer cell lines and their biochemical responses to two clinically relevant cancer drugs. The cell lines were chosen to represent different molecular subtypes of breast cancer, each of which having a different prognosis and treatment response50. The sensitivity of these cell lines towards lapatinib and selumetinib and the expected changes in major signalling pathways are known53 and thus served as benchmarks in this study. We demonstrated that RPPA technology can identify expected protein markers for response to treatment and resistance in a robust and platform-independent manner. Although we analysed cell lines herein to enable benchmarking, RPPA is routinely applied to clinical samples, including small biopsies, on the platforms used in this study. In such cases, the amount of lysis buffer is adapted to the amount and the cellularity of the tissue in order to obtain sufficiently concentrated samples. We therefore expect that RPPA can robustly assess response to treatment and resistance independently of the analysed samples. Understanding how resistance can be predicted and prevented is a major therapeutic challenge, and the use of proteomic approaches such as RPPA will, by defining the functional state of cells and tissues, enable the validation and assessment of resistance mechanisms in clinical samples54.There is an outstanding need for the development of robust markers of response and resistance to targeted cancer therapy55. The development of robust biomarkers is thus becoming an essential component of new clinical trial designs to guide patient selection, optimise dosing schedules and minimise ineffective or over-treatment. For many complex diseases, biomarkers at the genetic, proteomic, metabolomic and phenotypic levels are required to characterise individual patient disease and response to therapy sufficiently. Integration of multiple biomarker modalities with emerging computational and statistical approaches represents the future direction of personalised medicine strategies. However, the successful implementation of personalised medicine is dependent upon the validation and reproducibility of biomarker tests performed across distinct research centres and national boundaries.High attrition rates in clinical drug development present significant challenges to drug developers, with only one in eight oncology drugs that enter clinical development in phase 1 achieving US Food and Drug Administration approval, and a 1-in-15 success rate when these candidate drugs are under evaluation in secondary oncology indicationsOur data show that RPPA analyses of drug-treated breast cancer cells using distinct antibodies and different RPPA platforms can identify robust profiles of protein markers reporting signalling pathway responses to pharmacological inhibition. This suggests that the consistency of RPPA-based assays will enable the validation and assessment of treatment response and resistance mechanisms in clinical samples across international laboratories. These data provide, to our knowledge, the first extensive cross-platform validation of RPPA technology, which paves the way for further investigation and improvement of technology robustness.g, 10 min, room temperature). Clarified lysates were aliquoted and snap-frozen in liquid nitrogen prior to shipment to the various research sites. Protein concentration was determined using a reducing agent-compatible BCA kit (Pierce).MCF7, MDA-MB-231, MDA-MB-468, MDA-MB-453, HCC1954 and SKBR3 breast cancer cells were purchased from American Type Culture Collection and grown according to supplied instructions  were prepared as 10 mM stock solutions in DMSO. Cells were treated with 1 µM lapatinib, 1 µM selumetinib or DMSO in growth medium for 20 min or 24 h. Control cells were treated with DMSO in growth medium for 20 min.56. We confirmed that > 99% of data points derived from undiluted samples at the Heidelberg platform were within the dynamic range of the representative dilution series 58 and compiled using Illustrator (Adobe). Principal component analyses were performed using Python (version 3.7.4) or Perseus (version 1.5.2.6)59.Binary, agglomerative hierarchical cluster analyses of centred normalised abundances for proteins and phosphoproteins were performed using Cluster 3.0 For data-driven representation of antibody similarity across multiple RPPA platforms, we devised the clustered antibody antigen map. For all antibodies used in the integrative analysis, centred normalised abundances for proteins and phosphoproteins were clustered on the basis of Spearman rank correlation coefficient-based distance, Euclidean distance or Kendall tau coefficient-based distance using Cluster 3.0, computing distances with an average-linkage matrix. Clustered antibody (column) node memberships were stored for antibody mapping, and clustered antibodies were expanded in a second dimension according to antibody antigen classification. Antibodies targeting the same antigen class were indexed according to unique antibody identifier, enumerating from 1. A matrix of integer-indexed antibody identifier elements was generated according to clustered antibodies (columns) and corresponding antibody antigen classes (rows) .No statistical methods were used to pre-determine sample size. Spearman rank correlation coefficients were calculated for every pair-wise combination of antibody identifiers. Kernel density estimates were computed using R (version 3.4.1)Supplementary InformationSupplementary InformationSupplementary InformationSupplementary InformationSupplementary InformationSupplementary InformationSupplementary Information


Predicting cancer dependencies from molecular data can help stratify patients and identify novel therapeutic targets. Recently available data on large-scale cancer cell line dependency allow a systematic assessment of the predictive power of diverse molecular features; however, the protein expression data have not been rigorously evaluated. By using the protein expression data generated by reverse-phase protein arrays, we aimed to assess their predictive power in identifying cancer dependencies and to develop a related analytic tool for community use.By using a machine learning schema, we conducted an analysis of feature importance based on cancer dependency and multiomic data from the DepMap and Cancer Cell Line Encyclopedia projects. We assessed the consistency of cancer dependency data between CRISPR/Cas9 and short hairpin RNA–mediated perturbation platforms. For a fair comparison, we focused on a set of genes with robust dependency data and four available expression-related features  and performed the same-gene predictions of the cancer dependency using different molecular features.For the genes surveyed, we observed that the protein expression data contained substantial predictive power for cancer dependencies, and they were the best predictive feature for the CRISPR/Cas9-based dependency data. We also developed a user-friendly protein-dependency analytic module and integrated it with The Cancer Proteome Atlas; this module allows researchers to explore and analyze our results intuitively.This study provides a systematic assessment for predicting cancer dependencies of cell lines from different expression-related features of a gene. Our results suggest that protein expression data are a highly valuable information resource for understanding tumor vulnerabilities and identifying therapeutic opportunities. Several studies have assessed the possibility of predicting cancer dependency from genomic or transcriptomic features.4 Although proteins are basic functional units in most biologic processes and represent the vast majority of therapeutic targets, proteomic features have not been evaluated along with those DNA- or RNA-level features in such studies.Understanding the genotype-phenotype relationships of cancer cells is a central task for precision cancer medicine because it will help classify patients into different treatment groups and identify novel therapeutic targets. The recent genome-wide short hairpin RNA (shRNA) or CRISPR/Cas9-mediated cell viability screens provide a unique opportunity to systematically characterize cancer dependencies in human cancer cell lines.Key ObjectiveThis study aimed to systematically assess the predictive power of different expression-related features of a gene for its cancer dependency through a rigorous machine learning (ML)–based feature importance analysis and develop the related bioinformatics module for community use.Knowledge GeneratedReverse-phase protein array (RPPA)-based protein expression data contain substantial predictive power as messenger RNA (mRNA) expression for cancer dependencies. Through our newly developed analytic module, researchers can discover novel genotype-phenotype patterns, generate testable hypotheses, and interpret biologic findings in a tumor context–dependent manner.RelevanceThis is a systematic analysis that assesses the predictive power of protein expression in inferring gene dependencies across a large number of cell lines. The developed analytic module is a valuable informatics tool for understanding tumor vulnerabilities and identifying therapeutic opportunities.7 By using RPPAs, we have characterized a large number of patient and cell line samples through The Cancer Genome Atlas,9 Cancer Cell Line Encyclopedia (CCLE),13 and MD Anderson Cell Line projects.14 Furthermore, we have built an open-access, dedicated bioinformatics resource, The Cancer Proteome Atlas (TCPA), for the cancer research community to study these large-scale functional proteomic data in a rich context.17 Here, we used a rigorous machine learning (ML) schema to evaluate the cancer-dependency predictive power of the RPPA-based protein expression along with other expression-related molecular features . We also implemented a new protein-dependency analytic module in TCPA, thereby allowing users to explore, analyze, and visualize the relationships between protein expression and cancer dependency.RPPAs are a powerful approach to generate functional proteomics data. This quantitative antibody-based assay can assess a large number of protein markers in many samples in a cost-effective, sensitive, and high-throughput manner.13 which assayed 214 protein markers across 899 cell lines . We obtained cancer dependency data, including CRISPR/Cas9 (DepMap19Q1)18 and shRNA (DEMETER2)1 data sets, from the DepMap portal . We also collected CNA, DNA methylation, and mRNA expression data from CCLE .We downloaded the RPPA data from the CCLE,We considered a regression task in dependency scores (cell growth change) that experienced gene knockdown (shRNA) or knockout (CRISPR/Cas9). Specifically, the response variable (model outcome) is a vector of dependency scores for each gene across cell lines. A score of 0 indicates that a gene is not essential, whereas a score of –1 corresponds to the median value of all common essential genes. The explanatory variables (predictors) were the self-features that were related to gene expression. To ensure the quality of the model outcome, we first constructed a robust cancer dependency set by collecting genes and cell lines that showed high consistency between shRNA (DEMETER2) and CRISPR/Cas9 (DepMap19Q1) experiments. Next, this robust dependency set was overlapped with the cell lines and genes from CCLE to extract available data for model training . To condR2. We trained models for each gene dependency. A dependency was flagged as predictable if it had at least one classifier that outperformed (had lower RMSE than) the baseline model in both the training and testing predictions. For the genes with predictable dependencies, we selected the best classifier (with the highest R2) based on the testing results and used the selected classifier to retrain the model using all samples. Finally, to evaluate the individual contribution of each feature, we performed a feature importance analysis to identify the best predictor for each dependency. We implemented this ML schema in R v3.5.0 using the caret package19 with the ML methods of lm, rf, and cforest. In addition, we used the varImp function to estimate the feature importance.As shown in We used R and Python libraries to process and analyze the data. All the precomputed analytic results were converted into the JSON format and loaded into the CouchDB database for users to query and analyze. We used JavaScript D3 and the Angular library to construct the Web user interface of the protein-dependency analytic module. The module displays the table results by DataTables and the nested plots by HighCharts.http://tcpaportal.org/mclp).The data and results are available at the TCPA website . In contrast, only 862 genes (5.8%) showed significant positive correlations across cell lines . This pattern suggested that many of the dependencies resulted from random effects and thus could not be preserved across the platforms. In subsequent analyses, we retained only 402 cell lines and 862 genes that showed significant consistency between the shRNA and CRISPR/Cas9 platforms to reduce potential random noise.To ensure the data quality of the cancer dependency scores, we constructed a robust cancer dependency set. The shRNA (DEMETER2) and CRISPR/Cas9 (DepMap19Q1) platforms shared 403 cancer cell lines and 14,913 genes . We firsOn the basis of the designed ML schema , we asse14 for users to examine the protein-dependency relationships of interest with ease. The module provides a straightforward, intuitive table view so that users can investigate whether the expression level of a protein is a good predictor for the corresponding cancer dependency across cancer cell lines (tested by Pearson’s correlation; visualized by scatter plots; P value.The above results highlight the utility of RPPA-based proteomic data in understanding cancer phenotypes and identifying novel therapeutic targets. Therefore, we developed a protein-dependency analytic module and integrated it into the cell-line Web platform of TCPA. This user-friendly, interactive module allows researchers to explore, visualize, and analyze the relationships between the RPPA and cancer dependency data. We included two independent RPPA data setsIn this study, we assessed the potential of RPPA-based protein expression to infer cancer dependencies through a rigorous ML-based feature importance analysis. To the best of our knowledge, this is the first systematic analysis to elucidate the predictive power of protein expression in inferring gene dependencies across a large number of cell lines. Our findings provide a strong rationale for incorporating protein expression data into the prediction tasks of cancer dependencies. One limitation of this study is the relatively small number of genes and proteins assessed, which limits the statistical power compared with other expression-related features. Our RPPA platform covers only approximately 200 protein markers, and we are in the process of expanding the protein list to approximately 500 proteins. We will revisit this topic when a larger RPPA data set becomes available. In addition to the dependency of its gene, the protein level likely helps predict the effects of other genes, and for such an analysis, a similar ML strategy equipped with advanced feature selection techniques is warranted. We also implemented a new analytic module in TCPA that can be used to directly analyze and visualize the relationships between protein expression and cancer dependencies across cancer cell lines. This module will help researchers discover novel genotype-phenotype patterns, generate testable hypotheses, and interpret biologic findings in a tumor context–dependent manner. We expect it to be a valuable bioinformatics tool for the cancer research community.


With the explosion in the number of methods designed to analyze bulk and single-cell RNA-seq data, there is a growing need for approaches that assess and compare these methods. The usual technique is to compare methods on data simulated according to some theoretical model. However, as real data often exhibit violations from theoretical models, this can result in unsubstantiated claims of a method’s performance.Rather than generate data from a theoretical model, in this paper we develop methods to add signal to real RNA-seq datasets. Since the resulting simulated data are not generated from an unrealistic theoretical model, they exhibit realistic (annoying) attributes of real data. This lets RNA-seq methods developers assess their procedures in non-ideal (model-violating) scenarios. Our procedures may be applied to both single-cell and bulk RNA-seq. We show that our simulation method results in more realistic datasets and can alter the conclusions of a differential expression analysis study. We also demonstrate our approach by comparing various factor analysis techniques on RNA-seq datasets.https://cran.r-project.org/package=seqgendiff.Using data simulated from a theoretical model can substantially impact the results of a study. We developed more realistic simulation techniques for RNA-seq data. Our tools are available in the seqgendiff R package on the Comprehensive R Archive Network:  Due to its higher signal-to-noise ratio, larger range of detection, and its ability to measure a priori unknown genes, RNA-seq has surpassed microarrays as the technology of choice to measure gene expression . With thThe typical pipeline to evaluate a method is to first simulate data according to some theoretical model, then compare it to competing methods on these simulated data and show it to be superior in some fashion. This way of evaluation can be useful to see how a method works in ideal scenarios. However, real data rarely live in ideal scenarios. Real data often exhibit unwanted variation beyond that assumed by a model . TheoretIn this paper, we propose an alternative approach. Rather than generate data with a prespecified signal according to some modeling assumptions, we take a real RNA-seq dataset and add a prespecified signal to it. The main advantage of our approach is that any unwanted variation in the real data is maintained in the simulated data, and this unwanted variation need not be prespecified by the researcher. The way we add signal does carry assumptions, but they are flexible  have beOur paper is organized as follows. We first list the goals and assumptions of our simulation scheme “” sectionA), vectors by bold lowercase letters (a), and scalars by non-bold letters (a or A). Indices typically run from 1 to their uppercase version, e.g. a=1,2,…,A. Where there is no chance for confusion, we let non-bold versions of letters represent the scalar elements of matrices and vectors. So aij is the th element of A, while ai is the ith element of a. We let 1A denote the A-vector of 1’s and 1A×B the A×B matrix of 1’s. The matrix transpose is denoted by We adopt the following notation. We denote matrices by bold uppercase letters  elements:X2,X3.where Y such thatOur goal is to generate a matrix ΠX3 and the columns of Z, andwhereY in “Note 1: For simplicity we use the Poisson distribution in the main text (Eqs. (E[Y])=Θ, then E[Y] and Note 2: The Ω term in 1), we poOne of the more common applications of RNA-seq data is estimating and testing for differences in gene expression between two groups. Many packages and techniques exist to perform this task [b and x and evaluate a method’s ability to (i) estimate b and (ii) detect which genes have non-zero bg.and bg’s (the elements of b). Our software implementation allows for this. In addition, following [sg is the sample standard deviation of the gth row of log2(Y+0.5), and α is a user-specified constant. Allowing for α=0 corresponds to the scenario of specifying the distribution of the effects, while allowing for α=1 corresponds to specifying the p-value prior of [In many settings, a researcher may want to specify the distribution of the ollowing , we alloprior of .Though the two-group model is perhaps the most common scenario in differential expression analysis, our method also allows for arbitrary design matrices. Such design matrices have applications in many types of expression experiments , 42–44, Unobserved confounding / batch effects / surrogate variables / unwanted variation has been recognized as a serious impediment to scientific studies in the modern “omics” era . As suchTypically, the form and strength of any unobserved confounding is not known. So one way to assess different confounder adjustment methods would be to assume model  and add B3 is to the truth (assuming the researcher may use both X1 and ΠX3 to obtain this estimate). The researcher can control the correlation between the columns of ΠX3 and the columns of Z by specifying the distribution of Π  additional signal and (ii) a further amount of thinning on each sample by generating the following submodel of :7\documB3 under various amounts of library thinning .To evaluate the effectiveness of a library size adjustment procedure, researchers may observe the effects on the estimates of Factor analysis is a fundamental technique in every statistician’s arsenal. Since its creation by Spearman , literalWe suppose that the RNA-seq read-counts follow the following submodel of :9\documWe then suppose that the researcher generates a modified dataset that follows the following submodel of :10\docuK+P3 factors. That is, the researcher fits the following model,We assume that a researcher applies a factor analysis to  to estimΠX3. A researcher may evaluate their factor analysis byΠX3,Assessing if any of the columns of B3, andAssessing if any of the columns of ΠX3 is close to the column-space of Assessing if the column-space of analyses .with factor matrix B3 and ΠX3 and observe the robustness of their factor analysis methods to violations of their structural assumptions.In a factor analysis, the factors and loadings are only identifiable after imposing assumptions on their structure . Thus, researchers may vary the structure of We will now discuss the approach of obtaining  from 1)1). We wiy∼Poisson(a) and If Generalizations of Lemma Π is drawn uniformly from the space of permutation matrices, we have the simplified procedure described in Procedure 1. The validity of Procedure 1 follows directly from the modeling assumptions in (In the case when Θ from (Q from Step 4 of Procedure 1. Equation (μ−e.Equation  follows Θ from  and the Equation  follows There are two main reasons to subtract the row-wise maximum from each row in Step 4 of Procedure 1: (i) this ensures that the binomial probabilities (entclass1pt{minimaentclass1pt{minimaZ) will be independent of the user-specified covariates (ΠX3). To allow the user to control the level of association between the surrogate variables and the user-provided variables, we propose using Procedure 2 to choose Π, rather than drawing Π uniformly from the space of permutation matrices. In brief, the user specifies a “target correlation” matrix, rik is what the user desires to be the correlation between the ith column of ΠX3 and the kth column of Z. We then estimate the surrogate variables either using a factor analysis  or surrogate variable analysis [Z is only used to permute the rows of X3 and is otherwise not included in the simulated data. We then draw a new random matrix U and Z is jointly normal with covariance matrix (16), thus the correlation between the columns of U and Z will be approximately R. We then match the rows of X3 with the rows of U using the pair-wise matching algorithm of [ΠX3 is as close to U as possible. We denote the permutation matrix that matches the rows of X3 with the rows of U by Π.The main disadvantage to Procedure 1 is that the surrogate variables  used in Procedure 2 is not guaranteed to be positive semi-definite. Rather than demand the user specify an appropriate target correlation matrix , we modify the target correlation matrix using Procedure 3 to iteratively shrink niteness  is satisX3 and letting the user specify the correlation between the columns of ΠX3 and Z. A user might want to specify the correlation between ΠX3 and Z to evaluate factor analyses in the presence of correlated factors  log2-fold-change between groups to 25% of the genes. We similarly used the powsimR software [N log2-fold-change between the two groups in 25% of the genes. The parameters estimated and used by powsimR include the mean normalized read counts per gene, the estimated library size factor per sample, and a nonparametric estimate of the mean/dispersion relationship of the counts. powsimR uses the mean normalized read counts, the estimated size factors, and the user-provided log2-fold changes to provide a mean for the negative binomial distribution. Based on this mean, it uses the estimated mean/dispersion relationship to provide a dispersion parameter for the negative binomial distribution.We took the GTEx muscle data , and filsoftware  to generThe results below are from one simulation, but the results are robust and consistent across many datasets. The reader is encouraged to change the random seed in our code to explore the robustness of our conclusions.The structure of the powsimR dataset is very different from that observed in the seqgendiff and GTEx datasets. There seems to be more zeros in the powsimR dataset than in the seqgendiff and GTEx datasets  distribution for the non-null genes. We simulated 500 datasets this way using both seqgendiff and powsimR. Each replication, we applied DESeq2 [The differences in real versus simulated data  by the group membership for each gene, which we define asbg3 is log2-fold change for gene g, gth row of V(·) returns the empirical variance of a vector. When we looked at the median (over the non-null genes) PVE across the datasets, the seqgendiff datasets and powsimR datasets had the same median PVE on average, though there was higher variability in the median PVE among the seqgendiff datasets  ,Sparse singular value decomposition (SSVD) ,Independent component analysis (ICA) ,flash), an empirical Bayes matrix factorization approach proposed in [Factors and loadings by adaptive shrinkage  , a BayesWe chose to focus on the following methods based on (i) previous use in expression studies, (ii) software availability, (iii) popularity, and (iv) ease of use.All factor analysis methods were applied to the log2-counts after adding half a pseudo-count. To simulate RNA-seq data, we took the muscle GTEx data  and remox3 and the non-zero components of b3 from independent normal distributions. We varied the following parameters of the simulation study:N∈{10,20,40}The sample size: bg3’s) was set to one of {0.4,0.8}, with higher standard deviations corresponding to higher signal. These values were chosen to have the median PVE vary greatly between the two settings  that are 0 was set to one of {0,0.9}, andThe sparsity: the proportion of loadings (the r∈{0,0.5}.The target correlations of the added factor with the first unobserved factor: where we simulated the components of This resulted in 24 unique simulation parameter settings. We also used 1000 genes each replication. For each setting, we ran 100 replications of generating data from model , and fitΠx3 and the columns of ℓ2-norm of 1 prior to calculating the mean squared error. This measure is meant to evaluate if any of the estimated factors corresponds to the added factor.The minimum mean squared error between b3 and the columns of ℓ2-norm of 1.The minimum mean squared error between Πx3 and its projection onto the column space of Πx3 among its unidentified factors.The angle between We chose three metrics to evaluate the performance of the different factor analysis methods:PEER performs very poorly when either the sparsity is high or when there are few samples. It also performs less well when the factors are correlated. A possible explanation is that PEER assumes a normal distribution on the factors and loadings, which is violated in the high-sparsity regime and is observed in the low-sparsity regime. Though, this does not explain its poor performance in small sample size settings.SSVD estimates the loadings very poorly in low-sparsity regimes. This is to be as expected as SSVD assumes sparsity on the loadings. Surprisingly, though, it outperforms PCA in high sparsity regimes only when both the sample size and signal are also large.ICA performs very poorly in low sparsity regimes. This is to be as expected as the normal distributions placed on the factors and loadings are a worst-case scenario for ICA. However, there is no scenario where ICA performs significantly better than PCA.flash performs adequately in all scenarios and performs best in high-sparsity and high-signal regimes.PCA performs adequately in most scenarios, and is only truly outperformed in high sparsity high signal regimes.The results are presented in Supplementary Figures S9–S14 of Additional file flash.Based on these initial explorations, we would recommend users not use PEER, SSVD, or ICA and instead try either PCA or flash perform comparably in all simulation settings, while PEER and ICA have worse performance in some simulation settings. Though the results were less clear than when using the GTEx data.In Section S2.2 of Additional file Q matrix in Procedure 1. However, it is possible to replace Q by any arbitrary G×N matrix that has non-positive entries. This might be useful for simulations that study adjusting for non-linear effects, such as bias due to GC content [We have focused on a log-linear model because of the large number of applications this generates “” section content . This al content  and t-SN content , as thisOur simulation procedures may be applicable beyond evaluating competing methods. Vieth et al.  used theSimilarly,  already In this manuscript, we have discussed our simulation techniques in the context of RNA-seq. However, our techniques would also be applicable to the comparative analysis of metagenomics methods . Insteadflash had the most robust performances across a wide range of simulation settings.We developed a procedure to add a known amount of signal to any real RNA-seq dataset. We only assume that this signal comes in the form of a generalized linear model with a log-link function from a very flexible distribution. We demonstrated how real data contain features that are not captured by simulated data, and that this can cause important differences in the results of a simulation study. We highlighted our simulation approach by comparing a few popular factor analysis methods. We found that PCA and Additional file 1 This PDF file contains theoretical considerations, simulation summaries and figures, and additional simulation details.