Most biomedical knowledge is published as text, making it challenging to analyse using traditional statistical methods. In contrast, machine-interpretable data primarily comes from structured property databases, which represent only a fraction of the knowledge present in the biomedical literature. Crucial insights and inferences can be drawn from these publications by the scientific community. We trained language models on literature from different time periods to evaluate their ranking of prospective gene-disease associations and protein–protein interactions. Using 28 distinct historical text corpora of abstracts published between 1995 and 2022, we trained independent Word2Vec models to prioritise associations that were likely to be reported in future years. This study demonstrates that biomedical knowledge can be encoded as word embeddings without the need for human labelling or supervision. Language models effectively capture drug discovery concepts such as clinical tractability, disease associations, and biochemical pathways. Additionally, these models can prioritise hypotheses years before their initial reporting. Our findings underscore the potential for extracting yet-to-be-discovered relationships through data-driven approaches, leading to generalised biomedical literature mining for potential therapeutic drug targets. The Publication-Wide Association Study (PWAS) enables the prioritisation of under-explored targets and provides a scalable system for accelerating early-stage target ranking, irrespective of the specific disease of interest. Target identification and prioritisation represent critical initial steps in the drug discovery process. Historically, only 10% of drug targets successfully progress through clinical trials4, and the success rate appears even lower for novel targets6. Traditionally, individual researchers have relied on their expertise to analyse scientific literature and prioritise potential opportunities. Recently, numerous efforts have employed machine learning techniques to automate and prioritise gene-disease associations13. However, most of these studies have focused on training models and validating them on pre-existing therapeutic targets, rather than adopting a time-sliced approach to train on past data and evaluate the models' prospective predictions. Additionally, these approaches primarily rely on structured property databases as their main source of machine-interpretable data, which only encompass a limited portion of the knowledge found in the research literature. Paliwal et al. utilised a knowledge graph approach combined with link prediction methods to historically prioritise targets, claiming that their method successfully identified 1 in 4 therapeutic relationships that were eventually proven true.Numerous machine learning strategies have demonstrated the potential to accelerate drug discovery across various stages, such as drug repurposing, de novo drug design, clinical trial optimisation, and patient selection14 and PubMedBERT15, whereas GPT includes BioGPT16. These transformer language models have been pre-trained on biomedical literature. Typically, these models are evaluated and benchmarked on tasks such as entity recognition, relationship extraction, and question answering for known biomedical applications, following their training on the entirety of the biomedical literature. However, assessing their performance on genome-wide prioritisation of therapeutic targets presents a challenge, as these models often divide disease and gene synonyms into multiple tokens, complicating the process of consolidating them into a cohesive ranking.Scientific publications contain valuable relationships and inferences that the research community can interpret, with PubMed housing over 35 million publications as of 2023. The volume of scientific literature will only continue to grow, creating a vast repository of undiscovered public knowledge. The biomedical domain has seen a surge of interest in pre-trained language models, owing to their remarkable success in the general natural language domain. The pre-trained models can be classified into two main categories: BERT and GPT, along with their respective variants. BERT covers models like BioBERT17, and are eventually validated. Additionally, we examine the potential of historical Word2Vec models to prioritise yet-unknown hypotheses by utilising future test sets instead of relying on test datasets containing already established associations that the language models have already seen. By training unsupervised language models on historical subsets of biomedical literature, we explore whether these models could have effectively prioritised or suggested novel drug discovery scientific hypotheses in the past. To this end, we evaluate the language models on a range of drug discovery-related tasks, including target classification, gene-disease prioritisation, and protein–protein interaction prediction, within the context of a historical, retrospective study.In this study, we introduce a Publication-Wide Association Study (PWAS) to assess the ability of independently trained models using historical data to prioritise unstated hypotheses, which could be considered false positives or ‘hallucinations’18. Drug targets with genetic evidence are twice as likely to succeed in clinical trials20. However, most immune-related targets are found in the protein–protein interaction vicinity of genetically associated genes rather than being genetically associated themselves21. Therefore, it is essential to understand systems biology to propose novel targets. Moreover, validating protein–protein interactions can be time-consuming and costly, but computational tools offer a cost-effective alternative to prioritise novel protein–protein interactions18 and suggest new therapeutic targets. There have been numerous methods to predict protein–protein interactions using a mixture of traditional machine learning methods22 and deep learning models25 that include AlfaFold2 protein structure information. However, this work focuses on whether language models could have prioritised novel interactions with the limited data available in the past scientific literature.Protein–protein interactions are also crucial in drug discovery because data incompleteness in the human interactome hinders our understanding of the molecular roots of human diseaseIn this work, we introduce a Publication-Wide Association Study (PWAS) that examines whether language models can retrospectively prioritise gene-disease associations before their initial statements, target-disease indications before the conception or synthesis of drug-like molecules, and protein–protein interactions before their discovery.26 language models on an English corpus of 19.5 million biomedical titles and abstracts to generate embeddings for a vocabulary of approximately 2 million phrases  and Toll-like receptor 7 (TLR7) were ranked from the top-10000 to the top-1000 genes for SLE before the first publication implicating them in the pathology of lupus, in 2009 by Cohen37 and 2005 by Barrat et al.38, respectively. IRAK4 was consistently ranked in the top-1000 genes prior to the first PF06650833 inhibitor by Pfizer which dates back to 201439. Similarly, TLR7 was on the top ranking for several years before the first TLR7 inhibitor that entered the clinic, IMO3100, which was discontinued. Enpatoran is another TLR7 inhibitor currently in Phase 2 trials for lupus40. The contextual linking words indicate why the model suggested IRAK4 for lupus, due to its involvement in innate immunity through TLR7 that subsequently activates Myd8842  in 200745 and the AXL inhibitor bemcetinib in 201046. The 2006 language model was aware of their role as receptor tyrosine kinase oncogenes  association with types of NSCLC in 199551 but the oncogenic role in lung adenocarcinoma was not confirmed until 2008 by Ding et al.51. NRTK1 ranking was consistently in the top 100 genes , we uncovered hidden structures within the embeddings in this lower-dimensional space , 0.27 for kinases, 0.23 for transcription factors, and 0.38 for voltage-gated ion channels  and monoclonal antibody (AB) targets Fig. G. Figure60. For identifying potential therapeutic targets, it is essential to understand the pathways impacted by the disease and pinpoint nodes within the signalling network whose modulation can alleviate the disease state60. We investigated if language models prioritise protein–protein interactions (PPIs) before they are published. The historical data of PPIs were obtained from yeast-two-hybrid assays of the human interactome60, which consists of four datasets: H-I-05 from 2005, HI-II-14 from 2014, HuRI from 2020, and HI-union from 2020. A bona fide negative set was absent. Combinations of 'sticky' proteins, representing the top 10% of proteins with the highest number of interactions, served as negative examples, except when they overlapped with the positive set , clustered together with a mean cosine distance of 0.32, compared to 0.14 for any-to-any gene  for promiscuous compounds acting on three or more targets Table C. The H162.As demonstrated in Table We examined the genome-wide target ranking for metformin in comparison to genome-wide association data  is involved in converting citrate and coenzyme A into acetyl-Coenzyme A and oxaloacetate67. ACLY is deacetylated and inhibited by SIRT2, but not SIRT167, a putative metformin target. The Arg592Gln ACLY mutant may exhibit altered acetylation, ubiquitination, or phosphorylation upon metformin treatment. However, additional experimental work is necessary to confirm its connection with metformin.The Arg592Gln (rs147387411) mutation is located near the coenzyme A binding domain but not directly within it. Acetylation at three lysine residues  by lysine acetyltransferase 2B (KAT2B) has been shown to increase ACLY stability by blocking its ubiquitylation and promoting de-novo lipid synthesisThe Publication-Wide Association Study (PWAS) leverages language models to analyse 19.5 million publications, prioritising previously unpublished hypotheses across various contexts: gene-disease associations, target-disease clinical trials, protein–protein interactions, and unexplored yet plausible drug mechanisms of action. The study demonstrates that word embeddings, or vector representations of words, can effectively encode complex biomedical knowledge from published literature without requiring a priori biomedical knowledge. These embeddings capture intricate biomedical concepts, such as drug target classes, tractability, disease phenotype involvement, and protein–protein interactions. Furthermore, language models have successfully ranked novel hypotheses years before their publication in top-tier journals, suggesting that the foundations of future discoveries are already embedded within existing publications to some extent.To the best of our knowledge, this is the first attempt to utilise language models for prioritising drug discovery hypotheses prior to their publication. Language-based inference methods may serve as a promising new research field at the intersection of natural language processing and target identification, propelling language models beyond their traditional use in named entity recognition and normalisation tasks to harness the wealth of associations present in biomedical literature. The various tasks for language models will be discussed in the following sections.13 . These associations cannot be prioritised with language models unless scientists write paragraphs describing the gene function based on experimental work. Furthermore, PWAS is another association study far from prioritising causal genes. Paradigm shifts in molecular biology and spurious associations between genes and diseases will bias the models. Furthermore, scientific literature is noisy and often contradictory. Despite these problems, the PWAS could prioritise some targets several years before their first publication and before their first clinical trial. Future work will require training more complex transformer models at hypothesis generation and question answering.The vector representations of genes contain information about small molecule and antibody druggability, disease association, therapeutic target family, pathways and clinical precedence that can be generalised to unseen genes Fig. . These e18, as depicted in Fig. 25, the predictions in this study do not identify the interface residues essential for the interaction. Additionally, the current work heavily depends on high-throughput yeast-two-hybrid (Y2H) experiments, known for limited sensitivity and specificity60. Y2H experiments face a significant drawback, as they cannot capture PPIs requiring post-translational processing, a common occurrence in humans, and only focus on a single isoform per gene, excluding alternative splicing variants that may have crucial roles in protein function and interactions.In this study, the PWAS exhibited enhanced performance in comparison to L370 or STRING71. These resources offer valuable information on experimentally validated and predicted PPIs, and their inclusion could potentially enhance the comprehensiveness and accuracy of human interactome mapping.Moreover, this study did not incorporate several independent and competing PPI databases, such as IntActLanguage models demonstrated the ability to learn known target-drug relationships for specific compounds effectively. Notably, the putative targets of metformin prioritised by these models and the genome-wide ranking generated strongly correlated with independent small molecule tractability scores. It is hypothesised that the Arg592Gln ACLY mutant might undergo differential acetylation, ubiquitination, or phosphorylation in response to metformin treatment. However, this hypothesis necessitates further experimental validation to establish its veracity conclusively.Despite the vast potential of language models trained on biomedical literature, it is crucial to acknowledge their inherent limitations. One primary concern is the presence of biased, erroneous, irrelevant, and weak studies within the training data. These suboptimal sources may introduce noise and inaccuracies, thereby corrupting the model's knowledge base and impairing its ability to make accurate predictions or generate reliable insights. Consequently, the conclusions drawn from such models could be compromised, leading to misguided hypotheses and, ultimately, impeding scientific progress. Therefore, it is essential to exercise caution when interpreting the outputs of language models and to validate their findings through independent experimental approaches to ensure the veracity and robustness of the generated results.Named entity recognition (NER) errors can greatly affect the quality and dependability of a publication-wide association study (PWAS) in target-disease prioritisation. These inaccuracies may result in false or missed associations, misrepresented relationships, and diminished trust in the model outputs. While advanced NLP techniques and transformer models, such as BioBERT or GPT-based architectures, may offer improvements, they still face challenges in ranking extensive lists, like 20,000 human genes. As these models tokenise input sequences into smaller units, gene names or symbols can be divided into multiple tokens, complicating the generation of a precise genome-wide ranking approach. Future research will need to be made to replicate the PWAS work with transformer models and evaluate them in a retrospective analysis to prove they can prioritise novel preclinical targets that are eventually successful in the clinic.A further limitation concerning entity recognition lies in the representation of diseases as singular entities with a unified vector representation. However, diseases are heterogeneous, and gene alterations contribute variably to the onset and progression of the condition. Certain genes may serve as potential targets for the initial manifestation, while others play a role in exacerbating and aggravating the disease's progression. This study generates a single genome-wide ranking for each disease, regardless of its state or trajectory. Nonetheless, some disease stages are represented by distinct ontology terms within the MeSH ontology, such as the progression from NAFLD to non-alcoholic steatohepatitis and ultimately to hepatocellular carcinoma where the rankings will be slightly different. Moreover, target prioritisation encompasses a multi-objective ranking approach where the target needs to be safe, tractable and associated with the disease. Incorporating additional objectives or keywords could help prioritise targets for particular phenotypes of the disease. For example, combining associations to DNA-damage repair keywords with safety and tractability keywords along with cancer types to bias the initial ranking, similar to how one would pose a question with more detail to a transformer model.The findings of this study indicate that to some extent, latent information regarding future discoveries is embedded within past scientific publications. Furthermore, language models present a compelling method for prioritising unverified hypotheses derived from the continuously growing corpus of biomedical literature. To our knowledge, this work represents the first literature-wide association study that ranks therapeutic drug hypotheses for subsequent validation utilising language models trained on historical corpora.Efficient assimilation of existing knowledge, identification of promising research directions, and avoidance of redundant efforts are all vital components for scientific progress and hypothesis generation in target analysis. However, with the ever-increasing volume of biomedical literature, this task becomes progressively more challenging, if not unfeasible, for individual researchers.Our research has the potential to contribute to a novel paradigm wherein the vast amount of information contained in scientific publications becomes readily accessible to individual target analysts, thereby fostering machine-assisted breakthroughs in the identification of innovative targets. The Publication-Wide Association Study (PWAS) represents a scalable system that facilitates the prioritisation of under-explored targets, expediting early-stage drug target selection irrespective of the specific disease of interest. This method holds the potential to radically transform the way researchers navigate and leverage the extensive knowledge base within the biomedical field.The input data for this study comprised roughly 19.5 million English abstracts sourced from PubMed, which included mentions of any human gene or disease synonyms in their titles or abstracts. Abstracts classified as 'Commentary', 'Correction', and 'Corrigendum' in PubMed were excluded. In this study, we utilised a comprehensive set of 4,819 disease terms from the Medical Subject Headings (MeSH) ontology, which encompasses a diverse range of diseases, phenotypes, and symptoms. MeSH, developed by the National Library of Medicine, is a controlled vocabulary used for indexing and cataloguing biomedical literature in the MEDLINE/PubMed database. Furthermore, regarding the gene vocabulary, we have taken into account 19,229 human protein-coding genes.72 pipeline was employed to identify biomedical entities, specifically genes and diseases, within these abstracts. Standardised identifiers were utilised to normalise genes and diseases, a process that involves canonicalising all concept synonyms. For instance, 'Her2', 'ERBB2', and 'Neu' were normalised into their Ensembl identifiers, while 'non-insulin-dependent diabetes', 'diabetes mellitus, type II', and 'T2DM' were normalised to the corresponding Medical Subject Headings (MeSH) identifier for type 2 diabetes.The TrendyGenes69.Normalisation was implemented for several reasons. Firstly, it reduces the vocabulary size. Secondly, it accounts for different research groups using distinct gene synonyms in various research contexts . Canonicalization consolidates information into a single token, enabling the generation of a genome-wide ranking of 20,000 human protein-coding genes, rather than n_synonyms multiplied by the number of genes. Normalising biomedical entities led to a fivefold reduction in gene synonym mentions, from 102,719 to 19,229 human protein-coding gene identifiers. Similarly, disease synonym mentions were reduced tenfold, from 53,317 to 4819 human disease identifiers. Proper corpus preprocessing significantly enhances the performance of language modelsThe vocabulary for word2vec models included all words that appeared more than ten times and normalised gene and disease identifiers, regardless of the number of mentions. As a result, historical models featured different vocabularies. If a gene is never mentioned in the training corpus, it is excluded from the language model's vocabulary. For instance, 13,676 distinct human genes were mentioned in the training corpus up to 2005, while 16,533 different human genes were mentioned until the present, accounting for 86% of the total 19,229 human protein-coding genes. Normalised gene and disease mentions refer to the mapping of any textual references to a unique identifier from the HUGO or the Medical Subheadings Ontology (MeSH).Leading statements such as 'Background:', 'Abstract:', and 'Introduction:' were removed. Phrases were generated using a minimum phrase count of 10, normalised mutual information score greater than 0.7, and phrase depth up to three times. This phrasing process was repeated thrice, allowing the generation of up to 8-g. For example, the phrase 'g protein coupled receptor' appeared more than ten times in the corpus and had a normalised mutual information score greater than 0.7, which was then phrased into 'g_protein_coupled_receptor'. The text was converted to lowercase and deaccented. Floating decimals and percentage numbers were replaced with the word '< number >' using regular expressions to decrease vocabulary size. Stop words were retained as they constituted approximately 100 tokens within the half-million-token vocabulary.73, an embedding size of 255 minimised the Pairwise Inner Product loss, though we also conducted a grid search with sizes of 128, 256, and 512. The remaining hyperparameters were set to their default values.We employed the Word2Vec algorithm from the Gensim Python library for our analysis. A range of hyperparameter combinations were explored to identify the optimal configuration for capturing gene target to disease indication analogies Table . We sele−4 threshold, targeting approximately the 400 most frequent tokens in the vocabulary. Both Word2Vec variants, skip-gram and continuous bag of words (CBOW), were considered. The chosen model, which maximise performance on a curated dataset of target-to-disease analogies .Literature associations between genes and diseases were obtained from the TrendyGenes pipeline12 2020 publication: 200 diseases and all their targets were in the testing set, and the remaining disease and their targets were part of the training set. In the supplementary methodology section, Paliwal et al.12 provided supplementary information for only 40 out of the 200 diseases. The remaining 160 diseases were randomly selected to encompass a variety of disease types, including metabolic, immune, neurological, and oncological indications. To achieve this, diseases were first categorised into four groups . Only diseases with at least three targets in Informa were considered for inclusion in the testing set, while over-specific diseases situated at the leaf nodes of the ontology tree were excluded. Utilising a uniform distribution, 160 diseases were then randomly selected from each category, 40 diseases from each therapeutic area. All remaining diseases with at least one target were allocated to the training and validation sets.For Fig. 52 were obtained from Wang et al. 2021 data53. MAGENTA ran with default parameters with the summary statistics and all data in the genome build Genome Reference Consortium 37 (GRCh37). To run the Priority index21  scored positively (including 644 'Lead' and 6001 'Linkage Disequilibrium' genes); 714 nearby genes within 50,000 base pairs genomic distance window of 6129 SNPs 2045 expression Quantitative Trait Loci (eQTL) genes with expression modulated by 4677 SNPs; 912 HiC genes physically interacted with 4033 SNP; 2900 genes defined as seeds from 6645 SNPs; randomly walk the network  starting from 2900 seed genes with a restarting probability of 0.75. The Open Targets dataset only gives a score of 0 and 1 to targets with evidence for diseases. To generate a genome-wide ranking to calculate the z-scores, the remaining non-associated targets were given a random score (uniform distribution) between the minimum association score for the disease and zero. Z-scores were calculated with the quantile normalisation function in Scikit Learn.For Fig. The list of therapeutic targets in this section was downloaded from Pharmaprojects and included all therapeutic targets for systemic lupus erythematosus that had an active program in the following stages: Phase I Clinical Trial, Phase II Clinical Trial, Phase III Clinical Trial, Registered, and Launched. This included the following gene symbols: BTK, BTLA, CD200, CD200R1, CD22, CD28, CD38, CD40, CD40LG, CD6, CD79B, CLEC4C, CNR2, CRBN, CXCR5, FCGR2B, FCGRT, FKBP1A, ICOSLG, IFNA1, IFNAR1, IKBKE, IL12B, IL1RL2, IL2, IL21, IL23R, IL2RA, IL2RB, IL2RG, IRAK1, IRAK4, JAK1, JAK3, LANCL2, LGALS1, LGALS3, LILRA4, MALT1, MASP2, MC2R, MIF, MS4A1, NLRP3, NR3C1, PIK3CB, S1PR1, SIK2, SLC15A4, SNRNP70, SOCS1, STING1, SYK, TBK1, TLR7, TLR8, TNFRSF13B, TNFRSF13C, TNFRSF4, TNFSF13, TNFSF13B, TYK2, XPO1.54 and Open Targets13 datasets. The therapeutic target classes and the kinase families were downloaded from the Guide to Pharmacology database57. The Uniform Manifold Approximation and Projection (UMAP) low dimensional representation was achieved using the umap-learn package in Python with the following hyperparameters: 15 neighbours, cosine similarity as the metric, 1000 epochs, a repulsion strength of 12 and local connectivity of 3 neighbours. To measure the consistency of the clusters in Fig. The clinical stages for the targets were gathered from Pharmaprojects9 for the multilayer perceptron, a feed-forward neural network with a single layer and a balanced function implemented in SciKit Learn. The solver was Adam, and the network stopped if the performance did not improve for two epochs in a validation set. The input data was the gene embeddings from the language model from December 2016 to directly compare to Open Targets features at the time. The metrics displayed in Table 54 and Open Targets. The datasets in this section were randomly split into training and test sets, containing 80 and 20% of the observations , respectively.We adopted the same hyperparameters as Ferrero et al. in 201760, accounting for 64,000 human protein–protein interactions in its latest release. Self-interactions were discarded like self-phosphorylations in tyrosine kinase receptors. Protein interaction pairs were augmented by reversing the gene order (e.g. 'A1BG-ZNF44' =  > 'ZNF44-A1BG'). These interactions were positive examples. There was no bona fide negative set. Therefore, all combinations of the top 10% of proteins with most interactions were negative examples unless they intersected with the positive set. For example, even though the genes DVL2 and TRAF2 had 36 and 69 interaction pairs, the DVL2-TRAF2 pair was excluded from the negative pair set because it was in the H-I-05 positive pair set. However, the pair DVL2-NIF3L1, with 36 and 31 interactions respectively in H-I-05, was included in the set of negative pairs. An upset plot for the different protein–protein interactions coming from each of the historical datasets can be seen in Fig. We used all historical versions of the human interactome75 could prioritise future protein interaction pairs even if they were presented as negatives. The features were the concatenation of the two embeddings for the proteins in the pair. Language models were checkpointed each December. Embeddings trained until December 2005 were the features for the model trained on H-I-05 examples, embeddings from December 2014 embeddings to train with HI-II-14 positive examples and embeddings from December 2020 embeddings with positive examples from HuRI and HI-union.To test the ability of the models to eventually recover true protein–protein interactions, the newly positive interactions in 2020 were regarded as negative examples to the 2014 and 2005 models. For example, the CCHCR1-ZWINT interaction introduced in HuRI was a positive example for the 2020 experiment but negative for the 2014 and 2005 experiments. The idea was to test whether a positive-unlabelled strategyThe split into training, validation and test sets was done in two ways. There was a protein-based split for testing. A random split of 1:3 for the test and train sets at the protein but not the pair level. 25% of genes and all their partners are just on the test set and never seen during the training. Also, there was a time-based split for validation. The predictions were also evaluated prospectively in Fig. A grid search was performed for the hyperparameter tuning. Three hyperparameters were tuned: the number of hidden layers (1 or 2), the dropout  and the size of the hidden layers . This tuning generated 60 different models with different combinations of hyperparameters that resulted in the contour ternary plot in Fig. 76, comprising over two thousand curated, high-quality binary interactions involving human proteins. Positive examples were derived from this dataset, while negative examples were generated by combining the 400 most frequently occurring proteins with the 400 least specific compounds. These negative examples did not overlap with the positive set. This approach was adopted for three reasons: (i) to improve false association discrimination by incorporating non-specific drugs and targets in the negative sets, enabling models to better differentiate between true and false protein-drug associations so the models do not just predict that staurosporine, sunitinib and sorafenib inhibit all kinases; (ii) to enhance model robustness by introducing difficult negative examples, leading to more robust and generalisable models; and (iii) to ensure rigorous model evaluation by presenting a challenging negative set, characterised by a higher potential for false positives.The compound-protein interaction data was sourced from DrugBankMultiple machine learning classifiers were trained to prioritise positive and negative compound-target interactions, including random forest classifiers and logistic regression implemented in Scikit Learn and deep neural networks in TensorFlow. The neural networks have a variable number of layers, dropouts and sizes of their hidden layers. All models have a balanced loss function to account for the imbalance in the dependent variable. All models used are displayed in Table 77 for protein-motif links. InterPro classifies protein amino acid sequences into families and predicts the presence of functionally essential domains and sites. For the small molecule tractability, we followed the implementation of Berenstein et al.68, where each protein target had a tractability score. Each protein had the maximum tractability score from the p-value of a Fisher’s exact test for their constituent InterPro77 motifs. InterPro motifs were associated with compounds through a motif-protein-compound graph. This approach gives non-trivial scores to proteins with druggable motifs if another protein with the same motif has been drugged.We used the latest release of the InterPro database78. This dataset contains 1024 Scottish individuals with type 2 diabetes and two replication cohorts: 1113 individuals from the UK Prospective Diabetes Study and 1783 Scottish individuals. MyVariant79 was used as a service to query multiple variables regarding each single nucleotide polymorphism, including non-synonymous replacements and their functional consequences.The genome-wide association study in Fig.


Cells can die through a process called apoptosis in both pathological and healthy conditions. Cancer development and progression may result from abnormal apoptosis. The 78-kDa glucose-regulated protein (GRP78) is increased on the surface of cancer cells. Kringle 5, a cell apoptosis agent, is bound to GRP78 to induce cancer cell apoptosis. Kringle 5 was docked to GRP78 using ClusPro 2.0. The interaction between Kringle 5 and GRP78 was investigated.The interacting amino acids were found to be localized in three areas of Kringle 5. The proposed peptide is made up of secondary structure amino acids that contain Kringle 5 interaction residues. The 3D structure of the peptide model amino acids was created using the PEP-FOLD3 web tool.The proposed peptide completely binds to the GRP78 binding site on the Kringle 5, signaling that it might be effective in the apoptosis of cancer cells. Cancer is a major global public health concern with its rising incidence over the past few decades, and it has significant economic and social impacts , 2. ThisUtilizing computational analysis methods, specifically molecular dynamic simulation, is one method for addressing the problem of therapeutic cancer treatment –7. The cAnticancer therapies have been focused on their ability to destroy cancer cells while preserving healthy cells. Targets for anticancer treatments are molecules that are selective for cancer cells. Peptides that target cancer cells could be used in the development of anticancer drug delivery agents for localizing cancer sites , 11. On When natural ligands bind to GRP78 on the cell surface, they can cause a number of signaling responses. GRP78 can then bind to 2-macroglobulin and Cripto on tumor cells, causing cell survival and proliferation, or bind to Kringle 5 and Par-4, causing cell death , 19, 20.The current study employs computational docking screening to investigate potential Kringle 5 peptides that bind to GRP78 and cause cancer cell apoptosis. Docking of Kringle 5 to GRP78 was performed in silico, and the interacting peptides were studied.https://www.rcsb.org/structure/5HPG).The structure of Kringle 5 was obtained from the data bank of protein (PDB ID: 5HPG). The protein structures were downloaded from (https://cluspro.org), is commonly used for docking analysis of proteins [The webserver ClusPro 2.0 for protein docking interactions was used to dock Kringle 5 (5HPG: A) to GRP78. ClusPro 2.0 is a powerful tool for predicting the three-dimensional structure of protein–protein complexes through molecular docking simulations. It uses a rigid-body docking approach, in which the two protein structures are kept rigid while their orientations are adjusted to find the optimal binding configuration. The tool, available at (proteins –24. The proteins . The MM/https://espript.ibcp.fr/ESPript/ESPript), a web-based program for extracting and producing an automated analysis of protein structures, with high-quality representations of multiple sequence alignments of proteins. The program reveals sequence similarities, generates sequence logos, secondary structure, and residue conservation plots, and offers ways to enhance and improve the way they are represented [We used ESPript (resented . ESPripthttp://web.expasy.org/protparam) was employed to calculate the candidate peptide characterizations and hydropathicity index, including aliphatic index (pI), instability index, extinction coefficient, and molecular weight. ProtParam calculates several important protein properties, including the molecular weight, theoretical pI, amino acid composition, extinction coefficient, and instability index. It also provides information on the number of positively charged, negatively charged, and polar residues in the protein sequence, as well as the percentage of hydrophobic and hydrophilic residues [http://pepcalc.com) [The online program ProtParam  . Pepcalchttp://bioinf.cs.ucl.ac.uk/psipred) was used to determine the peptide’s secondary structure [The PSIPRED prediction method (tructure . PSIPREDhttps://bioserv.rpbs.univ-paris-diderot.fr/services/PEP-FOLD3) was utilized to predict the peptide’s three-dimensional (3D) structure, generating five possible models [https://spdbv.unil.ch). The peptide with the lowest free energy was chosen. Both PROCHECK webserver (https://www.ebi.ac.uk/thornton-srv/software/PROCHECK) [https://prosa.services.came.sbg.ac.at/prosa.php) [The online computational framework tool PEP-FOLD3   and ProSosa.php)  were usehttps://www.ebi.ac.uk/msd-srv/prot_int/cgi-bin/piserver) was used to investigate the interactions between the proposed peptide and GRP78. PDBePISA works by analyzing the 3D structures of proteins and other molecules in the PDB, and identifying the interfaces and surfaces that are involved in protein–protein, protein–ligand, and protein-DNA interactions. The program then calculates the binding energies and other physical and chemical properties of these interfaces and generates detailed reports and graphical representations of the results.The peptide model was docked to GRP78 once more using ClusPro 2.0. The MM/GBSA was used to rank the docking complexes based on their binding energy. The compound with the highest binding energy was selected and searched for interaction residues. The Protein Data Bank in Europe: Protein Interfaces, Surfaces, and Assemblies (PDBePISA), which is a webserver tool available at  and phi (Ф) twisting or dihedral angles. Ramachandran plot analysis’ tertiary structure validation revealed that the overall percentage of favorable and allowed region residues was 100 percent, as shown in Fig. ClusPro 2.0 was utilized to dock PEP 35 to GRP78. At the same location as Kringle 5, PEP 35 interacts with GRP78. GRP78 residues that interact with both PEP 35 and Kringle 5 are highlighted in grey as demonstrated in Fig. The docking mechanism formed binding between the legends and the protein, which could be unstable . The simThe rising incidence of cancer is a major global public health concern, and it has significant economic and social effects. We demonstrated using molecular docking and molecular dynamics simulations that GRP78 binds to Kringle 5 and triggers the death of tumor cells, indicating that it may be beneficial as a therapeutic agent to site-direct and localize cancer cells. The efficacy of the proposed peptide apoptotic pathway requires additional experimental investigation.


Proteins are the building blocks of life, carrying out fundamental functions in biology. In computational biology, an effective protein representation facilitates many important biological quantifications. Most existing protein representation methods are derived from self‐supervised language models designed for text analysis. Proteins, however, are more than linear sequences of amino acids. Here, a multimodal deep learning framework for incorporating ≈1 million protein sequence, structure, and functional annotation (MASSA) is proposed. A multitask learning process with five specific pretraining objectives is presented to extract a fine‐grained protein‐domain feature. Through pretraining, multimodal protein representation achieves state‐of‐the‐art performance in specific downstream tasks such as protein properties (stability and fluorescence), protein‒protein interactions (shs27k/shs148k/string/skempi), and protein‒ligand interactions , while achieving competitive results in secondary structure and remote homology tasks. Moreover, a novel optimal‐transport‐based metric with rich geometry awareness is introduced to quantify the dynamic transferability from the pretrained representation to the related downstream tasks, which provides a panoramic view of the step‐by‐step learning process. The pairwise distances between these downstream tasks are also calculated, and a strong correlation between the inter‐task feature space distributions and adaptability is observed. In this work, a comprehensive protein representation framework integrating sequence, structure, and function is introduced. This model excels in diverse applications. A novel spatial metric is also presented, assessing the dynamic transferability of pretrained protein representations to downstream tasks, offering insights into the step‐by‐step learning process. A natural protein, which is made up of a linear sequence of amino acids linked by peptide bonds, folds into its 3D or tertiary structure to perform its biological function. Understanding a protein from its sequence, structure, and function is one of the greatest scientific challenges of the 21st century, as it is essential for elucidating the mechanisms of disease and therapeutic development. In recent years, the exponential growth in available protein data  has provided fertile resources for studying proteins using computational methods, particularly artificial intelligence.1 A representation is, at its core, a distillation of raw data into a high‐level, typically low‐dimensional space that captures the underlying features of the complex raw data. This representation, usually a numerical vector or array, can then be utilized for a variety of purposes, including task‐specific prediction  and interpretable exploration. A good representation can be greatly helpful for improving the performance of a model. Recently, self‐supervised pretrained language models have emerged as a powerful paradigm for learning efficient representations from raw data in various fields, such as natural language processing (NLP) and computer vision (CV). The goal of a language model is to learn underlying semantics from large text corpora based on the distributional hypothesis: words that appear in similar contexts may have similar meanings.Artificial intelligence techniques have become mainstream in this area, and they usually start with representation learning.2 A re and thus, NLP language models can be well applied. The corresponding protein representations have shown excellent performance in a variety of protein‐related downstream applications, such as predictions of protein stability and mutational effects. However, proteins are more than simply linear sequences of amino acids, and it is difficult to infer the completeness of a protein from sequence data alone. The incorporation of protein structure or functional annotation into language models is a relatively new development. Although such approaches have enhanced the performance and application of models, challenges remain. Numerous distinctive and fine‐grained protein properties, such as functional unit domains, have not yet been properly incorporated into a pretrained model. In addition, there is still no metric for quantifying how well a protein representation has been pretrained and how suitable it is for downstream tasks.Proteins can be regarded as the natural language of biology, written with multiple amino acid words,1 and Figure  There are three notable improvements.1)The rational integration of three protein modalities. The heterogeneity of protein modalities hinders their efficient fusion. Common techniques, such as concatenating the embeddings of various modalities, disregard this heterogeneity and consequently result in the loss of modality‐specific features. To solve this problem, we have developed a hierarchical two‐step alignment method. Initially, each amino acid token embedding of the protein sequence and structure is aligned using token‐level self‐attention, yielding a sequence‐structure embedding containing both sequential evolutionary and spatial information. This representation is then globally aligned with GO (Gene Ontology) annotation embedding via a cross Transformer decoder.2) in which physical, biochemical, and biological knowledge of proteins is not fully considered. To extract high‐level and fine‐grained protein‐domain features, we present an equalized multitask loss function for five protein‐specific pretraining objectives, including masked amino acid/Gene Ontology prediction and domain/motif/region placement capture. Based on the model architecture and comprehensive pretraining objectives, the resulting protein representation outperformed other methods in specific tasks such as protein properties (stability and fluorescence), protein–protein interaction (PPI) (shs27k/shs148k/string/skempi), and protein–ligand interaction (PLI) , while achieving competitive results in secondary structure and remote homology.The elaborate design of pretraining objectives. Existing approaches frequently employ pretraining tasks similar to tasks from the NLP domain, such as next token prediction and masked language modeling ,10 in 3) However, this raises the question of whether such transferability can be quantified to determine how well a model has been pretrained and which downstream tasks benefit most from pretraining. Here, we introduce a novel spatial metric for quantifying the dynamic transferability from a pretrained protein representation to downstream tasks, which provides a panoramic view of the step‐by‐step learning process. With this metric, downstream task‐to‐task adaptability experiments are further explored, and a strong correlation is observed between task‐specific representation similarities and cross‐task adaptability.The quantification of pretraining‐to‐task and task‐to‐task transferability. Previous studies have demonstrated that downstream tasks often benefit from pretrained models.12 HowHere, we introduce MASSA, a multimodal protein representation framework that incorporates domain knowledge across protein sequence, structure, and functional annotation Figure 1. The r22.1www.uniprot.org), protein structures from RSCB PDB (www.rcsb.org) and the AlphaFold Protein Structure Database , GO annotations from the Gene Ontology Resource (geneontology.org), and protein motif, region, and domain information from UniProtKB (www.uniprot.org/uniprotkb). After preprocessing, the constructed multimodal set consists of approximately one million sequence, structure, GO annotation, protein region, motif, and domain samples. The quantity and raw format of these multimodal data are depicted in Figure We collected multiple datasets, including protein sequences from UniProt  is tagged as “O” (outside of entity). An entity that has only one token is tagged as “S” (single), such as S − Motif3.In Figure Following named entity preprocessing, we analyzed the distributions of the categories. There are 1364 categories of motifs, 3383 categories of domains, and 10628 categories of regions. All of these data, as depicted in Figure 2.2The MASSA model was then pretrained using the constructed multimodal dataset. The pretraining process consisted of three steps Figure : 1) Feat and GraphGO . To obtain the initial embedding of GO terms, we developed and trained a model called GraphGO Figure . Specifi After 150 epochs, pretraining was stopped, and the model was then evaluated on downstream tasks.MASSA was pretrained on the constructed multimodal dataset using an equalized multitask loss function for the five protein‐specific pretraining objectives. The detailed model architecture and parameters are exhibited in Extended Data Figure 12 Aft2.3 More details can be found in Supporting Information and source code.We evaluated and compared the pretrained MASSA model on a variety of protein‐related benchmark tasks, including protein properties, protein‐protein interactions (PPI), and protein‐ligand interactions (PLI). It is worth noting that our model doesn't necessarily require all of the sequence, structure, and function data for downstream application. When a protein sample contains three modalities, they are all taken as input, whereas for samples with missing modal (such as the protein property tasks), structure and GO term information were processed as masked tokens.3 More2.3.1 including secondary structure, remote homology, fluorescence, and stability benchmarks. On these sequence‐based datasets, we evaluated the performance of our model in comparison with other methods in two distinct ways: with or without pretraining objectives. “Without pretraining objectives” Random initialization: random vectors for the sequence and GO embedding.2)Random initialization (without pretraining): random vectors for the sequence and GO embeddings Random initialization of the GO embedding: without step 1–2.4)ESM‐MSA‐1b only: the ESM‐MAS‐1b sequence embedding only (step 1‐1).5)2‐layer FFN (feedforward network) only: the network used to predict downstream tasks from final protein representation .Next, we carried out an ablation study to assess the effectiveness of each module of the pretraining multi‐modal model Figure . There w multiple sequence alignment and masked language modeling were found to be most advantageous for protein‐structure‐related tasks, such as secondary structures and remote homology. In contrast, information obtained from our proposed multimodal fusion and fine‐grained pretraining objectives is more beneficial for biophysically related tasks such as stability and fluorescence. The gaps in the results between the ablation groups and the full model  and depth‐first search (DFS). Specifically, BFS and DFS generate unknown proteins Pu (used as the test set) in two distinct ways: 1) The Pu interact closely with one another and are grouped as clusters within the PPI network. 2) The Pu are widely dispersed in the PPI network and seldom interact with one another.The model was evaluated on several protein‒protein interaction benchmarks, including SHS27k, SHS148k, STRING, and SKEMPI. Among them, STRING, SHS27k, and SHS148k are multilabel classification benchmarks, while SKEMPI is built for regression. More details about these datasets can be found in the Experimental Section. Four methods, PIPR,17 GNN and the second‐best method OntoProtein by an average of 41.32 and 4.74 percent, respectively.As seen, our multimodal model outperformed the other methods on all these benchmarks. Our model outperformed the baseline method PIPR17 andFor these PPI benchmarks, the full model receives inputs from all three modalities , and an ablation study was conducted to assess the efficacy of each of these modalities. There were five experimental groups representing different combinations of modalities: 1) random initialization, 2) sequence only, 3) sequence + structure, 4) structure + GO, and 5) sequence + GO. As demonstrated, Groups 3–5 performed better than Group 2, suggesting that the incorporation of multiple modalities can be advantageous for all of these PPI datasets. Among these results, Groups 3 and 5, with sequence plus another modality achieved relatively superior outcomes, confirming the significance of sequence information. and 13 other protein language models. As shown in Figure   and the second‐best method ProtALBERT . These results suggest that the additional information provided by the rational integration of multiple modalities may be advantageous for PPI tasks.Many previous prediction models have been applied to SKEMPI, which is a regression dataset for predicting the influence of mutation‐based  changes in binding affinity on PPI. Our model was evaluated on this dataset using 10‐fold cross‐validation and three metrics, namely, Pearson's correlation (higher is better), mean squared error (MSE) (lower is better), and mean absolute error (MAE) (lower is better). The 10‐fold training process for the SKEMPI set is shown in detail in Figure 17 and2.3.3 (a graph‐attention‐network‐based method), and protein‒ligand interactions were processed by a 3‐layer FFN.We also evaluated our model on protein‒ligand interaction (PLI) tasks. Two PLI benchmarks were used: the Kinase set (sequence‐based), PDBbind v.2016 (structure‐based), and DUD‐E (structure‐based). Similar to the PPI experiments described previously, our model received inputs from all three modalities. The structures of proteins from the Kinase set used in this evaluation were predicted by Alphafold2. Another point to note is the PLI architecture; ligand embedding was provided by Attentive‐FP22 (a  (AUC = 0.598 and PRC = 0.280). For DUD‐E, the target‐clustered split technique is applied to avoid protein redundancy between the train and test sets. Our model is tested using 3‐fold cross validation, and proteins are grouped using global sequence alignment to ensure that targets with sequence identities exceeding 70% are included in the same fold during cross validation. The AUC results indicate that our model performs well on DUD‐E dataset, with a mean AUC of 0.972 for target clustering with 70% sequence identity.We split PDBbind into training/validation/test sets similarly to what was done for Pafnucy  and compared the results using two metrics: Pearson's correlation coefficient R and root mean square error (RMSE). As shown in Extended Data Figure 23 (AU2.3.41)Our model achieved SOTA results on most protein‐related benchmarks, particularly protein‒protein interactions (nearly 5% superior to the second‐best method). The PPI task is designed to discover the relationship between two proteins. A multimodal model can inject more domain‐specific information into the protein representation, thus enhancing the ability to discover such relationships. Specifically, the GO terms can provide information about protein functions, such as biological processes and cellular locations. When two proteins are found in the same cellular component and are involved in the same biological process, the likelihood of their interaction increases, and vice versa.2)The efficacy of each module was revealed. The effectiveness of the initial sequence and GO embeddings as well as various combinations of modalities was investigated in two ablation studies. The presented results indicate that the initial sequence embedding is especially useful for protein‐structure‐related tasks, such as secondary structures and remote homology, whereas biophysically related tasks, such as stability and fluorescence, benefit more from domain knowledge introduced through multimodal pretraining. For multiple modalities, the results suggest that combinations of sequence data with another modality (structure or GO) produce relatively superior results.3)Properly incorporating multiple modalities is important. Based on the findings, we conclude that the model can have the best effect when all multimodal data, including sequence, structure, and GO information, are provided as input. For instance, when only sequence information was used as input for the four protein property benchmarks while the structure and GO inputs to our model were masked, it was obvious that the model's performance was severely hampered by the lack of other modalities, as further evidenced by the PPI benchmark results. When all available multimodal data were used, our model outperformed other SOTA approaches on almost all PPI benchmarks. In the ablation study, model variants using fewer modalities exhibited worse performance under the same circumstances, further validating our conclusion. On the other hand, the PLI task may depend on the ligand representation and the protein‒ligand interaction module in addition to the protein representation. It is possible that a more complex interaction module  would improve the model performance on the PLI task.The results mentioned above can be summarized as follows.2.4 and conducted protein sequence homology ablation experiments with 25% sequence identity threshold.In order to make fair comparisons with other landmark methods, the data partitioning of all the applied downstream tasks were obtained from their original sources . However, the protein homologous issues within these benchmarks merit discussion. The presence of homologous proteins in both the training and test sets can easily lead to overfitting. Consequently, we examined the protein homology of each of these benchmarks using MMseqs224 andBriefly, we removed proteins from the train set whose sequence identity with the other sets exceeded 25%, and then we trained and evaluated the performance of the model. On these homology removal datasets, we then run our model MASSA and other methods (using their released codes and model weights on GitHub). As shown in Supporting Information, our model outperforms other methods on these homology removal datasets. Notably, all deep learning models have demonstrated significantly better performance than models with few parameters . These results indicate that our model was not overtrained and has superior predictive ability compared to other state‐of‐the‐art methods in the presence of few homologous training samples.2.5 However, there are still many challenges that need to be further addressed in this field. First, is it possible to quantify such transferability, thereby determining how well a model has been pretrained and the extent to which downstream tasks benefit from pretraining? Another interesting issue is the adaptability across downstream tasks. Specifically, what is the relationship between the distributions of the intertask feature spaces and their mutual transferability?The above results verify the effectiveness of transfer learning from a pretrained multimodal representation to downstream tasks, consistent with previous studies.6, 9Here, to measure the dynamic transferability of a multimodal representation to downstream tasks, we introduce a novel optimal‐transport‐based metric for quantifying the cross‐task learning process .2.5.1We expect that in a well‐formed feature space, there should be a high level of intratask distributional similarity and a low level of intertask distributional similarity. To quantify the dynamic transferability, we introduce a novel metric called the optimal‐transport‐based feature space representation measure (OTFRM). This metric is intuitively the ratio of the intratask similarity  to the average intertask similarity between the given task and other tasks . This value makes intuitive sense because a larger value indicates a closer spatial distribution for the given task and a greater spatial separation between the given task and other tasks.Figure As depicted in 2.5.2Based on the aforementioned findings, we hypothesized that the heterogeneity of the feature space distributions of tasks might affect how well the corresponding feature representations can be transferred between tasks. Accordingly, we investigated the adaptability across downstream tasks. Using the intertask distribution difference operator from the OTFRM, we first calculated the pairwise distances between the spatial distributions of different downstream tasks based on the feature space of the multimodal representation. Among these downstream tasks, we then conducted a number of adaptation experiments. For instance, we trained the refined model for task A on task B until the models converged . To determine whether the adaptive performance was improving or degrading, it was compared to the initial performance for task A. The relationship between the intertask distribution distances and the mutual adaptability of the tasks was then calculated.ρ = −0.488). In other words, the adaptation performance is more likely to decrease when tasks A and B are farther apart. For instance, there was a decline in performance of more than 6 points in the adaptation from PDBbind to stability. In addition to the protein representation, the PLI and PPI tasks may also rely on the interaction module, which may introduce additional noise. Therefore, an attempt was made to conduct experiments solely on protein property benchmarks. We performed the adaptation tasks five times with distinct random seeds for each pair of protein property benchmarks and computed the confidence intervals. As demonstrated, the Spearman correlation rose to −0.763. Collectively, these findings corroborate the utility of the OTFRM for elucidating dynamic transferability and suggest that it can be used to predict adaptability across downstream tasks as well.The pairwise distances between the tasks gradually increased as the phases progressed, consistent with the above results , in which a model for the target task B is directly fine‐tuned based on a model for the source task A, Adapted‐FT consists of three steps  Therefore, a multimodal protein model will have broader applicability. We showed that incorporating multiple protein modalities can result in better performance on a variety of downstream tasks. However, combining different modalities of protein‐related data is difficult due to the complex characteristics of these data, such as high‐dimensional sparse features. We extracted knowledge from domain/motif/region information using a named entity recognition strategy. For other types of protein data, however, such as post‐translational modification, additional techniques will be needed.Appropriate data type selection and preprocessing. The majority of protein representation methods are based on sequence data due to the massive available corpora. However, as a result of DeepMind's outstanding efforts, a wealth of information on protein structures is also available.15 The2) The heterogeneity of information across modalities may impede efficient fusion. Each protein modality is associated with a private subspace that captures domain‐specific features and a shared subspace that captures shared features. Performing multimodal fusion to early may result in the loss of modality‐specific features. Therefore, we extracted the features of each protein modality separately, including evolutionary and spatial information for sequence and structure data, respectively. Then, based on the characteristics of the different modalities, they were aligned at either the token or global level. This technique can preserve more modality‐specific features than conventional methods, such as concatenating the modal embeddings in a late stage without alignment. Under the assumption that there are both domain‐specific and shared features across modalities, more intriguing methods could be attempted. For instance, a better representation might be obtained by training a domain classifier to minimize domain‐shared features and maximize domain‐specific features.Elaborate model architecture. The alignment and fusion of different modalities are critical in multimodal machine learning.25 The3)Suitable training objectives. The relevant training objectives, which guide model optimization and feature extraction, should be given proper priority in representation learning. However, when defining pretraining objectives, existing approaches have relied significantly on experience obtained from natural language processing . We have introduced more protein‐specific training objectives to extract high‐level and fine‐grained protein‐domain features. Although their effectiveness has been confirmed, there is still considerable room for improvement. For instance, it is still unclear which features obtained based on which objectives contribute most to a particular biological task. The rational design of a series of training objectives to encapsulate the desired properties for a given task remains a challenge.4)Future research direction: rationally designing protein representation learning by quantifying transferability. It should be highlighted that different biological properties of a protein will place varying demands on the representations for various downstream tasks, and it is difficult to embed all desired properties in one representation. Therefore, it appears that using a single protein representation for all tasks, while appealing, is not feasible in reality. The relationship between biological properties and hidden features, as well as which components should be included in the pretraining phase in order to capture and transfer the most desirable biological properties for the given task, should be quantitatively analyzed. Here, we have presented a novel metric for quantifying the dynamic transferability of a pretrained protein representation to downstream tasks. We have shown that this metric can be utilized to evaluate the cross‐task learning process, predict adaptability and guide fine‐tuning across various tasks. Future work will focus on improving the transferability metric to guide the design of neural networks and training objectives for protein representation learning.In recent years, we have witnessed the emergence of an increased number of AI‐based computational approaches for learning protein representations, which are essential in downstream biological applications. Most existing methods are usually trained using only a single data format, such as protein sequences. However, protein knowledge can be accumulated from numerous types of biological experiments. In this study, we have proposed a multimodal protein representation framework for incorporating domain knowledge across protein sequence, structure, and function information. With careful design of the pretraining process, we were able to create a protein representation learning tool with broad applicability (github.com/SIAT‐code/MASSA). We have demonstrated that our model performs exceptionally well on all protein‐related tasks, exceeding state‐of‐the‐art levels, particularly when all multimodal data are used as inputs. Specifically, we argue that for effective protein representation learning, three critical factors should be considered, as follows:4Multimodal Pretraining Data: A multimodal protein dataset was constructed for pretraining the protein representation. Three types of data were used: sequence, structure, and functional annotation data.https://www.uniprot.org/) were used to compile the multimodal protein dataset. For 3D structure embedding, protein structures from the AlphaFold Protein Structure Database  and the RCSB PDB (https://www.rcsb.org/) were used. The structures predicted by AlphaFold2 may not be accurate. This had been examined in the Supporting Information. To obtain fine‐grained domain knowledge, protein data at the amino acid level, such as regions, motifs and domains, were extracted from UniProtKB as training objectives. Finally, the constructed multimodal protein dataset consists of nearly one million samples, each of which contains sequence and structure data and many of which contain GO annotations, regions, motifs, and domains . For the initial GO annotation encoder, GO terms and their relationships from the Gene Ontology Resource were used to build a GO graph containing 44733 nodes and 150322 edges.To achieve an efficient preliminary representation and to save training time, two initial encoders were utilized for the sequences and GO annotations before multimodal pretraining. Specifically, for the initial sequence encoder, ESM‐MAS‐1b was trained on a dataset of 26 million MSAs, in which the average depth was 1192. An MSA was generated for each UniRef50 sequence by searching UniClust30 with HHblits (detailed in14). FTo evaluate the pretrained multimodal model, It was further tested on three protein‐related tasks.Protein Property Benchmarks: Four protein property benchmarks covering stability, fluorescence, remote homology, and secondary structure were used. Among them, three datasets  were taken from TAPE, which was a set of semisupervised biological benchmarks for evaluating the performance of protein language models.Protein‒Protein Interaction Datasets: Four protein‒protein interaction datasets with three types of tasks were used in this study. was used for a regression task, which was constructed to investigate the influence of mutation‐based binding affinity changes on PPI. SKEMPI contains 3047 binding affinity changes based on amino acid substitutions within protein complexes. The unit of binding affinity is the equilibrium dissociation constant (Kd). The lower the Kd value is, the better the binding affinity. Chen et al. further averaged the Kd values of duplicate entries to obtain 2950 PPI samples, which had served as a comparative benchmark for many methods.The Structural Kinetic and Energetic database of Mutant Protein Interactions (SKEMPI)28 wasThe STRING database integrates the majority of publicly accessible protein‒protein interaction data to construct a global PPI network that includes both direct  and indirect  interactions. Here, the classification of PPI was focused by STRING for multiple types. This database categorizes PPI into the following seven types: reaction, binding, posttranslational modification (ptmod), activation, inhibition, catalysis, and expression. At least one of them was present in every pair of interacting proteins. In a previous study 1690 and 5189 proteins with 40% sequence identity were selected from the Homo sapiens subset of STRING to generate SHS27k and SHS148k, which contain 7624 and 44488 multilabel PPI, respectively. Simultaneously, all PPI of Homo sapiens were used as the third dataset, STRING, containing 15335 proteins and 593397 PPI.18Protein‒Ligand Interaction Datasets: Three protein‒ligand interaction datasets were used here, PDBbind,the Kinase set, and Directory of Useful Decoys Enhanced (DUD‐E). The PDBbind v2016 core set, with 290 diverse complexes covering all protein classes in the PDBbind refined set, was used as a test set for comparisons with other methods. The PDBbind dataset split was the same as in our previous study. First, the core set containing 290 protein‒ligand complexes was utilized as the test set. Second, 1000 complexes in the refined set were separated to be used as a validation set (same as Pafnucy). Third, the other complexes from the refined and general sets were combined for use as the training set. Thus, a total of 13196 complexes were used for model evaluation. The Kinase dataset was taken from TransformerCPI, which was derived from the KIBA set, containing 229 proteins, 1644 ligands, and 111237 interactions. The DUDE dataset includes 102 targets belonging to eight broad protein categories, such as kinases, proteases, nuclear receptors, etc., as well as 22886 active compounds and their affinities against 102 targets, with an average of 224 ligands per target. GVP‐GNN, and GraphGO . It consists of two inputs, scalar feature and geometric feature, which represent the attributes and 3D coordinates of amino acids, respectively. Scalar features provide rotational invariance of molecules, whereas geometric features provide an absolute direction for each amino acid node, as opposed to the relative direction between neighbors, making it easier for GNN to access the global geometric characteristics of the structure. Using GVP instead of multi‐layer perceptron (MLP) could preserve rotation invariance and global geometric properties during the graph propagation process. More details can be found in Supporting Information.GVP‐GNN module34 was which was designed to address the classification task for long‐tailed distributions. Typically, the multiclass cross‐entropy (CE) loss (where the label is 0 or 1) is defined as:p and q correspond to real and predicted distributions, respectively. Then, the equalized focal loss (EFL) was applied for multiclass classification with long‐tailed distributions. To address the issue of data imbalance, a focusing factor was added to the CE expression; thus, the EFL is defined as:pt ∈  represents the confidence score prediction for a given target and αt is a weighting factor for balancing positive and negative samples. Then, the focusing factor γj is further decomposed into two factors: γj=γb+γvj=γb+s(1−gj), where γb is a focusing factor that controls the basic behavior of the classifier in the balanced data scenario and γvj≥0 is a variable factor associated with the level of imbalance in the j‐th category. In addition, another set of weights is added to rebalance the loss contributions of different categories, where high weighting factor values are assigned to increase the loss contributions of rare categories while maintaining the weighting factors for common categories near 1. The weighting factor of the j‐th category is represented as γb+γvjγb. Finally, the EFL is defined as:The loss function minimized by the model during pretraining was a sum of a categorical cross‐entropy loss and an equalized focal loss,35 whiHere, a novel metric, namely, the optimal‐transport‐based feature space representation measure (OTFRM) was proposed, to measure the dynamic transferability from a multimodal representation to downstream tasks.Background on Optimal Transport: Optimal transport (OT) theory was first derived from the Monge problem, and later, the Kantorovich relaxed OT problem was proposed to find a transport plan describing how to move some measure to another measure of the same mass such that a certain cost function is minimized. OT theory was a powerful and principled method for comparing probability distributions due to its inherent properties. For a complete and separable metric space X, along with continuous or discrete probability measures α∈P(X) and β∈P(X), the Kantorovich OT problem is defined as:c:X×X→R+ is a cost function and Π is a set of couplings consisting of joint distributions over the product space X×X with marginal distributions α and β:dX; if the ground‐truth cost satisfies c=dXp for some p ≥ 1, then it have the p‐Wasserstein distance:α and β is almost infeasible in practice, discrete empirical measures α=∑i=1maiδxi and β=∑i=1nbiδyj are usually used instead, where a and b are vectors in the probability simplex. In this case, the pairwise costs in Equation .ned as:OTα,βΔ=mWith the addition of an entropy‐regularized term, the OT problem defined in Equation  becomes 39The OTFRM: Intuitively, a well‐formed feature space should have high intraclass similarity and low interclass similarity. Here, it was assumed that the cross‐task feature space possesses comparable qualities. Consequently, the OTFRM was proposed for evaluating the quality of feature representations by quantifying the level of intratask distributional similarity versus intertask distributional similarity based on the multimodal representation feature space.simCosine = 1 − distCosine, allowing to define similarity based on the corresponding distance. For the feature probability distributions α and β of two tasks Ck and Cl, using the cosine distance as the cost metric dX in the equation for the p‐Wasserstein distance, the OT similarity of the two tasks can be defined as:m and n are the numbers of samples in Ck and Cl, respectively; π is a coupling matrix of size m × n; and H(π)=−∑i=1m∑j=1nπijlogπij is an entropy‐regularized term with ε = 0.1.The cosine similarity was employed as the fundamental distance metric based on two considerations: 1) A high‐dimensional feature space necessitates a similarity that performs well in high dimensions. 2) The cosine similarity satisfies K tasks, where Ck is the inspected task, the intertask similarity could be defined as the sum of the OT similarities of all unique pairs between samples in Ck and samples outside of Ck, normalized by the number of unique pairs:For a feature space containing yi is the i‐th real value, ŷi is the i‐th predicted value, y^¯ is the average of the real values, and y¯ is the average of the predicted values.For regression tasks, Pearson's correlation coefficient R and the root mean square error (RMSE), which reflect the linear correlation and the difference between the predicted and real values, respectively, were used as evaluation metrics:For classification tasks, several metrics, including accuracy, precision, recall and the Matthews correlation coefficient (Mcc), were used to evaluate the model performance on an imbalanced dataset:Statistical Analysis: Results were presented as the mean ± SD. The Independent Sample t‐test was used to determine the statistical significance. All evaluation metrics were computed using the sklearn and scipy Python libraries.The authors declare no conflict of interest.F.H., Y.H., and W.Z. contributed equally to this work. F.H. and P.Y. led the research. F.H. and P.Y. contributed technical ideas. F.H., Y.H, and W.Z. developed the proposed method. F.H., Y.H., W.Z., and H.H. performed analysis and experiments. Y.P. provided evaluation and suggestions. All authors contributed to the manuscript.Supporting InformationClick here for additional data file.


The identification of human proteins that are amenable to pharmacologic modulation without significant off-target effects remains an important unsolved challenge. Computational methods have been devised to identify features which distinguish between “druggable” and “undruggable” proteins, finding that protein sequence, tissue and cellular localization, biological role, and position in the protein–protein interaction network are all important discriminant factors. However, many prior efforts to automate the assessment of protein druggability suffer from low performance or poor interpretability. We developed a neural network-based machine learning model capable of generating druggability sub-scores based on each of four distinct categories, combining them to form an overall druggability score. The model achieves an excellent performance in separating drugged and undrugged proteins in the human proteome, with an area under the receiver operating characteristic (AUC) of 0.95. Our use of multiple sub-scores allows the assessment of potential protein targets of interest based on distinct contributors to druggability, leading to a more interpretable and holistic model to identify novel targets.The online version contains supplementary material available at 10.1186/s13321-023-00735-7. The cost of developing new therapeutic drugs has risen significantly in recent years, with the average R&D (Research & Development) cost per new drug ranging between $314 million and $2.8 billion . Most ofDruggability is a poorly defined term; it can be used narrowly to refer only to a protein’s ability to bind an activity-modifying small molecule ligand, or more broadly to refer to a protein’s relevance as a therapeutic target in human disease. For this paper’s purposes, druggability encompasses the ability of a protein’s activity to be modulated for pharmacologic effect by a drug which gains regulatory approval. Undruggable proteins are those that cannot be influenced for therapeutic benefit, either because they lack disease relevance, are biologically essential, or cannot be targeted through any known drug modality. Throughout our work we will use these definitions.Recent advances in machine learning offer the potential for in silico feature identification of druggable proteins. This can facilitate computational evaluation of prospective targets prior to the initiation of expensive clinical trials. A variety of efforts in this area have taken different approaches, incorporating different predictors of druggability into their feature sets. Several groups have sought to solely use properties derived from the primary protein sequence, achieving impressive results in distinguishing drugged proteins from a select subset of difficult-to-drug proteins , 28, 35.Given the wide variety of features which may determine whether a protein can be categorized as druggable, it is likely that the most successful approach will incorporate a comprehensive range of properties, including physical and chemical attributes, expression profile, biological functions, and protein–protein interactions. Successful machine learning efforts in this area have utilized features from several of these domains , 15, 43.To our knowledge, all previously published machine learning models are trained to discriminate druggable from undruggable proteins with a single druggability score or binary classification. This approach lacks interpretability and wholeness, particularly when many distinct types of features are specifically and uniquely contributing to druggability. For example, a protein’s position in the protein–protein interaction network may have major implications for potential off-target effects during clinical trials but does not demonstrate its structural amenability to small molecule modulation. A classifier that separates distinct features into sub-scores prior to obtaining a total druggability score could output multiple types of pertinent information about whether a protein is druggable or undruggable. We created the Predictive Interpretable Neural Network for Druggability (PINNED), a deep learning model which divides its inputs into four distinct groups—sequence and structure, localization, biological functions, and network information—and generates interpretable sub-scores that contribute to a final druggability score.Many factors influence a protein’s druggability, including its effectiveness as a disease-modifying target and its propensity for causing undesired side-effects. A protein’s physical and chemical properties, such as amino acid composition, secondary structure, post-translational modification, and others, can determine whether it can be readily liganded by a drug-like molecule. Its position in the complex network of protein–protein interactions which occur within the human body can influence its role in disease and its potential for off-target effects. The biological function of a protein plays a significant role in whether it is a useful drug target; however, many proteins are involved in multiple different processes, disturbance of any can lead to unanticipated consequences for homeostasis and thus leading to off-target effects. Additionally, a protein’s expression profile across target and non-target tissues can have implications for its efficacy and safety.To incorporate all these contributions to druggability, we generated a feature set that contains a variety of data for 20,404 human proteins, including properties extracted from the protein sequence, tissue specificity, subcellular localization, biological functions, and position in the protein–protein interaction network . The feaSequence and structure properties included information about 52 physiochemical features, such as protein molecular weight and amino acid residues, charge and isoelectric points, extinction coefficients, predicted post-translational modifications, secondary structure, and solvent accessibility. Previous works indicate that the grouped dipeptide composition (GDPC) and pseudo amino acid composition (PAAC) of a protein may be useful characteristics in determining its druggability , 28, 35.AlphaFold is a deep learning network developed by DeepMind that can predict a protein’s structure from its three-dimensional amino acid sequence. The AlphaFold Protein Structure Database was established between AlphaFold and EMBL-EBI . This daThe Subcellular Localization Predictive System (CELLO) was used to predict subcellular localization for each protein in the dataset . We inclGene essentiality, assessed by lethality of mouse homozygous loss-of-function mutations  and enzyThe signaling pathways (“Maps”) score generated by Dezső et al. was included in the network information features. Degree, closeness, betweenness, eigen centrality, and PageRank of each protein in the protein–protein interaction network were calculated using information from the STRING database . These fThe National Center for Biotechnology Information (NCBI) Pharos database, a data repository of human protein properties and drugged status, identifies proteins as confirmed drug targets if they are “protein drug targets via which approved drugs act” (“Tclin”) . As of OAll other proteins are classified as one of three other categories: undrugged proteins which bind small molecules with high potency (“Tchem”), proteins with well-studied biology (“Tbio”), and proteins not meeting the criteria for any of the other categories (“Tdark”) . Of thesThe model architecture consisted of four separate deep neural networks, designated “sequence and structure,” “localization,” “biological functions,” and “network information.” Each network contained an input layer, a hidden layer with ReLU activation, and a single output neuron representing the network sub-score. The four sub-scores were summed, producing a logit which was passed through a sigmoid function to generate the final probability of druggability Fig. B.Prior to model tuning, 20% of the dataset was held out to form a separate test set, which was used to evaluate the model after the optimal architecture had been determined. The remaining data was divided into five equal groups, one of which was held out as a validation set, while the remaining four were combined to form the training set  Fig. A. It wasAfter hyperparameter optimization, a model was trained on the full training/validation set, with the held-out test data used as the final validation set. The complete model achieved an excellent AUC of 0.950 on the test set Fig. A, with tTo determine if the scores could predict success in clinical trials, we tested them against a dataset of successful and failed phase III clinical targets . We founReducing the druggability score required to consider a protein “druggable” can increase the sensitivity of the predictor. By default, this value was set as 0.5 during training, but may be changed to any arbitrary value during inference. At a threshold of 0.5, PINNED achieves excellent specificity but low sensitivity, with many drugged proteins in the test set being mistakenly classed as undruggable Fig. . At a reComparing PINNED’s performance to prior machine learning efforts to assess protein druggability is challenging due to the wide variety of datasets used and metrics reported. Many previous works exclude proteins with significant homology to drugged proteins from their undrugged sets , 14, eveOf the 696 drugged proteins in our dataset, 294 were affiliated with three protein families: ion channels (124 proteins), G-protein-coupled receptors, or GPCRs (102 proteins), and kinases (68 proteins). The PINNED architecture model of four subnetworks determines which individual members of protein families represent the most promising targets, not just a druggability assessment of entire families. To test the feature learning ability indicating druggability across the whole proteome and apply it to identify targets within unseen protein families, we excluded ion channels, GPCRs, or kinases, respectively, from the training data, and tested the models’ performance on these held-out families. In each case, a training and validation set was constructed consisting of the entire proteome except for the members of the held-out family and used to train five models applying cross-validation. Each model was then implemented to score all proteins within the held-out family, and the scores averaged to generate an ensemble score. The ensemble score AUC in distinguishing drugged and undrugged members of the held-out family was assessed for the overall druggability score and all four constituent sub-scores. This process was repeated for each of the three main drugged families. The PINNED framework maintained the ability to distinguish between drugged and undrugged proteins within each family with reasonable discriminatory power Table . AlthougAfter training the model and assessing it on the test set, we ablated each feature by randomly shuffling (permuting) the values among the protein test set and assessed the increase in test loss induced by the change. As loss is inversely related to the network’s performance, more prominent features will result in a higher increase in loss after being permuted. We found that features belonging to the biological function’s subnetwork comprised seven of the top 10 Table , consistTo generate druggability scores for the entire proteome, we split our entire dataset, including the training/validation and test sets into five parts. Each part was held out and the remaining four were used to train a classifier model. The scores for the held-out set were designated as the final druggability scores for the protein set. This process was repeated with each of the sets being held out once to generate scores for the proteins in the entire proteome. Of the 10 highest-scoring undrugged proteins in the proteome, all except TNFRSF11A are listed by Pharos as Tchem, having validated high-potency small molecule ligands Table . The mecOf the 20,412 proteins in the Pharos database, 5679 (28%) are designated as “Tdark”—having extremely limited data about their properties and functions. Considerable interest exists in exploring these understudied parts of the genome, particularly to discover novel therapeutic targets which have previously been overlooked . At leasTMPRSS11B’s sub-scores for sequence and structure, localization, biological functions, and network information, compared to the Tclin (drugged) proteins, were respectively in the 84th, 97th, 29th, and 1st percentiles  with the ability to estimate the druggability of over 20,000 proteins spanning the entire human proteome. PINNED can be used as a pre-screening tool to determine a protein’s amenability to drugging prior to the initiation of pre-clinical programs and identify weaknesses in the form of low sub-scores of top targets that do not necessarily score high in all four areas, providing room for insight and early remediation. This methodology enables the exploration of novel targets cost-effectively while improving the clinical phase success rate.Drugged and undrugged proteins and sequences were obtained from the Pharos database on October 12, 2022. Proteins categorized as Tclin were labeled as drugged, while proteins categorized as Tchem, Tbio, or Tdark were labeled as undrugged. Protein features were obtained from Dezső et al.’s features  and the All features generated by Dezső et al. were incorporated into our feature set and divided between the four subnetworks. These include characteristics calculated or predicted from the amino acid sequence, such as posttranslational modifications, enzyme classification, localization, secondary structure, and sequence motifs. Details on the generation of these features can be found in Dezső et al. . All numInformation about protein molecular weight and amino acid residues, charge and isoelectric points, extinction coefficients, predicted post-translational modifications, secondary structure, and solvent accessibility from Dezső et al.’s feature set were included as sequence and structure properties.Grouped dipeptide composition (GDPC) and pseudo amino acid composition (PAAC) were calculated using the iFeature toolkit . All selHuman protein structure predictions were acquired from AlphaFold (last modification on 05/05/2022). The structures were curated to run through Fpocket. Fpocket is an open-source protein prediction algorithm based on the Voronoi tessellation and the alpha sphere theory . FpocketProtein localization and tissue specificity data obtained from Dezső et al. was included in the localization data.GO terms were downloaded from the Target Central Resource Database (TCRD) on July 29, 2022, and separated into GO terms categorized as Components, Functions, or Processes. They were used to generate a one-hot encoded GO terms matrix that mapped each protein in the dataset. Terms mapped to less than 10 proteins were excluded. GO Components were included in the localization data, while Functions and Processes were included in the biological functions data (see below).Scores generated for each protein by Dezső et al. from the MetaCore database for “Biological Function,” and “Molecular Process” were standard-scaled and included in the “biological functions” sub-score. The enzyme classification and gene essentiality feature from Dezső et al. were included in the biological functions data.GO Functions and Processes were obtained and processed as described above and included in biological functions.The “Maps”  scores from Dezső et al. and calculated protein–protein interaction network features were used as the input to the network information subnetwork.Features for all four sub-scores were combined into a single feature matrix. 20% of the proteins were selected at random prior to model development and held out as a test set. Prior to training, the drugged proteins in the training set were randomly oversampled with replacement until the quantity was equal to the quantity of undrugged proteins. Oversampling by SMOTE, ADASYN, or applying different weights to positive and negative samples were evaluated, but performance was not improved.n inputs, a hidden layer with size 2i, where i is the largest integer such that 2i ≤ n, and an output layer of size 1, representing that network’s sub-score. ReLU activation was applied to the hidden layers, and an L2 penalty of 0.001 was applied to both the hidden and output layers. The four subnetwork output layers were summed to generate the logits of the overall druggability score. Different numbers of hidden layers, dropout for the input and hidden layers, learning rates, and L2 coefficients were tested, and the above values were found to lead to optimal AUC scores on validation sets.Our model was implemented in Python 3.7.13 using TensorFlow 2.11.0 and consisted of four densely connected neural networks, corresponding to the four sub-scores. Each consisted of a single input layer of size Support vector machine, logistic regression, XGBoost, and random forest models were also evaluated and found to deliver performance comparable or inferior to neural network.− 3.5, with a batch size of 32 and the binary cross entropy loss function.The model was trained using the Adam optimizer with TensorFlow default parameters at a learning rate of 10Additional file 1. Comparison to previous druggability classifiers.Additional file 2. Target classification of highest scoring undrugged proteins.Additional file 3. All protein scores.Additional file 4. Feature importance scores.Additional file 5. Results from evaluating PINNED scores on phase III clinical targets.