Inferring the structural properties of a protein from its amino acid sequence is a challenging yet important problem in biology. Structures are not known for the vast majority of protein sequences, but structure is critical for understanding function. Existing approaches for detecting structural similarity between proteins from sequence are unable to recognize and exploit structural patterns when sequences have diverged too far, limiting our ability to transfer knowledge between structurally related proteins. We newly approach this problem through the lens of representation learning. We introduce a framework that maps any protein sequence to a sequence of vector embeddings --- one per amino acid position --- that encode structural information. We train bidirectional long short-term memory (LSTM) models on protein sequences with a two-part feedback mechanism that incorporates information from (i) global structural similarity between proteins and (ii) pairwise residue contact maps for individual proteins. To enable learning from structural similarity information, we define a novel similarity measure between arbitrary-length sequences of vector embeddings based on a soft symmetric alignment (SSA) between them. Our method is able to learn useful position-specific embeddings despite lacking direct observations of position-level correspondence between sequences. We show empirically that our multi-task framework outperforms other sequence-based methods and even a top-performing structure-based alignment method when predicting structural similarity, our goal. Finally, we demonstrate that our learned embeddings can be transferred to other protein sequence problems, improving the state-of-the-art in transmembrane domain prediction. Proteins are linear chains of amino acid residues that fold into specific 3D conformations as a result of the physical properties of the amino acid sequence. These structures, in turn, determine the wide array of protein functions, from binding specificity to catalytic activity to localization within the cell. Information about structure is vital for studying the mechanisms of these molecular machines in health and disease, and for development of new therapeutics. However, experimental structure determination is costly and atomic structures have only been determined for a tiny fraction of known proteins. Methods for finding proteins with related structure directly from sequence are of considerable interest, but the problem is challenging, because sequence similarity and structural similarity are only loosely related BID0 BID1 BID2 BID3 , e.g. similar structural folds can be formed by diverse sequences. As a result, our ability to transfer knowledge between proteins with similar structures is limited.In this work, we address this problem by learning protein sequence embeddings using weak supervision from global structural similarity for the first time. Specifically, we aim to learn a bidirectional LSTM (biLSTM) embedding model, mapping sequences of amino acids to sequences of vector representations, such that residues occurring in similar structural contexts will be close in embedding space. This is difficult, because we have not observed position-level correspondences between sequences, only global sequence similarity. We solve this by defining a whole sequence similarity measure from sequences of vector embeddings. The measure decomposes into an alignment of the sequences and pairwise comparison of the aligned positions in embedding space. For the alignment, we propose a soft symmetric alignment (SSA) mechanism -a symmetrization of the directional alignment commonly used in attention mechanisms. Furthermore, in order to take advantage of information about local structural context within proteins, we extend this framework to include position-level supervision from contacts between residues in the individual protein structures. This multitask framework FIG0 ) allows us to newly leverage both global structural similarity between proteins and residue-residue contacts within proteins for training embedding models. The similarity prediction module takes pairs of proteins represented by their sequences of vector embeddings and predicts their shared SCOP level. Sequences are first aligned based on L1 distance between their vector embeddings using SSA. From the alignment, a similarity score is calculated and related to shared SCOP levels by ordinal regression. (3) The contact prediction module uses the sequence of vector embeddings to predict contacts between amino acid positions within each protein. The contact loss is calculated by comparing these predictions with contacts observed in the 3D structure of the protein. Error signal from both tasks is used to fit the parameters of the encoder.We first benchmark our model's ability to correctly predict structural similarity between pairs of sequences using the SCOPe ASTRAL dataset BID4 . This dataset contains protein domains manually classified into a hierarchy of structural categories (Appendix Figure 3) . We show that our model dramatically outperforms other sequence-based protein comparison methods when predicting comembership in the SCOP hierarchy. Remarkably, our model even outperforms TMalign BID5 , which requires structures as input and therefore structures must be known a priori. In contrast, our model uses only sequence as input. Next, we perform an ablation study to evaluate the importance of our modeling components for structural similarity prediction. We also consider an additional task, secondary structure prediction, to assess the model's ability to capture local structure features. We demonstrate that SSA outperforms alternative alignment methods for both of these tasks and that inclusion of the contact prediction training task further improves performance.Finally, we demonstrate that the embeddings learned by our model are generally applicable to other protein machine learning problems by leveraging our embeddings to improve the state-of-the-art in transmembrane prediction. This work presents the first attempt in learning protein sequence embeddings from structure and takes a step towards bridging the sequence-structure divide with representation learning. In this work, we proposed a novel alignment approach to learning contextual sequence embeddings with weak supervision from a global similarity measure. Our SSA model is fully differentiable, fast to compute, and can be augmented with position-level structural information. It outperforms competition in predicting protein structural similarity including, remarkably, structure alignment with TMalign. One consideration of training using SCOP, however, is that we focus exclusively on single-domain protein sequences. This means that the highly contextual embeddings given by the biLSTM encoder to single domains may differ from embeddings for the same domain in a multi-domain sequence. One interesting extension would thus be to modify the encoder architecture or training procedure to better model domains in multi-domain contexts. Nonetheless, the resulting embeddings are widely useful, allowing us to improve over the state-of-the-art in transmembrane region prediction, and can easily be applied to other protein prediction tasks such as predicting functional properties, active site locations, protein-protein interactions, etc. Most methods that use HMM sequence profiles or position-specific scoring matrices could be augmented with our embeddings. The broader framework extends to other related (non-biological) tasks.A APPENDIX Figure The bidirectional LSTM language model was trained on the full set of protein domain sequences in the Pfam database, 21,827,419 total sequences. The language model was trained to predict the amino acid at position i given observations of all amino acids before i and all amino acids after i by minimizing the cross entropy loss with log predicted log probabilities given by the sum of the forward and reverse LM direction predictions DISPLAYFORM0 where p F (x i ) is the probability given by the forward direction LSTM and p R (x i ) is the probability given by the reverse direction LSTM.The language model architecture consisted of a 2-layer LSTM with 1024 units in each layer followed by a linear transformation into the 20-d amino acid prediction. All parameters were shared between the forward and reverse direction components. The model was trained for a single epoch using ADAM with a learning rate of 0.001 and minibatch size of 32.
We present a method for learning protein sequence embedding models using structural information in the form of global structural similarity between proteins and within protein residue-residue contacts.
{"src": "scitldr-AIC_train", "paper_id": "SygLehCqtm"}