One of the fundamental tasks in understanding genomics is the problem of predicting Transcription Factor Binding Sites (TFBSs). With more than hundreds of Transcription Factors (TFs) as labels, genomic-sequence based TFBS prediction is a challenging multi-label classification task. There are two major biological mechanisms for TF binding: (1) sequence-specific binding patterns on genomes known as “motifs” and (2) interactions among TFs known as co-binding effects. In this paper, we propose a novel deep architecture, the Prototype Matching Network (PMN) to mimic the TF binding mechanisms. Our PMN model automatically extracts prototypes (“motif”-like features) for each TF through a novel prototype-matching loss. Borrowing ideas from few-shot matching models, we use the notion of support set of prototypes and an LSTM to learn how TFs interact and bind to genomic sequences. On a reference TFBS dataset with 2.1 million genomic sequences, PMN significantly outperforms baselines and validates our design choices empirically. To our knowledge, this is the first deep learning architecture that introduces prototype learning and considers TF-TF interactions for large scale TFBS prediction. Not only is the proposed architecture accurate, but it also models the underlying biology. Genomic sequences build the basis of a large body of research on understanding the biological processes in living organisms. Enabling machines to read and comprehend genomes is a longstanding and unfulfilled goal of computational biology. One of the fundamental task to understand genomes is the problem of predicting Transcription Factor Binding Sites (TFBSs), attracting much attention over the years BID5 . Transcription Factors (TFs) are proteins which bind (i.e., attach) to DNA and control whether a gene is expressed or not. Patterns of how different genes expressed or not expressed control many important biological phenomena, including diseases such as cancer. Therefore accurate models for identifying and describing the binding sites of TFs are essential in understanding cells.Owing to the development of chromatin immunoprecipitation and massively parallel DNA sequencing (ChIP-seq) technologies BID26 ), maps of genome-wide binding sites are currently available for multiple TFs in a few cell types across human and mouse genomes via the ENCODE BID5 database. However, ChIP-seq experiments are slow and expensive; they have not been performed for many important cell types or organisms. Therefore, computational methods to identify TFBS accurately remain essential for understanding the functioning and evolution of genomes.An important feature of TFs is that they typically bind to sequence-specific patterns on genomes, known as "motifs" BID25 . Motifs are essentially a blueprint, or a "prototype" which a TF searches for in order to bind. However, motifs are only one part in determining whether or not a TF will bind to specific locations. If a TF binds in the absence of its motif, or it does not bind in the presence of its motif, then it is likely there are some external causes such as an interaction with another TF, known as co-binding effects in biology BID46 . This indicates that when designing a genomic-sequence based TFBS predictor, we should consider two modeling challenges: (1) how to automatically extract "motifs"-like features and (2) how to model the co-binding patterns and consider such patterns in predicting TFBSs. In this paper, we address both proposing a novel deep-learning model: prototype matching network (PMN).To address the first challenge of motif learning and matching, many bioinformatics studies tried to predict TFBSs by constructing motifs using position weight matrices (PWMs) which best represented the positive binding sites. To test a sequence for binding, the sequence is compared against the PWMs to see if there is a close match BID37 . PWM-matching was later outperformed by convolutional neural network (CNN) and CNN-variant models that can learn PWM-like filters BID0 . Different from basic CNNs, our proposed PMN is inspired by the idea of "prototype-matching" BID44 BID14 . These studies refer to the CNN type of model as the "feature-matching" mode of pattern recognition. While pure feature matching has proven effective, studies have shown a "prototype effect" where objects are likely recognized as a whole using a similarity measure from a blurred prototype representation, and prototypes do not necessarily match the object precisely BID44 . It is plausible that humans use a combination of feature matching and prototype matching where feature-matching is used to construct a prototype for testing unseen samples BID14 . For TFBS prediction, the underlying biology evidently favors computation models that can learn "prototypes" (i.e. effective motifs). Although motifs are indirectly learned in convolutional layers, existing deep learning studies of TFBS (details in Section 3) have not considered the angle of "motif-matching" using a similarity measure. We, instead, propose a novel prototype-matching loss to learn prototype embedding automatically for each TF involved in the data.None of the previous deep-learning studies for TFBS predictions have considered tackling the second challenge of including the co-binding effects among TFs in data modeling. From a machine learning angle, the genomic sequence based TFBS prediction is a multi-label sequence classification task. Rather than learning a prediction model for each TF (i.e., each label) predicting if the TF will bind or not on input, a joint model is ideal for outputting how a genomic sequence input is attached by a set of TFs (i.e., labels). The so-called "co-binding effects" connect deeply to how to model the dependency and combinations of TFs (labels). Multi-label classification is receiving increasing attention in deep learning BID9 BID47 (detailed review in Section 3). Modeling the multi-label formulation for TFBS is an extremely challenging task because the number of labels (TFs) is in hundreds to thousands (e.g. 1,391 TFs in BID41 ). The classic solution for multi-label classification using the powerset idea (i.e., the set of all subsets of the label set) is clearly not feasible BID40 . Possible prior information about TF-TF interactions is unknown or limited in the biology literature.To tackle these obstacles, our proposed model PMN borrows ideas from the memory network and attention literature. BID43 proposed a "matching network" model where they train a differentiable nearest neighbor model to find the closest matching image from a support set on a new unseen image. They use a CNN to extract features and then match those features against the support set images. We replace this support set of images with a learned support set of prototypes from the large-scale training set of TFBS prediction, and we use this support set to match against a new test sample. The key difference is that our PMN model is not for few-shot learning and we seek to learn the support set (prototypes). BID43 uses an attentionLSTM to model how a test sample matches to different items in the support set through softmax based attention. Differently, we use what we call a combinationLSTM to model how the embedding of a test sample matches to a combination of relevant prototypes. Using multiple "hops", the combinationLSTM updates the embedding of the input sequence by searching for which TFs (prototypes) are more relevant in the label combination. Instead of explicitly modeling interactions among labels, we try to use the combinationLSTM to mimic the underlying biology. The combinationLSTM tries to learn prototype embedding and represent high-order label combinations through a weighted sum of prototype embedding. This weighted summation can model many "co-binding effects " reported in the biology literature BID46 ) (details in Section 2).In summary, we propose a novel PMN model by combining few-shot matching and prototype feature learning. To our knowledge, this is the first deep learning architecture to model TF-TF interactions in an end-to-end model. In addition, this is also the first paper to introduce large scale prototype learning using a deep learning architecture. On a reference TFBS dataset with 2.1 million genomic sequences, PMN significantly outperforms the state-of-the-art TFBS prediction baselines. We validate the learned prototypes through an existing database about TF-TF interactions. The TF groups obtained by clustering prototype embedding evidently captures the "cooperative effects" that has not been modeled by previous TFBS prediction works.The main contributions of our model are: DISPLAYFORM0 On the left is an overview of the model. The input sequence x is encoded asx using f (3-layer CNN).x is then matched against the learned prototypes using the combinationLSTM for K "hops" so that it can update its output based on TF interactions for this input sequence. The final outputŷ is based on a concatenation of the final updated sequence vector h K from the LSTM, and final read vector r K from the matching. On the right is a closer look at the internal aspects of the combinationLSTM.• We propose a novel model by combining few-shot matching with large-scale prototype feature learning.• We design a novel prototype-matching loss to learn "motif"-like features in deep learning , which is important for the TFBS prediction task.• We extend matching models from the few-shot single-label task to a large-scale multi-label task for genomic sequence classification.• We implement an attention LSTM module to model label interactions in a novel way.• Our model favors design choices mimicking the underlying biological processes. We think such modeling strategies are more fundamental especially on datasets from biology. Sequence analysis plays an important role in the field of bioinformatics. A prominent task is to understand how Transcription Factor proteins (TFs) bind to DNA. Researchers in biology hypothesize that each TF searches for certain sequence patterns on genome to bind to, known as "motifs". Accordingly we propose a novel prototype matching network (PMN) for learning motif-like prototype features. On a support set of learned prototypes, we use a combinationLSTM for modeling label dependencies. The combinationLSTM tries to learn and mimic the underlying biological effects among labels (e.g. co-binding). Our results on a dataset of 2.1 million genomic strings show that the prototype matching model outperforms baseline variations not having prototype-matching or not using the combinationLSTM. This empirically validates our design choices to favor those mimicking the underlying biological mechanisms.Our PMN model is a general classification approach and not tied to the TFBS applications. We show this generality by applying it on the MNIST dataset and obtain convincing results in Appendix Section 7.1. MNIST differs from TFBS prediction in its smaller training size as well as in its multi-class properties. We plan a few future directions to extend the PMN. First, TFBSs vary across different cell types, cell stages and genomes. Extending PMN for considering the knowledge transfer is especially important for unannotated cellular contexts (e.g., cell types of rare diseases or rare organisms). Another direction is to add more domain-specific features. While we show that using prototype matching and the combinationLSTM can help modelling TF combinations, there are additional raw feature extraction methods that we could add in order to obtain better representations of genomics sequences. These include reverse complement sequence inputs or convolutional parameter sharing BID32 , or an RNN to model lower level spatial interactions BID28 among motifs.
We combine the matching network framework for few shot learning into a large scale multi-label model for genomic sequence classification.
{"src": "scitldr-AIC_train", "paper_id": "S191YzbRZ"}