Click Through Rate (CTR) prediction is a critical task in industrial applications, especially for online social and commerce applications. It is challenging to find a proper way to automatically discover the effective cross features in CTR tasks. We propose a novel model for CTR tasks, called Deep neural networks with Encoder enhanced Factorization Machine (DeepEnFM). Instead of learning the cross features directly, DeepEnFM adopts the Transformer encoder as a backbone to align the feature embeddings with the clues of other fields. The embeddings generated from encoder are beneficial for the further feature interactions. Particularly, DeepEnFM utilizes a bilinear approach to generate different similarity functions with respect to different field pairs. Furthermore, the max-pooling method makes DeepEnFM feasible to capture both the supplementary and suppressing information among different attention heads. Our model is validated on the Criteo and Avazu datasets, and achieves state-of-art performance. This paper studies the problem of predicting the Click Through Rate (CTR), which is an essential task in industrial applications, such as online advertising, and e-commerce. To be exact, the advertisements of cost-per-click (CPC) advertising system are normally ranked by the eCPM (effective cost per mille), which is computed as the prodcut of bid price and CTR (click-through rate). To predict CTR precisely, feature representation is an important step in extracting the good, interpretable patterns from training data. For example, the co-occurrence of "Valentine's Day", "chocolate" and "male" can be viewed as one meaningful indicator/feature for the recommendation. Such handcrafted feature type is predominant in CTR prediction (Lee et al., 2012) , until the renaissance of Deep Neural Networks (DNNs). Recently, a more effective manner, i.e., representation learning has been investigated in CTR prediction with some works (Guo et al., 2017; Qu et al., 2016; Wang et al., 2017; Lian et al., 2018; Song et al., 2018) , which implicitly or explicitly learn the embeddings of high-order feature extractions among neurons or input elements by the expressive power of DNNs or FM. Despite their noticeable performance improvement, DNNs and explicit high order feature-based methods (Wang et al., 2017; Guo et al., 2017; Lian et al., 2018) seek better feature interactions merely based on the naive feature embeddings. Few efforts have been made in addressing the task of holistically understanding and learning representations of inputs. This leads to many practical problems, such as "polysemy" in the learned feature embeddings existed in previous works. For example, the input feature 'chocolate' is much closer to the 'snack' than 'gift' in normal cases, while we believe 'chocolate' should be better paired with 'gift' if given the occurrence input as "Valentine's Day". This is one common polysemy problem in CTR prediction. Towards fully understanding the inputs, we re-introduce to CTR, the idea of Transformer encoder (Vaswani et al., 2017) , which is oriented in Natural Language Processing (NLP). Such an encoder can efficiently accumulate and extract patterns from contextual word embeddings in NLP, and thus potentially would be very useful in holistically representation learning in CTR. Critically, the Transformer encoder has seldom been applied to CTR prediction with the only one exception arxiv paper AutoInt (Song et al., 2018) , which, however, simply implements the multi-head selfattention (MHSA) mechanism of encoders, to directly extract high-order feature interactions. We argue that the output of MHSA/encoder should be still considered as first-order embedding influenced by the other fields, rather than a high-order interaction feature. To this end, our main idea is to apply the encoder to learn a context-aware feature embedding, which contains the clues from the content of other features. Thus the "polysemy" problem can be solved naturally, and the second-order interaction of such features can represent more meaning. Contrast to AutoInt (Song et al., 2018) , which feeds the output of encoder directly to the prediction layer or a DNN, our work not only improves the encoder to be more suitable for CTR task, but also feeds the encoder output to FM, since both our encoder and FM are based on vector-wise learning mechanism. And we adopt DNN to learn the bit-wise high-order feature interactions in a parallel way, which avoids interweaving the vector-wise and bit-wise interactions in a stacked way. Formally, we propose a novel framework -Deep neural networks with Encoder enhanced Factorization Machine (DeepEnFM). DeepEnFM focuses on generating better contextual aligned vectors for FM and uses DNN as a bit-wise information supplement. The architecture adopting both Deep and FM part is inspired by DeepFM (Guo et al., 2017) . The encoder is endowed with bilinear attention and max-pooling power. First, we observed that unlike the random order of words in a sentence, the features in a transaction are in a fixed order of fields. For example, the fields of features are arranged in an order of {Gender, Age, Price ...}. When the features are embedded in dense vectors, the first and second vectors in a transaction always represent the field "Gender" and "Age". To make use of this advantage, we add a bilinear mechanism to the Transformer encoder. We use bilinear functions to replace the simple dot product in attention. In this way, feature similarity of different field pairs is modeled with different functions. The embedding size in CTR tasks is usually around 10, which allows the application of bilinear functions without unbearable computing complexity. Second, the original multi-head outputs are merged by concatenation, which considers the outputs are complementary to each other. We argue that there are also suppressing information between different heads. We apply a max-pooling merge mechanism to extract both complementary and suppressing information from the multi-head outputs. Experimental results on Criteo and Avazu datasets have demonstrated the efficacy of our proposed model. In this paper, we propose a novel framework named Deep neural networks with Encoder enhanced Factorization Machine (DeepEnFM), which aims to learn a better aligned vector embedding through the encoder. The encoder combines the bilinear attention and max-pooling method to gather both the complementary and suppressing information from the content of other fields. The extensive experiments demonstrate that our approach achieves state-of-art performance on Criteo and Avazu dataset.
DNN and Encoder enhanced FM with bilinear attention and max-pooling for CTR
{"src": "scitldr-AIC_train", "paper_id": "SJlyta4YPS"}