I am implementing a matching model for sentence-pairs following this paper. For those who may haven't read this paper yet, I give a brief introduction here: given a sentence pair x and y, a 2d matrix z can be constructed, where z_{ij}=[x_{i},y_{j}]. I have implemented it. For simplicity, there is just one CNN layer and one pooling layer. The output layer is a softmax layer. I have also tried removing pooling layer. The optimizer is Adam with 1e-4 learning rate. The word embedding is initialized with normal distribution with 1e-4 stddev. But just after one step, the output from the conv layer is nearly all zeros. Seems that a negative gradient always occur in the first training step and cause the deading relu problem. Do you have any experience on training 2D CNN on text? Could you tell me how do you successfully train it? TIA!
How to train the 2D-CNN for matching sentence-pairs properly?
machine learning;deep learning;nlp;tensorflow;convnet