I have setup a four layer CNN designed to predict two classes. The two classes are more or less in the same ratio. The negative class is 55% of the data and the positive class makes up the remaining. When I train the network on the data and look at the final results, I see two problems: The validation accuracy fluctuates. In fact, it can be thought of as oscillating. The validation loss increases and the accuracy however keeps hovering.The final results indicate a possible reason for this behaviour. The negative classes are very well classified. However, the postive classes are very poorly classified. My question is why is the network exhbiting this behaviour. I've enough training data, ~10k images in training, and the classes are more or less balanced (5400 in the negative and 4500 in the positive). The issue raised in point 2 increases with an increase in the number of epochs. I've tried decreasing the learning rate, different optimizers, increased the number of layers, decreased the number of layers, used transfer learning (VGG-16). I've also tried reducing the number of epochs, but this leads to poor classification in the negative class as well. 
Neural net learning only one class?
deep learning;convnet;keras