I have a very small dataset (40 training examples, 10 validation examples, 120 classes) for which I'm getting very high accuracies with a very simple model in Keras (batchnorm, flatten, and dense layers only).My training accuracy is 94-95% and validation accuracy is 76-78%. I know it's overfitting and I have tried a few things. The data is not images, so I cannot augment the data. I also cannot add data because it's a specific type. I'm using two dropout layers with 0.5 levels, and the architecture is very simple so I don't think I can reduce the architecture complexity. I can paste the model if anyone likes.My question is: Is there ever a situation where validation accuracy cannot be as high as the training accuracy? Is there a limitation based on the size of the dataset? Or is it ALWAYS possible for validation accuracies to match training accuracies and the network just needs the right parameters?
Is it always possible for validation accuracy to be as high as training accuracy?
deep learning;keras