So I wrote a vanilla 1 hidden layer neural net in Java and with the goal of training it to recognize handwritten digits from the MNIST database. However, on the first run, it didn't work, performing no better than random guessing. I went back into the code to debug to find where I wen't wrong, and one of the things I tried was only training one layer at a time. This is where the weirdness happened. When I train only the connections into the output layer, the neural net converges to about 80% really quickly, then slowly degrades back to about random. When I train only the connections from the input layer to the hidden layer, it converges to about 80% really slowly, then degrades back to random rather rapidly.I decided to look at the actual output values produced in these cases (the ten values corresponding to the confidence that the neural net has towards each of the ten possible digits), and found that when I trained only the first layer, all values tended towards one, with the expected one converging faster. On the other hand, when I trained only the final layer, all values tended towards zero, with the expected one converging slower. This goes against all my sensibilities of how the back-propagation algorithm should work mathematically, and I can't seem to work out what's wrong despite hours of effort.     run();     int sum = 0;     for(int ii = 0; ii < outputLayer.size(); ii++)     {        Node n = outputLayer.get(ii);        for(Connection c : n.connections)        {           c.origin.adjustSum += c.destination.value * (1-c.destination.value) * (c.destination.value - expected[ii])*c.weight;           c.weight -= learningRate * c.origin.value * c.destination.value * (1-c.destination.value) * (c.destination.value - expected[ii]);        }     }     for(Node n : hiddenLayer)     {        for(Connection c : n.connections)        {           c.weight -= learningRate * c.origin.value * c.destination.value * (1-c.destination.value) * n.adjustSum;        }     }Upon request here is node:public class Node implements Serializable{   public ArrayList<Connection> connections;   public double value;    public double adjustSum;   /**   * The constructor for the node object   * Initializes the instance variables   */      public Node()   {        connections = new ArrayList<Connection>();    }   /**   * A method that finds the value of a node   * It sums up the values of the product of each of its inputs and the connections weight   * then it passes this weight through the logistic function and sets that as teh value of the current node   */      public void findValue()   {      double totalInput = 0;      for(Connection c : connections)      {         totalInput += (c.weight*c.origin.value);      }      value = logisticCurve(totalInput);   }                                                /**   * This method calculates the output of the function 1/(1+e^-x) for a given input   * @param- the input(x) value   * @return- The output(y) value   */      public double logisticCurve(double xVal)   {      double yVal = 1/(1+Math.exp(-1*xVal));      return yVal;                                             }}The neural net uses a sigmoid activation function and size 786-256-10. Connection weights are initialized in the following way:weight = (Math.random()*2/Math.sqrt(numInputs))-(1/Math.sqrt(numInputs));Nothing else in my unit testing has seemed odd or off, but if you have any tests you want me to try, or need to see or describe any more code, just ask. If you have any advice at all, it would be greatly appreciated, I am absolutely stuck.
Training each layer in a neural net works individually, but fails as a whole
machine learning;neural network;classification;backpropagation