I have found the following example online:from keras.datasets import mnistfrom keras.models import Sequential from keras.layers.core import Dense, Activationfrom keras.utils import np_utils(X_train, Y_train), (X_test, Y_test) = mnist.load_data()X_train = X_train.reshape(60000, 784)     X_test = X_test.reshape(10000, 784)Y_train = np_utils.to_categorical(Y_train, classes)     Y_test = np_utils.to_categorical(Y_test, classes)batch_size = 100      epochs = 15model = Sequential()     model.add(Dense(100, input_dim=784)) model.add(Activation('sigmoid'))     model.add(Dense(10)) model.add(Activation('softmax'))model.compile(loss='categorical_crossentropy', metrics=['accuracy'], optimizer='sgd')model.fit(X_train, Y_train, batch_size=batch_size, epochs=epochs, verbose=1)score = model.evaluate(X_test, Y_test, verbose=1)print('Test accuracy:', score[1])This gives about 95% accuracy, but if I change the sigmoid with the ReLU, I get less than 50% accuracy. Why is that? 
ReLU vs sigmoid in mnist example
python;neural network;keras;image classification