batch = tf.Variable(0)
learning_rate = tf.train.exponential_decay(0.01, batch * BATCH_SIZE,
    train_size, 0.95, staircase=True)
optimizer = tf.train.MomentumOptimizer(learning_rate, 0.9).minimize(loss,
    global_step=batch)
How to set adaptive learning rate for GradientDescentOptimizer?