Q:

Model seems to be overfitting with Optimizer.minimize() but not tf.contrib.layers.optimize_loss()

When I create train_op like this:
train_op = tf.contrib.layers.optimize_loss(
    loss=loss,
    global_step=tf.contrib.framework.get_global_step(),
    learning_rate=params['learning_rate'],
    optimizer='Adam'
)

I get a working network that performs well on validation and test sets.
If I just use minimize() method like this:
optimizer = tf.train.AdamOptimizer(learning_rate=params['learning_rate'])
train_op = optimizer.minimize(
    loss=loss,
    global_step=tf.train.get_global_step()
)

I get much worse results (precision, recall, loss) even on the first validation after 1000 steps, and after a while it seems like it completely overfitted (loss on validation is more or less constant and is 100x train loss, but precision and recall crash)
I created a function that is cleaned-up version of contrib one, that differs from straight Optimizer.minimize() in two marked places:
def make_train_op(loss, optimizer, global_step):
    with tf.variable_scope(None, "OptimizeLoss", [loss, global_step]):

        # ==========================================
        # this part is extra comparing to minimize()
        update_ops = set(tf.get_collection(tf.GraphKeys.UPDATE_OPS))
        if update_ops:
            with tf.control_dependencies([update_ops]):
                loss = tf.identity(loss)
        # ==========================================

        gradients = optimizer.calculate_gradients(
            loss,
            tf.trainable_variables()
        )

        grad_updates = optimizer.apply_gradients(
            gradients,
            global_step=global_step,
            name="train")

        # ==========================================
        # so is this one
        with tf.control_dependencies([grad_updates]):
            train_op = tf.identity(loss)
        # ==========================================
        return train_op

And validation performs well again. Training in all cases look more or less same (and healthy). Network is relatively straightforward CNN/batchnorm/dropout/maxpool mix with cross-entropy loss.
The way I understand this is that there are some operations that are part of a graph that don't appear as dependencies for loss, but that are needed to calculate gradients. How is that even possible? If this is a normal situation, why aren't those two snippets part of a core? Should I have done something different while building a model to avoid the need for this dependency forcing?

A:

The issue is with batchnorm update operations, and it's actually documented:

Note: when training, the moving_mean and moving_variance need to be updated. By default the update ops are placed in tf.GraphKeys.UPDATE_OPS, so they need to be added as a dependency to the train_op. For example:

update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
with tf.control_dependencies(update_ops):
    train_op = optimizer.minimize(loss)

