Evaluation in style transfer is a difficult problem, and an open question without widely accepted standards. Given that rigorously measuring style and content is a central challenge of the field, employing automatic metrics is problematic. When this is done it typically involves measuring an algorithm's output's score under the objective function (which measures style loss and/or content loss between the output image and style or content image respectively). Since this is the objective the output was chosen to minimize, this doesn't necessarily measure stylization quality so much as the behaviour of the optimization procedure. An alternate, potentially more meaningful, automatic metric proposed by [1]} is to train a classification model on different artists' work, and evaluate style transfer outputs based on whether they are classified in the same way as the target style.
