The most common form of evaluation [1]}, [2]}, [3]}, [4]}, [5]}, [6]}, [7]}, [8]}, [9]}, [10]}, [11]}, [12]}, [13]}, [14]}, [15]} is simply showing qualitative comparisons between the proposed algorithm and prior work on several input pairs (typically 5-30 within a paper). Some work has proposed more rigorous evaluation protocols based on user studies that report human's perceptual judgement. One popular way to frame these studies is as measuring output's overall aesthetic quality: In [16]} study participants are asked to give an absolute score for output's aesthetic quality, in [17]} participants are shown the outputs of all benchmarked algorithms simultaneously and asked to rank them based on aesthetic quality, and in [18]}, [19]} participants were similarly shown all outputs but instead asked to choose a single best result. While most studies use participants drawn from the general population of mechanical turk workers, some studies have leveraged the help of experts (holders of advanced degrees in art history) in evaluating style transfer. Sanakoyeu et al. [20]} asked experts to choose which algorithms' output best match the style of the input image. Kotovenko et al. [21]} conducted a similar study, but with styles from well known artists. Experts were shown zoomed in crops of output images, and asked which crop best matched the style of the desired artist (but where not shown the style input), they also proposed a more challenging variant where a crop from a real style input was an additional option that could be selected as the 'best'.
