To quantitatively compare STROTSS with prior work we
performed several studies using Amazon Mechanical Turk (AMT) across the three regimes. An example of the workers' interface is shown in
Figure REF . Images A and B are the product of two different style transfer algorithms given the same inputs. We consider the algorithms proposed
in [1]},[2]}, [3]}, [4]},
and STROTSS. In Figure REF  image C is the corresponding
style image, and workers were asked to choose whether the style of
image is best matched by: 'A', 'B', 'Both Equally', or 'Neither'. If
image C is a content image, workers are posed the same question with
'content' replacing 'style'. For each competing
algorithm except [2]} we test three sets of
hyper-parameters, the defaults recommended by the authors, the same
with \(\frac{1}{4}\)  of the content weight (high stylization), and the
same with double the content weight (low stylization). Because these modifications to content weight did not alter the behavior of  [1]} significantly we also tested  [1]} with \(\frac{1}{100}\)  and \(100\times \)  the default content weight. We also test our method with \(4\times \)  the content weight. We could only test the default hyper-parameters of  [2]} because
the author's code did not expose content weight as a
parameter to users. We test all possible pairings of A and B between
different algorithms and their hyperparameters (i.e. we do not compare
an algorithm against itself with different hyperparameters, but do
compare it to all hyperparameter settings of other algorithms). In
each presentation, the order of output (assignment of methods to A or B in
the interface) was randomized. Each pairing was voted on by an average
of 4.98 different workers (minimum 4, maximum 5), 662 workers in
total. On average, 3.7 workers agreed with the majority vote for each
pairing.
