Generative models often use human evaluations to determine and justify progress. Unfortunately, existing human evaluation methods are ad-hoc: there is currently no standardized, validated evaluation that: (1) measures perceptual fidelity, (2) is reliable, (3) separates models into clear rank order, and (4) ensures high-quality measurement without intractable cost. In response, we construct Human-eYe Perceptual Evaluation (HYPE), a human metric that is (1) grounded in psychophysics research in perception, (2) reliable across different sets of randomly sampled outputs from a model, (3) results in separable model performances, and (4) efficient in cost and time. We introduce two methods. The first, HYPE-Time, measures visual perception under adaptive time constraints to determine the minimum length of time (e.g., 250ms) that model output such as a generated face needs to be visible for people to distinguish it as real or fake. The second, HYPE-Infinity, measures human error rate on fake and real images with no time constraints, maintaining stability and drastically reducing time and cost. We test HYPE across four state-of-the-art generative adversarial networks (GANs) on unconditional image generation using two datasets, the popular CelebA and the newer higher-resolution FFHQ, and two sampling techniques of model outputs. By simulating HYPE's evaluation multiple times, we demonstrate consistent ranking of different models, identifying StyleGAN with truncation trick sampling (27.6% HYPE-Infinity deception rate, with roughly one quarter of images being misclassified by humans) as superior to StyleGAN without truncation (19.0%) on FFHQ. Historically, likelihood-based estimation techniques served as the de-facto evaluation metric for generative models BID18 BID5 . But recently, with the application of generative models to complex tasks such as image and text generation BID14 BID34 , likelihood or density estimation grew no longer tractable BID46 . Moreover, for high-dimensional problems, even likelihood-based evaluation has been called into question BID46 . Consequently, most generative tasks today resort to analyzing model outputs BID41 BID43 BID11 BID21 BID7 BID37 . These output evaluation metrics consist of either automatic algorithms that do not reach the ideals of likelihood-based estimation, or ad-hoc human-derived methods that are unreliable and inconsistent BID41 BID11 .Consider the well-examined and popular computer vision task of realistic face generation BID14 . Automatic algorithms used for this task include Inception Score (IS) BID43 and Fréchet Inception Distance (FID) BID17 . Both have been discredited for evaluation on non-ImageNet datasets such as faces BID2 BID40 BID6 BID38 . They are also much more sensitive to visual corruptions such as salt and pepper noise than to semantic distortions such as swirled images BID17 . So, while automatic metrics are consistent and standardized, they cannot fully capture the semantic side of perceptual fidelity BID6 .Realizing the constraints of the available automatic metrics, many generative modeling challenges resort to summative assessments that are completely human BID41 BID43 BID11 . These human measures are (1) ad-hoc, each executed in idiosyncrasy without proof of reliability or grounding to theory, and (2) high variance in their estimates BID43 BID11 BID33 . These characteristics combine to a lack of reliability, and downstream, (3) a lack of clear separability between models. Theoretically, given sufficiently large sample sizes of human evaluators and model outputs, the law of large numbers would smooth out the variance and reach eventual convergence; but this would occur at (4) a high cost and a long delay.In this paper, we present HYPE (HUMAN EYE PERCEPTUAL EVALUATION) that addresses these criteria in turn. It: (1) measures the perceptual fidelity of generative model outputs via a grounded method inspired by psychophysics methods in perceptual psychology, (2) is a reliable and consistent estimator, (3) is statistically separable to enable a comparative ranking, and (4) ensures a cost and time efficient method through modern crowdsourcing techniques such as training and aggregation. We present two methods of evaluation. The first, called HYPE time , is drawn directly from psychophysics literature BID22 ) and displays images using adaptive time constraints to determine the time-limited perceptual threshold a person needs to distinguish real from fake BID9 . The HYPE time score is understood as the minimum time, in milliseconds, that a person needs to see the model's output before they can distinguish it as real or fake. Small HYPE time scores indicate that model outputs can be identified even at a glance; large scores suggest that people need to dedicate substantial time and attention. The second method, called HYPE ∞ , is derived from the first to make it simpler, faster, and cheaper while maintaining reliability. It measures human deception from fake images with no time constraints. The HYPE ∞ score is interpretable as the rate at which people mistake fake images and real images, given unlimited time to make their decisions.We demonstrate HYPE's performance on unconditional generation of human faces using generative adversarial networks (GANs) BID14 . We evaluate four state-of-the-art GANs: WGAN-GP BID16 , BEGAN BID4 , ProGAN BID20 , and the most recent StyleGAN BID21 . First, we track progress across the years on the popular CelebA dataset BID28 . We derive a ranking based on perception (HYPE time , in milliseconds) and error rate (HYPE ∞ , as a percentage) as follows: StyleGAN (439.4ms, 50.7%), ProGAN (363.7ms, 40.3%), BEGAN (111.1ms, 10.0%), WGAN-GP (100.0ms, 3.8%). A score of 500ms on HYPE time indicates that outputs from the model become indistinguishable from real, when shown for 500ms or less, but any more would start to reveal notable differences. A score of 50% on HYPE ∞ represents indistinguishable results from real, conditioned on the real training set, while a score above 50% through 100% represents hyper-realism in which generated images appear more real than real ones when drawn from a mixed pool of both. Next, we test StyleGAN trained on the newer FFHQ dataset BID21 , comparing between outputs generated when sampled with and without the truncation trick, a technique used to prune low-fidelity generated images BID7 BID21 . We find that outputs generated with the truncation trick (363.2ms, 27.6%) significantly outperforms those without it (240.7ms, 19.0%), which runs counter to scores reported by FID.HYPE indicates that GANs have clear, measurable perceptual differences between them. HYPE produces identical rankings between HYPE time and HYPE ∞ . We also find that even the best eval- Images on the right exhibit the highest HYPE scores, the highest human perceptual fidelity. uated model, StyleGAN trained on FFHQ and sampled with the truncation trick, only performs at 27.6% HYPE ∞ , suggesting substantial opportunity for improvement. Finally, we show that we can reliably reproduce these results with 95 % confidence intervals using 30 human evaluators at $60 in a task that takes 10 minutes. While important measures, we do not focus on diversity, overfitting, entanglement, training stability, and computational and sample efficiency of the model BID6 BID29 and instead aim to construct the gold standard for human perceptual fidelity.We deploy HYPE as a rapid solution for researchers to measure their generative models, requiring just a single click to produce reliable scores and measure progress. We deploy HYPE at https://hype.stanford.edu, where researchers can upload a model and retrieve a HYPE score in 10 minutes for $60. Future work would extend HYPE to adapt to other generative tasks such as text generation or abstractive summarization.
HYPE is a reliable human evaluation metric for scoring generative models, starting with human face generation across 4 GANs.
{"src": "scitldr-AIC_train", "paper_id": "SJgZSULYdN"}