A Study on the Evaluation of Generative Models
Implicit generative models, which do not return likelihood values, such as
generative adversarial networks and diffusion models, have become prevalent in
recent years. While it is true that these models have shown remarkable results,
evaluating their performance is challenging. This issue is of vital importance
to push research forward and identify meaningful gains from random noise.
Currently, heuristic metrics such as the Inception score (IS) and Frechet
Inception Distance (FID) are the most common evaluation metrics, but what they
measure is not entirely clear. Additionally, there are questions regarding how
meaningful their score actually is. In this work, we study the evaluation
metrics of generative models by generating a high-quality synthetic dataset on
which we can estimate classical metrics for comparison. Our study shows that
while FID and IS do correlate to several f-divergences, their ranking of close
models can vary considerably making them problematic when used for fain-grained
comparison. We further used this experimental setting to study which evaluation
metric best correlates with our probabilistic metrics. Lastly, we look into the
base features used for metrics such as FID.