DESCQA: An Automated Validation Framework for Synthetic Sky Catalogs
  The use of high-quality simulated sky catalogs is essential for the success
of cosmological surveys. The catalogs have diverse applications, such as
investigating signatures of fundamental physics in cosmological observables,
understanding the effect of systematic uncertainties on measured signals and
testing mitigation strategies for reducing these uncertainties, aiding analysis
pipeline development and testing, and survey strategy optimization. The list of
applications is growing with improvements in the quality of the catalogs and
the details that they can provide. Given the importance of simulated catalogs,
it is critical to provide rigorous validation protocols that enable both
catalog providers and users to assess the quality of the catalogs in a
straightforward and comprehensive way. For this purpose, we have developed the
DESCQA framework for the Large Synoptic Survey Telescope Dark Energy Science
Collaboration as well as for the broader community. The goal of DESCQA is to
enable the inspection, validation, and comparison of an inhomogeneous set of
synthetic catalogs via the provision of a common interface within an automated
framework. In this paper, we present the design concept and first
implementation of DESCQA. In order to establish and demonstrate its full
functionality we use a set of interim catalogs and validation tests. We
highlight several important aspects, both technical and scientific, that
require thoughtful consideration when designing a validation framework,
including validation metrics and how these metrics impose requirements on the
synthetic sky catalogs.
