<commit_before>\documentclass[12pt, letterpaper, preprint]{aastex}

\newcommand{\setof}[1]{\left\{{#1}\right\}}

\begin{document}

\title{Inference of variances:\\
       Implications for cosmological large-scale structure experiments}
\author{
  David~W.~Hogg\altaffilmark{1,2}
}
\affil{Simons Center for Data Analysis,
       New York}
\altaffiltext{1}{also: Center for Cosmology and Particle Physics,
                       Department of Physics,
                       New York University}
\altaffiltext{2}{also: Center for Data Science,
                       New York University}
\email{<EMAIL>}

\begin{abstract}
The study of large-scale structure is (at present) primarily the study
of variances, since the power spectrum or correlation function are
two-point statistics.
Present-day large-scale structure experiments often perform
cosmological inferences using a pseudo-likelihood that compares (by something like chi-squared) a
point estimate of the two-point function to an equivalent estimate
from a simulation or theory.
Here we show---with toy problems---that this procedure in general
produces incorrect posterior inferences about the variance (and
therefore cosmological parameters).
However, we also show that if the same pseudo-likelihood (or even something simpler) is used
\emph{not} as a likelihood in a standard probabilistic inference but
rather as a \emph{distance metric} in an approximate Bayesian
computation (ABC) method, correct posterior inferences can be drawn.
We discuss the relationships between the toy problems shown here and
the standard cosmological inferences, and try to suggest places where
a change from the standard incorrect method to the ABC method would
have the biggest impacts.
We argue that this is likely to be important at the baryon acoustic
scale.
\end{abstract}

Present-day large-scale structure experiments are fundamentally
projects to measure the variance of cosmological fields, such as the
galaxy, quasar, intergalactic medium, or mass fields.
Indeed, if the initial conditions of the Universe are very close to
Gaussian (and they are) and if the evolution of the Universe is very
close to linear (as it is on large scales), then the mean and variance
of the density field (on some scale) are sufficient statistics to
infer everything about the initial conditions (at that scale).
Thus, the measurement or inference of variance as a function of
scale---or, equivalently, inference of the power spectrum or the
two-point auto-correlation function---is critical to inference of the
cosmological initial conditions and the cosmological parameters.
There is an enormous literature on making these cosmological
measurements!
Here we won't address this literature in detail.
We are going to step back and look at issues of principle in measuring
variances, and see if there are small changes to be made to
present-day cosmological practice that might improve our cosmological
inferences.

DWH: Introduce the idea of measuring a variance of a
process. Introduce the scalar problem and consider the Gaussian
pseudo-likelihood. Why is this patently absurd?

One critical property of the \emph{variance} of a Gaussian (or any other
distribution) is that it is a \emph{non-negative} quantity.
How does this relate to the properties of a two-point correlation
function (for, say, galaxies, the temperature map of the cosmic
microwave background, or the initial conditions of the density field)?
The answer is slightly non-trivial:
It is that the correlation function (what ought to be called the
``covariance function'') be a non-negative semi-definite function, or
that the correlation function obey what's called Mercer's condition,
or that any covariance matrix made up by any evaluation of the
correlation function on any grid of points be non-negative
semi-definite.
These (identical) conditions are best explained with equations.

First, define the correlation function $\xi(|r|)$ as the covariance of
the overdensity field $\delta(x)$, where $r$ is a vector displacement
in three-space (and $|r|$ is a scalar separation) and $x$ is
three-dimensional position:
\begin{eqnarray}
\xi(|r|) &=& E[\delta(x)\,\delta(x+r)]
\\
\delta(x) &\equiv& \frac{\rho(x)}{\bar{\rho}} - 1
\quad ,
\end{eqnarray}
where $E[q]$ is the expectation value of $q$ (implicitly taken as an integral
over all of space),
$\rho$ is the density field,
and $\bar{\rho}$ is the mean density.
By assuming that $\xi(\cdot)$ depends only on $|r|$ we are effectively
assuming statistical isotropy.
Now choose a spatial set of points $\setof{x_i}$ in three-dimensional space
and construct the matrix $C$ such that the elements $C_{ij}$ of $C$
are given by
\begin{eqnarray}
C_{ij} &\equiv& \xi(|x_i - x_j|)
\quad .
\end{eqnarray}
This matrix is manifestly symmetric, but if $\xi(|r|)$ is the true
covariance function of the density field, then this matrix must also
be non-negative semi-definite.
That is, it must have eigenvalues that are real and non-negative.
That this be true for any choice of points $\setof{x_i}$ is equivalent
(for our purposes) to Mercer's condition.

\acknowledgements
It is a pleasure to thank
  Brendon~J.~Brewer (Auckland),
  Chang~Hoon~Hahn (NYU),
  Dan~Foreman-Mackey (UW),
  M.~J.~Vakili (NYU),
  and the Blanton--Hogg group meeting at NYU
for valuable discussions.

\end{document}
<commit_msg>a bit more<commit_after>\documentclass[12pt, letterpaper, preprint]{aastex}

\newcommand{\setof}[1]{\left\{{#1}\right\}}

\begin{document}

\title{Inference of variances:\\
       Implications for cosmological large-scale structure experiments}
\author{
  David~W.~Hogg\altaffilmark{1,2}
}
\affil{Simons Center for Data Analysis,
       New York}
\altaffiltext{1}{also: Center for Cosmology and Particle Physics,
                       Department of Physics,
                       New York University}
\altaffiltext{2}{also: Center for Data Science,
                       New York University}
\email{<EMAIL>}

\begin{abstract}
The study of large-scale structure is (at present) primarily the study
of variances, since the power spectrum or correlation function are
two-point statistics.
Present-day large-scale structure experiments often perform
cosmological inferences using a pseudo-likelihood that compares (by something like chi-squared) a
point estimate of the two-point function to an equivalent estimate
from a simulation or theory.
Here we show---with toy problems---that this procedure in general
produces incorrect posterior inferences about the variance (and
therefore cosmological parameters).
However, we also show that if the same pseudo-likelihood (or even something simpler) is used
\emph{not} as a likelihood in a standard probabilistic inference but
rather as a \emph{distance metric} in an approximate Bayesian
computation (ABC) method, correct posterior inferences can be drawn.
We discuss the relationships between the toy problems shown here and
the standard cosmological inferences, and try to suggest places where
a change from the standard incorrect method to the ABC method would
have the biggest impacts.
We argue that this is likely to be important at the baryon acoustic
scale.
\end{abstract}

Present-day large-scale structure experiments are fundamentally
projects to measure the variance of cosmological fields, such as the
galaxy, quasar, intergalactic medium, or mass fields.
Indeed, if the initial conditions of the Universe are very close to
Gaussian (and they are) and if the evolution of the Universe is very
close to linear (as it is on large scales), then the mean and variance
of the density field (on some scale) are sufficient statistics to
infer everything about the initial conditions (at that scale).
Thus, the measurement or inference of variance as a function of
scale---or, equivalently, inference of the power spectrum or the
two-point auto-correlation function---is critical to inference of the
cosmological initial conditions and the cosmological parameters.
There is an enormous literature on making these cosmological
measurements!
Here we won't address this literature in detail.
We are going to step back and look at issues of principle in measuring
variances, and see if there are small changes to be made to
present-day cosmological practice that might improve future cosmological
inferences.

DWH: Introduce the idea of measuring a variance of a
process. Introduce the scalar problem and consider the Gaussian
pseudo-likelihood. Why is this patently absurd?

One critical property of the \emph{variance} of a Gaussian (or any other
distribution) is that it is a \emph{non-negative} quantity.
How does this relate to the properties of a two-point correlation
function (for, say, galaxies, the temperature map of the cosmic
microwave background, or the initial conditions of the density field)?
The answer is slightly non-trivial:
It is that the correlation function (what ought to be called the
``covariance function'') be a non-negative semi-definite function, or
that the correlation function obey what's called Mercer's condition,
or that any covariance matrix made up by any evaluation of the
correlation function on any grid of points be non-negative
semi-definite.
These (identical) conditions are best explained with equations.

First, define the correlation function $\xi(|r|)$ as the covariance of
the overdensity field $\delta(x)$, where $r$ is a vector displacement
in three-space (and $|r|$ is a scalar separation) and $x$ is
three-dimensional position:
\begin{eqnarray}
\xi(|r|) &=& E[\delta(x)\,\delta(x+r)]
\\
\delta(x) &\equiv& \frac{\rho(x)}{\bar{\rho}} - 1
\quad ,
\end{eqnarray}
where $E[q]$ is the expectation value of $q$ (implicitly taken as an integral
over all of space),
$\rho$ is the density field,
and $\bar{\rho}$ is the mean density.
By assuming that $\xi(\cdot)$ depends only on $|r|$ we are effectively
assuming statistical isotropy.
Now choose a spatial set of points $\setof{x_i}$ in three-dimensional space
and construct the matrix $C$ such that the elements $C_{ij}$ of $C$
are given by
\begin{eqnarray}
C_{ij} &\equiv& \xi(|x_i - x_j|)
\quad .
\end{eqnarray}
This matrix is manifestly symmetric, but if $\xi(|r|)$ is the true
covariance function of the density field, then this matrix must also
be non-negative semi-definite.
That is, it must have eigenvalues that are real and non-negative.
That this be true for any choice of points $\setof{x_i}$ is equivalent
(for our purposes) to Mercer's condition.

That is, there is no requirement that the correlation function (really
covariance function) be positive everywhere; it can go negative!
But it must be such that it always produces non-negative semi-definite
covariance tensors.

\acknowledgements
It is a pleasure to thank
  Brendon~J.~Brewer (Auckland),
  Chang~Hoon~Hahn (NYU),
  Dan~Foreman-Mackey (UW),
  M.~J.~Vakili (NYU),
  and the Blanton--Hogg group meeting at NYU
for valuable discussions.

\end{document}

{'fraction_non_alphanumeric': 0.07205720572057206, 'fraction_numerical': 0.0016501650165016502, 'mean_word_length': 4.543191056910569, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6148295', 'n_tokens_mistral': 2927, 'n_tokens_neox': 2773, 'n_words': 1485}