Scaling pair count to next galaxy surveys
  Counting pairs of galaxies or stars according to their distance is at the
core of real-space correlation analyzes performed in astrophysics and
cosmology. Upcoming galaxy surveys (LSST, Euclid) will measure properties of
billions of galaxies challenging our ability to perform such counting in a
minute-scale time relevant for the usage of simulations. The problem is only
limited by efficient access to the data, hence belongs to the big data
category. We use the popular Apache Spark framework to address it and design an
efficient high-throughput algorithm to deal with hundreds of millions to
billions of input data. To optimize it, we revisit the question of
nonhierarchical sphere pixelization based on cube symmetries and develop a new
one dubbed the "Similar Radius Sphere Pixelization" (SARSPix) with very close
to square pixels. It provides the most adapted indexing over the sphere for all
distance-related computations. Using LSST-like fast simulations, we compute
autocorrelation functions on tomographic bins containing between a hundred
million to one billion data points. In each case we achieve the construction of
a standard pair-distance histogram in about 2 minutes, using a simple algorithm
that is shown to scale, over a moderate number of nodes (16 to 64). This
illustrates the potential of this new techniques in the field of astronomy
where data access is becoming the main bottleneck. They can be easily adapted
to other use-cases as nearest-neighbors search, catalog cross-match or cluster
finding. The software is publicly available from
https://github.com/astrolabsoftware/SparkCorr.
