<commit_before>Galaxy Mass Proxy Fitting
=========================

This code implements a "regression" on galaxy mass proxies.  Given
observations of galaxy mass and a suitable proxy for mass, with
associated errors, it fits a model to the data and returns posteriors
on model parameters.  Using these posteriors, subsequent observations
of either the proxy, the mass, or both can be turned into a posterior
on the true mass of the galaxy; in other words, the regression can be
"inverted" to predict the quantity of interest (mass), exploiting all
the available data (mass/proxy observations if available).

The Model
---------

The model that is fit is the following.  Given observations of Mobs
and proxy with errors dM and dp, we assume that these are generated by
the following model:

log(Mobs) ~ N(log(Mtrue), dM/Mobs)

log(proxy) ~ N(log(alpha*Mtrue + beta), sqrt(dp^2 + sigma^2)/proxy)

with Mtrue, alpha, beta, sigma parameters.  In words: the observed
mass is normally distributed about the true mass with width equal to
the quoted observational errors; the observed proxy is normally
distributed about the linear relationship alpha*Mtrue + beta with
width given by a combination of intrinsic scatter (sigma) and
observational error (dp).

When fitting, we assume a prior on log(Mtrue) that is uniform.  Priors
on alpha, beta, and sigma are Jeffreys.

Installation/Running
--------------------

The model is encoded in posterior.py's Posterior class.  It is
intended for use with the [emcee](http://dan.iel.fm/emcee/current/)
sampler.  Note that Posterior includes routines for computing the
expected value of Mtrue given parameters and a subsequent observation
of Mobs and proxy with errors.  In other words, once a relation has
been fit, it can be exploited to improve the accuracy of subsequent
observations, in a manner consistent with the posterior on model
parameters.

The code in plots.py permits to generate plots illustrating the fit.

License
-------

This code is released under the [MIT
License](http://opensource.org/licenses/MIT).  See the file LICENSE
for details.<commit_msg>Updates to README.md describing operation of the new version of the code.<commit_after>Galaxy Mass Proxy Fitting
=========================

This code implements a "regression" on galaxy mass from observations
of mass and a mass proxy.  Using the regression posterior, subsequent
observations of the proxy can be converted to a posterior estimate of
the associated mass and its uncertainty.

Preliminaries
-------------

To use the code, you will need to install 

* [emcee](http://dan.iel.fm/emcee/current/), <NAME>-Mackey's
  excellent MCMC sampler.  The sampler is used to draw from the
  posterior of the regression model.

* [plotutils](https://github.com/farr/plotutils), a collection of
  routines to support managing emcee runs and the plotting of results.

Both of these libraries require [numpy](http://www.numpy.org/), and
plotutils requires [scipy](http://www.scipy.org/) and
[matplotlib](http://matplotlib.org/), too.

Basic Mode of Operation
-----------------------

First, one calibrates a mass-proxy relationship using 

       calibrate.py --data observational-data.dat --outdir output-directory

The file of observational data should be a text (or gzipped/bzipped
text) file with columns.  The first line is a header line, and should
contain at least the <code>mass</code>, <code>proxy</code>,
<code>dm</code>, and <code>dp</code> giving the measured mass and
proxy values, together with the (relative) observational uncertainties
in these quantities.  Various output files will be placed in the
indicated directory that will permit subsequent estimation of masses.
You may want to examine the "fit" displayed in <code>fit.pdf</code>,
which shows the data, the average "regression" line, and 10 draws from
the regression posterior.

Alternately, the package comes with several pre-calibrated mass-proxy
relationships, as described in
[arXiv:XXXX.XXXX](http://arxiv.org/abs/XXXX.XXXX); these can be found
in subdirectories of the <code>data</code> directory.

Once one has a directory of calbration results, observations of the
proxy only can be turned into mass estimates using 

      estimate.py --caldir calibration-directory --proxyfile proxy-data.dat --output mass-output.dat.bz2

The proxy data should be a text (or gzipped/bzipped text) file with
columns.  The first line is a header line, and should contain at least
the colums <code>proxy</code> and <code>dp</code> giving the observed
proxy values and (relative) uncertainties.  The results will be placed
in the output file (which will always be compressed with bzip2); each
line of that file contains draws from the posterior on the true mass
for each corresponding proxy measurement.  

The Model
---------

The model used for the regression is described in detail in
[arXiv:XXXX.XXXX](http://arxiv.org/abs/XXXX.XXXX), and implemented in
the extensively-commented <code>posterior.py</code>.  The model
assumes that the true masses and proxies are drawn from a correlated,
multivariate log-normal distribution; additionally, observed values
are assumed to distribute log-normally about the true values with
width equal to the quoted observational uncertainty.  The model fits
for the mean and covariance matrix of the log-normal, and then
combines the posterior on these parameters with subsequent
observations of proxy values to draw posterior estimates of masses.

The code in <code>plots.py</code> permits to generate plots
illustrating various aspects of the fit.

The code in <code>paper_test.py</code> implements the synthetic data
set and tests described in
[arXiv:XXXX.XXXX](http://arxiv.org/abs/XXXX.XXXX).

License
-------

This code is released under the [MIT
License](http://opensource.org/licenses/MIT).  See the file LICENSE
for details.
{'fraction_non_alphanumeric': 0.08709069089662266, 'fraction_numerical': 0.001028630207440425, 'mean_word_length': 4.298819255222525, 'pattern_counts': {'":': 0, '<': 26, '<?xml version=': 0, '>': 26, 'https://': 1, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4342582', 'n_tokens_mistral': 1568, 'n_tokens_neox': 1484, 'n_words': 769}