<commit_before>Background and Background Noise Estimation
==========================================

Introduction
------------

To accurately measure the photometry and morphological properties of
astronomical sources, one requires an accurate estimate of the
background, which can be from both the sky and the detector.
Similarly, having an accurate estimate of the background rms noise is
important for determining the significance of source detections and
for estimating photometric errors.

Unfortunately, accurate background and background noise estimation is
a difficult task.  Further, because astronomical images can cover a
wide variety of scenes, there is not a single background estimation
method that will always be applicable.  Photutils provides some tools
for estimating the background and background noise in your data, but
ultimately you have the flexibility of determining the background most
appropriate for your data.


Scalar Background Level and Noise Estimation
--------------------------------------------

Simple Statistics
^^^^^^^^^^^^^^^^^

If the background level and noise are relatively constant across an
image, the simplest way to estimate these values is to derive scalar
values for these quantities using simple approximations.  Of course,
when computing the image statistics one must take into account the
astronomical sources present in the images, which add a positive tail
to the distribution of pixel intensities.  For example, one may
consider using the image median as the background level and the image
standard deviation as the 1-sigma background noise, but the resulting
values are obviously biased by the presence of real sources.

A slightly better method involves using statistics that are robust
against the presence of outliers, such as the biweight location for
the background level and biweight midvariance or `median absolute
deviation (MAD)
<http://en.wikipedia.org/wiki/Median_absolute_deviation>`_ for the
background noise estimation.  However, for most astronomical scenes
these methods will also be biased by the presence of astronomical
sources in the image.

As an example, we load a synthetic image comprised of 100 sources with
a Gaussian-distributed background whose mean is 5 and standard
deviation is 2::

    >>> from photutils.datasets import make_100gaussians_image
    >>> data = make_100gaussians_image()

Let's plot the image:

.. doctest-skip::

    >>> from photutils.extern.imageutils.visualization import SqrtStretch
    >>> from photutils.extern.imageutils.visualization.mpl_normalize import ImageNormalize
    >>> import matplotlib.pylab as plt
    >>> norm = ImageNormalize(stretch=SqrtStretch())
    >>> plt.imshow(data, origin='lower', cmap='Greys_r', norm=norm)

.. plot::

    from photutils.datasets import make_100gaussians_image
    from photutils.extern.imageutils.visualization import SqrtStretch
    from photutils.extern.imageutils.visualization.mpl_normalize import ImageNormalize
    import matplotlib.pylab as plt
    data = make_100gaussians_image()
    norm = ImageNormalize(stretch=SqrtStretch())
    plt.imshow(data, origin='lower', cmap='Greys_r', norm=norm)

The image median and biweight location are both larger than the true
background level of 5::

    >>> import numpy as np
    >>> from astropy.stats import biweight_location
    >>> print(np.median(data))
    5.2255295184
    >>> print(biweight_location(data))
    5.1867597555

Similarly, using the biweight midvariance and median absolute
deviation to estimate the background noise level give values that are
larger than the true value of 2::

    >>> from astropy.stats import biweight_midvariance
    >>> from photutils.extern.stats import mad_std
    >>> print(biweight_midvariance(data))
    2.22011175104
    >>> print(mad_std(data))    # doctest: +FLOAT_CMP
    2.1443728009


Sigma-Clipped Statistics
^^^^^^^^^^^^^^^^^^^^^^^^

The most widely used technique to remove the sources from the image
statistics is called sigma clipping.  Briefly, pixels that are above
or below a specified sigma level from the median are discarded and the
statistics are recalculated.  The procedure is typically repeated over
a number of iterations or until convergence is reached.  This method
provides a better estimate of the background and background noise
levels::

    >>> from photutils.extern.imageutils.stats import sigma_clipped_stats
    >>> mean, median, std = sigma_clipped_stats(data, sigma=3.0)
    >>> print(median, median, std)    # doctest: +FLOAT_CMP
    (5.15557244947, 5.15557244947, 2.09393539966)


Masking Sources
^^^^^^^^^^^^^^^

An even better method is to exclude the sources in the image by
masking them.  Of course, this technique requires one to `identify the
sources in the data <detection.html>`_, which in turn depends on the
background and background noise.  Therefore, this method for
estimating the background and background rms requires an iterative
procedure.  We start by using the sigma-clipped statistics as the
first estimate of the background and noise levels for the source<commit_msg>Glitch in background estimation docs<commit_after>Background and Background Noise Estimation
==========================================

Introduction
------------

To accurately measure the photometry and morphological properties of
astronomical sources, one requires an accurate estimate of the
background, which can be from both the sky and the detector.
Similarly, having an accurate estimate of the background rms noise is
important for determining the significance of source detections and
for estimating photometric errors.

Unfortunately, accurate background and background noise estimation is
a difficult task.  Further, because astronomical images can cover a
wide variety of scenes, there is not a single background estimation
method that will always be applicable.  Photutils provides some tools
for estimating the background and background noise in your data, but
ultimately you have the flexibility of determining the background most
appropriate for your data.


Scalar Background Level and Noise Estimation
--------------------------------------------

Simple Statistics
^^^^^^^^^^^^^^^^^

If the background level and noise are relatively constant across an
image, the simplest way to estimate these values is to derive scalar
values for these quantities using simple approximations.  Of course,
when computing the image statistics one must take into account the
astronomical sources present in the images, which add a positive tail
to the distribution of pixel intensities.  For example, one may
consider using the image median as the background level and the image
standard deviation as the 1-sigma background noise, but the resulting
values are obviously biased by the presence of real sources.

A slightly better method involves using statistics that are robust
against the presence of outliers, such as the biweight location for
the background level and biweight midvariance or `median absolute
deviation (MAD)
<http://en.wikipedia.org/wiki/Median_absolute_deviation>`_ for the
background noise estimation.  However, for most astronomical scenes
these methods will also be biased by the presence of astronomical
sources in the image.

As an example, we load a synthetic image comprised of 100 sources with
a Gaussian-distributed background whose mean is 5 and standard
deviation is 2::

    >>> from photutils.datasets import make_100gaussians_image
    >>> data = make_100gaussians_image()

Let's plot the image:

.. doctest-skip::

    >>> from photutils.extern.imageutils.visualization import SqrtStretch
    >>> from photutils.extern.imageutils.visualization.mpl_normalize import ImageNormalize
    >>> import matplotlib.pylab as plt
    >>> norm = ImageNormalize(stretch=SqrtStretch())
    >>> plt.imshow(data, origin='lower', cmap='Greys_r', norm=norm)

.. plot::

    from photutils.datasets import make_100gaussians_image
    from photutils.extern.imageutils.visualization import SqrtStretch
    from photutils.extern.imageutils.visualization.mpl_normalize import ImageNormalize
    import matplotlib.pylab as plt
    data = make_100gaussians_image()
    norm = ImageNormalize(stretch=SqrtStretch())
    plt.imshow(data, origin='lower', cmap='Greys_r', norm=norm)

The image median and biweight location are both larger than the true
background level of 5::

    >>> import numpy as np
    >>> from astropy.stats import biweight_location
    >>> print(np.median(data))
    5.2255295184
    >>> print(biweight_location(data))
    5.1867597555

Similarly, using the biweight midvariance and median absolute
deviation to estimate the background noise level give values that are
larger than the true value of 2::

    >>> from astropy.stats import biweight_midvariance
    >>> from photutils.extern.stats import mad_std
    >>> print(biweight_midvariance(data))
    2.22011175104
    >>> print(mad_std(data))    # doctest: +FLOAT_CMP
    2.1443728009


Sigma-Clipped Statistics
^^^^^^^^^^^^^^^^^^^^^^^^

The most widely used technique to remove the sources from the image
statistics is called sigma clipping.  Briefly, pixels that are above
or below a specified sigma level from the median are discarded and the
statistics are recalculated.  The procedure is typically repeated over
a number of iterations or until convergence is reached.  This method
provides a better estimate of the background and background noise
levels::

    >>> from photutils.extern.imageutils.stats import sigma_clipped_stats
    >>> mean, median, std = sigma_clipped_stats(data, sigma=3.0)
    >>> print(mean, median, std)    # doctest: +FLOAT_CMP
    (5.19888175541, 5.15557244947, 2.09393539966)


Masking Sources
^^^^^^^^^^^^^^^

An even better method is to exclude the sources in the image by
masking them.  Of course, this technique requires one to `identify the
sources in the data <detection.html>`_, which in turn depends on the
background and background noise.  Therefore, this method for
estimating the background and background rms requires an iterative
procedure.  We start by using the sigma-clipped statistics as the
first estimate of the background and noise levels for the source
{'fraction_non_alphanumeric': 0.07926049759071689, 'fraction_numerical': 0.020257645786213, 'mean_word_length': 4.566502463054187, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 115, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5753710', 'n_tokens_mistral': 2748, 'n_tokens_neox': 2484, 'n_words': 1187}