<jupyter_start><jupyter_text>Makers - Data reduction IntroductionThe `gammapy.makers` sub-package contains classes to perform data reduction tasksfrom DL3 data to binned datasets.In the data reduction step the DL3 data is prepared for modeling and fitting,by binning events into a counts map and interpolating the exposure, background,psf and energy dispersion on the chosen analysis geometry. Setup<jupyter_code>from gammapy.makers import (
    MapDatasetMaker,
    FoVBackgroundMaker,
    SafeMaskMaker,
    SpectrumDatasetMaker,
    ReflectedRegionsBackgroundMaker,
    DatasetsMaker
)
from gammapy.datasets import MapDataset, SpectrumDataset, Datasets
from gammapy.data import DataStore
from gammapy.maps import MapAxis, WcsGeom, RegionGeom
from regions import CircleSkyRegion
from astropy import units as u
import numpy as np<jupyter_output><empty_output><jupyter_text>DatasetThe counts, exposure,background and IRF maps are bundled together in a data structure named `MapDataset`.To handle on-off observations Gammapy also features a `MapDatasetOnOff` class, whichstores in addition the `counts_off`, `acceptance` and `acceptance_off` data.The first step of the data reduction is to create an empty dataset. A `MapDataset` can be created from any `WcsGeom` object. This is illustrated in the following example:<jupyter_code>energy_axis = MapAxis.from_bounds(
    1, 10, nbin=11, name="energy", unit="TeV", interp="log"
)
geom = WcsGeom.create(
    skydir=(83.63, 22.01),
    axes=[energy_axis],
    width=5 * u.deg,
    binsz=0.05 * u.deg,
    frame="icrs",
)
dataset_empty = MapDataset.create(geom=geom)
print(dataset_empty)<jupyter_output><empty_output><jupyter_text>It is possible to compute the instrument response functions with different spatial and energy binnings as compared to the counts and background maps. For example, one can specify a true energy axis which defines the energy binning of the IRFs:<jupyter_code>energy_axis_true = MapAxis.from_bounds(
    0.3, 10, nbin=31, name="energy_true", unit="TeV", interp="log"
)
dataset_empty = MapDataset.create(geom=geom, energy_axis_true=energy_axis_true)<jupyter_output><empty_output><jupyter_text>For the detail of the other options availables, you can always call the help:<jupyter_code>help(MapDataset.create)<jupyter_output><empty_output><jupyter_text>Once this empty "reference" dataset is defined, it can be filled with observationaldata using the `MapDatasetMaker`:<jupyter_code># get observation
data_store = DataStore.from_dir("$GAMMAPY_DATA/hess-dl3-dr1")
obs = data_store.get_observations([23592])[0]

# fill dataset
maker = MapDatasetMaker()
dataset = maker.run(dataset_empty, obs)
print(dataset)
dataset.counts.sum_over_axes().plot(stretch="sqrt", add_cbar=True);<jupyter_output><empty_output><jupyter_text>The `MapDatasetMaker` fills the corresponding `counts`, `exposure`, `background`,`psf` and `edisp` map per observation. The `MapDatasetMaker` has a`selection` parameter, in case some of the maps should not be computed. There is also a `background_oversampling` parameter that defines the oversampling factor in energy used to compute the bakcground (default is None). Safe data range handlingTo exclude the data range from a `MapDataset`, that is associated withhigh systematics on instrument response functions, a `mask_safe`can be defined. The `mask_safe` is a `Map` object with `bool` datatype, which indicates for each pixel, whether it should be includedin the analysis. The convention is that a value of `True` or `1`includes the pixel, while a value of `False` or `0` excludes a pixelsfrom the analysis. To compute safe data range masks according to certaincriteria, Gammapy provides a `SafeMaskMaker` class. The different criteria are given by the `methods`argument, available options are :- aeff-default, uses the energy ranged specified in the DL3 data files, if available.- aeff-max, the lower energy threshold is determined such as the effective area is above a given percentage of its maximum- edisp-bias, the lower energy threshold is determined such as the energy bias is below a given percentage- offset-max, the data beyond a given offset radius from the observation center are excluded- bkg-peak, the energy threshold is defined as the upper edge of the energy bin with the highest predicted background rate. This method was introduced in the HESS DL3 validation paper: https://arxiv.org/pdf/1910.08088.pdfMultiple methods can be combined. Here is an example :<jupyter_code>safe_mask_maker = SafeMaskMaker(
    methods=["aeff-default", "offset-max"], offset_max="3 deg"
)

dataset = maker.run(dataset_empty, obs)
dataset = safe_mask_maker.run(dataset, obs)
print(dataset.mask_safe)
dataset.mask_safe.sum_over_axes().plot();<jupyter_output><empty_output><jupyter_text>The `SafeMaskMaker` does not modify any data, but only defines the`MapDataset.mask_safe` attribute. This means that the safe datarange can be defined and modified in between the data reductionand stacking and fitting. For a joint-likelihood analysis of multipleobservations the safe mask is applied to the counts and predictednumber of counts map during fitting. This correctly accounts forcontributions (spill-over) by the PSF from outside the field of view. Background estimationThe background computed by the `MapDatasetMaker` gives the number of counts predictedby the background IRF of the observation. Because its actual normalization, or even itsspectral shape, might be poorly constrained, it is necessary to correct it with the datathemselves. This is the role of background estimation Makers.  FoV backgroundIf the background energy dependent morphology is well reproduced by the background modelstored in the IRF, it might be that its normalization is incorrect and that some spectralcorrections are necessary. This is made possible thanks to the `~gammapy.makers.FoVBackgroundMaker`.This technique is recommended in most 3D data reductions. For more details and usage, see [fov_background](https://docs.gammapy.org/dev/makers/fov.html).Here we are going to use a `~gammapy.makers.FoVBackgroundMaker` that will rescale the background model to the data excluding the region where a known source is present. For more details on the way to create exclusion masks see the [mask maps](mask_maps.ipynb) notebook.<jupyter_code>circle = CircleSkyRegion(center=geom.center_skydir, radius=0.2 * u.deg)
exclusion_mask = geom.region_mask([circle], inside=False)

fov_bkg_maker = FoVBackgroundMaker(
    method="scale", exclusion_mask=exclusion_mask
)
dataset = fov_bkg_maker.run(dataset)<jupyter_output><empty_output><jupyter_text>Other backgrounds production methods are available as listed below. Ring backgroundIf the background model does not reproduce well the morphology, a classical approach consistsin applying local corrections by smoothing the data with a ring kernel. This allows to build a setof OFF counts taking into account the inperfect knowledge of the background. This is implementedin the `~gammapy.makers.RingBackgroundMaker` which transforms the Dataset in a `MapDatasetOnOff`.This technique is mostly used for imaging, and should not be applied for 3D modeling and fitting.For more details and usage, see [ring_background](https://docs.gammapy.org/dev/makers/ring.html). Reflected regions backgroundIn the absence of a solid background model, a classical technique in Cherenkov astronomy for 1Dspectral analysis is to estimate the background in a number of OFF regions. When the backgroundcan be safely estimated as radially symmetric w.r.t. the pointing direction, one can apply thereflected regions background technique.This is implemented in the `~gammapy.makers.ReflectedRegionsBackgroundMaker` which transforms a`SpectrumDataset` in a `SpectrumDatasetOnOff`. This technique is only used for 1D spectralanalysis.For more details and usage, see [reflected_background](https://docs.gammapy.org/dev/makers/reflected.html). Data reduction loopThe data reduction steps can be combined in a single loop to runa full data reduction chain. For this the `MapDatasetMaker` is runfirst and the output dataset is the passed on to the next maker step.Finally the dataset per observation is stacked into a larger map.<jupyter_code>data_store = DataStore.from_dir("$GAMMAPY_DATA/hess-dl3-dr1")
observations = data_store.get_observations([23523, 23592, 23526, 23559])

energy_axis = MapAxis.from_bounds(
    1, 10, nbin=11, name="energy", unit="TeV", interp="log"
)
geom = WcsGeom.create(
    skydir=(83.63, 22.01), axes=[energy_axis], width=5, binsz=0.02
)

dataset_maker = MapDatasetMaker()
safe_mask_maker = SafeMaskMaker(
    methods=["aeff-default", "offset-max"], offset_max="3 deg"
)

stacked = MapDataset.create(geom)

for obs in observations:
    local_dataset = stacked.cutout(obs.pointing_radec, width="6 deg")
    dataset = dataset_maker.run(local_dataset, obs)
    dataset = safe_mask_maker.run(dataset, obs)
    dataset = fov_bkg_maker.run(dataset)
    stacked.stack(dataset)

print(stacked)<jupyter_output><empty_output><jupyter_text>To maintain good performance it is always recommended to do a cutoutof the `MapDataset` as shown above. In case you want to increase theoffset-cut later, you can also choose a larger width of the cutoutthan `2 * offset_max`.Note that we stack the individual `MapDataset`, which are computedper observation into a larger dataset. During the stacking the safe datarange mask (`MapDataset.mask_safe`) is applied by setting data outside tozero, then data is added to the larger map dataset. To stack multipleobservations, the larger dataset must be created first.The data reduction loop shown above can be done throught the `DatasetsMaker` class that take as argument a list of makers. **Note that the order of the makers list is important as it determines their execution order.** Moreover the `stack_datasets` option offers the possibily to stack or not the output datasets, and the `n_jobs` option allow to use multiple processes on run.<jupyter_code>global_dataset = MapDataset.create(geom)
makers = [dataset_maker, safe_mask_maker, fov_bkg_maker] #the order matter
datasets_maker = DatasetsMaker(makers, stack_datasets=False, n_jobs=1)
datasets = datasets_maker.run(global_dataset, observations)
print(datasets)<jupyter_output><empty_output><jupyter_text>Spectrum datasetThe spectrum datasets represent 1D spectra along an energy axis whitin a given on region. The `SpectrumDataset` contains a counts spectrum, and a background model. The `SpectrumDatasetOnOff` contains ON and OFF count spectra, background is implicitly modeled via the OFF counts spectrum. The `SpectrumDatasetMaker` make spectrum dataset for a single observation.In that case the irfs and background are computed at a single fixed offset, which is recommend only for point-sources.Here is an example of data reduction loop to create `SpectrumDatasetOnOff` datasets:<jupyter_code># on region is given by the CircleSkyRegion previously defined
geom = RegionGeom.create(region=circle, axes=[energy_axis])
exclusion_mask_2d = exclusion_mask.reduce_over_axes(
    np.logical_or, keepdims=False
)

spectrum_dataset_empty = SpectrumDataset.create(
    geom=geom, energy_axis_true=energy_axis_true
)

spectrum_dataset_maker = SpectrumDatasetMaker(
    containment_correction=False, selection=["counts", "exposure", "edisp"]
)
reflected_bkg_maker = ReflectedRegionsBackgroundMaker(
    exclusion_mask=exclusion_mask_2d
)
safe_mask_masker = SafeMaskMaker(methods=["aeff-max"], aeff_percent=10)

datasets = Datasets()

for observation in observations:
    dataset = spectrum_dataset_maker.run(
        spectrum_dataset_empty.copy(name=f"obs-{observation.obs_id}"),
        observation,
    )
    dataset_on_off = reflected_bkg_maker.run(dataset, observation)
    dataset_on_off = safe_mask_masker.run(dataset_on_off, observation)
    datasets.append(dataset_on_off)
print(datasets)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06974789915966387, 'fraction_numerical': 0.008487394957983193, 'mean_word_length': 6.000588235294118, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 41, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '28701', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 3413, 'n_tokens_neox': 3313, 'n_words': 1289, 'path': 'docs/tutorials/api/makers.ipynb', 'repo_name': 'isu-veritas/gammapy', 'size': '11900'}