<jupyter_start><jupyter_text>GaussPy+ tutorial for a test field of the Galactic Ring SurveyThis notebook is intended to guide users through the typical GaussPy+ procedures for the decomposition of a position-position-velocity (PPV) dataset. For more information exceeding this tutorial we recommend taking a look at the following papers and documents:- For a description about the GaussPy+ decomposition package see:> - [Riener et al. 2019](https://arxiv.org/abs/1906.10506)- For a description about the GaussPy algorithm see: > - [GaussPy documentation](https://gausspy.readthedocs.io/en/latest/)> - [Lindner et al. 2015](https://arxiv.org/abs/1409.2840)<jupyter_code># import necessary libraries and plotting functions

import os

from astropy.io import fits

%matplotlib inline
import matplotlib
import matplotlib.pyplot as plt
from pylab import cm

from astropy.wcs import WCS

from gausspyplus.plotting import get_points_for_colormap, shiftedColorMap


def get_cmap_rchi2(vmin, vmax):
    orig_cmap = matplotlib.cm.RdBu_r
    start, stop = get_points_for_colormap(vmin, vmax, central_val=1.)
    midpoint = (1 - vmin) / (vmax - vmin)
    return shiftedColorMap(orig_cmap, start=0., midpoint=midpoint, stop=stop)


def add_style(ax):
    ax.set_xlabel('Galactic Longitude')
    ax.set_ylabel('Galactic Latitude')


if not os.path.exists('decomposition_grs'):
    !mkdir decomposition_grs<jupyter_output><empty_output><jupyter_text>OverviewThis tutorial consists of the following steps:|        |  || :-      |:-                                                || 0.       | [Create a configuration file (optional)](step0) || 1.       | [Create a training set from the data](step1) || 2.       | [Find the optimal values for the smoothing parameters $\alpha_{1}$ and $\alpha_{2}$](step2) || 3.       | [Prepare the data for the decomposition](step3) || 4.       | [Decomposition of the data](step4) || 5.       | [Spatially coherent refitting - phase 1](step5) || 6.       | [Spatially coherent refitting - phase 2](step6) || 7.       | [Finalize the decomposition results](step7) ||          | [Example: Create FITS files of fit parameters](example) |The directory `gausspyplus/data` contains the FITS cube `grs-test_field.fits`, which is a subset of the Galactic Ring Survey ([Jackson et al. 2006](https://ui.adsabs.harvard.edu/abs/2006ApJS..163..145J/abstract)) that we will use for this tutorial. This is the same dataset that was used as a test field in Riener+ 2019.  0. Create a configuration file (optional)We start by generating a default configuration file. The next code creates a configuration file called 'gausspy+.ini' in the current working directory.<jupyter_code>import gausspyplus.config_file as cf
cf.make()<jupyter_output><empty_output><jupyter_text>This configuration file only contains the most essential parameters, which in many cases should already be sufficient to create first good results. To take a look or access the full range of keywords that can be changed by the user change the above command to `cf.make(all_keywords=True)`.It is not necessary to create a configuration file for running `GaussPy+`. If no configuration file is supplied `GaussPy+` will resort to the default value for the parameters. Parameters can also be supplied or changed later on as will be done in the scripts we call further below.   1. Create a training set from the dataNow we create a training set from the FITS cube `grs-test_field.fits`. For this we execute the `step_1-training_set--grs.py` script contained in the `example` directory. See the `step_1-training_set--grs.py` script for more comments.NOTE: Running this script will use 75% of all CPUs on the machine you are running it unless the `use_ncpus` parameter is specified.Depending on the number of available CPUs the execution of this script might take a couple of minutes.<jupyter_code>#  run the script
!python step_1-training_set--grs.py<jupyter_output><empty_output><jupyter_text>ResultsAfter this script was successfully executed, the `decomposition_grs` directory should contain a folder named `gpy_training` with two files:- `grs-test_field-training_set_100_spectra.pickle`: a pickled dictionary of the decomposition results of the training set- `grs-test_field-training_set_100_spectra_plots.pdf`: plots of the decomposition results. We recommend to always plot a subsample of the training set to check whether the fitting of the training set worked out well.Since this example serves only to illustrate how training sets can be created we kept the number of spectra of the training set deliberately low. We recommend to include at least 200 spectra in the training set to get good results for the smoothing parameters.  2. Find the optimal values for the smoothing parameters $\alpha_{1}$ and $\alpha_{2}$After we checked that the decomposition of the training set gave good results, we can supply it to the machine learning procedure that GaussPy employs to find the best smoothing parameters values $\alpha_{1}$ and $\alpha_{2}$. For this we execute the `step_2-train--grs.py` script contained in the `example` directory. See the `step_2-train--grs.py` script for more comments.NOTE: Running this script will use 75% of all CPUs on the machine you are running it unless the `use_ncpus` parameter is specified.Depending on the number of available CPUs the execution of this script might take a couple of minutes.<jupyter_code>#  run the script
!python step_2-train--grs.py<jupyter_output><empty_output><jupyter_text>ResultsIn our case, we needed 47 iterations until a stable convergence was achieved. This yielded values for the smoothing parameters of $\alpha_{1} = 2.58$ and $\alpha_{2} = 5.14$.Note that the `decomposition_grs` directory now contains a new folder named `gpy_log`, which contains a log of the training results.   3. Prepare the data for the decompositionNext, we have to prepare our data cube for the decomposition. Executing the script `step_3-prepare--grs.py` will automatically:- estimate the root-mean-square noise $\sigma_{\mathrm{rms}}$ of the spectra- determine regions in the spectra that are likely to contain signal peaks- mask out negative noise spikes (by default all data peaks with a minimum $< -5 \times \sigma_{\mathrm{rms}}$)- produce a pickled dictionary containing all necessary information for the decompositionNOTE: Running this script will use 75% of all CPUs on the machine you are running it unless the `use_ncpus` parameter is specified.<jupyter_code>#  run the script
!python step_3-prepare--grs.py<jupyter_output><empty_output><jupyter_text>ResultsAfter this script was successfully executed, the `decomposition_grs` directory should contain a folder named `gpy_prepared` that contains `grs-test_field.pickle`, which is a pickled dictionary of the prepared data cube.The `decomposition_grs` directory should also contain a new folder named `gpy_maps` that contains the file `grs-test_field_noise_map.fits`, which is a map of the estimated rms noise values of our GRS test field.Lets take a look at the noise map:<jupyter_code>filepath = os.path.join('decomposition_grs', 'gpy_maps', 'grs-test_field_noise_map.fits')
noise = fits.getdata(filepath)
wcs = WCS(fits.getheader(filepath))

fig, ax = plt.subplots(figsize=(6, 4), subplot_kw=dict(projection=wcs))

img_noise = ax.imshow(noise, cmap='plasma_r', vmin=0.075, vmax=0.375)
fig.colorbar(img_noise, ax=ax, extend='max')
ax.set_title('Noise map')
add_style(ax)

plt.show()<jupyter_output><empty_output><jupyter_text>The `step_3-prepare--grs.py` script also plotted all spectra within pixel ranges $30 \leq \text{X} \leq 34$ and $25 \leq \text{Y} \leq 29$ and saved it as `grs-test_field_plots.pdf` in the `gpy_plots` directory.The spectra are plotted in the correct spatial order to aid in comparing differences between neighboring spectra. The shaded red areas indicate the regions of the spectrum that were identified to contain signal (goodness of fit calculations are restricted to these areas). The dotted horizontal red lines indicate the estimated noise values of $\pm \sigma_{\mathrm{rms}}$ and the horizontal dashed line marks a S/N ratio of 3. The title of each subplot contains information about the location in terms of pixels and the index of the spectrum in the `grs-test_field.pickle` dictionary.  4. Decomposition of the dataAfter the successful preparation of the data, we can proceed to the decomposition of the data. The following script will run an improved fitting routine on top of the original GaussPy decomposition routine; for more details see Sect. 3.2. in [Riener et al. 2019](https://arxiv.org/abs/1906.10506).NOTE: Running this script will use 75% of all CPUs on the machine you are running it unless the `use_ncpus` parameter is specified.Depending on the number of available CPUs the execution of this script might take a couple of minutes.<jupyter_code>#  run the script
!python step_4-decompose--grs.py<jupyter_output><empty_output><jupyter_text>ResultsAfter this script was successfully executed, the `decomposition_grs` directory should contain a folder named `gpy_decomposed` that contains `grs-test_field_g+_fit_fin.pickle`, which is a pickled dictionary of the decomposition results.In addition, we also produced a map showing the number of fitted components and a map showing the $\chi_{\mathrm{red}}^{2}$ values of the fit. Lets take a look at both of these maps:<jupyter_code>filepath = os.path.join('decomposition_grs', 'gpy_maps', 'grs-test_field_g+_rchi2_map.fits')
rchi2 = fits.getdata(filepath)
wcs = WCS(fits.getheader(filepath))

fig, axes = plt.subplots(ncols=2, figsize=(12, 4), subplot_kw=dict(projection=wcs))

ax = axes.flatten()[0]

vmin = min(rchi2.flatten())
vmax = 2.5
new_cmap = get_cmap_rchi2(vmin, vmax)

img_rchi2 = ax.imshow(rchi2, cmap=new_cmap, vmin=vmin, vmax=vmax)
fig.colorbar(img_rchi2, ax=ax, extend='max')
ax.set_title('$\chi_{\mathrm{red}}^{2}$ map')
add_style(ax)

ax = axes.flatten()[1]

ncomps = fits.getdata(os.path.join(
    'decomposition_grs', 'gpy_maps', 'grs-test_field_g+_component_map.fits'))

vmax = 6
new_cmap = cm.get_cmap('Spectral_r', vmax + 1)

img_ncomps = ax.imshow(ncomps, cmap=new_cmap, vmin=0, vmax=vmax)
fig.colorbar(img_ncomps, ax=ax)
ax.set_title('Number of fitted components')
add_style(ax)

fig.suptitle('After first decomposition with improved fitting routine')

plt.show()<jupyter_output><empty_output><jupyter_text>5. Spatially coherent refitting - phase 1Next we will try to improve upon the decomposition results obtained in the last step. For this, we try to refit spectra that were flagged as not satisfying our chosen quality criteria. For more details see Sect.$\,$3.3.1. in [Riener et al. 2019](https://arxiv.org/abs/1906.10506) for more details.For this we execute the `step_5-spatial_refitting-p1--grs.py` script contained in the `example` directory. See the `step_5-spatial_refitting-p1--grs.py` script for more comments.NOTE: Running this script will use 75% of all CPUs on the machine you are running it unless the `use_ncpus` parameter is specified.Depending on the number of available CPUs the execution of this script might take a couple of minutes.<jupyter_code>#  run the script
!python step_5-spatial_refitting-p1--grs.py<jupyter_output><empty_output><jupyter_text>ResultsAfter this script was successfully executed, the `gpy_decomposed` folder contains the new file `grs-test_field_g+_fit_fin_sf-p1.pickle`, which is a pickled dictionary of the new decomposition results.In addition, we also produced new corresponding mapa of the number of fitted components and $\chi_{\mathrm{red}}^{2}$ values of the fit. Lets take a look again at both of these maps:<jupyter_code>filepath = os.path.join(
    'decomposition_grs', 'gpy_maps', 'grs-test_field_g+_fit_fin_sf-p1_rchi2_map.fits')
rchi2 = fits.getdata(filepath)
header = fits.getheader(filepath)
wcs = WCS(header)

fig, axes = plt.subplots(ncols=2, figsize=(12, 4), subplot_kw=dict(projection=wcs))

ax = axes.flatten()[0]

vmin = min(rchi2.flatten())
vmax = 2.5
new_cmap = get_cmap_rchi2(vmin, vmax)

img_rchi2 = ax.imshow(rchi2, cmap=new_cmap, vmin=vmin, vmax=vmax)
fig.colorbar(img_rchi2, ax=ax, extend='max')
ax.set_title('$\chi_{\mathrm{red}}^{2}$ map')
add_style(ax)

ax = axes.flatten()[1]

ncomps = fits.getdata(os.path.join(
    'decomposition_grs', 'gpy_maps', 'grs-test_field_g+_fit_fin_sf-p1_component_map.fits'))

vmax = 6
new_cmap = cm.get_cmap('Spectral_r', vmax + 1)

img_ncomps = ax.imshow(ncomps, cmap=new_cmap, vmin=0, vmax=vmax)
fig.colorbar(img_ncomps, ax=ax)
ax.set_title('Number of fitted components')
add_style(ax)

fig.suptitle('After spatially coherent refitting - phase 1')

plt.show()<jupyter_output><empty_output><jupyter_text>We can see that the $\chi_{\mathrm{red}}^{2}$ values of the fit did not change significantly; however the map of the number of fitted components shows already more spatial coherence. We can also check the new fit results for our 25 neighboring spectra that are plotted in `grs-test_field_g+_fit_fin_sf-p1_plots.pdf`. These plots also confirm that we gained a significant improvement in terms of spatial coherence of the fit results.  6. Spatially coherent refitting - phase 2In the last step, we try to further improve upon the decomposition results obtained in the last step by checking the coherence of the centroid positions of the fitted Gaussian components. See Sect.$\,$3.3.2. in [Riener et al. 2019](https://arxiv.org/abs/1906.10506) for more details.For this we execute the `step_6-spatial_refitting-p2--grs.py` script contained in the `example` directory. See the `step_6-spatial_refitting-p2--grs.py` script for more comments.NOTE: Running this script will use 75% of all CPUs on the machine you are running it unless the `use_ncpus` parameter is specified.Depending on the number of available CPUs the execution of this script might take a couple of minutes.<jupyter_code>#  run the script
!python step_6-spatial_refitting-p2--grs.py<jupyter_output><empty_output><jupyter_text>ResultsAfter this script was successfully executed, the `gpy_decomposed` folder contains the new file `grs-test_field_g+_fit_fin_sf-p2.pickle`, which is a pickled dictionary of the new decomposition results.In addition, we also produced new corresponding maps of the number of fitted components and $\chi_{\mathrm{red}}^{2}$ values of the fit. Lets take a look again at both of these maps:<jupyter_code>filepath = os.path.join(
    'decomposition_grs', 'gpy_maps', 'grs-test_field_g+_fit_fin_sf-p2_rchi2_map.fits')
rchi2 = fits.getdata(filepath)
wcs = WCS(fits.getheader(filepath))

fig, axes = plt.subplots(ncols=2, figsize=(12, 4), subplot_kw=dict(projection=wcs))

ax = axes.flatten()[0]

vmin = min(rchi2.flatten())
vmax = 2.5
new_cmap = get_cmap_rchi2(vmin, vmax)

img_rchi2 = ax.imshow(rchi2, cmap=new_cmap, vmin=vmin, vmax=vmax)
fig.colorbar(img_rchi2, ax=ax, extend='max')
ax.set_title('$\chi_{\mathrm{red}}^{2}$ map')
add_style(ax)

ax = axes.flatten()[1]

ncomps = fits.getdata(os.path.join(
    'decomposition_grs', 'gpy_maps', 'grs-test_field_g+_fit_fin_sf-p2_component_map.fits'))

vmax = 6
new_cmap = cm.get_cmap('Spectral_r', vmax + 1)

img_ncomps = ax.imshow(ncomps, cmap=new_cmap, vmin=0, vmax=vmax)
fig.colorbar(img_ncomps, ax=ax)
ax.set_title('Number of fitted components')
add_style(ax)

fig.suptitle('After spatially coherent refitting - phase 2')

plt.show()<jupyter_output><empty_output><jupyter_text>Judging by the map showing the number of fitted components we seem to have gained more spatial coherence. The new and final fit results for our 25 neighboring spectra are plotted in `grs-test_field_g+_fit_fin_sf-p2_plots.pdf`.  7. Finalize the decomposition resultsFinally, we can produce a table containing the final fit results from the second phase of the spatially coherent refitting stage.NOTE: Running this script will use 75% of all CPUs on the machine you are running it unless the `use_ncpus` parameter is specified.<jupyter_code>#  run the script
!python step_7-finalize--grs.py<jupyter_output><empty_output><jupyter_text>ResultsAfter this script was successfully executed, the `gpy_decomposed` folder contains the new file `grs-test_field_g+_fit_fin_sf-p2_finalized.dat`, which is a table containing the entire decomposition results.By default the table is similar to Table 1 presented in [Riener et al. 2020](https://arxiv.org/abs/1911.06860) (see Sect. 3.1 in [Riener et al. 2020](https://arxiv.org/abs/1911.06860) for a description of the table columns).  Example: Create FITS files of fit parametersFinally, we show an example of how it is possible to extract information on individual fit components. GaussPy+ saves all the relevant data as pickled dictionaries in the ``gpy_prepared`` and ``gpy_decomposed`` directories. By executing the following script, we create FITS files of the fit parameters (amplitude, mean position, FWHM) of the component with the lowest fitted mean position per spectrum.<jupyter_code>#  run the script
!python example_fits_first_component--grs.py<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08960741740508187, 'fraction_numerical': 0.01813273868904407, 'mean_word_length': 5.2424908424908425, 'pattern_counts': {'":': 0, '<': 58, '<?xml version=': 0, '>': 60, 'https://': 9, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '356764', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 5511, 'n_tokens_neox': 5180, 'n_words': 1953, 'path': 'example/Tutorial_example-GRS.ipynb', 'repo_name': 'cmurray-astro/gausspyplus', 'size': '17041'}