<jupyter_start><jupyter_text>.. _data-format:<jupyter_code># Data format and SED data<jupyter_output><empty_output><jupyter_text>The data are treated using two classes from the module :mod:`.data_loader`- :class:`jetset.data_loader.Data` class - :class:`jetset.data_loader.ObsData` class The class :class:`jetset.data_loader.Data` is in charge of storing the data, giving access to the I/O functionalities, and provides an interface  to the astropy Table class (see the  `astropy table `_. documentation, for further information)The class :class:`jetset.data_loader.ObsData` uses the information stored in :class:`jetset.data_loader.Data`, and can perform several operations This is the class to use for model fitting and in general in Jetset  - rebinning (grouping) of the data - selection of time ranges - selection of datasets - transformation from linear to logarithmic representation - handling of errors and systematics .. note::    old version of data file in non `ecsv` astropy table format, can be easily converted by importing them using the method described     in :ref:`import` and saving them<jupyter_code>import jetset
print('tested on jetset',jetset.__version__)
import warnings
warnings.filterwarnings('ignore')

import matplotlib
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Data format for Data objectThe SED data are internally stored as astropy tables, but it is very easy to import from 1. ascii files 2. numpy array in general once that is clear the data format. The easiest way to understand the data format is to build an empty table to have a look at the structure of the table:<jupyter_code>from jetset.data_loader import Data
data=Data(n_rows=10)<jupyter_output><empty_output><jupyter_text>we can easily access the astropy table<jupyter_code>data.table<jupyter_output><empty_output><jupyter_text>- ``x`` column is reserved to frequencies (mandatory)- ``y`` columm is reserved to fluxes  (mandatory)- ``dx`` columm is reserved to the error on the frequency,or bin width- ``dy`` columm is reserved to the  error on the fluxes- ``UL`` columm is reserved to the flag for Upper Limit - ``T_start`` and ``T_stop`` are used to identify the time range to select data using the  class `ObsData`- ``data_set``<jupyter_code>data.table['x']<jupyter_output><empty_output><jupyter_text>columns with units are implemented using the `Units` module of astropy (https://docs.astropy.org/en/stable/units/). and we can easily access the metadata<jupyter_code>data.metadata<jupyter_output><empty_output><jupyter_text>- ``z``: the redshift of the object- ``UL_CL``: the CL for the UL- ``restframe``:  possible values``obs`` or ``src``, indicating if the data are observed flux, or luminosities, respectively- ``data_scale``: possible values``lin-lin`` or ``log-log``,  indicating if the data are in linear or logarithmic scale, respectively- ``obj_name``: the name of the object<jupyter_code>.. note::
   starting from version 1.1.0 ``src`` to ``obs`` transformation is available<jupyter_output><empty_output><jupyter_text>Loading from astropy table you can use the default SEDs distributed with the package to get familiar with data handling<jupyter_code>from jetset.test_data_helper import  test_SEDs
test_SEDs<jupyter_output><empty_output><jupyter_text>As you can see there are three 3 files. We use in this example the file for Mrk 421, and we use class:`jetset.data_loader.Data` class<jupyter_code>from jetset.data_loader import Data
data=Data.from_file(data_table=test_SEDs[1])
data.table
data.metadata<jupyter_output><empty_output><jupyter_text>this is an extract of the  astropy table saved in the format ``ascii.ecsv``<jupyter_code>```
# %ECSV 0.9
# ---
# datatype:
# - {name: x, unit: Hz, datatype: float64}
# - {name: dx, unit: Hz, datatype: float64}
# - {name: y, unit: erg / (cm2 s), datatype: float64}
# - {name: dy, unit: erg / (cm2 s), datatype: float64}
# - {name: T_start, unit: MJD, datatype: float64}
# - {name: T_stop, unit: MJD, datatype: float64}
# - {name: UL, datatype: bool}
# - {name: data_set, datatype: string}
# meta: !!omap
# - {z: 0.0308}
# - {restframe: obs}
# - {data_scale: lin-lin}
# - {obj_name: 'J1104+3812,Mrk421'}
# schema: astropy-2.0
x dx y dy T_start T_stop UL data_set
2299540000.0 0.0 1.3409e-14 3.91e-16 0.0 0.0 False campaing-2009
2639697000.0 0.0 1.793088e-14 3.231099e-26 0.0 0.0 False campaing-2009
4799040000.0 0.0 2.3136e-14 2.4e-16 0.0 0.0 False campaing-2009
```

### Saving Data object to a file<jupyter_output><empty_output><jupyter_text>data.save_file('test.ecsv')<jupyter_code>the data can be loaded from the saved table<jupyter_output><empty_output><jupyter_text>data=Data.from_file('test.ecsv') data.table<jupyter_code>.. _import:<jupyter_output><empty_output><jupyter_text>Importing  from an arbitrary ascii file or numpy array to Data object Assume that your data are stored in an ASCII file named 'test-ascii.txt', with- ``x`` in the first column with frequency in ``Hz`` , - ``y`` in the second column with fluxes in erg ``cm-2 s-1``, - ``dy`` in the third column with the same units as ``y``- the data are  in ``log-log`` scale**of course the column number depends on the file that you are using, this is only an example**<jupyter_code>from jetset.data_loader import Data
import numpy as np

d=np.genfromtxt('test-ascii.txt')
data=Data(n_rows=d.shape[0])
data.set_field('x',d[:,0])
data.set_field('y',d[:,1])
data.set_field('dy',value=d[:,2])<jupyter_output><empty_output><jupyter_text>then you can set the meatdata as follows<jupyter_code>data.set_meta_data('z',1.02)
data.set_meta_data('restframe','obs')
data.set_meta_data('data_scale','log-log')<jupyter_output><empty_output><jupyter_text>of course this method applies if you have a generic 2-dim numpy array.<jupyter_code>data.table<jupyter_output><empty_output><jupyter_text>Importing to Data object from a generic astropy table mapping columns  If you want to use a ``TABLE`` with arbitrary column names, you can use an import dictionary, mapping the input name to the target.E.g. assume that you column in the input table column named ``freq`` that should target the ``x`` column, and another named ``freq err`` associated to ``dx`` you can simply pass the dictionary to the ``from_file`` method: ```pythondata=Data.from_file(data_table='your-file',import_dictionary={'freq':'x','freq err':'dx'})```  Importing from the ASI ssdc sedtool to  Data object To import data from a data file downloaded from the asi ssdc sedtool: https://tools.ssdc.asi.it/SED/<jupyter_code>we can use the importing tool in the :class:`jetset.data_loader.Data`. We just need to have the file downloaded from the asi ssdc sedtool, and to know the redshift of the object, the scale we selected (lin-lin, or log-log).
Assume that we downloaded the data for Mrk421, in observed fluxes and linear scale, and the data are saved in the file 'MRK421_asdc.txt', we only have to do:<jupyter_output><empty_output><jupyter_text>from jetset.data_loader import Datadata=Data.from_asdc(asdc_sed_file='MRK421_asdc.txt',obj_name='Mrk421',restframe='obs',data_scale='lin-lin',z=0.038)<jupyter_code>.. note::
   starting from version 1.1.0 ``src`` to ``obs`` transformation is available<jupyter_output><empty_output><jupyter_text>data.table<jupyter_code>.. note::
   When importing data from the ``src`` frame, the Data constructor will not convert units, but will assume that input units are erg/s. If this is not the case an error message will be displayed<jupyter_output><empty_output><jupyter_text>Building the SED the ObsData object<jupyter_code>Once we have a data table built with the class:`jetset.data_loader.Data`, following  one of the method described above, you can create  SED data using the  :class:`jetset.data_loader.ObsData` class.
In the example we use one of the test SEDs provided by the package:
We start to loading  the SED of  Mrk 421, and we pass to ``ObsData`` directly the path to the file, because this is already in the format that we need and that we have discussed above.<jupyter_output><empty_output><jupyter_text>from jetset.data_loader import Datafrom jetset.data_loader import ObsDatafrom jetset.test_data_helper import  test_SEDsdata_table=Data.from_file(test_SEDs[1])sed_data=ObsData(data_table=data_table)<jupyter_code>if you want to use a cosmology model different from the default one please read the :ref:`cosmology` section<jupyter_output><empty_output><jupyter_text>As you can see all the meta-data have been properly sourced from the SED file header. You also get information on the length of the data, before and after elimination of duplicated entries, and upper limits<jupyter_code>sed_data.table
sed_data.metadata<jupyter_output>z :  0.0308
obj_name :  J1104+3812,Mrk421
restframe :  obs
data_scale :  lin-lin
UL_CL :  0.95<jupyter_text>Plotting ObsData<jupyter_code>We can now plot our SED using the :class:`BlazarSEDFit.plot_sedfit.Plot` class<jupyter_output><empty_output><jupyter_text>from jetset.plot_sedfit import PlotSEDmyPlot=PlotSED(sed_data)<jupyter_code>or you can create the object to plot on the fly in this way<jupyter_output><empty_output><jupyter_text>myPlot=sed_data.plot_sed()<jupyter_code>you can rescale your plot<jupyter_output><empty_output><jupyter_text>myPlot=sed_data.plot_sed()myPlot.rescale(x_min=7,x_max=28,y_min=-15,y_max=-9)<jupyter_code>plotting in the ``src`` restframe<jupyter_output><empty_output><jupyter_text>myPlot=sed_data.plot_sed(frame='src')myPlot.rescale(x_min=7,x_max=28,y_min=40,y_max=46)<jupyter_code>**to have interactive plot in jupyter**

if you want to to have interacitve plot in a jupyter notebook use:

.. code-block:: no

    %matplotlib notebook

to have interactive plot in jupyter lab use:

.. code-block:: no
 
    %matplotlib widget

### Grouping data

As you can see, due to the overlapping of different instruments and to different time snapshots, 
some points have multiple values. Although this is not a problem for the fit process, you might 
want to rebin (group) your data. This can be obtained with the following command:<jupyter_output><empty_output><jupyter_text>%matplotlib inlinemyPlot=sed_data.plot_sed()myPlot.rescale(y_min=-15)sed_data.group_data(bin_width=0.2)myPlot.add_data_plot(sed_data,label='rebinned')<jupyter_code>### Handling errors and systematics

Another important issue when dealing with fitting of data, is the proper handling of errors.
Typically one might need to add systematics for different reasons:
 
 - data are not really simultaneous, and you want to add systematics to take this into account
 - data (typically IR up to UV), might have very small errors compared to those at higher energies.
   This might bias the minimizer to accommodate the parameters  in order to fit 'better' the low
   frequencies branch.
  
For these reasons the package offer the possibility to add systematics<jupyter_output><empty_output><jupyter_text>sed_data.add_systematics(0.2,[10.**6,10.**29])myPlot=sed_data.plot_sed()myPlot.rescale(y_min=-15)<jupyter_code>with this command we add 20% systematics for data between :math:`10^{6}<\nu<10^{29}` Hz<jupyter_output><empty_output><jupyter_text>Filtering data sets  we use the `show_data_sets()`  method to have know wich data sets are defined in our table<jupyter_code>sed_data.show_data_sets()<jupyter_output>current datasets
dataset 0.0<jupyter_text>we use `show_dataset=True` to have the legend of all the datasets<jupyter_code>data=Data.from_file(test_SEDs[0])
sed_data=ObsData(data_table=data)
%matplotlib inline
p=sed_data.plot_sed(show_dataset=True)
sed_data.show_data_sets()<jupyter_output>current datasets
dataset -1
dataset 0
dataset 1
dataset 2<jupyter_text>we filter out the data set `-1` using the `filter_data_set()` method. Please not with `exclude=True` we exclude dataset in `filters`<jupyter_code>sed_data.filter_data_set(filters='-1',exclude=True)
sed_data.show_data_sets()
p=sed_data.plot_sed(show_dataset=True)<jupyter_output>---> excluding  data_set/s ['-1']
filter -1 192
current datasets
dataset 0
dataset 1
dataset 2
---> data sets left after filtering None
---> data len after filtering=192
current datasets
dataset 0
dataset 1
dataset 2<jupyter_text>we can pass more datasets, comma separated<jupyter_code>sed_data.filter_data_set(filters='-1,0',exclude=True)
sed_data.show_data_sets()
p=sed_data.plot_sed(show_dataset=True)<jupyter_output>---> excluding  data_set/s ['-1', '0']
filter -1 192
filter 0 57
current datasets
dataset 1
dataset 2
---> data sets left after filtering None
---> data len after filtering=57
current datasets
dataset 1
dataset 2<jupyter_text>we can also use `filter_data_set` to exclude *only* the datasets in `filters` with `exclude=False`<jupyter_code>sed_data.filter_data_set(filters='-1',exclude=True)
sed_data.show_data_sets()
p=sed_data.plot_sed(show_dataset=True)<jupyter_output>---> excluding  data_set/s ['-1']
filter -1 192
current datasets
dataset 0
dataset 1
dataset 2
---> data sets left after filtering None
---> data len after filtering=192
current datasets
dataset 0
dataset 1
dataset 2<jupyter_text>we can revert `sed_data` to the original state with the `reset_data()` method<jupyter_code>sed_data.reset_data()
sed_data.show_data_sets()
p=sed_data.plot_sed(show_dataset=True)<jupyter_output>current datasets
dataset -1
dataset 0
dataset 1
dataset 2<jupyter_text>Saving sed_data and loading you can save and relaod you sed_data<jupyter_code>sed_data.save('3C454_data.pkl')
sed_data=ObsData.load('3C454_data.pkl')
p=sed_data.plot_sed()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10075975510806226, 'fraction_numerical': 0.022571365346315556, 'mean_word_length': 6.154617414248021, 'pattern_counts': {'":': 0, '<': 136, '<?xml version=': 0, '>': 143, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '35', 'dir': 'jupyter-structured-clean-dedup', 'id': '149370', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 4701, 'n_tokens_neox': 4420, 'n_words': 1445, 'path': 'doc/documentation_notebooks/load_data/Jet_example_load_data.ipynb', 'repo_name': 'andreatramacere/jetset', 'size': '13557'}