<jupyter_start><jupyter_text>Source The `Source` class provides an interface and predefined methods for reading rawdata from file. If `source.use_nexus == True` the raw data will be saved andread preferentially from a generated `NeXus` file, which results in a singledata file with high compression and fast data access.The `Source` class provides a `dict()` of all scans, called `scan_dict`.The items in this dictionary are instances of the `Scan` class, which definesthe common data structure. However, the access to the scan data should **NOT**be done via the `scan_dict` directly, but instead for a deticated scan via`Source.get_scan_data(scan_number)` or for a list of scans via`Source.get_scan_list_data(scan_number_list)`. By this, all features of the`Source` class can be utilized.One key concept of the `pyEvalData` is the idea, that `Scan` instances mustonly provide the meta information of the scan but not necessarily its data.By default, the scan data will only be read on request and is then stored inthe `Scan` object for later use. It is possible, however, to read all datato each scan directly on first parsing by `Source.read_all_data = Ture`.On the other hand, if the data is allocating too much memory, it is possibleto clear the data from the `Scan` object, directly after accessing it, via`Source.read_and_forget = True`. The flag `Source.update_before_read` enablesparsing the raw source file to search and add new scans before accessing anyother scans. By default the last `Scan` in the `scan_dict` will always bere-created in case new data was added. The flag `Source.force_overwrite` willforce a full parsing of the raw source file and a complete overwrite of the`NeXus` file on every `update()`.  With these set of parameters the user is very flexible in how and if the rawdata is read and stored and can adapt the `Source` behaviour for differentsituations such as *on-line* anaylsis during a beamtime or *post* analysislater at home.The following examples are meant for introducing the low-level access to rawdata via the `Source` layer. This can be helpful for direct access to raw dataand integration into existing scripts and applications.  However, user will generally access the `Source` via the `Evaluation` classas described in the [upcoming examples](./evaluation).  SetupHere we do the necessary import for this example<jupyter_code>import matplotlib.pyplot as plt
import numpy as np

import pyEvalData as ped
# define the path for the example data
example_data_path = '../../../example_data/'<jupyter_output><empty_output><jupyter_text>SPECA very common data source are SPEC files from the original[Certif spec application](https://certif.com/content/spec/) as well as from theopen-source alternative [Sardana](https://sardana-controls.org).The `Spec` source relies on the great parser provided by[xrayutilities](https://xrayutilities.sourceforge.io).<jupyter_code>spec = ped.io.Spec(file_name='certif_xu.spec',
                   file_path=example_data_path,
                   use_nexus=True,
                   force_overwrite=False,
                   update_before_read=False,
                   read_and_forget=True,
                   nexus_file_path='./')<jupyter_output><empty_output><jupyter_text>As described above, the scan data and meta information should be accessed via`spec.get_scan_data(scan_number)`.  Here, `data` is returned as `numpy.recarray` and `meta` is a `dict()`.<jupyter_code>data, meta = spec.get_scan_data(33)

plt.figure()
plt.imshow(np.log10(data['MCA']))
plt.title(meta['cmd'])
plt.show()<jupyter_output><empty_output><jupyter_text>One can also directly work with the `Scan` objects, which provides nearly thesame functionality as above but misses to automatically `read_and_forget` thedata. Below, the example from above is reproduces with the `Scan`. Here one canalso easily access the meta information as attributes of the `Scan` instance.<jupyter_code>scan = spec.get_scan(33)

plt.figure()
plt.imshow(np.log10(scan.MCA))
plt.title(scan.cmd)
plt.show()<jupyter_output><empty_output><jupyter_text>It is also possible to access the scan directly as attribute of the source whichis fully aquivalent to the last example.<jupyter_code>plt.figure()
plt.imshow(np.log10(spec.scan33.MCA))
plt.title(spec.scan33.cmd)
plt.show()<jupyter_output><empty_output><jupyter_text>Sardana NeXusThe [NeXus file format](https://www.nexusformat.org) is a common standard inscience for storing and exchanging data. It is based on[hdf5](https://www.hdfgroup.org/solutions/hdf5/) files with a hierarchicalstructure. As an example we read the NeXus files as created by Sardana.As we can directly access the data from the NeXus files, there is no need toenable additional NeXus export so one can set `source.use_nexus = False`.<jupyter_code>sarnxs = ped.io.SardanaNeXus(file_name='sardana_nexus.h5',
                             file_path=example_data_path,
                             use_nexus=False,
                             force_overwrite=False,
                             update_before_read=False,
                             read_and_forget=True)
plt.figure()
plt.plot(sarnxs.scan435.mhor, sarnxs.scan435.pmPump)
plt.show()<jupyter_output><empty_output><jupyter_text>hdf5 - PAL FEL[hdf5](https://www.hdfgroup.org/solutions/hdf5/) is also a very common dataformat. In this example the raw data was measured at PAL FEL in South Korea.The individual files are grouped in a single folder per scan. This is handledby the `fprint`-statement for the `file_name` is is automatically evaluatedto provide the correct file and folder name for a given scan number.<jupyter_code>pal = ped.io.PalH5(name='2020_12_Schick',
                   file_name='{0:07d}',
                   file_path=example_data_path+'/pal_fel',
                   use_nexus=False,
                   force_overwrite=False,
                   update_before_read=True,
                   read_and_forget=True,
                   nexus_file_path='./',
                   nexus_file_name='2020_12_Schick')<jupyter_output><empty_output><jupyter_text>Again we have easy access to the raw data similar to the former examples.<jupyter_code>plt.figure()
plt.plot(pal.scan9.th, pal.scan9.digi1_1, '.')
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08101629913710451, 'fraction_numerical': 0.007350591243208693, 'mean_word_length': 4.084484159220146, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 6, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '599749', 'license': 'MIT', 'n_tokens_mistral': 1864, 'n_tokens_neox': 1785, 'n_words': 654, 'path': 'docs/source/examples/source.ipynb', 'repo_name': 'steinbachfe/pyEvalData', 'size': '6258'}