<jupyter_start><jupyter_text>Exploring dataWhen working with a dataset, the first step is usually to understand what data and metadata it contains.In this chapter we explore how scipp supports this.This tutorial contains exercises, but solutions are included directly.We encourage you to download this notebook and run through it step by step before looking at the solutions.We recommend to use a recent version of *JupyterLab*:The solutions are included as hidden cells and shown only on demand.First, in addition to importing `scipp`, we import `scippneutron` since this is required for loading Nexus files:<jupyter_code>import scipp as sc
import scippneutron as scn<jupyter_output><empty_output><jupyter_text>We  start by loading some data, in this case measured with a prototype of the [LoKI](https://europeanspallationsource.se/instruments/loki) detectors at the [LARMOR beamline](https://www.isis.stfc.ac.uk/Pages/Larmor.aspx):<jupyter_code>data = scn.data.tutorial_dense_data()<jupyter_output><empty_output><jupyter_text>In practice, we would use [scippneutron.load](https://scipp.github.io/scippneutron/generated/scippneutron.load.html?highlight=loadscippneutron.load) or [scippneutron.load_nexus](https://scipp.github.io/scippneutron/generated/scippneutron.load_nexus.html?highlight=load_nexusscippneutron.load_nexus) to load the data from a NeXus file, but the tutorial data comes bundled with scippneutron to make it easily available.Note that the exercises in the following are fictional and do not represent the actual SANS data reduction workflow.  Step 1: Use the HTML representation to see what the loaded data containsThe HTML representation is what Jupyter displays for a scipp object.- Take some time to explore this view and try to understand all the information (dimensions, dtypes, units, ...).- Note that sections can be expanded, and values can shown by clicking the icons to the right.<jupyter_code>data<jupyter_output><empty_output><jupyter_text>Whenever you get stuck with one of the exercises below we recommend consulting the HTML representations of the objects you are working with.  Step 2: Plot the dataScipp objects (variables, data arrays, or datasets) can be plotted using the `plot()` method.Alternatively `sc.plot(obj)` can be used, e.g., when `obj` is a Python `dict` of scipp data arrays.Since this is neutron-scattering data, we can also use the "instrument view", provided by `scn.instrument_view(obj)` (assuming `scippneutron` was imported as `scn`). Exercise- Plot the loaded data and familiarize yourself with the controls.- Create the instrument view and familiarize yourself with the controls. Solution<jupyter_code>data.plot()
scn.instrument_view(data)<jupyter_output><empty_output><jupyter_text>Step 3: Exploring meta dataAbove we saw that many attributes are scalar variables with `dtype=DataArray`.The single value in a scalar variable is accessed using the `value` property.Compare:<jupyter_code>data.attrs['proton_charge_by_period']
data.attrs['proton_charge_by_period'].value<jupyter_output><empty_output><jupyter_text>Exercise1. Find some attributes of `data` with `dtype=DataArray` and plot their `value`.   Also try `sc.table(attr.value)` to show a table representation (where `attr` is an attribute of your choice).2. Find and plot a monitor.3. Try to normalize `data` to monitor 1.   Why does this fail?4. Plot all the monitors on the same plot.   Note that `sc.plot()` can be used with a Python `dict` for this purpose: `sc.plot({'a':something, 'b':else})`.5. Convert all the monitors from `'tof'` to `'wavelength'` using, e.g., `mon1_wav = scn.convert(mon1, 'tof', 'wavelength', scatter=False)`.6. Inspect the HTML view and note how the "unit conversion" changed the dimensions and units.7. Re-plot all the monitors on the same plot, now in `'wavelength'`. Solution<jupyter_code>sc.table(data.attrs['DCMagField2'].value)
try:
    data / data.attrs['monitor1'].value
except sc.CoordError:
    print('Data and monitor are in unit TOF, but pixels and monitors are at different position, so data is not comparable')<jupyter_output><empty_output><jupyter_text>A monitor can be converted to wavelength as follows:<jupyter_code>mon1 = data.attrs['monitor1'].value
scn.convert(mon1, 'tof', 'wavelength', scatter=False)
sc.plot({f'monitor{i}':data.attrs[f'monitor{i}'].value for i in [1,2,3,4,5]}, norm='log')
sc.plot({f'monitor{i}':scn.convert(data.attrs[f'monitor{i}'].value, 'tof', 'wavelength', scatter=False) for i in [1,2,3,4,5]}, norm='log')<jupyter_output><empty_output><jupyter_text>Step 4: Fixing metadata ExerciseConsider the following (hypothetical) problems with the metadata stored in `data`:1. The `sample_position` coord (`data.coords['sample_position']`) is wrong, shift the sample by `delta = sc.vector(value=np.array([0.01,0.01,0.04]), unit='m')`.2. Because of a glitch in the timing system the time-of-flight has an offset of $2.3~\mu s$.   Fix the corresponding coordinate.3. Use the HTML view of `data` to verify that you applied the corrections/calibrations there, rather than in a copy. Solution<jupyter_code>data.coords['sample_position'] += sc.vector(value=[0.01,0.01,0.04], unit='m')
data.coords['tof'] += 2.3 * sc.Unit('us')  # note how we forgot to fix the monitor's TOF
data<jupyter_output><empty_output><jupyter_text>Note how adding such offsets fails if we fail to specify a unit:<jupyter_code>try:
    data.coords['tof'] += 2.3
except sc.UnitError as e:
    print(e)<jupyter_output><empty_output><jupyter_text>This has several advantages:- We are protected from accidential errors.  If someone changes the unit of data or metatdata without our knowledge, e.g., from `us` to `ms` this mechanism protects us from silent errors corrupting the data.- It makes the code clearer and more readable, both for others as well as for our future selves. The offset to the sample could also be done component-wise using the special `fields` property of variables with "vector" dtype, e.g.,<jupyter_code>data.coords['sample_position'].fields.z += 0.001 * sc.Unit('m')<jupyter_output><empty_output><jupyter_text>Step 5: A closer look at the dataThe 2-D plot we obtain above by default is often not very enlightening.Define:<jupyter_code>counts = data.sum('tof')<jupyter_output><empty_output><jupyter_text>Exercise1. Create a plot of `counts` and also try the instrument view.2. How many counts are there in total, in all spectra combined?3. Plot a single spectrum of `data` as a 1-D plot using the slicing syntax `array[dim_name, integer_index]` to access the spectrum. Solution<jupyter_code># slice is optional, making plot more readable in the documentation
counts['spectrum', 56000:62000].plot()
scn.instrument_view(counts, norm='log')
# counts.sum('spectrum') # would be another solution
data.sum().value
data['spectrum',10000].plot()<jupyter_output><empty_output><jupyter_text>Volumetric detectorsAs seen in the instrument view the detectors consist of 4 layers of tubes, each containing 7 straws.Let us try to split up our data, so we can compare layers.There are other (and probably better) ways to do this, but here we try to define an integer variable containing a layer index:<jupyter_code>z = data.coords['position'].fields.z
near = z.min()
far = z.max()
layer = ((z-near)*400).astype('int32')
layer.unit = ''
layer.plot()<jupyter_output><empty_output><jupyter_text>Exercise- Change the magic parameter `400` in the cell above until pixels fall cleanly into layers, either 4 layers of tubes or 12 layers of straws.- Store `layer` as a new coord in `data`.- Use `data.groupby(group='layer').sum('spectrum')` to group spectra into layers.- Inspect and understand the HTML view of the result.- Plot the result.  There are two options:  - Use `plot` with `projection='1d'`  - Use `sc.plot` after collapsing dimensions, `sc.collapse(grouped, keep='tof')`- Bonus: When grouping by straw layers, there is a different number of straws in the center layer of each tube (3 instead of 2) due to the flower-pattern arrangement of straws.  Define a helper data array with data set to 1 for each spectrum (using, e.g., `norm = sc.DataArray(data=sc.ones_like(layer), coords={'layer':layer})`), group by layers and sum over spectrum as above, and use this result to normalize the layer-grouped data from above to spectrum count. Solution<jupyter_code># NOTE:
# - set magic factor to, e.g., 150 to group by straw layer
# - set magic factor to, e.g., 40 to group by tube layer
layer = ((z-near)*150).astype(sc.dtype.int32)
layer.unit = ''
data.coords['layer'] = layer
grouped = data.groupby(group='layer').sum('spectrum')
grouped.plot(projection='1d')
sc.plot(sc.collapse(grouped, keep='tof'))
norm = sc.DataArray(data=sc.ones_like(layer), coords={'layer':layer})
norm = norm.groupby(group='layer').sum('spectrum')
sc.plot(sc.collapse(grouped/norm, keep='tof'))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10583982830678866, 'fraction_numerical': 0.012312210550096013, 'mean_word_length': 5.758778625954198, 'pattern_counts': {'":': 0, '<': 57, '<?xml version=': 0, '>': 57, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '18415', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 2751, 'n_tokens_neox': 2641, 'n_words': 1008, 'path': 'docs/tutorials/1_exploring-data.ipynb', 'repo_name': 'arm61/scippneutron', 'size': '8853'}