<jupyter_start><jupyter_text>Reading Binary Data with Numpy<NAME> (<EMAIL>)The latest version of this notebook is available at [https://github.com/escape2020/school2021](https://github.com/escape2020/school2021)<jupyter_code>import numpy as np
import sys

print(f"Python version:  {sys.version}\n"
      f"NumPy version:   {np.__version__}")
%matplotlib inline
import matplotlib.pyplot as plt
plt.rcParams['figure.figsize'] = (16, 5)
plt.rcParams['figure.dpi'] = 300<jupyter_output><empty_output><jupyter_text>Exercise: Read a KM3NeT Event File and create a histogram of the PMT hit signalsUse `numpy.fromfile()` and custom `dtype`s to read an event from `School2019/numpy/IO_EVT.dat`The KM3NeT DAQ dataformat for storing an event consists of a header and two sets of hits (triggered hits and snapshot hits). The header has been skipped, so `IO_EVT.dat` only contains the **triggered** and **snapshot** hits. Triggered hits:- n_hits `[int32, little endian]`- n_hits * triggered_hit_struct    - optical module ID `[int32, little endian]`, example 808476737    - PMT ID `[unsigned char (byte)]`, value between 0 and 30    - time in nanoseconds `[uint32, big endian]`, example 90544845    - ToT (time over threshold in nanoseconds) `[unsigned byte]`, value between 0 and 255    - trigger mask `[uint64, little endian]`, bitmask, typical values are 1, 3, 4, 6 Snapshot hits: same as triggered hits but without the `trigger mask` field  Solution We can use the `xxd` command to have a quick look at the binary data. If we don't know the structure, this might be a good starting point to identify some strings or recognise numbers from a proiri knowledge.<jupyter_code>!xxd IO_EVT.dat |head -n 10<jupyter_output>00000000: 0f00 0000 4160 3030 0205 659a 101d 0400  ....A`00..e.....
00000010: 0000 0000 0000 4160 3030 0305 659a 2515  ......A`00..e.%.
00000020: 0400 0000 0000 0000 5e7b 3030 0005 659a  ........^{00..e.
00000030: 6821 0400 0000 0000 0000 5e7b 3030 0105  h!........^{00..
00000040: 659a 541b 0600 0000 0000 0000 5e7b 3030  e.T.........^{00
00000050: 0a05 659a 6511 0600 0000 0000 0000 5e7b  ..e.e.........^{
00000060: 3030 1005 659a 5c1b 0600 0000 0000 0000  00..e.\.........
00000070: 5e7b 3030 1405 659a 5619 0600 0000 0000  ^{00..e.V.......
00000080: 0000 4887 3730 0105 6599 cf1a 0600 0000  ..H.70..e.......
00000090: 0000 0000 4887 3730 0b05 6599 d613 0600  ....H.70..e.....<jupyter_text>The hit `dtype` We define our custom `dtype` for the hits and use the `dtype.descr` attribute as a base `dtype` for triggered hits, extended with the `triggermask` field.<jupyter_code>hit_dtype = np.dtype([
    ("dom_id", "<i"),
    ("pmt_id", "B"),
    ("time", ">I"),
    ("tot", "B"),
])

trig_hit_dtype = np.dtype(hit_dtype.descr + [('triggermask', '<Q')])<jupyter_output><empty_output><jupyter_text>The file `IO_EVT.dat` contains a single event. Opened in binary-read mode (`"rb"`), the `fobj` behaves like a stream. `np.fromfile` will call the `.read()` method with the number of bytes calculated from the given `dtype`.According to the data format specification, the first integer (represented by `dtype='<i'` where `<` indicates that it's little endian) is the number of triggered hits.To read the array of triggered hits, the `trig_hit_dtype` is used and the `count=n_trig_hits` argument is passed, otherwise `numpy` will read till the end of the file.We repeat the same process for the regular (snapshot) hits.  Parsing the binary data<jupyter_code>with open("IO_EVT.dat", "rb") as fobj:
    n_trig_hits = np.fromfile(fobj, dtype='<i', count=1)[0]
    trig_hits = np.fromfile(fobj, dtype=trig_hit_dtype, count=n_trig_hits)
    
    n_hits = np.fromfile(fobj, dtype='<i', count=1)[0]
    hits = np.fromfile(fobj, dtype=hit_dtype, count=n_hits)<jupyter_output><empty_output><jupyter_text>Let's see what we got:<jupyter_code>trig_hits
hits<jupyter_output><empty_output><jupyter_text>The overall ToT distributionWe can easily access a specific attribute of all hits using dictionary-notation.<jupyter_code>hits['tot']
fig, ax = plt.subplots()
ax.hist(hits['tot'])
ax.set_xlabel('ToT [ns]')
ax.set_yscale('log')
ax.set_ylabel('count');<jupyter_output><empty_output><jupyter_text>Solution (binary file)<jupyter_code>def extract_hits(filename):
    """Extract the hits from a binary dump"""
    fobj = open(filename, 'rb')
    hits = []
    triggered_hits = []
    while fobj:
        header = fobj.read(48)  # skip the first 48 bytes
        if not header:
            break  # end of file

        n_trig_hits = np.frombuffer(fobj.read(4), dtype='<i', count=1)[0]
        _triggered_hits = np.frombuffer(
            fobj.read(trig_hit_dtype.itemsize * n_trig_hits),
            dtype=trig_hit_dtype
        )
        triggered_hits.append(_triggered_hits)
        
        n_hits = np.frombuffer(fobj.read(4), dtype='<i', count=1)[0]
        _hits = np.frombuffer(
            fobj.read(hit_dtype.itemsize * n_hits),
            dtype=hit_dtype
        )
        hits.append(_hits)
        
    
    fobj.close()
    return trig_hits, hits<jupyter_output><empty_output><jupyter_text>Extracting hits (binary file)<jupyter_code>triggered_hits, hits = extract_hits("events.dat")
tots = [h['tot'] for h in  hits]
fig, ax = plt.subplots()
ax.hist(np.concatenate(tots).ravel(), bins=100)
ax.set_xlabel('ToT [ns]')
ax.set_yscale('log')
ax.set_ylabel('count');<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13049935028772972, 'fraction_numerical': 0.08780397252645257, 'mean_word_length': 4.030812324929972, 'pattern_counts': {'":': 0, '<': 42, '<?xml version=': 0, '>': 35, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '586143', 'license': 'MIT', 'n_tokens_mistral': 2270, 'n_tokens_neox': 1895, 'n_words': 568, 'path': 'numpy/4. Binary Data.ipynb', 'repo_name': 'vishu-byte/school2021', 'size': '5399'}