<jupyter_start><jupyter_text>Data import and explorationNew pulsemaker files  First, look at the dataid, samples, amplitude1, rise1, decay1, position1, offset, amplitude2, rise2, decay2, position2, val1, val2, ...The id is either 1 for a single pulse or 2 for a double pulse. In the single pulse case all the pulse parameters for the second pulse are 0.Open up the file and look at it. In most linux terminals you can also use the command`head` to display a set number of lines from the start of the file. We'll start with the first line.Usually you will also have some information about the file formatting.You can run shell commands inside the notebook by starting the line with an exclamation mark.<jupyter_code>!head -1 ../data/training_pm_nosat_150k.dat<jupyter_output><empty_output><jupyter_text>Data importThis is a bit of a "hands-on" approach that focuses on using base functions in python andNumpy to read the file and create some arrays we can work with.Reading the file is pretty straightforward, but some challenges appear when the file is large.We will prepare for that scenario from the get-go.training_pm_nosat.dat contains 1M mixed single and double events.\We set a datapath and filename. Note that we use the 'relative' path to our data, from thelocation of the notebook. This works well in a github repo where you control the folder structure.And if someone else will clone or fork the repo, they don't need to replace the paths.For small datafiles you can read the entire file into memory, store it as a list where each element is a line in the file, and work from there.```pythonwith open(DATA_PATH + fname, "r") as datafile:    data = datafile.readlines()```You now have a list `data` containing all the lines in the file. Using the `with` statement is a shortcut sowe don't need to manually close the file after opening it.However, reading the entire file into memory might not be possible, so we will read it line by line instead,looping over the file. This read only one line into memory at a time.Some resources for this exercise* [Python I/O File handling](https://docs.python.org/3/tutorial/inputoutput.htmlreading-and-writing-files)* [Numpy array indexing](https://numpy.org/doc/1.18/reference/arrays.indexing.html)* [Numpy's fromstring() function](https://numpy.org/doc/1.18/reference/generated/numpy.fromstring.html)<jupyter_code># Imports
import numpy as np
import matplotlib.pyplot as plt
# Set path to data and filename. You can also store it a single variable
# DATA_PATH = "../data/CeBr10k_1.txt" if you prefer. Here we expect to use at least the path itself later,
# so we separate them.
DATA_PATH = "../data/"
fname = "training_pm_nosat_150k.dat"

# Number of lines to import
num_lines = 10
# Lists to store each image, energy, position, and labels. We know the filesize, so we could create
# arrays that perfectly match the data, but let's assume we don't know how many lines we're going to
# read. 
labels = []
amplitudes = []
rises = []
decays = []
positions = []
offsets = []
samples = []

# Open the file
with open(DATA_PATH + fname, "r") as datafile:
    # Loop over the file, line by line
    linecount = 0
    for line in datafile:
        if linecount >= num_lines:
            break
        # The line is still a string when read from the file. We use numpys fromstring()
        # to convert the line to a numpy array, specifying that each element is separated
        # by a space. This does not convert the line in the file, only the "copy" that we have
        # read into memory. fromstring() also removes any trailing newline ('\n') characters
        # so we don't have to worry about that. The values will be interpreted as floats.
        line = np.fromstring(line, sep=' ')
        
        # id, samples, amplitude1, rise1, decay1, position1, offset, amplitude2, rise2, decay2, position2, val1, val2,
        # First we grab the known variables for the data, and set labels to 0 / 1 for single / double
        label = 0 if line[0] == 1 else 1
        amplitude = np.array((line[2], line[7]))
        rise = np.array((line[3], line[8]))
        decay = np.array((line[4], line[9]))
        pos = np.array((line[5], line[10]))
        offset = line[6]
        
        # Grab the sample values, located at index 11 to the end of the line
        sample = np.array(line[11:])

        # Finally, we take the separated arrays and add them to their respective "storage" lists.
        labels.append(label)
        amplitudes.append(amplitude)
        rises.append(rise)
        decays.append(decay)
        positions.append(pos)
        offsets.append(offset)
        samples.append(sample)
        
        linecount += 1

        
# We've now looped over the entire file. The only thing that remains is to convert the lists
# to numpy arrays.
labels = np.array(labels)
amplitudes = np.array(amplitudes)
rises = np.array(rises)
decays = np.array(decays)
positions = np.array(positions)
offsets = np.array(offsets)
samples = np.array(samples)
# We print the shape of the arrays we've made as a quick check using
# the shape property of numpy arrays.
print("Labels shape, dtype:", labels.shape, labels.dtype)
print("Amplitues shape:", amplitudes.shape, amplitudes.dtype)
print("Rises shape:", rises.shape, rises.dtype)
print("Decays shape:", decays.shape, decays.dtype)
print("Positions shape:", positions.shape, positions.dtype)
print("Offsets shape:", offsets.shape, offsets.dtype)
print("Samples shape:", samples.shape, samples.dtype)<jupyter_output><empty_output><jupyter_text>The output above tells us that we've got 10000 images, each of length 256, as well as two energy values for each image, and four positions. The images, energies, and positions arrays have two dimensions, while the labels arrayonly has one.  Inspection of imported dataLet's plot a single and a double event to see what we're working with.We can extract the indices of singles and double using the labels array, and a numpyfunction called np.where.Resources:* [Matplotlib tutorial](https://matplotlib.org/tutorials/introductory/pyplot.html)* [Numpy np.where() doc](https://numpy.org/doc/stable/reference/generated/numpy.where.html)<jupyter_code># np.where returns a tuple where the first (and in this case only) element is our indices
singles = np.where(labels == 0)[0]
doubles = np.where(labels == 1)[0]
print("Num imported single events: ", singles.shape)
print("Num imported double events: ", doubles.shape)
# We can now grab a single and a double event by using the index arrays we made
# Ex. to get the sample values for the first single event: samples[singles[0]]

# Plot a single and double event side by side
# Add red dotted lines for where the positions are.
fig, ax = plt.subplots(1, 2, figsize=(12, 5))
ax[0].plot(samples[singles[0]])
ax[0].axvline(positions[singles[0], 0], color='r', linestyle='--')
ax[0].set_title("Single event ")
ax[0].set_xlabel("Sample number")
ax[0].set_ylabel("Amplitude")

ax[1].plot(samples[doubles[0]])
ax[1].axvline(positions[doubles[0], 0], color='r', linestyle='--')
ax[1].axvline(positions[doubles[0], 1], color='r', linestyle='--')
ax[1].set_title("Double event")
ax[1].set_xlabel("Sample number")
ax[1].set_ylabel("Amplitude")
plt.show()

# Print the additional information we have about the events.
print("============ Single event info")
print("Amplitude:", amplitudes[singles[0]])
print("Rise:", rises[singles[0]])
print("Decay:", decays[singles[0]])
print("Positions:", positions[singles[0]])
print("Offset:", offsets[singles[0]])

print("============ Double event info")
print("Amplitude:", amplitudes[doubles[0]])
print("Rise:", rises[doubles[0]])
print("Decay:", decays[doubles[0]])
print("Positions:", positions[doubles[0]])
print("Offset:", offsets[doubles[0]])
# Plotting with annotations
fig, ax = plt.subplots(1, 2, figsize=(12, 5))
ax[0].plot(samples[singles[0]])
ax[0].plot(positions[singles[0], 0], amplitudes[singles[0], 0], 'rx')
ax[0].set_title("Single event")
ax[0].set_xlabel("Sample number")
ax[0].set_ylabel("Amplitude")

ax[1].plot(samples[doubles[0]])
ax[1].plot(positions[doubles[0], 0], amplitudes[doubles[0], 0], 'rx')
ax[1].plot(positions[doubles[0], 1], amplitudes[doubles[0], 1], 'rx')
ax[1].set_title("Double event")
ax[1].set_xlabel("Sample number")
ax[1].set_ylabel("Amplitude")
plt.show()<jupyter_output><empty_output><jupyter_text>Separate and save the dataFirst, we separate the data into a *training set* and a *test set*. The test set will be used to give us an "out-of-sample"accuracy. To do this properly, the model will never see the test set in any way, shape, or form before we've trained, evaluated, and optimized it as far as we'd like to go. In other words, we use the training set to make the model as good as possible, and only then do we predict on the test set and report out-of-sample metrics.You can do the splitting of data manually by slicing the arrays, but scikit-learn has it's own function for doing this,[train_test_split](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html?highlight=train_test_splitsklearn.model_selection.train_test_split).Instead of creating copies of the dataset, filling up often precious memory, we will work with indices that wepass around. This also makes it easier to trace any results back to the original inputs.  Numpy format<jupyter_code>from sklearn.model_selection import train_test_split
# Indices for all data
x_idx = np.arange(images.shape[0])

# Split the indices into training and test sets (take out 10% of the data as test)
train_idx, test_idx, not_used1, not_used2 = train_test_split(x_idx, x_idx, test_size = 0.1)

# Save the training and test data in the data folder
# We also need to save the labels, energies, and positions. This allows us to
# quickly load it if we need it.

# Save the training data. np.save adds a ".npy" file extension to the provided filename.
# We save the files in the same folder as the original datafile.
np.save(DATA_PATH + "images_training", images[train_idx])
np.save(DATA_PATH + "energies_training", energies[train_idx])
np.save(DATA_PATH + "positions_training", positions[train_idx])
np.save(DATA_PATH + "labels_training", labels[train_idx])

# Save the test data
np.save(DATA_PATH + "images_test", images[test_idx])
np.save(DATA_PATH + "energies_test", energies[test_idx])
np.save(DATA_PATH + "positions_test", positions[test_idx])
np.save(DATA_PATH + "labels_test", labels[test_idx])<jupyter_output><empty_output><jupyter_text>You could also opt to store the entire training and test sets as two files, by concatenating them.This is just personal preference.  hdf5 format<jupyter_code>import h5py
# Open separate files for saving training and test datasets
data_training = h5py.File(DATA_PATH + "data_training.hdf5", "w")
data_test = h5py.File(DATA_PATH + "data_test.hdf5", "w")

# Training data
data_training.create_dataset('images', data=images[train_idx])
data_training.create_dataset('energies', data=energies[train_idx])
data_training.create_dataset('positions', data=positions[train_idx])
data_training.create_dataset('labels', data=labels[train_idx])
data_training.close()

# Test data
data_test.create_dataset('images', data=images[test_idx])
data_test.create_dataset('energies', data=energies[test_idx])
data_test.create_dataset('positions', data=positions[test_idx])
data_test.create_dataset('labels', data=labels[test_idx])
data_test.close()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11052861511577108, 'fraction_numerical': 0.013543031891655745, 'mean_word_length': 4.887860082304527, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 22, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '255613', 'license': 'MIT', 'n_tokens_mistral': 3546, 'n_tokens_neox': 3407, 'n_words': 1295, 'path': 'notebooks/data_import_exploration_traces.ipynb', 'repo_name': 'OceanNuclear/event_classification_example', 'size': '11445'}