<jupyter_start><jupyter_text>Writing DataThis notebook demonstrates how to use DL1 Data Handler to write/read MAGIC data for use in machine learning analysis in Python. ctapipe-extra need to be installed on top of dl1-data-handler to read MAGIC data from ROOT files. Please do "conda install ctapipe-extra".<jupyter_code>from dl1_data_handler.writer import DL1DataWriter, CTAMLDataDumper<jupyter_output><empty_output><jupyter_text>Please change the path to your directory hosting the data!<jupyter_code>data_dir = "/Users/tmiener/deeplearning/magic_school/"

calibrated_root_file = data_dir + "GA_M*_za05to35_8_821319_Y_w0.root"
superstar_root_file = data_dir + "GA_za05to35_8_821319_S_w0.root"

calibrated_hdf5_file = data_dir + "GA_za05to35_8_821319_Y_w0_dl1dh_v0.9.0.h5"
superstar_hdf5_file = data_dir + "GA_za05to35_8_821319_S_w0_dl1dh_v0.9.0.h5"<jupyter_output><empty_output><jupyter_text>This is a walk-through with example files. For a dataset production, please set up the runlist automatically using scripts/generate_runlist.py and execute on the command line:```bashpython scripts/write_data.py [runlist] [--config_file CONFIG_FILE_PATH] [--output_dir OUTPUT_DIR] [--debug]```<jupyter_code>runlist_calibrated = [
    {
        'inputs': [calibrated_root_file],
        'target': calibrated_hdf5_file
    },
]

# If you would like to use the DL1DH with superstar files,
# make sure you had run "star" with flags "-saveimages -saveimagesclean -savecerevt".
# i.e.:
# $ star -b -f -mc -q -saveimages -saveimagesclean -savecerevt --config=mrcfiles/star_M{1,2}_OSA.rc --ind="/home/tjark/MAGIC_files/cal/*M{1,2}*_Y_*.root" --out="/home/tjark/MAGIC_files/starM{1,2}/" --log=/home/tjark/MAGIC_files/starM{1,2}/LogFile.txt
# $ superstar -q -b -f -mc --config=mrcfiles/superstar.rc --ind1=/home/tjark/MAGIC_files/starM1/GA_M1_za05to35_8_*_I_w0.root --ind2=/home/tjark/MAGIC_files/starM2/GA_M2_za05to35_8_*_I_w0.root --out=/home/tjark/MAGIC_files/superstar/ --log=/home/tjark/MAGIC_files/superstar/logfile.txt
runlist_superstar = [
    {
        'inputs': [superstar_root_file],
        'target': superstar_hdf5_file
    },
]



print("Number of input files in runlist: {}".format(
    len([input_file for run in runlist_calibrated for input_file in run['inputs']])))
print("Number of output files requested: {}".format(
    len(runlist_calibrated)))<jupyter_output><empty_output><jupyter_text>Firstly, for each event, data is read from the ROOT files in the DL1DataWriter. Then, this data is sent to the CTAMLDataDumper to dump it to a HDF5 file with the DL1DH data format.<jupyter_code>event_src_settings = {}
data_dumper_class = CTAMLDataDumper

dumper_settings = {
    'filter_settings': {
        'complib': 'lzo',
        'complevel': 1
    },
    'expected_tel_types': 1,
    'expected_tels': 2,
    'expected_events': 300,
    'expected_images_per_event': {
        'MAGIC:MAGICCam': 2.0
    },
    'index_columns': [
        ['/Events', 'mc_energy'],
        ['/Events', 'alt'],
        ['/Events', 'az'],
        ['tel', 'event_index']
    ],
    # Various hillas parameters caculation via ctapipe are stored additionally in the same file.
    'cleaning_settings': [
        {'algorithm': 'tailcuts_clean',
        'args': {
            'picture_thresh': 10,
            'boundary_thresh': 5}},
        
        {'algorithm': 'tailcuts_clean',
        'args': {
            'picture_thresh': 20,
            'boundary_thresh': 5}}
    ]
}
# Some basic writer settinngs
writer_settings = {
    # Parallel for doing everything via python multiprocessing
    'write_mode': 'serial',
    'output_file_size': 1378500,
    'events_per_file': 300,
    # Selected the telescopes, which should be dumped into the hdf5. M1 and M2 for MAGIC.
    'selected_telescope_ids': [1, 2],
    # This is used for the main hillas parameter table.
    # If the code is called with the superstar files,
    # it will ignore this dict and write the MARS
    # hillas parameters to the main parameter table.
    'cleaning_settings': {
        'algorithm': 'tailcuts_clean',
        'args': {
            'picture_thresh': 6,
            'boundary_thresh': 3}
    }
}<jupyter_output><empty_output><jupyter_text>Now, we instantiate our DL1DataWriter and then call process_data with our runlist. After a brief wait, the output files we requested in our runlist should be written.<jupyter_code>data_writer = DL1DataWriter(event_source_class=None,
                            event_source_settings=event_src_settings,
                            data_dumper_class=data_dumper_class,
                            data_dumper_settings=dumper_settings,
                            write_mode = writer_settings['write_mode'],
                            output_file_size = writer_settings['output_file_size'],
                            events_per_file = writer_settings['events_per_file'],
                            selected_telescope_ids = writer_settings['selected_telescope_ids'],
                            cleaning_settings = writer_settings['cleaning_settings'],
                        )

data_writer.process_data(runlist_calibrated)
data_writer.process_data(runlist_superstar)<jupyter_output><empty_output><jupyter_text>We can cross check the size of the output files created now. The MAGIC superstar were produced using the standard input_cards, so there are the MAGIC standard cuts applied, i.e. the file size of the superstar DL1DH hdf5 should be much smaller.<jupyter_code>import os

for run in runlist_calibrated:
    size = os.path.getsize(run['target'])
    print("File: {}, Size: {}".format(run['target'], size))
    

for run in runlist_superstar:
    size = os.path.getsize(run['target'])
    print("File: {}, Size: {}".format(run['target'], size))<jupyter_output><empty_output><jupyter_text>Reading Data Now that we have created the HDF5 files, we can easily use DL1DataReader.<jupyter_code>from dl1_data_handler.reader import *
import matplotlib.pyplot as plt
%matplotlib inline<jupyter_output><empty_output><jupyter_text>We can create a reader by passing the path to HDF5 file. Hence, using ImageMapper, we can plot the gamma images and verify the transformation is correct<jupyter_code>parameter_selection = [{'col_name': "hillas_intensity", 'min_value':1000.0}]
selection_string = '(mc_energy > 1.0)'
reader = DL1DataReaderDL1DH([calibrated_hdf5_file],
                            mode='mono',
                            selected_telescope_types=["LST_MAGIC_MAGICCam"],
                            selection_string = selection_string,
                            parameter_selection = parameter_selection,
                            image_channels = ['charge'])
NUM_IMAGES_TO_PLOT = 10
i = 0
while i < NUM_IMAGES_TO_PLOT:
    example = reader[i]
    image = example[0]
    plt.figure()
    plt.pcolor(image[:,:,0],cmap='viridis')
    plt.axes().set_aspect('equal')
    plt.show()
    plt.close()
    i+=1
reader = DL1DataReaderDL1DH([superstar_hdf5_file],
                            mode='stereo',
                            #selected_telescope_types=["LST_MAGIC_MAGICCam"],
                            #selected_telescope_ids = {"LST_MAGIC_MAGICCam": [1]},
                            image_channels = ['charge', 'peak_time', 'image_mask'],
                            parameter_list = ['hillas_intensity', 'hillas_x'],
                            event_info = ["mc_energy", "alt", "az"])

NUM_IMAGES_TO_PLOT = 10
i = 0
while i < NUM_IMAGES_TO_PLOT:
    print("Event nr. {}".format(i+1))
    example = reader[i]
    image = example[1]
    plt.figure()
    plt.pcolor(image[0,:,:,0],cmap='viridis')
    plt.axes().set_aspect('equal')
    plt.show()
    plt.close()
    plt.pcolor(image[1,:,:,0],cmap='viridis')
    plt.axes().set_aspect('equal')
    plt.show()
    plt.close()
    plt.pcolor(image[0,:,:,1],cmap='viridis')
    plt.axes().set_aspect('equal')
    plt.show()
    plt.close()
    plt.pcolor(image[1,:,:,1],cmap='viridis')
    plt.axes().set_aspect('equal')
    plt.show()
    plt.close()
    plt.pcolor(image[0,:,:,2],cmap='viridis')
    plt.axes().set_aspect('equal')
    plt.show()
    plt.close()
    plt.pcolor(image[1,:,:,2],cmap='viridis')
    plt.axes().set_aspect('equal')
    plt.show()
    plt.close()
    i+=1<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11925495887760039, 'fraction_numerical': 0.02092404450895017, 'mean_word_length': 3.277806518365235, 'pattern_counts': {'":': 1, '<': 35, '<?xml version=': 0, '>': 34, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '551452', 'license': 'MIT', 'n_tokens_mistral': 2893, 'n_tokens_neox': 2754, 'n_words': 586, 'path': 'notebooks/MAGIC_data_demo.ipynb', 'repo_name': 'bryankim96/ROOT-imageExtractor', 'size': '8268'}