<jupyter_start><jupyter_text>Model Training and Evaluation Install `abwoc` and `sunpy````pip install git+https://github.com/PHYS477677/abwocpip install sunpy```And obtain the sample dataset with```curl -LJO https://github.com/PHYS477677/abwoc/raw/master/sample-data/script_testing_set*.npy yourDirectory```or download directly from https://github.com/PHYS477677/abwoc/blob/master/sample-data/script_testing_set_171_512_normalized.npy Import the necessary packages:<jupyter_code>import abwoc.xray_model as xray_model<jupyter_output><empty_output><jupyter_text>Model Training Train the model on a given dataset, assuming the sample data was downloaded to the '/sample-data/' directory this can be accomplished through the command below.Generally, the model can train on a `npy` file that is provided, as long as there is also a corresponding labels `npy` file in the same directory. i.e. for this file there must also be `script_testing_set_171_512_normalized_labels.npy` in the '/sample-data/' directory<jupyter_code>dataset_file = './sample-data/script_testing_set_171_512_normalized.npy'<jupyter_output><empty_output><jupyter_text>Specify the wavelength to use with the model as a string. The allowed options are '131', '171', and '211' for the individual wavelength models, or 'all' for the three wavelength model. Using a single wavelength for this example:<jupyter_code>wavelength = '171'<jupyter_output><empty_output><jupyter_text>Also specify the structure of the model (how many convolution layers are used). This can be '1layer', '2layer', or 'ctmodel' for 1, 2, and 3 layer models respectively.<jupyter_code>model_struct = '1layer'<jupyter_output><empty_output><jupyter_text>Enter the number of epochs to train for. Note that if the model is resuming from a checkpoint at the fifth epoch, and this value is set to five, then the model will train for five additional epochs, ending with a checkpoint file for the tenth epoch.<jupyter_code>epochs = 5<jupyter_output><empty_output><jupyter_text>Enter the directory to save the checkpoint files to. The checkpoint files will be saved using the name of this directory, so it is advised to have separate directories for each different model.<jupyter_code>checkpoint_path = './sample-data/checkpoint_testing/'<jupyter_output><empty_output><jupyter_text>Finally, choose whether to load checkpoints, denoted by a boolean, and what epoch to start training from.<jupyter_code>load_checkpoint = False
initial_epoch = 0<jupyter_output><empty_output><jupyter_text>Now the desired model can be trained, from all of the inputs above this model will:Train on the './sample-data/script_testing_set_171_512_normalized.npy' datasetCreate a single wavelength, single layer modelTrain on the data for five epochsSave model checkpoints in the directory: './sample-data/checkpoint_testing/'And begin training the model from scratchThe model will train for five epochs and output the saved weights in the checkpoint directoryA 'npy' file will also be created saving the performance of the model in loss and accuracy vs. epoch. These metrics will also be plotted<jupyter_code>xray_model.model_learn(dataset_file, wavelength, model_struct, epochs, checkpoint_path, load_checkpoint, initial_epoch)<jupyter_output>Dataset loaded successfully
Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d (Conv2D)              (None, 510, 510, 128)     1280      
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 51, 51, 128)       0         
_________________________________________________________________
flatten (Flatten)            (None, 332928)            0         
_________________________________________________________________
dense (Dense)                (None, 1)                 332929    
=================================================================
Total params: 334,209
Trainable params: 334,209
Non-trainable params: 0
_________________________________________________________________

model name is: checkpoint_testing

Epoch 1/5
9/9 [==============================] - 3s [...]<jupyter_text>Now to train the model for five more epochs, loading the most recent checkpoint:<jupyter_code>load_checkpoint = True
initial_epoch = 5

xray_model.model_learn(dataset_file, wavelength, model_struct, epochs, checkpoint_path, load_checkpoint, initial_epoch)<jupyter_output>Dataset loaded successfully
Model: "sequential_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_1 (Conv2D)            (None, 510, 510, 128)     1280      
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 51, 51, 128)       0         
_________________________________________________________________
flatten_1 (Flatten)          (None, 332928)            0         
_________________________________________________________________
dense_1 (Dense)              (None, 1)                 332929    
=================================================================
Total params: 334,209
Trainable params: 334,209
Non-trainable params: 0
_________________________________________________________________

model name is: checkpoint_testing

Epoch 6/10
9/9 [==============================] - [...]<jupyter_text>Model Evaluation The performance of the model can then be evaluated using a given dataset and checkpoint.Using the model tested above, many of the required inputs are the same as training the model.<jupyter_code>dataset_file = './sample-data/script_testing_set_171_512_normalized.npy'
wavelength = '171'
model_struct = '1layer'
checkpoint_path = './sample-data/checkpoint_testing/'<jupyter_output><empty_output><jupyter_text>To evaluate the performance of the model at a single epoch, the start and end epochs must be defined as the same value. Entering -1 as the epoch will evaluate the performance of the most recently saved checkpoint.<jupyter_code>start_epoch = -1
end_epoch = -1<jupyter_output><empty_output><jupyter_text>The below function will then output the accuracy and loss metrics for the most recently saved checkpoint.<jupyter_code>xray_model.model_evaluate(dataset_file, wavelength, model_struct, checkpoint_path, start_epoch, end_epoch)<jupyter_output>Dataset loaded successfully
Model: "sequential_2"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_2 (Conv2D)            (None, 510, 510, 128)     1280      
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 51, 51, 128)       0         
_________________________________________________________________
flatten_2 (Flatten)          (None, 332928)            0         
_________________________________________________________________
dense_2 (Dense)              (None, 1)                 332929    
=================================================================
Total params: 334,209
Trainable params: 334,209
Non-trainable params: 0
_________________________________________________________________

model name is: checkpoint_testing

12/12 - 1s - loss: 0.6136 - accuracy: 1.0000
model[...]<jupyter_text>To instead evaluate the performance of a range of checkpoint files, the start and end epochs can be defined<jupyter_code>start_epoch = 6
end_epoch = 8<jupyter_output><empty_output><jupyter_text>Now, the ```model_evaluate()``` function will output a plot showing the accuracy and loss metrics for checkpoints saved in the third through the seventh epochs<jupyter_code>xray_model.model_evaluate(dataset_file, wavelength, model_struct, checkpoint_path, start_epoch, end_epoch)<jupyter_output>Dataset loaded successfully
Model: "sequential_3"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_3 (Conv2D)            (None, 510, 510, 128)     1280      
_________________________________________________________________
max_pooling2d_3 (MaxPooling2 (None, 51, 51, 128)       0         
_________________________________________________________________
flatten_3 (Flatten)          (None, 332928)            0         
_________________________________________________________________
dense_3 (Dense)              (None, 1)                 332929    
=================================================================
Total params: 334,209
Trainable params: 334,209
Non-trainable params: 0
_________________________________________________________________

model name is: checkpoint_testing

12/12 - 1s - loss: 0.6925 - accuracy: 0.5000
12/12[...]
{'fraction_non_alphanumeric': 0.12441886207659951, 'fraction_numerical': 0.038189063537746294, 'mean_word_length': 5.226740179186768, 'pattern_counts': {'":': 0, '<': 53, '<?xml version=': 0, '>': 53, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '427533', 'license': 'Apache-2.0', 'n_tokens_mistral': 2467, 'n_tokens_neox': 2150, 'n_words': 729, 'path': 'demos/model_train_eval.ipynb', 'repo_name': 'PHYS477677/Astro-team', 'size': '9034'}