<jupyter_start><jupyter_text>Introduction While a script to run the classification is available in this folder, this notebook performs nonetheless a classification. The idea is to have an environment to comment the choices and the results obtained.  Classification on Simulated Since the only data available to train a classifier come from the [ExtremeWeather dataset](https://extremeweatherdataset.github.io), I shall us it. The preparation for the train dataset is not performed here though.This data loaded here are two sets of $120\times120$ images, depicting either a cyclon (positive) or not a cyclon (negative). A notice: the actual ExtremeWeather dataset distinguishes between land and seas cyclones, while here the distinction is not taken. For more information as to the data was prepared, that is how the images were selected starting from the whole globe image, see the script ```prepare_data.py```. **NB** For some reason, if I run the training using the notebook, the model does not learn. I thus decided just run the train script from the notebook. The different settings can be implemented using the ```config_var.yml``` available in this folder.<jupyter_code># slicing on 15
!python3 train_classifier.py
# slicing on 12
!python3 train_classifier.py
# slicing on 18
!python3 train_classifier.py
# slicing on 10
!python3 train_classifier.py
# slicing on 5
!python3 train_classifier.py
# slicing on 2
!python3 train_classifier.py<jupyter_output>[INFO] Loading vars
[INFO] Loading data from hdf5 file.
[INFO] RAM usage for data array: 10.54GB
[INFO] Preparing train-test split.
[INFO] Loading base VGG model.
[INFO] One-dimensional channel mode activated
2021-04-28 14:42:31.396473: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-04-28 14:42:31.397183: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-04-28 14:42:31.414666: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:196] None of the MLIR optimization passes are enabled (registered 0 passes)
[INFO] Using VGG as feature extractor and then training new model.
train_classifier.py:128: UserWarning: This method uses more memory. If you want to [...]<jupyter_text>So, apparently a very high slicing is the thing that works best.<jupyter_code># slicing on 0
!python3 train_classifier.py<jupyter_output>[INFO] Loading vars
[INFO] Loading data from hdf5 file.
[INFO] RAM usage for data array: 10.54GB
[INFO] Preparing train-test split.
[INFO] Loading base VGG model.
[INFO] One-dimensional channel mode activated
2021-04-28 14:48:10.467511: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-04-28 14:48:10.467698: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-04-28 14:48:10.478491: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:196] None of the MLIR optimization passes are enabled (registered 0 passes)
[INFO] Using VGG as feature extractor and then training new model.
train_classifier.py:128: UserWarning: This method uses more memory. If you want to [...]<jupyter_text>Adding too many layers is the lower accuracy that I have obtained.<jupyter_code># slicing on 4
!python3 train_classifier.py<jupyter_output>[INFO] Loading vars
[INFO] Loading data from hdf5 file.
[INFO] RAM usage for data array: 10.54GB
[INFO] Preparing train-test split.
[INFO] Loading base VGG model.
[INFO] One-dimensional channel mode activated
2021-04-28 14:54:27.027826: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-04-28 14:54:27.028099: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-04-28 14:54:27.043763: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:196] None of the MLIR optimization passes are enabled (registered 0 passes)
[INFO] Using VGG as feature extractor and then training new model.
train_classifier.py:128: UserWarning: This method uses more memory. If you want to [...]<jupyter_text>-----From these results, it may appear that the best approach would be to slice VGG at the end either of the first or second stack of convolutional operations, that is at position "4". Thus, this will be selected as the best model to perform subsequent operations.  Test on Simulated (2004) In order to see the classifier's ability to actually predict cyclones, I decide to test it using another year (2004) of the same data. In this context, the test shall be performed on the classifier trained just with **U850** variable (so data with just one channel).<jupyter_code>import numpy as np
import h5py
import matplotlib.pyplot as plt
import cartopy.crs as ccrs
import seaborn as sns
import pandas as pd
import matplotlib as mpl
# import cv2
from datetime import datetime

from sklearn.metrics import (roc_curve, auc, classification_report,
                             confusion_matrix)

from tensorflow.keras.models import load_model, Model
from tensorflow.keras.applications.vgg16 import VGG16
from tensorflow.keras.applications.vgg16 import preprocess_input
from tensorflow.keras.layers import Input, Concatenate

# this way I do not have to reshape the data
from keras.backend import set_image_data_format, image_data_format
from tensorflow.compat.v1 import disable_eager_execution
from tensorflow import executing_eagerly
from tensorflow.python.compiler.mlcompute import mlcompute

# import jtplot submodule from jupyterthemes
from jupyterthemes import jtplot

set_image_data_format('channels_first')
assert image_data_format(
) == 'channels_first', "Plase set keras backend as channels first."

# APPLE SPECIFIC IMPLEMENTATION
disable_eager_execution()
assert executing_eagerly(
) is False, "Please do not set to eager execution for this program."

# Import mlcompute module to use the optional set_mlc_device API for device selection with ML Compute.
mlcompute.set_mlc_device(
    device_name='gpu')  # Available options are 'cpu', 'gpu', and 'any'.

# currently installed theme will be used to
# set plot style if no arguments provided
# jtplot.style()
test_paths = ["./data.nosync/train_pos_2.h5", "./data.nosync/train_neg_2.h5"]
test_data = np.vstack((np.array(h5py.File(test_paths[0])['positive']),
                          np.array(h5py.File(test_paths[1])['negative'])))

test_labels = np.append(np.zeros(len(h5py.File(test_paths[0])['positive'])) + 1,
    np.zeros(len(h5py.File(test_paths[1])['negative'])))<jupyter_output><empty_output><jupyter_text>Since I am only testing, there is no need to shuffle the data.<jupyter_code># select only U850
test_data = test_data[:,[0], :,:]
# this few layer ahead are needed, since the VGG16 classifier takes only 3 channel images
# (I just take 3 identical channels)
print("Loading VGG (with concatenation of the single channel)")
new_input = Input(shape=test_data.shape[1:])
img_conc = Concatenate(axis = 1)([new_input, new_input, new_input])  
VGG_model = VGG16(include_top=False, input_tensor=img_conc)
base_model = Model(inputs = VGG_model.input, outputs = VGG_model.layers[15].output)

# the model is loaded. This is the part trained locally
print("Load pre-trained CNN classifier for cyclons (U850 specific)")
U850_model = load_model("./trained_models/model_just_850_4")

print(
    "Since the real data depicts all cyclons, we are sure the correct class to be '1'."
)

print("Accuracy on test set")
print("ACC: %.3f" % (U850_model.evaluate(base_model.predict(test_data), test_labels))[-1])<jupyter_output>Accuracy on test set<jupyter_text>Even on new data (and from a different year), and with just the usage of a single channel, the model manages to achieve very high accuracies. Some plots shall be given.<jupyter_code>predicted_test = U850_model.predict(base_model.predict(test_data))
fpr_keras, tpr_keras, thresholds_keras = roc_curve(test_labels, predicted_test)
auc_keras = auc(fpr_keras, tpr_keras)
print("ROC curve plot")
plt.figure(figsize = (8,8))
plt.plot([0, 1], [0, 1], 'k--')
plt.plot(fpr_keras, tpr_keras, label='U850 classifier (area = {:.3f})'.format(auc_keras))
plt.xlabel('False positive rate')
plt.ylabel('True positive rate')
plt.title('ROC curve')
plt.legend(loc='best')

plt.savefig("./images/ROC.pdf")
plt.show()
plt.close()
print("Test set size: %i" %(test_data.shape[0]))
optimal_idx = np.argmax(tpr_keras - fpr_keras)
optimal_threshold = thresholds_keras[optimal_idx]
print("Optimal Threshold according to Youden's J-score: %.3f" %(optimal_threshold))
cm = confusion_matrix(test_labels, (predicted_test + optimal_threshold).astype(int))
classes = ['Cyclone', 'No Cyclone']

fig, ax = plt.subplots()
im = ax.imshow(cm, cmap=plt.cm.Blues)

ax.figure.colorbar(im, ax=ax)
ax.set(xticks=np.arange(cm.shape[1]),
       yticks=np.arange(cm.shape[0]),
       xticklabels=classes,
       yticklabels=classes,
       ylabel='True label',
       xlabel='Predicted label')

for i in range(cm.shape[0]):
    for j in range(cm.shape[1]):
        ax.text(j, i, format(cm[i, j], 'd'),
                ha="center", va="center",
                color="white" if cm[i, j] > cm.max() / 2. else "black")
plt.savefig("./images/confusion_matrix_test.pdf")
plt.show()
plt.close()<jupyter_output><empty_output><jupyter_text>As can be seen from the confusion matrix, the algorithm classifies non-cyclones are cyclones far more likely than the opposite, that is the amount of **false negative** is much smaller than that of **false positives**. Given the task at hand, this result is quite remarkable, since it is much safer to classify a non-cyclone image as having a cyclone than the opposite. Also, in a real-world scenario, the usage of image-recognition techniques, e.g. see if the cyclone is detected in consecutive days, could be applied.<jupyter_code>print(classification_report(test_labels, (predicted_test + optimal_threshold).astype(int)))<jupyter_output>precision    recall  f1-score   support

         0.0       0.88      0.94      0.91      6337
         1.0       0.93      0.87      0.90      6044

    accuracy                           0.90     12381
   macro avg       0.91      0.90      0.90     12381
weighted avg       0.91      0.90      0.90     12381
{'fraction_non_alphanumeric': 0.0889132821075741, 'fraction_numerical': 0.037596048298573, 'mean_word_length': 4.890625, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 25, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '255831', 'license': 'MIT', 'n_tokens_mistral': 3636, 'n_tokens_neox': 3318, 'n_words': 1210, 'path': 'Model_Traning&Test.ipynb', 'repo_name': 'LeonardoAlchieri/TranferLearningWeather', 'size': '10932'}