<jupyter_start><jupyter_text>Notebook for analyzing synthetic radar resultsThis notebook makes use of resources from the github profile of MIT AI Accelerator available at https://github.com/MIT-AI-Accelerator/neurips-2020-sevir.git. We will be cloning this repository and accessing its contents for running this notebook as shown in the following cells.<jupyter_code>!git clone https://github.com/MIT-AI-Accelerator/neurips-2020-sevir.git<jupyter_output>Cloning into 'neurips-2020-sevir'...
remote: Enumerating objects: 386, done.[K
remote: Counting objects: 100% (386/386), done.[K
remote: Compressing objects: 100% (205/205), done.[K
remote: Total 386 (delta 181), reused 369 (delta 169), pack-reused 0[K
Receiving objects: 100% (386/386), 1.58 MiB | 4.96 MiB/s, done.
Resolving deltas: 100% (181/181), done.<jupyter_text>We will moving the files accessed from this github profile into our google cloud storage bucket by authenticating it as shown below.<jupyter_code>from google.colab import auth
auth.authenticate_user()

# https://cloud.google.com/resource-manager/docs/creating-managing-projects
project_id = 'sevir-data-pipeline'
!gcloud config set project {project_id}<jupyter_output>Updated property [core/project].<jupyter_text>Intializing the bucket name and project name<jupyter_code>bucket_name = 'sevir-generated-files'<jupyter_output><empty_output><jupyter_text>The gsutil function is used to access the google storage bucket and access the files moved into the bucket earlier<jupyter_code>#We have already created a GCP bucket with the below
bucket_name = 'sevir-generated-files'
#Copying changed src files and others to GCP Bucket
!gsutil cp -r /content/neurips-2020-sevir gs://{bucket_name}/
  

#The model weights are directly uploaded on the bucket - so we will access that here

!gsutil cp -r gs://{bucket_name}/neurips-2020-sevir/models/ /content/neurips-2020-sevir/models/
import os
os.environ["HDF5_USE_FILE_LOCKING"]='FALSE'
import sys
sys.path.append('../src/')
import h5py
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd<jupyter_output>/usr/local/lib/python3.7/dist-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.8) or chardet (3.0.4) doesn't match a supported version!
  RequestsDependencyWarning)<jupyter_text>Plot metricsWe will be plotting the metrics for Synrad use case below:<jupyter_code># Read the metrics output during training
log_files= {
    'mse':'/content/neurips-2020-sevir/logs/sample-mse/metrics.csv',
    'mse+vgg':'/content/neurips-2020-sevir/logs/sample-mse-vgg/metrics.csv',
    'gan':'/content//neurips-2020-sevir/logs/sample-gan-mae/metrics.csv'
}
metrics={ k:pd.read_csv(v) for k,v in log_files.items()}

# MSE
fig,ax=plt.subplots(2,3,figsize=(15,6))
def plot_metrics_row(df,metric_names,ax):
    for k,m in enumerate(metric_names):
        df[[m,'val_'+m]].plot(ax=ax[k])
        ax[k].grid(True,alpha=.25)

plot_metrics_row(metrics['mse'],['pod74','sucr74','csi74'],ax[0])
plot_metrics_row(metrics['mse'],['pod133','sucr133','csi133'],ax[1])<jupyter_output><empty_output><jupyter_text>With MSE loss, the model converges in relatively few epochs when using the full SEVIR training dataset.  In this case, each epoch represents 75 batches of 32 SEVIR samples.  In this case, roughly 20 epochs seems like enough for peak performance on the validation set (orange line) in a variety of metrics.  The training loss continues to improve over time, but this is likely overfitting based on the stagnant validation curve.<jupyter_code># mse+mse
fig,ax=plt.subplots(2,3,figsize=(15,6))
def plot_metrics_row(df,metric_names,ax):
    for k,m in enumerate(metric_names):
        df[[m,'val_'+m]].plot(ax=ax[k])
        ax[k].grid(True,alpha=.25)

plot_metrics_row(metrics['mse+vgg'],['output_layer_1_pod74','output_layer_1_sucr74','output_layer_1_csi74'],ax[0])
plot_metrics_row(metrics['mse+vgg'],['output_layer_1_pod133','output_layer_1_sucr133','output_layer_1_csi133'],ax[1])<jupyter_output><empty_output><jupyter_text>This model also converged in ~20 epochs.<jupyter_code># gan
fig,ax=plt.subplots(1,4,figsize=(15,4))
def plot_metrics_row(df,metrics,ax):
    for k,m in enumerate(metrics):
        df[[m]].plot(ax=ax[k])
        ax[k].grid(True,alpha=.25)

plot_metrics_row(metrics['gan'],['gen_total_loss', 'gen_gan_loss', 'gen_l1_loss', 'disc_loss'],ax)<jupyter_output><empty_output><jupyter_text>The four losses involved with training the cGAN version of the loss.  No validation set was used for this experiement.  At around spoch 50, the discriminator started competiting with the generator.     Generate sample imagesWe will now be using the Synrad generator to create a sample dataset and use that to predict images for Synrad use case.<jupyter_code>pip install boto3
import boto3
from botocore.handlers import disable_signing
resource = boto3.resource('s3')
resource.meta.client.meta.events.register('choose-signer.s3.*', disable_signing)
bucket=resource.Bucket('sevir')<jupyter_output><empty_output><jupyter_text>We will use Boto to download SEVIR storm events .h5 files.<jupyter_code>bucket.download_file('data/vil/2019/SEVIR_VIL_STORMEVENTS_2019_0101_0630.h5','/content/neurips-2020-sevir/data/vil/2019/SEVIR_VIL_STORMEVENTS_2019_0101_0630.h5')
bucket.download_file('data/ir069/2019/SEVIR_IR069_STORMEVENTS_2019_0101_0630.h5','/content/neurips-2020-sevir/data/ir069/2019/SEVIR_IR069_STORMEVENTS_2019_0101_0630.h5')
bucket.download_file('data/ir107/2019/SEVIR_IR107_STORMEVENTS_2019_0101_0630.h5','/content/neurips-2020-sevir/data/ir107/2019/SEVIR_IR107_STORMEVENTS_2019_0101_0630.h5')
bucket.download_file('data/vis/2019/SEVIR_VIS_STORMEVENTS_2019_0601_0630.h5','/content/neurips-2020-sevir/data/vis/2019/SEVIR_VIS_STORMEVENTS_2019_0601_0630.h5')
bucket.download_file('data/lght/2019/SEVIR_LGHT_ALLEVENTS_2019_0601_0701.h5','/content/neurips-2020-sevir/data/lght/2019/SEVIR_LGHT_ALLEVENTS_2019_0601_0701.h5')<jupyter_output><empty_output><jupyter_text>We will move these files into Google storage buckets.<jupyter_code>!gsutil cp /content/neurips-2020-sevir/data/ir069/2019/SEVIR_IR069_STORMEVENTS_2019_0101_0630.h5 gs://{bucket_name}/neurips-2020-sevir/data/ir069/2019/
!gsutil cp /content/neurips-2020-sevir/data/ir107/2019/SEVIR_IR107_STORMEVENTS_2019_0101_0630.h5 gs://{bucket_name}/neurips-2020-sevir/data/ir107/2019/
!gsutil cp /content/neurips-2020-sevir/data/vis/2019/SEVIR_VIS_STORMEVENTS_2019_0601_0630.h5 gs://{bucket_name}/neurips-2020-sevir/data/vis/2019/
!gsutil cp /content/neurips-2020-sevir/data/vil/2019/SEVIR_VIL_STORMEVENTS_2019_0101_0630.h5 gs://{bucket_name}/neurips-2020-sevir/data/vil/2019/
!gsutil cp /content/neurips-2020-sevir/data/lght/2019/SEVIR_LGHT_ALLEVENTS_2019_0601_0701.h5 gs://{bucket_name}/neurips-2020-sevir/data/lght/2019/<jupyter_output>Copying file:///content/neurips-2020-sevir/data/ir069/2019/SEVIR_IR069_STORMEVENTS_2019_0101_0630.h5 [Content-Type=application/octet-stream]...
/ [0 files][    0.0 B/  2.4 GiB]                                                
==> NOTE: You are uploading one or more large file(s), which would run
significantly faster if you enable parallel composite uploads. This
feature can be enabled by editing the
"parallel_composite_upload_threshold" value in your .boto
configuration file. However, note that if you do this large files will
be uploaded as `composite objects
<https://cloud.google.com/storage/docs/composite-objects>`_,which
means that any user who downloads such objects will need to have a
compiled crcmod installed (see "gsutil help crcmod"). This is because
without a compiled crcmod, computing checksums on composite objects is
so slow that gsutil disables downloads of composite objects.

\
Operation completed over 1 objects/2.4 GiB.<jupyter_text>This runs the trained model on samples from the test set.  We will use basic color maps for this demo.<jupyter_code>#CATALOG.csv was directly uploaded on the bucket - so we will access that here
!gsutil cp gs://{bucket_name}/neurips-2020-sevir/data/CATALOG.csv /content/neurips-2020-sevir/data/
cd /content/neurips-2020-sevir/src/data/<jupyter_output>/content/neurips-2020-sevir/src/data<jupyter_text>We will now run the make_synrad_dataset function which wil create the synrad dataset and save it to the runtime. We will move it to GCP buckets to so as to access it later in our prediction.<jupyter_code>!python make_synrad_dataset.py --sevir_data '/content/neurips-2020-sevir/data/' --sevir_catalog '/content/neurips-2020-sevir/data/CATALOG.csv' --output_location '/content/neurips-2020-sevir/data/interim/'<jupyter_output>/usr/local/lib/python3.7/dist-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.8) or chardet (3.0.4) doesn't match a supported version!
  RequestsDependencyWarning)
2022-02-23 19:07:08,771 - __main__ - INFO - making final data set from raw data
2022-02-23 19:07:09,459 - __main__ - INFO - Reading/writing training data to /content/neurips-2020-sevir/data/interim//synrad_training.h5
2022-02-23 19:07:09,459 - __main__ - INFO - Reading/writing testing data to /content/neurips-2020-sevir/data/interim//synrad_testing.h5
2022-02-23 19:07:09,459 - __main__ - INFO - Gathering chunk 0/20:
100% 3/3 [00:00<00:00,  3.06it/s]
2022-02-23 19:07:10,510 - __main__ - INFO - Gathering chunk 1/20:
100% 3/3 [00:00<00:00,  3.06it/s]
2022-02-23 19:07:11,548 - __main__ - INFO - Gathering chunk 2/20:
100% 3/3 [00:00<00:00,  3.35it/s]
2022-02-23 19:07:12,496 - __main__ - INFO - Gathering chunk 3/20:
100% 3/3 [00:00<00:00,  3.21it/s]
2022-02-23 19:07:13,482 - __main__ - INFO - Gathering c[...]<jupyter_text>We will do the same with nowcast and generate the dataset for nowcast use case<jupyter_code>!python make_nowcast_dataset.py --sevir_data '/content/neurips-2020-sevir/data/' --sevir_catalog '/content/neurips-2020-sevir/data/CATALOG.csv' --output_location '/content/neurips-2020-sevir/data/interim/'
# Load weights from best model on val set
mse_weights_file = '/content/neurips-2020-sevir/models/models/mse_weights.h5'
mse_model = tf.keras.models.load_model(mse_weights_file,compile=False,custom_objects={"tf": tf})

mse_vgg_weights_file = '/content/neurips-2020-sevir/models/models/mse_vgg_weights.h5'
mse_vgg_model = tf.keras.models.load_model(mse_vgg_weights_file,compile=False,custom_objects={"tf": tf})

gan_weights_file = '/content/neurips-2020-sevir/models/models/gan_mae_weights.h5'
gan_model = tf.keras.models.load_model(gan_weights_file,compile=False,custom_objects={"tf": tf})
#Moving the generated data files to GCP Bucket

!gsutil cp -r /content/neurips-2020-sevir/data/interim/ gs://{bucket_name}/neurips-2020-sevir/data/interim/<jupyter_output>Copying file:///content/neurips-2020-sevir/data/interim/.gitkeep [Content-Type=application/octet-stream]...
Copying file:///content/neurips-2020-sevir/data/interim/nowcast_testing.h5 [Content-Type=application/octet-stream]...
==> NOTE: You are uploading one or more large file(s), which would run
significantly faster if you enable parallel composite uploads. This
feature can be enabled by editing the
"parallel_composite_upload_threshold" value in your .boto
configuration file. However, note that if you do this large files will
be uploaded as `composite objects
<https://cloud.google.com/storage/docs/composite-objects>`_,which
means that any user who downloads such objects will need to have a
compiled crcmod installed (see "gsutil help crcmod"). This is because
without a compiled crcmod, computing checksums on composite objects is
so slow that gsutil disables downloads of composite objects.

Copying file:///content/neurips-2020-sevir/data/interim/synrad_testing.h5 [Content-Type=applicatio[...]<jupyter_text>Load sample test dataWe will now use the test data that we generated and use that to predict the data for synrad and nowcast dataset<jupyter_code>!python /content/neurips-2020-sevir/src/readers/synrad_reader.py
cd /content/neurips-2020-sevir/src/readers/
# Load a part of the test dataset
from synrad_reader import read_data
x_test,y_test = read_data('/content/neurips-2020-sevir/data/interim/synrad_testing.h5',end=1000)<jupyter_output><empty_output><jupyter_text>Visualize results on some test samplesWe will be visualizing some of the test samples here<jupyter_code># Run model on test set
def run_synrad(model,x_test,batch_size=32):
    return model.predict([x_test[k] for k in ['ir069','ir107','lght']],batch_size=batch_size)
y_pred_mse     = run_synrad(mse_model,x_test)
y_pred_mse_vgg = run_synrad(mse_vgg_model,x_test)
y_pred_gan     = run_synrad(gan_model,x_test)
!python /content/neurips-2020-sevir/src/display/display.py
cd /content/neurips-2020-sevir/src/display/
# Plot using default cmap
from display import get_cmap
def visualize_result(y_test,y_preds,idx,ax):
    cmap_dict = lambda s: {'cmap':get_cmap(s,encoded=True)[0], 'norm':get_cmap(s,encoded=True)[1],
                           'vmin':get_cmap(s,encoded=True)[2], 'vmax':get_cmap(s,encoded=True)[3]}
    ax[0].imshow(x_test['ir069'][idx,:,:,0],**cmap_dict('ir069'))
    ax[1].imshow(x_test['ir107'][idx,:,:,0],**cmap_dict('ir107'))
    ax[2].imshow(x_test['lght'][idx,:,:,0],cmap='hot',vmin=0,vmax=10)
    ax[3].imshow(y_test['vil'][idx,:,:,0],**cmap_dict('vil'))
    for k in range(len(y_preds)):
        if isinstance(y_preds[k],(list,)):
            yp=y_preds[k][0]
        else:
            yp=y_preds[k]
        ax[4+k].imshow(yp[idx,:,:,0],**cmap_dict('vil'))
    for i in range(len(ax)):
        ax[i].xaxis.set_ticks([])
        ax[i].yaxis.set_ticks([])
    
test_idx = [123,456,789]
N=len(test_idx)
fig,ax = plt.subplots(N,7,figsize=(12,4))
for k,i in enumerate(test_idx):
    visualize_result(y_test,[y_pred_mse,y_pred_mse_vgg,y_pred_gan], i, ax[k] )

ax[0][0].set_title('Input ir069')
ax[0][1].set_title('Input ir107')
ax[0][2].set_title('Input lght')
ax[0][3].set_title('Truth')
ax[0][4].set_title('Output\nMSE Loss')
ax[0][5].set_title('Output\nMSE+VGG Loss')
ax[0][6].set_title('Output\nGAN+MAE Loss')
plt.subplots_adjust(top=0.92, bottom=0.08, left=0.10, right=0.95, hspace=0.05,
                    wspace=0.35)

import os
os.environ["HDF5_USE_FILE_LOCKING"]='FALSE'
import sys
sys.path.append('../src/')
import h5py
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
from matplotlib import cm
from matplotlib.colors import ListedColormap
from matplotlib.patches import Patch
from matplotlib.lines import Line2D
import matplotlib.patches as patches
import pandas as pd
#from display.display import get_cmap
# Load pretrained nowcasting models
mse_file  = '/content/neurips-2020-sevir/models/models/mse_model.h5'
mse_model = tf.keras.models.load_model(mse_file,compile=False,custom_objects={"tf": tf})

style_file = '/content/neurips-2020-sevir/models/models/style_model.h5'
style_model = tf.keras.models.load_model(style_file,compile=False,custom_objects={"tf": tf})

mse_style_file = '/content/neurips-2020-sevir/models/models/mse_and_style.h5'
mse_style_model = tf.keras.models.load_model(mse_style_file,compile=False,custom_objects={"tf": tf})

gan_file = '/content/neurips-2020-sevir/models/models/gan_generator.h5'
gan_model = tf.keras.models.load_model(gan_file,compile=False,custom_objects={"tf": tf})
!gsutil cp -r gs://{bucket_name}/neurips-2020-sevir/data/interim/ /content/neurips-2020-sevir/data/interim/
!python /content/neurips-2020-sevir/src/readers/nowcast_reader.py
cd /content/neurips-2020-sevir/src/readers/
# Load a part of the test dataset
from nowcast_reader import read_data
x_test,y_test = read_data('/content/neurips-2020-sevir/data/interim/interim/interim/nowcast_testing.h5',end=50)
## 
# Functions for plotting results
##

norm = {'scale':47.54,'shift':33.44}
hmf_colors = np.array( [
    [82,82,82], 
    [252,141,89],
    [255,255,191],
    [145,191,219]
])/255

# Model that implements persistence forecast that just repeasts last frame of input
class persistence:
    def predict(self,x_test):
        return np.tile(x_test[:,:,:,-1:],[1,1,1,12])

def plot_hit_miss_fa(ax,y_true,y_pred,thres):
    mask = np.zeros_like(y_true)
    mask[np.logical_and(y_true>=thres,y_pred>=thres)]=4
    mask[np.logical_and(y_true>=thres,y_pred<thres)]=3
    mask[np.logical_and(y_true<thres,y_pred>=thres)]=2
    mask[np.logical_and(y_true<thres,y_pred<thres)]=1
    cmap=ListedColormap(hmf_colors)
    ax.imshow(mask,cmap=cmap)


def visualize_result(models,x_test,y_test,idx,ax,labels):
    fs=10
    cmap_dict = lambda s: {'cmap':get_cmap(s,encoded=True)[0],
                           'norm':get_cmap(s,encoded=True)[1],
                           'vmin':get_cmap(s,encoded=True)[2],
                           'vmax':get_cmap(s,encoded=True)[3]}
    for i in range(1,13,3):
        xt = x_test[idx,:,:,i]*norm['scale']+norm['shift']
        ax[(i-1)//3][0].imshow(xt,**cmap_dict('vil'))
    ax[0][0].set_title('Inputs',fontsize=fs)
    
    pers = persistence().predict(x_test[idx:idx+1])
    pers = pers*norm['scale']+norm['shift']
    x_test = x_test[idx:idx+1]
    y_test = y_test[idx:idx+1]*norm['scale']+norm['shift']
    y_preds=[]
    for i,m in enumerate(models):
        yp = m.predict(x_test)
        if isinstance(yp,(list,)):
            yp=yp[0]
        y_preds.append(yp*norm['scale']+norm['shift'])
    
    for i in range(0,12,3):
        ax[i//3][2].imshow(y_test[0,:,:,i],**cmap_dict('vil'))
    ax[0][2].set_title('Target',fontsize=fs)
    
    # Plot Persistence
    for i in range(0,12,3):
        plot_hit_miss_fa(ax[i//3][4],y_test[0,:,:,i],pers[0,:,:,i],74)
    ax[0][4].set_title('Persistence\nScores',fontsize=fs)
    
    for k,m in enumerate(models):
        for i in range(0,12,3):
            ax[i//3][5+2*k].imshow(y_preds[k][0,:,:,i],**cmap_dict('vil'))
            plot_hit_miss_fa(ax[i//3][5+2*k+1],y_test[0,:,:,i],y_preds[k][0,:,:,i],74)

        ax[0][5+2*k].set_title(labels[k],fontsize=fs)
        ax[0][5+2*k+1].set_title(labels[k]+'\nScores',fontsize=fs)
        
    for j in range(len(ax)):
        for i in range(len(ax[j])):
            ax[j][i].xaxis.set_ticks([])
            ax[j][i].yaxis.set_ticks([])
    for i in range(4):
        ax[i][1].set_visible(False)
    for i in range(4):
        ax[i][3].set_visible(False)
    ax[0][0].set_ylabel('-45 Minutes')
    ax[1][0].set_ylabel('-30 Minutes')
    ax[2][0].set_ylabel('-15 Minutes')
    ax[3][0].set_ylabel('  0 Minutes')
    ax[0][2].set_ylabel('+15 Minutes')
    ax[1][2].set_ylabel('+30 Minutes')
    ax[2][2].set_ylabel('+45 Minutes')
    ax[3][2].set_ylabel('+60 Minutes')
    
    legend_elements = [Patch(facecolor=hmf_colors[1], edgecolor='k', label='False Alarm'),
                   Patch(facecolor=hmf_colors[2], edgecolor='k', label='Miss'),
                   Patch(facecolor=hmf_colors[3], edgecolor='k', label='Hit')]
    ax[-1][-1].legend(handles=legend_elements, loc='lower right', bbox_to_anchor= (-5.4, -.35), 
                           ncol=5, borderaxespad=0, frameon=False, fontsize='16')
    plt.subplots_adjust(hspace=0.05, wspace=0.05)
    
from display import get_cmap
idx=25 # adjust this to pick a case
fig,ax = plt.subplots(4,13,figsize=(24,8), gridspec_kw={'width_ratios': [1,.2,1,.2,1,1,1,1,1,1,1,1,1]})
visualize_result([mse_model,style_model,mse_style_model,gan_model],x_test,y_test,idx,ax,labels=['MSE','SC','MSE+SC','cGAN+MAE'])
idx=45 # adjust this to pick a case
fig,ax = plt.subplots(4,13,figsize=(24,8), gridspec_kw={'width_ratios': [1,.2,1,.2,1,1,1,1,1,1,1,1,1]})
visualize_result([mse_model,style_model,mse_style_model,gan_model],x_test,y_test,idx,ax,labels=['MSE','SC','MSE+SC','cGAN+MAE'])
idx=32 # adjust this to pick a case
fig,ax = plt.subplots(4,13,figsize=(24,8), gridspec_kw={'width_ratios': [1,.2,1,.2,1,1,1,1,1,1,1,1,1]})
visualize_result([mse_model,style_model,mse_style_model,gan_model],x_test,y_test,idx,ax,labels=['MSE','SC','MSE+SC','cGAN+MAE'])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14532366857948253, 'fraction_numerical': 0.06553911205073996, 'mean_word_length': 5.206497969384567, 'pattern_counts': {'":': 7, '<': 64, '<?xml version=': 0, '>': 62, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '349754', 'license': 'MIT', 'n_tokens_mistral': 8291, 'n_tokens_neox': 7452, 'n_words': 1358, 'path': 'Assignment2/notebooks/Analyzing_SEVIR_UseCases.ipynb', 'repo_name': 'shikashyam/BigDataSystemsCoursework', 'size': '19866'}