<jupyter_start><jupyter_text>Deep Learning in Earth Observation: RNN Cosine Demo   <NAME>, German Aerospace Center; <NAME>, German Aerospace Center & Technical University Munich    This notebook introduces you to the basic concepts of Deep Learning in Earth Observation. Specifically, it uses the simple example of learning the temporal pattern of a cosine curve to demonstrate the concepts of Recurrent Neural Networks (RNNs). The notebook let's you experiment with several hyper-parameters needed for training Deep Learning Networks such as RNNs, CNNs, or similar.    This notebook will introduce the following data analysis concepts:- How to set up a recurrent deep network within the Python-based keras/tensorflow environment- How to create an LSTM (long-term/short-term memory) recurrent network - How to optimize hyper-parameters when training a deep neural network  Important Note about JupyterHub  Your JupyterHub server will automatically shutdown when left idle for more than 1 hour. Your notebooks will not be lost but you will have to restart their kernels and re-run them from the beginning. You will not be able to seamlessly continue running a partially run notebook.<jupyter_code>%%javascript
var kernel = Jupyter.notebook.kernel;
var command = ["notebookUrl = ",
               "'", window.location, "'" ].join('')
kernel.execute(command)
from IPython.display import Markdown
from IPython.display import display

user = !echo $JUPYTERHUB_USER
env = !echo $CONDA_PREFIX
if env[0] == '':
    env[0] = 'Python 3 (base)'
if env[0] != '/home/jovyan/.local/envs/machine_learning':
    display(Markdown(f'<text style=color:red><strong>WARNING:</strong></text>'))
    display(Markdown(f'<text style=color:red>This notebook should be run using the "machine_learning" conda environment.</text>'))
    display(Markdown(f'<text style=color:red>It is currently using the "{env[0].split("/")[-1]}" environment.</text>'))
    display(Markdown(f'<text style=color:red>Select "machine_learning" from the "Change Kernel" submenu of the "Kernel" menu.</text>'))
    display(Markdown(f'<text style=color:red>If the "machine_learning" environment is not present, use <a href="{notebookUrl.split("/user")[0]}/user/{user[0]}/notebooks/conda_environments/Create_OSL_Conda_Environments.ipynb"> Create_OSL_Conda_Environments.ipynb </a> to create it.</text>'))
    display(Markdown(f'<text style=color:red>Note that you must restart your server after creating a new environment before it is usable by notebooks.</text>'))<jupyter_output><empty_output><jupyter_text>Predict a cosine wave using RNNs* A simple tutorial on LSTM and GRU to perdict a trigonometric wave.* Data noise can be added to test the robustness of the model.* Hyperparamters of the RNNs can be tweaked  0. Importing Relevant Python Packages  Our first step is to import the necessary python libraries into your Jupyter Notebook.<jupyter_code>%%capture
import os
from PIL import Image
import glob

import numpy as np
import matplotlib.pyplot as plt
from keras.models import Sequential
from keras.layers import Dense, Activation, LSTM, GRU, TimeDistributed
from keras.optimizers import RMSprop

from asf_notebook import new_directory<jupyter_output><empty_output><jupyter_text>1. Create a working directory for the analysis and change into it:<jupyter_code>base_path = "/home/jovyan/notebooks/SAR_Training/English/data_RNN_cosine"
new_directory(base_path)
os.chdir(base_path)
print(f"Current working directory: {os.getcwd()}")<jupyter_output><empty_output><jupyter_text>2. Get cosine dataData to train and evaluate the RNN:- Start, end and step define the range of the data series.- Sequence length defines the series to look back to train the model- Noisy data can be added to make the training data imperfect.Write a function to define training data:<jupyter_code># Takes: starting point, end point, number of steps between points
# number of steps to backpropagate through time, and noise to create imperfect data
# Returns: X,Y data


def cosine_data(start, end, step, sequence_length, noise_level=0):

    t = np.arange(start, end, step)
    cosine = np.cos(2 * np.pi * t) + noise_level * \
        np.random.normal(0, 1, np.shape(t))
    cosine = cosine.reshape((cosine.shape[0], 1))

    dX, dY = [], []
    for i in range(len(cosine) - 2*sequence_length):
        dX.append(cosine[i:i + sequence_length])
        dY.append(cosine[i + sequence_length:i + 2*sequence_length])
    dataX = np.array(dX)
    dataY = np.array(dY)
    return dataX, dataY<jupyter_output><empty_output><jupyter_text>3. Create an LSTM model- Linear activation- Loss in mean squared error Write a function that creates an LSTM model:<jupyter_code># Takes: number of neurons to train a GRU network, number of features to predict, and learning_rate
# Returns: model for training


def LSTM_(hidden_neurons, feature_count, learning_rate):
    model = Sequential()
    model.add(LSTM(hidden_neurons, return_sequences=True))
    model.add(TimeDistributed(Dense(feature_count)))
    model.add(Activation('linear'))
    optimizer = RMSprop(lr=learning_rate)
    model.compile(loss='mean_squared_error',
                  optimizer=optimizer, metrics=['mse'])
    return model<jupyter_output><empty_output><jupyter_text>3. Create a GRU model- Linear activation- Get loss using a mean squared errorWrite a function that creates a GRU model:<jupyter_code># Takes number of neurons to train a GRU network, number of features to predict, and learning_rate
# Returns: model for training


def GRU_(hidden_neurons, feature_count, learning_rate):
    model = Sequential()
    model.add(GRU(input_dim=feature_count,
                  output_dim=hidden_neurons, return_sequences=True))
    model.add(TimeDistributed(Dense(feature_count)))
    model.add(Activation('linear'))
    optimizer = RMSprop(lr=learning_rate)
    model.compile(loss='mean_squared_error',
                  optimizer=optimizer, metrics=['mse'])
    return model<jupyter_output><empty_output><jupyter_text>4. Write a function to train an RNN model:<jupyter_code># Takes: load RNN model, X cosine train data, Y cosine train data,
# number of samples to be propagated through the network, and
# number of time dataset is processed
# Returns: training and validation loss


def train_cosine(model, dataX, dataY, batch_size, epoch_count, count):

    history = model.fit(dataX, dataY, batch_size=batch_size,
                        epochs=epoch_count, validation_split=0.05)
    loss_history = history.history['loss']
    loss_history = np.array(loss_history)
    #np.savetxt("loss_history.txt", numpy_loss_history, delimiter=",")
    val_loss_history = history.history['val_loss']
    val_loss_history = np.array(val_loss_history)
    #np.savetxt("val_loss_history.txt", numpy_loss_history, delimiter=",")
    loss = history.history['loss']
    loss_val = history.history['val_loss']
    plt.rcParams.update({'font.size': 18})
    fig = plt.figure(figsize=(8, 7))
    ax = fig.add_subplot(1, 1, 1)
    plt.plot(loss)
    plt.plot(loss_val)
    plt.ylabel('loss')
    plt.xlabel('epoch')
    plt.legend(['train', 'val'], loc='upper right')
    plt.savefig(f"{os.getcwd()}/loss_{count}.png", dpi=72)
    plt.show()

    return loss_history, val_loss_history<jupyter_output><empty_output><jupyter_text>5. Write a function to run an RNN model:<jupyter_code># Takes: number of Epochs, noise level in training data, sequence length available to train a RNN,
# learning_rate, batch_size, nb_units, boolean if results should be plotted
# Return: Loss and Plot


def test_cosine(EPOCHS, count, noise_level=0.3, sequence_length=100, learning_rate=1e-3, batch_size=16, nb_units=32, plot_results=False):

    dataX, dataY = cosine_data(
        0.0, 10, 0.02, sequence_length, noise_level)  # 4.0
    # create and fit the LSTM network
    print('creating model...')

    # Choose RNN to train
    model = LSTM_(nb_units, 1, learning_rate)
    #model = GRU_(nb_units, 1, learning_rate)

    # Train RNN model
    tr_loss, val_loss = train_cosine(model, dataX, dataY, batch_size, EPOCHS, count)

    # now test
    dataX1, dataY1 = cosine_data(15.0, 21.0, 0.02, sequence_length)
    predict = model.predict(dataX1)
    if plot_results:
        plot_RNN_results(dataX, dataX1, dataY1, predict, sequence_length, count)

    return tr_loss, val_loss<jupyter_output><empty_output><jupyter_text>6. Write a function to plot RNN results:<jupyter_code>def plot_RNN_results(dataX, dataX1, dataY1, predict, sequence_length, count):
    nan_array = np.empty((sequence_length - 1))
    nan_array.fill(np.nan)
    nan_array2 = np.empty(sequence_length)
    nan_array2.fill(np.nan)
    ind = np.arange(2*sequence_length)
    plt.rcParams.update({'font.size': 18})
    fig = plt.figure(figsize=(8, 7))
    ax = fig.add_subplot(1, 1, 1)
    forecasts = np.concatenate(
        (nan_array, dataX1[0, -1:, 0], predict[0, :, 0]))
    ground_truth = np.concatenate(
        (nan_array, dataX1[0, -1:, 0], dataY1[0, :, 0]))
    network_input = np.concatenate((dataX[0, :, 0], nan_array2))

    ax.plot(ind, network_input, 'b-x', label='Network input')
    ax.plot(ind, forecasts, 'r-x', label='Many to many model forecast')
    ax.plot(ind, ground_truth, 'g-x', label='Ground truth')
    handles, labels = ax.get_legend_handles_labels()
    plt.xlabel('t')
    plt.ylabel('cos(t)')
    plt.title('Cosine Many to Many Forecast')
    text = ax.text(-0.2,1.05, " ", transform=ax.transAxes) #this is dummy text, needed by bbox_inches='tight', which requires >1 artist 
    lgd = ax.legend(handles, labels, bbox_to_anchor=(0.5, -0.1), loc='upper center')
    plt.savefig(f"{os.getcwd()}/cosine_wave_{count}.png", dpi=72, bbox_extra_artists=(lgd, text), bbox_inches='tight')
    plt.show()<jupyter_output><empty_output><jupyter_text>7. Write functions to group and save the plots created by train_cosine() and plot_RNN_results():Write a function to group the paths to the plots that will be concatenated:<jupyter_code>def group_plot_paths(count):
    paths = []
    for i in range (0, count):
        paths.append(glob.glob(f"*_{i}.*"))
    for i in range (0, len(paths)):
        paths[i].sort()
    return paths<jupyter_output><empty_output><jupyter_text>Write a function to concatenate and save the plots:<jupyter_code>def concat_plots(count):
    plot_pairs = group_plot_paths(count)
    for i in range(0, len(plot_pairs)):
        images = list(map(Image.open, plot_pairs[i]))
        widths, heights = zip(*(x.size for x in images))
        total_width = sum(widths)
        max_height = max(heights)
        x_offset = 0
        new_image = Image.new('RGBA', (total_width, max_height))
        for im in images:
            new_image.paste(im, (x_offset, 0))
            x_offset += im.size[0]
        new_image.save(f"cosine_wave_loss_{i}.png", "png")
    delete_files(plot_pairs)<jupyter_output><empty_output><jupyter_text>Write a function to delete the original seperate image files after they have been concatenated:<jupyter_code>def delete_files(files_list):
    assert type(files_list) == list
    assert type(files_list[0]) == list
    for files in files_list:
        for file in files:
            try:
                os.remove(file)
            except:
                FileNotFoundError<jupyter_output><empty_output><jupyter_text>8. Train and run the RNN model:<jupyter_code>nb_epochs = 10
noise_level = 0.0
sequence_length = 100
'''
learning_rate = 1e-3  # 1e-1, 1e-2, 1e-3, 1e-4, 1e-5
batch_size = 16  # 1, 2, 4, 8, 16
nb_units = 32  # 8, 16, 32, 64, 128

_, _ = test_cosine(EPOCHS = nb_epochs, noise_level =noise_level, 
                   plot_results=True)
'''
# try with different noise
'''
noise_level_range = [0, 0.1, 0.2, 0.3, 0.4, 0.5]
for nl in noise_level_range:
    tr_loss, val_loss = test_cosine(EPOCHS = nb_epochs, noise_level=nl)
    print('tr_loss:', tr_loss)
    print('val_loss:', val_loss)
'''

# hyperparameter: 1) lr
learning_rate_range = [1e-1, 1e-2, 1e-3, 1e-4, 1e-5]
count = 0
for lr in learning_rate_range:
    tr_loss, val_loss = test_cosine(nb_epochs, count, learning_rate=lr, plot_results=True)
    count += 1
    print('tr_loss:', tr_loss)
    print('val_loss:', val_loss)
concat_plots(len(learning_rate_range))

'''
# hyperparameter: 2) batch size
batch_size_range = [2, 4, 8, 16, 32]
for bs in batch_size_range:
    tr_loss, val_loss = test_cosine(EPOCHS = nb_epochs, batch_size=bs, plot_results=True)
    print('tr_loss:', tr_loss)
    print('val_loss:', val_loss)
'''
'''
# hyperparameter: 3) sequence length
sequence_length_range = [20, 50, 100, 200, 500]
for sl in sequence_length_range:
    tr_loss, val_loss = test_cosine(EPOCHS = nb_epochs, sequence_length=sl, plot_results=True)
    print('tr_loss:', tr_loss)
    print('val_loss:', val_loss)
'''
'''
# hyperparameter: 4) nb_units
nb_units_range = [8, 16, 32, 64, 128]
for nu in nb_units_range:
    tr_loss, val_loss = test_cosine(EPOCHS = nb_epochs, nb_units=nu, plot_results=True)
    print('tr_loss:', tr_loss)
    print('val_loss:', val_loss)
'''<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11351434269059672, 'fraction_numerical': 0.01679705476300046, 'mean_word_length': 4.48087431693989, 'pattern_counts': {'":': 0, '<': 71, '<?xml version=': 0, '>': 72, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '186495', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 4557, 'n_tokens_neox': 4355, 'n_words': 1201, 'path': 'SAR_Training/English/Master/RNN_cosine.ipynb', 'repo_name': 'danielbeny/opensarlab-notebooks', 'size': '13049'}