<jupyter_start><jupyter_text>Example Burgers' equation In this notebook we provide a simple example of the DeepMoD algorithm by applying it on the Burgers' equation. We start by importing the required libraries and setting the plotting style:<jupyter_code>import numpy as np
import pandas as pd 


from deepymod.DeepMoD import DeepMoD
from deepymod.library_functions import library_1D
from deepymod.utilities import library_matrix_mat, print_PDE, tensorboard_to_dataframe

import matplotlib.pyplot as plt
plt.style.use('seaborn-notebook')

np.random.seed(42)  # setting seed for randomisation<jupyter_output><empty_output><jupyter_text>Prepare the data Next, we prepare the dataset.<jupyter_code>data = np.load('data/burgers.npy', allow_pickle=True).item()
print('Shape of grid:', data['x'].shape)<jupyter_output>Shape of grid: (256, 101)<jupyter_text>Let's plot it to get an idea of the data:<jupyter_code>fig, ax = plt.subplots()
im = ax.contourf(data['x'], data['t'], np.real(data['u']))
ax.set_xlabel('x')
ax.set_ylabel('t')
fig.colorbar(mappable=im)

plt.show()<jupyter_output><empty_output><jupyter_text>First, we flatten it to give it the right dimensions for feeding it to the network:<jupyter_code>X = np.transpose((data['x'].flatten(), data['t'].flatten()))
y = np.real(data['u']).reshape((data['u'].size, 1))

print(X.shape, y.shape)<jupyter_output>(25856, 2) (25856, 1)<jupyter_text>As we can see, $X$ has 2 dimensions, $\{x, t\}$, while $y$ has only one, $\{u\}$. Always explicity set the shape (i.e. $N\times 1$, not $N$) or you'll get errors. This dataset is noiseless, so let's add $5\%$ noise:<jupyter_code>noise_level = 0.05
y_noisy = y + noise_level * np.std(y) * np.random.randn(y.size, 1)<jupyter_output><empty_output><jupyter_text>The dataset is also much larger than needed, so let's hussle it and pick out a 1000 samples:<jupyter_code>number_of_samples = 1000

idx = np.random.permutation(y.size)
X_train = X[idx, :][:number_of_samples]
y_train = y_noisy[idx, :][:number_of_samples]<jupyter_output><empty_output><jupyter_text>We now have a dataset which we can use. Let's plot, for a final time, the original dataset, the noisy set and the samples points:<jupyter_code>fig, axes = plt.subplots(ncols=3, figsize=(15, 4))

im0 = axes[0].contourf(data['x'], data['t'], np.real(data['u']), cmap='coolwarm')
axes[0].set_xlabel('x')
axes[0].set_ylabel('t')
axes[0].set_title('Ground truth')

im1 = axes[1].contourf(data['x'], data['t'], y_noisy.reshape(data['x'].shape), cmap='coolwarm')
axes[1].set_xlabel('x')
axes[1].set_title('Noisy')

sampled = np.array([y_noisy[index, 0] if index in idx[:number_of_samples] else np.nan for index in np.arange(data['x'].size)])
sampled = np.rot90(sampled.reshape(data['x'].shape)) #array needs to be rotated because of imshow

im2 = axes[2].imshow(sampled, aspect='auto', cmap='coolwarm')
axes[2].set_xlabel('x')
axes[2].set_title('Sampled')

fig.colorbar(im1, ax=axes.ravel().tolist())

plt.show()<jupyter_output><empty_output><jupyter_text>Configure the neural network We now setup the options for DeepMoD. There's as few dictionaries to fill, starting with 'config', which sets up the the general network layers and L$_1$ penalty. We use the following:<jupyter_code>config = {'layers': [2, 20, 20, 20, 20, 20, 1], 'lambda': 10e-6}<jupyter_output><empty_output><jupyter_text>Next we set the training options. Usually we only change the maximum iterations and the grad_tol, which sets the convergence criterion;<jupyter_code>train_opts = {'max_iterations': 10000, 'grad_tol':10**-6, 'learning_rate': 0.002, 'beta1': 0.99, 'beta2': 0.999, 'epsilon': 10**-8}<jupyter_output><empty_output><jupyter_text>DeepMoD accepts any arbitrary library function and any options for it can be given through *the library_config*. The library function for the burgers case accepts a maximum order for the polynomial and derivative terms. DeepMoD also needs to know the total number of terms upfront. We can calculate that by making a list of the polynomial and derivative terms and getting all the terms by feeding them into the library_matrix_mat function. Its output will be used later to print the found PDE.<jupyter_code>u = ['1', 'u', 'uˆ2']
du = ['1', 'u_{x}', 'u_{xx}', 'u_{xxx}']
coeffs_list = library_matrix_mat(u, du)

print(coeffs_list)

library_config = {'total_terms': len(coeffs_list), 'deriv_order': 3, 'poly_order': 2}<jupyter_output>['1', 'u_{x}', 'u_{xx}', 'u_{xxx}', 'u', 'uu_{x}', 'uu_{xx}', 'uu_{xxx}', 'uˆ2', 'uˆ2u_{x}', 'uˆ2u_{xx}', 'uˆ2u_{xxx}']<jupyter_text>The last configuration we need to fill is the ouput_opts. It contains an output_directory and X_predict field. We've build a custom tensorboard so you can follow the progress of the run.  Output_directory sets where the files are saved, then simply run `tensorboard --logdir $[OUTPUT_DIRECTORY]`in a terminal to open tensorboard. It shows the value of the coefficients, scaled coefficients and all possible costs. Note that the runs are timestamped in output_directory, so you'll have to add it! We can also use the output of tensorboard to analyze deepmod after. We show this below. The last option is X_predict. As DeepMoD also denoises the data, use this option to denoise some dataset X after DeepMoD has converged.<jupyter_code>output_opts = {'output_directory': 'output/burgers/', 'X_predict': X}<jupyter_output><empty_output><jupyter_text>Run DeepMoD  We can now run DeepMoD using all the options we have set and the training data. We also need to specify which library function we wish to use. You can build any library you want and just pass is through this command:<jupyter_code>sparse_vectors, denoised = DeepMoD(X_train, y_train, config, library_1D, library_config, train_opts, output_opts)<jupyter_output>WARNING:tensorflow:From /anaconda3/envs/deepmod/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
WARNING:tensorflow:From /Users/gert-janboth/Documents/PhD/Software/DeePyMoD/src/deepymod/graphs.py:29: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.dense instead.
WARNING:tensorflow:From /anaconda3/envs/deepmod/lib/python3.7/site-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
Epoch | Total loss | Loss gradient | MSE | PI | L1<jupyter_text>Now that DeepMoD has converged, it has found the following equation:<jupyter_code>print('Inferred equation:')
print_PDE(sparse_vectors[0], coeffs_list, PDE_term='u_t')<jupyter_output><empty_output><jupyter_text>which is correct! Let's analyse a few more results in the next section.  Analysis Let's first see how well the denoising did:<jupyter_code>data_dict = {'x_grid': X[:, 0], 't_grid': X[:, 1], 'ground truth': y, 'noisy': y_noisy, 'inferred': denoised}
for key in data_dict:
    data_dict[key] = np.squeeze(data_dict[key])  # pandas doesn't like higher dimensional arrays
df = pd.DataFrame(data_dict)
fig, axes = plt.subplots(ncols=4, figsize=(20, 4))

im0 = axes[0].contourf(df.pivot(index='t_grid', columns='x_grid', values='ground truth'), cmap='coolwarm')
axes[0].set_xlabel('x')
axes[0].set_ylabel('t')
axes[0].set_title('Ground truth')

im1 = axes[1].contourf(df.pivot(index='t_grid', columns='x_grid', values='noisy'), cmap='coolwarm')
axes[1].set_xlabel('x')
axes[1].set_title('Noisy')

sampled = np.array([y_noisy[index] if index in idx[:number_of_samples] else np.nan for index in np.arange(data['x'].size)])
sampled = np.rot90(sampled.reshape(data['x'].shape)) #array needs to be rotated because of imshow

im2 = axes[2].imshow(sampled, aspect='auto', cmap='coolwarm')
axes[2].set_xlabel('x')
axes[2].set_title('Sampled')

im1 = axes[3].contourf(df.pivot(index='t_grid', columns='x_grid', values='inferred'), cmap='coolwarm')
axes[3].set_xlabel('x')
axes[3].set_title('Reconstructed')

fig.colorbar(im1, ax=axes.ravel().tolist())

plt.show()<jupyter_output><empty_output><jupyter_text>We also include a parsing script for the tensorboard events which can be used to plot the coefficients and cost during the training. Just point it to the right event file and it returns a dataframe with all the tracked parameters:<jupyter_code>event_path = 'output/burgers/20190423_175345/iteration_0/events.out.tfevents.1556034829.C02QP0AZG8WL.local'
df_tb = tensorboard_to_dataframe(event_path)<jupyter_output><empty_output><jupyter_text>Now we can plot all the costs of as a function of the epoch:<jupyter_code>fig, ax1 = plt.subplots(ncols=1, nrows=1, figsize=(15, 8.0))

ax1.semilogy(df_tb['epoch'], df_tb['MSE_cost_0'], label='MSE')
ax1.semilogy(df_tb['epoch'], df_tb['PI_cost_0'], label='Reg')
ax1.semilogy(df_tb['epoch'], df_tb['L1_cost_0'], label=r'L$_1$')
ax1.semilogy(df_tb['epoch'], df_tb['Total_cost_1'], label='Total cost')
ax1.set(ylabel='Cost')
ax1.set_xlabel('Epoch')
ax1.legend(loc='best')

plt.show()<jupyter_output><empty_output><jupyter_text>To plot the coeffs, we need to do some slight parsing:<jupyter_code>coeffs = []
for coeff_vec in np.array(df_tb['coeffs'].values):
        coeffs.append(np.expand_dims(coeff_vec, axis=1))
coeffs = np.array(coeffs).squeeze()
fig, ax1 = plt.subplots(ncols=1, nrows=1, figsize=(15, 8.0))

for i in np.arange(12):
    if i != 2 and i != 5:
        ax1.plot(df_tb['epoch'], coeffs[:, i])

ax1.plot(df_tb['epoch'], coeffs[:, 2], linewidth=3.0, linestyle='-', label=r'$u_{xx}$')
ax1.plot(df_tb['epoch'], coeffs[:, 5], linewidth=3.0, linestyle='-', label=r'$uu_{x}$')

ax1.set(xlabel=r'Epochs', ylabel='Coefficient')
ax1.legend()

plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14036687949731427, 'fraction_numerical': 0.022904631600283776, 'mean_word_length': 6.62596599690881, 'pattern_counts': {'":': 0, '<': 65, '<?xml version=': 0, '>': 65, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '179189', 'license': 'MIT', 'n_tokens_mistral': 3674, 'n_tokens_neox': 3494, 'n_words': 953, 'path': 'Examples/Burgers.ipynb', 'repo_name': 'PhIMaL/DeePyMoD_tensorflow', 'size': '9867'}