<jupyter_start><jupyter_text>IMAGE INPAINTING WITH FLUID DYNAMICS Image inpainting aims to remove damage from an image. There are various techniques for image inpainting, and here we focus on solving a fluid-type PDE denoted as the Cahn-Hilliard equation.The three take-home messages from this notebook are that:1. Image inpainting can be solved with efficient and parallelizable finite-volume schemes2. The classification accuracy of neural networks is affected by the presence of damage 3. The application of image inpainting in damaged images improves their classification accuracy&nbsp; Damaged image: Restored image: As an example we take the MNIST dataset, which consists of binary images of handwritten digits:<jupyter_code>test_images = mnist.test_images() # Load MNIST test set
test_images = test_images.reshape((-1,784)) # Flatten
test_images = (test_images / 255) *2-1 # Normalize between -1 and 1
example = test_images[0,:] # Select 1 image
plots.plot_image(example) # Plot image<jupyter_output><empty_output><jupyter_text>The MNIST dataset is corrupted by adding different types of damage to it:<jupyter_code>intensity = 0.5 # elect % of damaged pixels

damage = np.random.choice(np.arange(example.size), replace=False, 
                          size=int(example.size * intensity)) # Create random damage
damaged_example = example.copy() # Generate damaged example
damaged_example[damage] = 0 # Turn damaged pixels to 0
plots.plot_image(damaged_example) # Plot image<jupyter_output><empty_output><jupyter_text>Finite volumes for image inpainting With image inpainting we aim to recover the original image. There are various methods to conduct image inpainting, and here I solve a modified Cahn-Hilliard equation via finite-volume schemes:$$\frac{\partial \phi (x,t)}{\partial t}= -\nabla^{2} \left(\epsilon^2 \nabla^{2} \phi -  H'(\phi) \right) + \lambda(x)\left(\phi (x,t=0) - \phi\right)$$As a baseline let's solve this equation with a simple finite-volume scheme:<jupyter_code>start = time.time() # Start time
restored_example = fv.temporal_loop(damaged_example, damage) # Run finite-volume scheme
print("Total time: {:.2f}".format(time.time()-start)) # Print spent time
plots.plot_image(restored_example) # Plot image<jupyter_output>--------------------
Time:  0.0
['L1 norm of the difference between the new and old state: ', 293.5527206136236]
--------------------
Time:  0.1
['L1 norm of the difference between the new and old state: ', 70.80765875785028]
--------------------
Time:  0.2
['L1 norm of the difference between the new and old state: ', 29.013627612885912]
--------------------
Time:  0.30000000000000004
['L1 norm of the difference between the new and old state: ', 11.724573659317787]
--------------------
Time:  0.4
['L1 norm of the difference between the new and old state: ', 5.837640815931027]
--------------------
Time:  0.5
['L1 norm of the difference between the new and old state: ', 2.8892061603180688]
--------------------
Time:  0.6
['L1 norm of the difference between the new and old state: ', 1.6380886623751225]
--------------------
Time:  0.7
['L1 norm of the difference between the new and old state: ', 0.9633544505193916]
--------------------
Time:  0.7999999999999999
['L1 norm of the difference [...]<jupyter_text>Let's compare the restored image with respect to the original image:<jupyter_code>plots.plot_3images(example, damaged_example, restored_example)  # Plot 3 images<jupyter_output><empty_output><jupyter_text>The computational cost of finite-volume scheme can be reduced by:1. Applying a dimensional-splitting technique and solving row by row and column by column2. Parallelizing the code and solving rows/columns simultaneouslyThe simple finite-volume scheme has taken 40s to run. Let's compare it with the dimensional-splitting code:fully parallelized code:<jupyter_code>start = time.time() # Start time
restored_example = fvs.temporal_loop_split(damaged_example, damage) # Run finite-volume scheme
print("Total time: {:.2f}".format(time.time()-start)) # Print spent time
plots.plot_image(restored_example) # Plot image<jupyter_output>--------------------
Time:  0.0
['L1 norm of the difference between the new and old state: ', 262.0339086632083]
--------------------
Time:  0.1
['L1 norm of the difference between the new and old state: ', 58.06955572215037]
--------------------
Time:  0.2
['L1 norm of the difference between the new and old state: ', 22.048729656053148]
--------------------
Time:  0.30000000000000004
['L1 norm of the difference between the new and old state: ', 9.789847177849662]
--------------------
Time:  0.4
['L1 norm of the difference between the new and old state: ', 4.825914027673576]
--------------------
Time:  0.5
['L1 norm of the difference between the new and old state: ', 2.592470093879773]
--------------------
Time:  0.6
['L1 norm of the difference between the new and old state: ', 1.489734930859102]
--------------------
Time:  0.7
['L1 norm of the difference between the new and old state: ', 0.9160257254952464]
--------------------
Time:  0.7999999999999999
['L1 norm of the difference bet[...]<jupyter_text>By dimensionally splitting the code we have reduced the computational time from 40s to 8s!Can we reduce that time by parallelizing?<jupyter_code>num_proc = 8 # Number of processors
start = time.time() # Start time
restored_example = fvp.temporal_loop_par(damaged_example, damage, num_proc) # Run finite-volume scheme
print("Total time: {:.2f}".format(time.time()-start)) # Print spent time
plots.plot_image(restored_example) # Plot image<jupyter_output>--------------------
Time:  0.0
['L1 norm of the difference between the new and old state: ', 262.42842549085543]
--------------------
Time:  0.1
['L1 norm of the difference between the new and old state: ', 56.71599373578927]
--------------------
Time:  0.2
['L1 norm of the difference between the new and old state: ', 21.793754162627046]
--------------------
Time:  0.30000000000000004
['L1 norm of the difference between the new and old state: ', 9.818795483300335]
--------------------
Time:  0.4
['L1 norm of the difference between the new and old state: ', 4.849590210655972]
--------------------
Time:  0.5
['L1 norm of the difference between the new and old state: ', 2.5932277591482533]
--------------------
Time:  0.6
['L1 norm of the difference between the new and old state: ', 1.4909819342148425]
--------------------
Time:  0.7
['L1 norm of the difference between the new and old state: ', 0.9169305811075479]
--------------------
Time:  0.7999999999999999
['L1 norm of the difference [...]<jupyter_text>The parallel code takes 15 seconds, which is a higher than the non-parallel one. Parallelizing the code does not reduce that time since MNIST images are only 28x28. However, for high-dimensional images it has a clear benefit.  Neural network for classification ![title](images/NN.png) The neural network is trained with the undamaged training dataset. Then we compare its accuracy for the test images with and without damage:<jupyter_code>train_images = mnist.train_images() # Load training set
train_labels = mnist.train_labels() # Load training labels
train_images = (train_images / 255) *2-1 # Normalize between -1 and 1
train_images = train_images.reshape((-1,784)) # Flatten

model, history = nn.training(train_images, train_labels) # Train the neural network
plots.loss_acc_plots(history) # Plot loss and accuracy

test_labels = mnist.test_labels() # Load test labels
print("Validation of undamaged test set:")
test_loss, test_accuracy = model.evaluate(test_images, to_categorical(test_labels), 
                                          verbose=2) # Print test loss and acc<jupyter_output>Epoch 1/8
1875/1875 [==============================] - 2s 1ms/step - loss: 0.5239 - accuracy: 0.8349
Epoch 2/8
1875/1875 [==============================] - 2s 1ms/step - loss: 0.1838 - accuracy: 0.9433
Epoch 3/8
1875/1875 [==============================] - 2s 1ms/step - loss: 0.1414 - accuracy: 0.9556
Epoch 4/8
1875/1875 [==============================] - 2s 1ms/step - loss: 0.1173 - accuracy: 0.9640
Epoch 5/8
1875/1875 [==============================] - 2s 1ms/step - loss: 0.1052 - accuracy: 0.9666
Epoch 6/8
1875/1875 [==============================] - 2s 1ms/step - loss: 0.0891 - accuracy: 0.9721
Epoch 7/8
1875/1875 [==============================] - 2s 1ms/step - loss: 0.0836 - accuracy: 0.9735
Epoch 8/8
1875/1875 [==============================] - 2s 1ms/step - loss: 0.0753 - accuracy: 0.9761: 1s - loss: 0.0731 -  - ETA: 
Validation of undamaged test set:
313/313 - 0s - loss: 0.1094 - accuracy: 0.9681<jupyter_text>The accuracy for the test dataset is quite high: 97%. This accuracy drops as we include damage in the test images. For instance, with an intensity of 80% the accuracy is 55%. Can we recover the accuracy by firstly applying image inpainting?  Image inpainting prior to classifying damaged images Let's select a group of 5 images to add damage:<jupyter_code>n_images = 5 # Number of images
indices_images = range(5) # Select indices
examples = test_images[indices_images,:].copy() # Choose examples from test set

intensity = 0.8 # Damage intensity
# damages = np.zeros((len(indices_images), int(examples.shape[1] * intensity)), dtype=int) # Instantiate damage matrices
# damaged_examples = examples.copy() # Instantiate damaged examples

damages = np.load("data/damages.npy") # Load a previously saved damage matrix

for i in range(len(indices_images)): # Loop over examples t introduce damage
#     damages[i, :] = np.random.choice(np.arange(examples.shape[1]), replace=False, 
#                                      size=int(examples.shape[1] * intensity)) # Choose random damage
    damaged_examples[i, damages[i, :]] = 0 # Turn damaged pixels to 0

plots.plot_image(damaged_examples[1,:]) # Plot one of the damaged examples<jupyter_output><empty_output><jupyter_text>We proceed to restore those 5 images:<jupyter_code>restored_examples = np.zeros(examples.shape) # Instantiate restored examples

for i in range(n_images): # Loop over damaged imaged
    restored_examples[i,:] = fvs.temporal_loop_split(
                                damaged_examples[i, :], damages[i, :])

plots.plot_3images(examples[1,:], damaged_examples[1,:], restored_examples[1,:])<jupyter_output>--------------------
Time:  0.0
['L1 norm of the difference between the new and old state: ', 227.90905061891908]
--------------------
Time:  0.1
['L1 norm of the difference between the new and old state: ', 92.73590898992742]
--------------------
Time:  0.2
['L1 norm of the difference between the new and old state: ', 54.91625640714811]
--------------------
Time:  0.30000000000000004
['L1 norm of the difference between the new and old state: ', 38.1183076633621]
--------------------
Time:  0.4
['L1 norm of the difference between the new and old state: ', 28.356995096230435]
--------------------
Time:  0.5
['L1 norm of the difference between the new and old state: ', 21.932264197866672]
--------------------
Time:  0.6
['L1 norm of the difference between the new and old state: ', 17.378866264948705]
--------------------
Time:  0.7
['L1 norm of the difference between the new and old state: ', 14.002056381181685]
--------------------
Time:  0.7999999999999999
['L1 norm of the difference b[...]<jupyter_text>We can now compare the ground truth with the predicted labels for the damaged and restore images:<jupyter_code>predictions_damaged = np.argmax(model.predict(damaged_examples), axis=1)  
predictions_restored = np.argmax(model.predict(restored_examples), axis=1)    

print("Ground truth: ", test_labels[indices_images])
print("Damaged images: ", predictions_damaged)
print("Restored images: ", predictions_restored)<jupyter_output>Ground truth:  [7 2 1 0 4]
Damaged images:  [7 2 3 0 4]
Restored images:  [7 2 1 0 4]
{'fraction_non_alphanumeric': 0.17049873203719357, 'fraction_numerical': 0.08901098901098901, 'mean_word_length': 3.4393996247654783, 'pattern_counts': {'":': 0, '<': 35, '<?xml version=': 0, '>': 35, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 8, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '499893', 'license': 'MIT', 'n_tokens_mistral': 4204, 'n_tokens_neox': 3503, 'n_words': 1252, 'path': 'Example.ipynb', 'repo_name': 'sergiopperez/Image_Inpainting', 'size': '11830'}