<jupyter_start><jupyter_text>Landscape Colorization EvaluationThis notebook Is focused at the evaluation of the colorization of the Landscapes dataset, as a POC forthe hyper-spectral colorization concept.  Auxiliary Functions and Imports:<jupyter_code>from pathlib import Path
import sys
sys.path.append(str(Path.cwd().parent.parent))

import pandas as pd
from landscapes import *<jupyter_output><empty_output><jupyter_text>Background: DatasetThe Landscapes dataset (https://www.kaggle.com/arnaud58/landscape-pictures) is a data-set made of 4315 naturalsceneries of:* landscapes (900 pictures)* landscapes montain (900 pictures)* landscapes desert (100 pictures)* landscapes sea (500 pictures)* landscapes beach (500 pictures)* landscapes island (500 pictures)* landscapes japan (900 pictures)Most images are 8-bit RGB, and those that aren't were discarded for the purpose of this solution. In addition, allimages were re-scaled to a spatial resolution of 256x256 pixels (using bicubic interpolation) to mimic the resolution ofthe TAU camera, which will be used to collect the real-world dataset for the purpose of our research.  Training:In order to train 3 Cycle-Gans (1 Cycle-Gan per color-channel), the images of the data-set were randomly divided into5 mutually-exclusive categories:1. Red (~1000 images)2. Green (~1000 images)image_name3. Blue (~1000 images)4. Panchromatic (~1000 images)5. Validation (~315 images)Each Cycle-Gan was trained to learn the transformation from the panchromatic image set to either color image set. Theresidual validation images are later used to test the obtained solutions for the learned transformation of all colors.This random-division of the dataset was repeated 5 times to assemble a cross-validation for the training procedure, i.e.The 3 Cycle-Gans are to be trained 5 consequent times, each time with a different set of images.<jupyter_code>partition_log = pd.read_pickle(getLatestFileVer(data_dir,"partition_log_crop"))
# partition_log = pd.read_pickle(getLatestFileVer(data_dir, '_'.join(['partition',  'log', 'crop'])))
partition_log = partition_log.rename_axis("Image Name", axis=1)
partition_log.head()<jupyter_output><empty_output><jupyter_text>Evaluation:The evaluation of the overall solution is based on the pixel-wise comparison between the original ground-truth $I$ imageand it's color-wise reconstruction $\hat{I}$ by the 3 trained Cycle-Gans.<jupyter_code>## Set Path for test-images
exp_name = "_".join(["landscapes", "crop", "cv", str(0)])
test_path = os.path.join(data_dir, "testA")
img_names = getUniqeFileNames(test_path)
uint8tofloat = lambda x: x.astype(float) / 255
## Reconstruct a test-image
eg_img = next(iter(img_names))
# eg_img = "00000597.png"
image_gt, image_rec = getImageGtRec(data_dir, eg_img, exp_name)
image_gt, image_rec = uint8tofloat(image_gt), uint8tofloat(image_rec)

## Display GT VS Reconstruction
fig, ax = plt.subplots(3, 2, dpi=150)
for i in range(ax.shape[0]):
    for j in range(ax.shape[1]):
        ax[i, j].set_yticks([])
        ax[i, j].set_xticks([])
        ax[i, j].set_frame_on(False)

ax[0, 0].set_title("Color")
ax[0, 1].set_title("Pan")
ax[0, 0].set_ylabel("GT")
ax[1, 0].set_ylabel("Reconstructed")
ax[2, 0].set_ylabel("Diff")

ax[0, 0].imshow(image_gt / rgb2grey)
ax[0, 1].imshow(image_gt.sum(axis=2), cmap="gray")
ax[1, 0].imshow(image_rec / rgb2grey)
ax[1, 1].imshow(image_rec.sum(axis=2), cmap="gray")
ax[2, 0].imshow(np.abs(image_rec - image_gt) / rgb2grey)
ax[2, 1].imshow(np.abs(image_rec - image_gt).sum(axis=2), cmap="gray")

plt.tight_layout()<jupyter_output>Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).
Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).<jupyter_text>For the purpose of numerical evaluation, 2 types of error are being examined:* Per-color: $e_c = I_c - \hat{I_c}$* Panchromatic: $e_{pan} = \sum_{c\in {red, green, blue}} I_c - \sum_{c\in {red, green, blue}} \hat{I_c}$  Evaluation Metrics:To quantify the performance of the solution, the following metrics were used:* RMSE (Root Mean Squared Error)* PSNR* SSIM (Structural Similarity )<jupyter_code>from pytorch_msssim import ssim
from tqdm import tqdm

colors = ["red", "green", "blue", "pan"]
metrics = ["rmse", "psnr", "ssim"]
metrics_df = pd.DataFrame(index=img_names, columns=pd.MultiIndex.from_product([colors, metrics]))

img2tens = lambda img: transforms.ToTensor()(img.astype(float)).unsqueeze(dim=0)

for image_name in tqdm(img_names, desc="Calculating Metrics"):
    image_orig, image_rec = getImageGtRec(data_dir, image_name, exp_name)
    for i, color in enumerate(colors):
        if color == "pan":
            I = image_orig.astype(float).sum(axis=2)
            I_hat = image_rec.astype(float).sum(axis=2)
        else:
            I = image_orig[:, :, i].astype(float)
            I_hat = image_rec[:, :, i].astype(float)
        err = I -  I_hat
        rmse = np.sqrt((err ** 2).mean())
        metrics_df.loc[image_name, (color, "rmse")] = rmse
        metrics_df.loc[image_name, (color, "psnr")] = 20 * np.log10(255.0 / rmse)
        metrics_df.loc[image_name, (color, "ssim")] = ssim(img2tens(I), img2tens(I_hat)).numpy()

metrics_df.head()
print(f"Average Values: \n {metrics_df.mean()}")<jupyter_output>Average Values: 
 red    rmse     9.495230
       psnr    30.014601
       ssim     0.871619
green  rmse    16.096464
       psnr    25.173724
       ssim     0.814527
blue   rmse     9.027971
       psnr    29.305423
       ssim     0.851735
pan    rmse    25.095615
       psnr    21.447322
       ssim     0.841736
dtype: float64
{'fraction_non_alphanumeric': 0.12036710201200142, 'fraction_numerical': 0.04165195905400635, 'mean_word_length': 4.583251231527093, 'pattern_counts': {'":': 1, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '453715', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 2089, 'n_tokens_neox': 1904, 'n_words': 563, 'path': 'datasets/landscapes/landscapes.ipynb', 'repo_name': 'bermanz/pytorch-CycleGAN-and-pix2pix', 'size': '5666'}