<jupyter_start><jupyter_text>PREPROCESSING I want to avoid the problem I had while training the autoencoder, which is that I would have to re-shuffle the training and validation set everytime I closed the notebook, so Instead I will do that once here and save the shuffled data + labels as .npy files<jupyter_code>import numpy as np
import random
import os<jupyter_output><empty_output><jupyter_text>Load redshift data labels<jupyter_code>os.listdir()
labels_all = np.load("esc_frac_0070_labels.npy")
images = np.load("esc_frac_0070_data.npy")

print(images.shape)
print(labels_all.shape)

images = list(images)
labels_all = list(labels_all)<jupyter_output>(3400, 200, 200)
(3400,)<jupyter_text>Shuffle<jupyter_code>shuffled_images = []
shuffled_labels_all = []
while images:
    i = random.randrange(len(images))
    shuffled_images.append(images[i])
    del images[i]
    shuffled_labels_all.append(labels_all[i])
    del labels_all[i]
    
images = np.array(shuffled_images[:])
labels_all = np.array(shuffled_labels_all[:])
print(images.shape)<jupyter_output>(3400, 200, 200)<jupyter_text>while im here i might aswell make the labels so that the model can read them... and do the preprocessing<jupyter_code># normalizes images
images_n = images / 70.0 # n for normalized
print("max : ", np.max(images_n))
print("min : ", np.min(images_n))
images.shape<jupyter_output>max :  0.9531142
min :  0.0<jupyter_text>Reshape the images for feeding through the encoder<jupyter_code>im_reshape = np.array(images_n[:])
im_reshape = np.expand_dims(im_reshape, axis = 3)
im_reshape = np.array(im_reshape)
print("this is the shape", im_reshape.shape)
for i in range(15):
    print(labels_all[i])<jupyter_output>6.5
8.5
13.0
8.5
10.5
12.0
11.5
6.0
6.5
8.0
6.5
7.5
10.5
6.0
8.0<jupyter_text>Format the labels so the network can read them<jupyter_code># display the labels
l = []
for i in labels_all:
    if i not in l: l.append(i)
l.sort()
print(l, "\nThere are", len(l), "labels. So the want",
     len(l), "classes")
# define the train labels in len 17 arrays
labels=[]
for i in labels_all:
    arr = np.zeros(len(l))
    index = l.index(i)
    arr[index] = 1
    labels.append(arr[:])
    
labels = np.array(labels)
print("train_labels shape =", labels.shape,
     "\n\nFirst 3 entries:")
print(labels[:3])<jupyter_output>train_labels shape = (3400, 17) 

First 3 entries:
[[0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
 [0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
 [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1.]]<jupyter_text>normalize<jupyter_code># normalize images
images_n = images / 70.0 # for some reason this one is 40...
# it's the data not me
print("max :", np.max(images_n))
print("min :", np.min(images_n))<jupyter_output>max : 0.9531142
min : 0.0<jupyter_text>augmentation dimension<jupyter_code>im_reshape = np.array(images_n[:])
im_reshape = np.expand_dims(im_reshape, axis=3)
im_reshape = np.array(im_reshape)
print("this is the shape", im_reshape.shape)<jupyter_output>this is the shape (3400, 200, 200, 1)<jupyter_text>train test split<jupyter_code>ratios = [0.75,0.85]
cut1,cut2 = int(len(im_reshape)*ratios[0]),int(len(im_reshape)*ratios[1])
images_train = im_reshape[:cut1]
images_val = im_reshape[cut1:cut2]
images_test = im_reshape[cut2:]
labels_train = labels[:cut1]
labels_val = labels[cut1:cut2]
labels_test = labels[cut2:]
images_train.shape<jupyter_output><empty_output><jupyter_text>save them<jupyter_code>np.save("esc0070_data_train.npy", images_train)
np.save("esc0070_data_val.npy", images_val)
np.save("esc0070_data_test.npy", images_test)
np.save("esc0070_labels_train.npy", labels_train)
np.save("esc0070_labels_val.npy", labels_val)
np.save("esc0070_labels_test.npy", labels_test)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1394729837636412, 'fraction_numerical': 0.057226510513707744, 'mean_word_length': 5.130505709624796, 'pattern_counts': {'":': 0, '<': 34, '<?xml version=': 0, '>': 34, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '611395', 'license': 'MIT', 'n_tokens_mistral': 1638, 'n_tokens_neox': 1490, 'n_words': 354, 'path': 'CNN/ESC_FRAC_0070/preprocessing.ipynb', 'repo_name': 'dcxSt/MAIS202---21cm-ionization-maps-of-the-universe', 'size': '3757'}