<jupyter_start><jupyter_text>The first big question: What library to use to read in images. Some options are: 1. Either use matplotlibs imread, directly loads an image as numpy array and installed everywhere2. Use pythons standard PIL (Pillow) library3. Use tf.keras.preprocessing.image.ImageDataGenerator4. Use skimage Each lib has its pros and cons. In the end the main focus here will be a CNN, so TF functionality will be important. Even for the other ML algorithms it may be wise to use the Convolutional Layers of the CNN to reduce the dimensionality of the data. (Note: TF internally uses Pillow.) First, lets get familiar with the dataset here, look at the dimensions of the images (without reading them in)<jupyter_code>SOURCE_DIR = "photoz_images/"
imgs = [PIL.Image.open(SOURCE_DIR+filename).size for filename in os.listdir(SOURCE_DIR)]

# see how many images of which size we have
sizes, counts = np.unique(imgs, axis=0, return_counts=True)
sizes, counts<jupyter_output><empty_output><jupyter_text>As all images are of the same size, we won't need heavy preprosessing, as we can use the original size, since 72x72 is quite handable. (Let's hope I don't regret saying that later)<jupyter_code>RE = re.compile(r"^(?P<ID>\d+)_z_(?P<redshift>\d+\.\d*(e-\d+)?).jpg$")
def load_image(source_dir, filename):
    img = img_to_array(load_img(source_dir+filename), dtype=np.float16)/255
    matches = RE.match(filename)
    return img, float(matches.group("redshift"))

dirlist = list(os.listdir(SOURCE_DIR))
images = np.empty((len(dirlist), 72, 72, 3), np.float16)
redshifts = np.empty((len(dirlist),), np.float32)
for idx, filename in enumerate(tqdm(dirlist)):
    image, redshift = load_image(SOURCE_DIR, filename)
    images[idx] = image
    redshifts[idx] = redshift
X_train, X_test, y_train, y_test = train_test_split(images, redshifts)
# test CNN
model = models.Sequential()
model.add(layers.Conv2D(32, (3, 3), activation=activations.swish, input_shape=(72, 72, 3)))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(32, (3, 3), activation=activations.swish))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(16, (3, 3), activation=activations.swish))
model.add(layers.MaxPooling2D((3, 3)))
model.add(layers.Flatten())
model.add(layers.Dense(60, activation=activations.swish))
model.add(layers.Dropout(0.8))
model.add(layers.Dense(60, activation=activations.swish))
model.add(layers.Dense(1, activation=activations.linear))
model.compile(optimizer=optimizers.Adam(learning_rate=1e-3),
              loss=losses.MeanSquaredError(),
              metrics=[metrics.MeanAbsoluteError()])
model.summary()
generator = preprocessing.image.ImageDataGenerator()
model.fit(generator.flow(X_train, y_train))
model.predict(X_test[:5]), y_test[:5]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12714285714285714, 'fraction_numerical': 0.0225, 'mean_word_length': 5.9503722084367245, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '646736', 'license': 'MIT', 'n_tokens_mistral': 970, 'n_tokens_neox': 922, 'n_words': 253, 'path': 'project/Photometric Redshift.ipynb', 'repo_name': 'The-Ludwig/mlfp', 'size': '2800'}