<jupyter_start><jupyter_text>feature reductionData can be loaded downloaded [here](https://drive.google.com/drive/folders/1yZI5v3ws3b8GZMl_ACe4TO_qebdS2fUz?usp=sharing). The data is contained in the `srp_raw01.zip` and has to be moved to `/data/raw`.The resulting folder structure looks like this:`/data/raw/n`, `/data/raw/o` and `/data/raw/x`.In this notebook the feature space of the data is reduced.By reducing redundancy in data the model decreases in size and may improve in performace, because less weights and biases have to be adjusted.<jupyter_code>from os.path import join

raw = join('data', 'raw')
processed = join('data', 'processed')

from src.utils import reset_and_distribute_data

reset_and_distribute_data(raw, processed, [400,0,100])<jupyter_output><empty_output><jupyter_text>In essence the only change undertaken in this notebook is the `color_mode` parameter given to the `flow_from_directory` function.<jupyter_code>from tensorflow.keras.preprocessing.image import ImageDataGenerator

def create_generator(data_dir, batch_size, datagen):
    full_path = join(processed, data_dir)
    return datagen.flow_from_directory(
        full_path,
        target_size=(32, 32),
        color_mode='grayscale',
        batch_size=batch_size,
        class_mode='binary')

train_datagen = ImageDataGenerator(
        rescale = 1./255,
        rotation_range=360,
        horizontal_flip=True,
        vertical_flip=True)

test_datagen = ImageDataGenerator(rescale = 1./255)

train_generator = create_generator('train', 20, train_datagen)
test_generator = create_generator('test', 10, test_datagen)<jupyter_output>Found 1200 images belonging to 3 classes.
Found 300 images belonging to 3 classes.<jupyter_text>The `model.summary` shows that the number of parameters is almost divided by a factor of three (was 99491 in previous models). This is translating directly into the size of the saved model.<jupyter_code>from tensorflow.keras import layers
from tensorflow.keras import models

model = models.Sequential()
model.add(layers.Flatten(input_shape=(32, 32, 1)))
model.add(layers.Dense(32,'relu'))
model.add(layers.Dense(32,'relu'))
model.add(layers.Dense(3, 'softmax'))

model.summary()
from tensorflow.keras.optimizers import SGD

optimizer = SGD(lr=0.005, momentum=0.9, nesterov=True)

model.compile(loss='sparse_categorical_crossentropy', optimizer=optimizer, metrics=['acc'])
from tensorflow.keras.callbacks import TensorBoard
import numpy as np
from datetime import datetime
from os import mkdir

log_dir = join('logs', 'srp62', datetime.now().strftime("%Y-%m-%dT%H-%M-%S"))
mkdir(log_dir)

callbacks = [ TensorBoard(
    log_dir=log_dir,
    histogram_freq=1,
    embeddings_freq=1) ]

history = model.fit_generator(
    train_generator,
    steps_per_epoch=20,
    epochs=20,
    callbacks=callbacks)
model.evaluate_generator(test_generator)<jupyter_output><empty_output><jupyter_text>The resulting model has 295kb, whereas the old models were 807kb big. The model is much smaller now without loosing any measurable data.<jupyter_code>model_path = join('models', 'devel', 'srp62.h5')
model.save(model_path)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10231232182451695, 'fraction_numerical': 0.025974025974025976, 'mean_word_length': 5.676532769556025, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '33101', 'license': 'MIT', 'n_tokens_mistral': 1072, 'n_tokens_neox': 1000, 'n_words': 263, 'path': 'reports/srp/notebooks/6.2-feature_reduction.ipynb', 'repo_name': 'klawr/deepmech', 'size': '3157'}