<jupyter_start><jupyter_text>Set input arguments for training:&emsp;- PERIOD: Number of LSTM units connected in series&emsp;- UNITS: Number of hidden layers in the lSTM&emsp;- SCALER: A custom sklearn scaler can be used, if necessary, to normalize the data. If None is provided, then a scaler is initialized.<jupyter_code>path_to_output = '../output'
YEAR = 2016
XTAL = 54000
PERIOD = 35
UNITS = 25
EPOCHS = 1
BATCH_SIZE = 1
SEED = 7
SCALER = 'None'
SPLIT = 0.8<jupyter_output><empty_output><jupyter_text>During training, the dataset is shuffled in a randomized fashion. To make the results reproduciable, initialize the seed.<jupyter_code>reset_random_seed(SEED)<jupyter_output><empty_output><jupyter_text>Import data for a given crystal from the csv file. Class lstm_data_processor can be used to construct normalized arrays that can be used for training. The dataset is split into training and testing based on a given split provided.<jupyter_code>X = None
Y = None

from src.python.tf.data_processor import lstm_data_processor
prsr = lstm_data_processor(XTAL, YEAR, PERIOD)
prsr.prepare_dataset_from_csv('{}/data/interim/'.format(PATH_TO_REPO), SCALER)
prsr.save_to_pickle('{}/data/processed/'.format(PATH_TO_REPO))
X = prsr.train_x
Y = prsr.train_y


trainmax = int(SPLIT*len(X))
index = np.arange(0, len(X)-1)
random.shuffle(index)


train_x = X[index][0:trainmax]
train_y = Y[index][0:trainmax]
valid_x = X[index][trainmax:]
valid_y = Y[index][trainmax:]<jupyter_output>Using self-scaling ...<jupyter_text>Import a model architecture and train the model.<jupyter_code>from src.models.tf.lstm import lstm
model = lstm(PERIOD, UNITS)
model.model.summary()
model.model.compile(loss=tf.keras.losses.MeanSquaredError(), optimizer='adam')

print('Model optimizer')
print(model.model.optimizer.get_config())

training_output = model.model.fit(train_x, train_y,
                            epochs=EPOCHS, batch_size=BATCH_SIZE, verbose=2,
                            validation_data=(valid_x, valid_y),
                            shuffle = True)
model_name = '{}/models/lstm_{}_period_{}_calib_inst_lumi_deltat_xtal_{}_epochs_{}_batch_{}_seed_{}'.format(path_to_output, UNITS, PERIOD, XTAL, EPOCHS, BATCH_SIZE, SEED)
model.model.save(model_name)
path_to_output = model_name
os.system('mkdir {}/plots/'.format(path_to_output))

# plot the training loss
plt.figure(figsize=(6.6,6.6))
plt.plot(training_output.epoch, training_output.history['loss'], color='orange', label='loss')
plt.plot(training_output.epoch, training_output.history['val_loss'], color='blue', label='validation loss')
plt.ylabel('loss', fontsize=14)
plt.xlabel('epochs', fontsize=14)
plt.legend()
plt.savefig('{}/plots/training_loss.png'.format(path_to_output))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12327773749093546, 'fraction_numerical': 0.011240029006526469, 'mean_word_length': 5.7128953771289535, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '399764', 'license': 'MIT', 'n_tokens_mistral': 1013, 'n_tokens_neox': 972, 'n_words': 214, 'path': 'notebooks/LSTM_tf_train_model.ipynb', 'repo_name': 'bmjoshi/fair_ecal_monitoring', 'size': '2758'}