<jupyter_start><jupyter_text>First try (with all variables and Albert model output)<jupyter_code>history = test_model.fit(train_ds, epochs=5, validation_data=val_ds, verbose=1, callbacks=callbacks)<jupyter_output>Epoch 1/2
INFO:tensorflow:batch_all_reduce: 14 all-reduces with algorithm = nccl, num_packs = 1
INFO:tensorflow:batch_all_reduce: 14 all-reduces with algorithm = nccl, num_packs = 1
74215/74215 [==============================] - 9759s 131ms/step - loss: 6.2373 - accuracy: 0.3410 - recall: 0.0135 - precision: 0.0280 - val_loss: 4.1578 - val_accuracy: 0.4481 - val_recall: 0.0933 - val_precision: 0.5296
INFO:tensorflow:Assets written to: ./models/baseline/first_try/assets
Epoch 2/2
74215/74215 [==============================] - 9745s 131ms/step - loss: 3.8262 - accuracy: 0.4863 - recall: 0.0540 - precision: 0.7250 - val_loss: 4.6855 - val_accuracy: 0.3344 - val_recall: 0.1575 - val_precision: 0.4799
INFO:tensorflow:Assets written to: ./models/baseline/first_try/assets<jupyter_text>ARCHIVE: Baseline Second Try (trainable embeddings)<jupyter_code>history = test_model.fit(train_ds, epochs=5, validation_data=val_ds, verbose=1, callbacks=callbacks)<jupyter_output>Epoch 1/5
INFO:tensorflow:batch_all_reduce: 14 all-reduces with algorithm = nccl, num_packs = 1
WARNING:tensorflow:Efficient allreduce is not supported for 1 IndexedSlices
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:GPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:GPU:0', '/job:localhost/replica:0/task:0/device:GPU:1', '/job:localhost/replica:0/task:0/device:GPU:2', '/job:localhost/replica:0/task:0/device:GPU:3').
INFO:tensorflow:batch_all_reduce: 14 all-reduces with algorithm = nccl, num_packs = 1
WARNING:tensorflow:Efficient allreduce is not supported for 1 IndexedSlices
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:GPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:GPU:0', '/job:localhost/replica:0/task:0/device:GPU:1', '/job:localhost/replica:0/task:0/device:GPU:2', '/job:localhost/replica:0/task:0/device:GPU:3').
74215/74215 [==============================] - 12309s 165ms/step - loss: 6.1006 - accuracy: 0.3679 -[...]
{'fraction_non_alphanumeric': 0.17413162705667276, 'fraction_numerical': 0.08957952468007313, 'mean_word_length': 6.445578231292517, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '274147', 'license': 'MIT', 'n_tokens_mistral': 915, 'n_tokens_neox': 802, 'n_words': 149, 'path': 'POC/mag_model_iteration_1.ipynb', 'repo_name': 'jpbarrett13/MAG-model-code', 'size': '2188'}