<jupyter_start><jupyter_text>Inspecting gradientsThis example assumes you've read `advanced.ipynb`, and covers:   - Inspecting gradients per layer   - Estimating good values of gradient clipping threshold<jupyter_code>import deeptrain
deeptrain.util.misc.append_examples_dir_to_sys_path()

from utils import make_autoencoder, init_session
from utils import AE_CONFIGS as C

from tensorflow.keras.optimizers import Adam
import numpy as np<jupyter_output><empty_output><jupyter_text>Configure training<jupyter_code>C['traingen']['iter_verbosity'] = 0  # silence iteration printing since currently irrelevant
tg = init_session(C, make_autoencoder)<jupyter_output>[31mWARNING:[0m multiple file extensions found in `path`; only .npy will be used 
Discovered 48 files with matching format
48 set nums inferred; if more are expected, ensure file names contain a common substring w/ a number (e.g. 'train1.npy', 'train2.npy', etc)
DataGenerator initiated

[31mWARNING:[0m multiple file extensions found in `path`; only .npy will be used 
Discovered 36 files with matching format
36 set nums inferred; if more are expected, ensure file names contain a common substring w/ a number (e.g. 'train1.npy', 'train2.npy', etc)
DataGenerator initiated

[34mNOTE:[0m will exclude `labels` from saving when `input_as_labels=True`; to keep 'labels', add '{labels}'to `saveskip_list` instead
Preloading superbatch ... [31mWARNING:[0m multiple file extensions found in `path`; only .npy will be used 
Discovered 48 files with matching format
................................................ finished, w/ 6144 total samples
Train initial data prepared
Preloading superbatch .[...]<jupyter_text>Expected gradient norm estimationWe iterate over entire train dataset, gathering gradients from every fit and computing and storing their L2-norms.<jupyter_code>grad_norms, *_ = tg.gradient_norm_over_dataset()<jupyter_output>Computing gradient l2-norm over datagen batches, in inference mode
[31mWARNING:[0m datagen states will be reset
'.' = slice processed, '|' = batch processed

Data set_nums shuffled

||||||||||10||||||||||20||||||||||30||||||||||40||||||||
Data set_nums shuffled


GRADIENT L2-NORM (AVG, MAX) = (0.003, 0.003), computed over 48 batches, 48 datagen updates<jupyter_text>We can now restart training with setting optimizer `clipnorm` to 1.5x average value, avoiding extreme gradients while not clipping most standard gradients<jupyter_code>C['model']['optimizer'] = Adam(clipnorm=1.5 * np.mean(grad_norms))
tg = init_session(C, make_autoencoder)
tg.epochs = 1  # train just for demo
tg.train()<jupyter_output>[31mWARNING:[0m multiple file extensions found in `path`; only .npy will be used 
Discovered 48 files with matching format
48 set nums inferred; if more are expected, ensure file names contain a common substring w/ a number (e.g. 'train1.npy', 'train2.npy', etc)
DataGenerator initiated

[31mWARNING:[0m multiple file extensions found in `path`; only .npy will be used 
Discovered 36 files with matching format
36 set nums inferred; if more are expected, ensure file names contain a common substring w/ a number (e.g. 'train1.npy', 'train2.npy', etc)
DataGenerator initiated

[34mNOTE:[0m will exclude `labels` from saving when `input_as_labels=True`; to keep 'labels', add '{labels}'to `saveskip_list` instead
Preloading superbatch ... [31mWARNING:[0m multiple file extensions found in `path`; only .npy will be used 
Discovered 48 files with matching format
................................................ finished, w/ 6144 total samples
Train initial data prepared
Preloading superbatch .[...]<jupyter_text>Complete gradient sumThis time we run a cumulative sum over actual gradient tensors, preserving and returning their shapes, allowing per-weight visualization<jupyter_code>plot_kw = {'h': 2}  # double default height since we expect many weights
grads_sum, *_ = tg.gradient_sum_over_dataset(plot_kw=plot_kw)<jupyter_output>Computing gradients sum over datagen batches, in inference mode
[31mWARNING:[0m datagen states will be reset
'.' = slice processed, '|' = batch processed

Data set_nums shuffled

||||||||||10||||||||||20||||||||||30||||||||||40||||||||
Data set_nums shuffled


GRADIENTS SUM computed over 48 batches, 48 datagen updates:
{'fraction_non_alphanumeric': 0.12497074654809268, 'fraction_numerical': 0.025274982447928855, 'mean_word_length': 5.2212518195050945, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '135296', 'license': 'MIT', 'n_tokens_mistral': 1373, 'n_tokens_neox': 1261, 'n_words': 469, 'path': 'examples/introspection/gradients.ipynb', 'repo_name': 'Mario-Kart-Felix/deeptrain', 'size': '4273'}