Footnote 3: During gradient clipping all of the model’s parameters are treated as a single big vector (as if all parameters were flattened and concatenated together), and we take the \(\ell_{2}\)-norm over the corresponding single gradient vector. This is the standard way to clip gradients in PyTorch.