The Maximal Update Parametrization (µP)  aims to make the optimal hyperparameters (HPs)  of a model independent of its size, allowing them to be swept using a cheap proxy model rather than the full-size target model.
We present a new scheme, u-µP, which improves upon µP by combining it with Unit Scaling, a method for designing models that makes them easy to train in low-precision.
The two techniques have a natural affinity: µP ensures that the scale of activations is independent of model size, and Unit Scaling ensures that activations, weights and gradients begin training with a scale of one.
This synthesis opens the door to a simpler scheme, whose default values are near-optimal. This in turn facilitates a more efficient sweeping strategy, with u-µP models reaching a lower loss than comparable µP models and working out-of-the-box in FP8.