The logical process of an AutoEncoder is that the original input (set to x) is weighted and mapped to y, which is then inversely weighted and mapped back to z. If through iterative training the loss function L(H) is minimised, i.e. z is as close to x as possible, i.e. x is perfectly reconstructed, then it can be said that forward weighting is a successful way of learning the key features of the input. AutoEncoder language models take their name from the Denoising AutoEncoder (DAE) (Vincent et al., 2008), which is used to predict tokens that are [masked] by contextual words (these [masked] words are actually noise added at the input, typical of thinking). DAE is a technique that involves adding random noise to the input layer of data. This helps to learn more robust features when using an unsupervised approach to pre-train the weights of a deep network in a hierarchical manner.