We construct a simple experiment to demonstrate the amount of information LM logits may convey about the input. Given 100 text inputs from Wikipedia, we substitute a single word in the first sentence with a synonym2. Let \(\hat{x}_{s}\) be the synonym of word \(x_{s}\). To measure the change in language model output between the original sequence (containing \(x_{s}\)) and the new sequence (containing \(\hat{x_{s}}\)), we compute two quantities: the KL divergence between the probability output of \(p\) for the original and synonym-swapped sequences, and the bit-level Hamming Distance between the two distributions when represented at 16-bit precision.