A common approach to improve diversity and quality of generation is to introduce randomness by sampling [1]}. Instead of always choosing the most likely token(s) at each time step, the decoding algorithm samples from the probability distribution over the whole vocabulary. The shape of the distribution can be controlled using the temperature parameter. Setting the temperature to (0,1) shifts the probability mass towards the more likely tokens. Lowering the temperature improves the generation quality at the cost of decreasing diversity [2]}.
