<jupyter_start><jupyter_text>Softmax and temperature scaling In this notebook we will take a look at the softmax function and its behavior for different values of the logits.We will also study the impact of temperature scaling over the final probabilities<jupyter_code>%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
def softmax(X, scale = 1.0, axis = None):
    """
    Compute the softmax of each element along an axis of X.

    Parameters
    ----------
    X: ND-Array. Probably should be floats.
    scale (optional): float parameter, used as a multiplier
        prior to exponentiation. Default = 1.0
    axis (optional): axis to compute values along. Default is the
        first non-singleton axis.

    Returns an array the same size as X. The result will sum to 1
    along the specified axis.
    """

    # make X at least 2d
    y = np.atleast_2d(X)

    # find axis
    if axis is None:
        axis = next(j[0] for j in enumerate(y.shape) if j[1] > 1)

    # multiply y against the theta parameter,
    y = y * float(scale)

    # exponentiate y
    y = np.exp(y)

    # take the sum along the specified axis
    ax_sum = np.expand_dims(np.sum(y, axis = axis), axis)

    # finally: divide elementwise
    p = y / ax_sum

    # flatten if X was 1D
    if len(X.shape) == 1: p = p.flatten()

    return p<jupyter_output><empty_output><jupyter_text>Generate a range of logits<jupyter_code>x = np.arange(0, 1, 0.2)
logits_1 = x 
logits_2 = -2*x - 2
logits_3 = -3*x + 1<jupyter_output><empty_output><jupyter_text>There are two parameters that you can adjust:- `logits_multiplier`: controlling the magnitudes of the logits- `temperature`: the coefficient used for adjusting the softmax values  ExerciseTry out different values for `logits_multiplier` in the range $[1,50]$ and for the `temperature` in the range $[0.1,200]$.What do you notice?<jupyter_code>logits_multiplier = 1
temperature = 200

logits = np.array([logits_1, logits_2, logits_3])* logits_multiplier
probas = softmax(logits, scale = 1, axis = 0)
probas_scaled = softmax(logits, scale = 1/temperature, axis = 0)

plt.figure(figsize=(15,3))
plt.subplot(1,3,1)

plt.plot(x, logits.T)
plt.legend(['logits_1','logits_2','logits_3'])
plt.title('logits')

plt.subplot(1,3,2)
plt.plot(x, probas.T)
plt.title('probas')

plt.subplot(1,3,3)
plt.plot(x, probas_scaled.T)
plt.title('probas_scaled')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10802980132450331, 'fraction_numerical': 0.02359271523178808, 'mean_word_length': 3.73921568627451, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 14, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '172783', 'license': 'Apache-2.0', 'n_tokens_mistral': 879, 'n_tokens_neox': 823, 'n_words': 272, 'path': 'CEA_EDF_INRIA/softmax_temperature_colab.ipynb', 'repo_name': 'lasofivec/dataflowr', 'size': '2416'}