<jupyter_start><jupyter_text>Translator: Go from moment representation to ExpFam-representationThis task is about taking a distribution represented by its moment-parameters, translate it into the exponential family representation, and compare the two by plotting them. We will focus on Gaussians and Gamma-distributed variables here, but this exercise an be done for any exp.fam. distribution. **Imports**<jupyter_code>import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import norm, gamma
from scipy.special import gammaln<jupyter_output><empty_output><jupyter_text>**Helper function**Take representation of an exp.fam. distribution and a function to plot the moment-pdf<jupyter_code>def plot_pdf(x, h, T, eta, A, moment_pdf):
    """
    Put the pieces of an exp.fam.representation together and compare with the moment-representation.

    :param x: The values for which we will evaluate the two functions
                -- a list of x-values for which the pdf's are calculated
    :param h: The log-base measure. This is a *function* that can be evaluated at any x, and for each x return a scalar
    :param T: The sufficient statistics function. This is again a *Æfunction*. Takes a vector as input and
                return a matrix (the vector of sufficient statistics for each value of the x-vector)
    :param eta: Natural parameters. This is a vector, constant in x so not a function.
    :param A: The log partition function. This is defined by the parameterization,
                hence not a function in the implementation.
    :param moment_pdf: A function that generates the pdf using moment parameters (so, a "pointer" to a built-in)
    :return: Nothing.
    """
    log_exp_fam = h(x) + np.matmul(eta, T(x)) - A   # Put together the log-pdf of the exp.fam. distribution
    plt.plot(x, np.exp(log_exp_fam), "b-")  # Plot it
    plt.plot(x, moment_pdf(x), "r--")   # Plot the "gold-standard"
    plt.title('Moment PDF (red) and ExpFam pdf (blue). Hopefully identical')
    plt.show()<jupyter_output><empty_output><jupyter_text>EXAMPLE: GAUSSIAN **Define the starting-point**, that is, define the moment-parameters, the range for which 98% of the probability mass resides, and the pdf-function<jupyter_code># Moment parameters
sigma = 3.
mu = -1.
# Choose x to cover most of the are where the distribution has probability mass
x = np.linspace(norm.ppf(0.01, loc=mu, scale=sigma),
                norm.ppf(0.99, loc=mu, scale=sigma), 25)

# The comparison: Gauss pdf using the moment parameters
def normal_pdf_function(x):
    return norm.pdf(x, loc=mu, scale=sigma)<jupyter_output><empty_output><jupyter_text>Define the exp.fam. representation **Log base measure:** For the Gaussian, $h(x) = -.5\log(2\pi)$. It is constant in $x$ for the Gaussian distribution,but is defined as a function in the implementation anyway, because the log base measure is a function in general.<jupyter_code>def log_base_measure_func(x):
    return -.5 * np.log(2*np.pi)<jupyter_output><empty_output><jupyter_text>**Sufficient statistics:** For the gaussian,  $T(x) = [x, x^2]$. Note that if `x` is a vector/list this function should return an array<jupyter_code>def T_func(x):
    # Define storage space
    ans = np.zeros((2, x.shape[0]))
    # Fill in values
    ans[0, :] = x
    ans[1, :] = x * x
    return ans<jupyter_output><empty_output><jupyter_text>**Natural parameters:** Defined from the moment parameters, and for the Gaussian it is     $[\frac{\mu}{\sigma^2}, -\frac{1}{2\sigma^2}]$.<jupyter_code>natural_parameters = np.array([mu/(sigma*sigma), -.5/(sigma*sigma)])<jupyter_output><empty_output><jupyter_text>**Log partition function:** Can be defined from moment parameters $(\mu, \sigma)$or alternatively from natural parameters. Here we use $A=\frac{\mu^2}{2\sigma^2} + \log|\sigma|$.<jupyter_code>log_partition = mu*mu/(2*sigma*sigma) + np.log(sigma)<jupyter_output><empty_output><jupyter_text>Plot the exp-fam we generated, and compare to the moment-parameterized PDFNote how functions are passed on, e.g. `h=log_base_measure_func` will ensure that `h(x)`can be called from `plot_pdf` with the `x` that is defined in the scope fo that function.<jupyter_code>plot_pdf(x=x, h=log_base_measure_func, T=T_func, 
         eta=natural_parameters, A=log_partition, 
         moment_pdf=normal_pdf_function)<jupyter_output><empty_output><jupyter_text>And now the Gamma-distruibutionStart by setting up. We can later play with alpha and beta<jupyter_code># Moment parameters
alpha = 2.
beta = 3.
# Choose x as the range that covers 98% prob.mass
x = np.linspace(gamma.ppf(0.01, a=alpha, scale=1/beta),
                gamma.ppf(0.99, a=alpha, scale=1/beta), 25)

# Calculate moment-parameter pdf
def gamma_pdf_func(x):
    return gamma.pdf(x, a=alpha, scale=1/beta)<jupyter_output><empty_output><jupyter_text>Make the ExpFam representationInformation about the Gamma can be found [here](https://en.wikipedia.org/wiki/Exponential_familyTable_of_distributions "here") or in the slides.<jupyter_code># Log base measure: $h(x) = 0$. It is constant in x, but is defined as a function anyway
def log_base_measure_func(x):
    return 0

# Sufficient statistics: $T(x) = [\log(x), x]$.
# If x is an array this function should return an array
def T_func(x):
    # Define storage space
    ans = np.zeros((2, x.shape[0]))
    # Fill in values
    ans[0, :] = np.log(x)
    ans[1, :] = x
    return ans

# Natural parameters: Defined from the moment parameters
natural_parameters = np.array([alpha - 1, -beta])

# Log partition function. Can be defined from moment parameters alpha, beta
# or alternatively from natural parameters [eta1, eta2]
log_partition = gammaln(alpha) - alpha * np.log(beta)<jupyter_output><empty_output><jupyter_text>**Test by generating a plot**<jupyter_code>plot_pdf(x=x, h=log_base_measure_func, T=T_func, 
         eta=natural_parameters, A=log_partition, moment_pdf=gamma_pdf_func)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10818074920208298, 'fraction_numerical': 0.009071056610112548, 'mean_word_length': 4.512962962962963, 'pattern_counts': {'":': 0, '<': 45, '<?xml version=': 0, '>': 45, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '24793', 'license': 'Apache-2.0', 'n_tokens_mistral': 1938, 'n_tokens_neox': 1869, 'n_words': 641, 'path': 'Python/solution_translator.ipynb', 'repo_name': 'HelgeLangseth/AlmeriaVB', 'size': '5953'}