<jupyter_start><jupyter_text>Data For illustrating purposes, the MNIST dataset containg handwritten digits will be used. In particular we might obtain the data from the ``inferpy`` package (and hence from keras):<jupyter_code># number of observations (dataset size)
N = 1000
# digits considered
DIG = [0,1,2]

# load the data
(x_train, y_train), _ = mnist.load_data(num_instances=N, digits=DIG)

# plot the digits
mnist.plot_digits(x_train, grid=[5,5])<jupyter_output><empty_output><jupyter_text>Model definitionThe implementation for the generative model for a NLPCA model (Algorithm 3) is defined below. The input parameters are: `k` is the latent dimension, `d` is the data-dimension and `N`the number of samples or data instances.<jupyter_code># Model constants
k, d0, d1 = 2, 100, np.shape(x_train)[-1]

# initial values
loc_init = 0.001
scale_init = 1

@inf.probmodel
def nlpca(k, d0, d1, decoder):

    with inf.datamodel():
        # Define local latent variables
        z = inf.Normal(loc=tf.ones([k]), scale=1, name="z")

        output = decoder(d0, d1, z)

        # Define the observed variables
        x = inf.Normal(loc=output, scale=1., name="x")


def decoder(d0, d1, z):
    return inf.keras.Sequential([
        tfp.layers.DenseFlipout(d0, activation=tf.nn.relu),
        tfp.layers.DenseFlipout(d1),
    ])(z)

print(nlpca)
print(decoder)<jupyter_output><function nlpca at 0x12f7cee18>
<function decoder at 0x12f7ced90><jupyter_text>This is a latent varable model (LVM) containing DNNs where the latent representation $\boldsymbol{z}$ is known as the scores, and the affine transformation is performed using a DNN parametrized by $\boldsymbol{\beta}$ and $\boldsymbol{\alpha}$.  InferenceVariational inference is a deterministic technique that finds a tractable approximation to an intractable (posterior) distribution. We will use $q$ to denote the approximation, and use $p$ to signify the true distribution (like $p(\boldsymbol{\beta, \alpha},\boldsymbol{z}|\boldsymbol{x})$ in the example above).More specifically, ${\cal Q}$ will denote a set of possible approximations $q$.In practice, we define a generative model for sampling from $q(\boldsymbol{\beta, \alpha},\boldsymbol{z} | \boldsymbol{\lambda}, \boldsymbol{\phi})$, where $\boldsymbol{\lambda}, \boldsymbol{\phi}$ are the variational parameters to optimise.<jupyter_code>@inf.probmodel
def Q(k, d0, d1):

    with inf.datamodel():
        qz_loc = inf.Parameter(tf.ones([k]), name="qz_loc")
        qz_scale = tf.math.softplus(inf.Parameter(tf.ones([k]), name="qz_scale"))
        qz = inf.Normal(loc=qz_loc, scale=qz_scale, name="z")

print(Q)<jupyter_output><function Q at 0x1363f9bf8><jupyter_text>Variational methods adjusts the parameters by maximizing the ELBO (Evidence LOwer Bound) denoted $\cal{L}$ and expressed as $\cal{L}(\boldsymbol{\lambda},\boldsymbol{\phi})  = \mathbb{E}_q [\ln p(\boldsymbol{x}, \boldsymbol{z}, \boldsymbol{\beta, \alpha})] - \mathbb{E}_q [\ln q(\boldsymbol{\beta, \alpha},\boldsymbol{z}|\boldsymbol{\lambda},\boldsymbol{\phi})]$In InferPy, this is transparent to the user: it is only required to create the instances of the P and Q models, the optimizer and inference method objects.<jupyter_code># create an instance of the P model and the Q model
m = nlpca(k, d0, d1, decoder)
q = Q(k,d0,d1)

# load the data
(x_train, y_train), _ = mnist.load_data(num_instances=N, digits=DIG)

optimizer = tf.train.AdamOptimizer(learning_rate)
VI = inf.inference.VI(q, optimizer=optimizer, epochs=2000)<jupyter_output>WARNING: Logging before flag parsing goes to stderr.
W0108 10:50:49.479316 4429538752 deprecation_wrapper.py:119] From ../../inferpy/models/prob_model.py:63: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

W0108 10:50:49.491589 4429538752 deprecation_wrapper.py:119] From ../../inferpy/util/tf_graph.py:63: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.

W0108 10:50:49.503413 4429538752 deprecation_wrapper.py:119] From ../../inferpy/util/interceptor.py:142: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead.<jupyter_text>Finally, the ELBO function is maximized.<jupyter_code>m.fit({"x": x_train}, VI)<jupyter_output>W0108 10:50:50.777243 4429538752 deprecation.py:323] From ../../inferpy/util/interceptor.py:34: Variable.load (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Prefer Variable.assign which has equivalent behavior in 2.X.
W0108 10:50:51.458394 4429538752 deprecation.py:323] From /Users/rcabanas/venv/InferPy/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py:1354: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W0108 10:50:52.114441 4429538752 deprecation_wrapper.py:119] From ../../inferpy/inference/variational/vi.py:187: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.<jupyter_text>After the inference, we can plot the hidden representation:<jupyter_code>post = {"z": m.posterior("z", data={"x": x_train}).sample()}

markers = ["x", "+", "o"]
colors = [plt.get_cmap("gist_rainbow")(0.05),
          plt.get_cmap("gnuplot2")(0.08),
          plt.get_cmap("gist_rainbow")(0.33)]
transp = [0.9, 0.9, 0.5]

fig = plt.figure()

for c in range(0, len(DIG)):
    col = colors[c]
    plt.scatter(post["z"][y_train == DIG[c], 0], post["z"][y_train == DIG[c], 1], color=col,
                label=DIG[c], marker=markers[c], alpha=transp[c], s=60)
    plt.legend(loc='upper right',framealpha=1)<jupyter_output><empty_output><jupyter_text>TestFor testing our model, we will generate samples of $\boldsymbol{x}$ given the infered posterior distributions.<jupyter_code>x_gen = m.posterior_predictive('x', data=post).sample()

# plot the digits
mnist.plot_digits(x_gen, grid=[5,5])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1321716258425119, 'fraction_numerical': 0.04668748972546441, 'mean_word_length': 5.139253279515641, 'pattern_counts': {'":': 3, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '230621', 'license': 'Apache-2.0', 'n_tokens_mistral': 2252, 'n_tokens_neox': 2015, 'n_words': 560, 'path': 'notebooks/ProbModelingDNNs/NLPCA-Inferpy-Bayesian-Layers.ipynb', 'repo_name': 'PGM-Lab/InferPy', 'size': '6083'}