<jupyter_start><jupyter_text>The structure of GPflow--*<NAME>, 2016**Small edits <NAME> 2017*In this document I'll try to give some insight into what's happening under the hood in GPflow. First I'll explain how the hierarchical structure works: GPflow models, kernels, likelihoods and parameters are arranged in a tree. Next I'll attempt to explain what happens when we run `compile`, which builds a tensorflow graph ready for execution. Lots of these ideas are taken from GPy, where I was strongly influenced by Max Zwiesselle's work. You can find some of that in the [paramz project](http://github.com/sods/paramz).  Tree structureGPflow has some core classes. They are- The `Param` class (`gpflow.param.Param`) which is used to represent parameters.- The DataHolder class (`gpflow.data_holders.DataDict`) which is used similarly to represent data.- The `Parameterized` class (`gpflow.param.Parameterized`) which is a container for `Param`s. Subclasses include  - Kernels (`gpflow.kernels.Kern`)    - Likelihoods (`gpflow.likelihoods.Likelihood`)    - Mean functions (`gpflow.mean_functions.MeanFunction`)  - The model class (`gpflow.model.Model`) which, which is a special class of `Parameterized`. Together these form a tree: the model class is the trunk, `Parameterized` classes are on branches, and `Param` and `DataHolder` classes are at the leaves. It possible to have as many layers (branches) in this structure as you want. Here's a silly example.<jupyter_code>from __future__ import print_function
import gpflow
import numpy as np
m = gpflow.model.Model()
m.p = gpflow.param.Parameterized()
m.p.p = gpflow.param.Parameterized()
m.p.p.p = gpflow.param.Parameterized()
m.p.p.p.p = gpflow.param.Parameterized()
m.p.p.p.p.foo = gpflow.param.Param(np.array([7, 12]))
print(m)<jupyter_output>model.p.p.p.p.[1mfoo[0m transform:(none) prior:None
[  7.  12.]<jupyter_text>Going down the treeThe tree is implicitely defined by the attributes of each class. To find all the child nodes of a node, we just search through the `__dict__` of the node, looking for `Parameterized` or `Param` objects. There's a built-in `sorted_params` property, which does this, and ensures that they always apear in the same order. A good example of a function that makes use of this is the `build_prior` function. In `Parameterized` this just calls `build_prior` on all the child nodes. In a `Param` object, this computes the log prior density, if applicable (else 0). Another good example that we'll see shortly is the `make_tf_array` function.   Going up the treeAll the elements of this tree inherrit from `Parentable`. This base class doesn't do much, apart from point 'up' the tree. Each element in the tree has a `_parent` attribute, which points at the member in the layer above. It's only possible to have one parent: this is a tree. The model (trunk) has a `None` parent. It's therefore possible to get a pointer to the trunk by recursing the `_parent` attribute, which is how we have defined `highest_parent`. This is used to tell the model if something has changed (like if a parameter has been swwitched to `fixed`).  What happens in `compile()` ? To understand the ideas behind the stucture of GPflow, let's first have a look at how to compute things in tensorflow.  Tensorflow works by building a symbolic graph, and then executing the graph using `session().run(graph)`. Data are fed into the graph using a dictionary in the form `session().run(graph, feed_dict={X_tf:X_np}`, where `X_tf` is a tensorflow placeholder, and `X_np` is a numpy array containing the data to be fed in. Let's have a look at a simple Linear regression example, without any GPlow machinery. We'll write a build_likelihood() function which will construct the tensorflow graph, and then we'll feed the graph with some data.<jupyter_code>import numpy as np
import tensorflow as tf

# generate a dataset
X = np.random.randn(10, 2)
Y = np.dot(X, np.array([2.3, -1.2])) + np.random.randn(10, 1) * 0.2

# placeholders for the data and coefficients
X_tf = tf.placeholder(tf.float64)
Y_tf = tf.placeholder(tf.float64)
coeffs_tf = tf.placeholder(tf.float64)
noise_var_tf = tf.placeholder(tf.float64)

def build_likelihood():
    err = Y_tf - tf.matmul(X_tf, coeffs_tf)
    log_likelihood = -0.5*tf.log(noise_var_tf) -0.5 / noise_var_tf * tf.reduce_sum(tf.square(err))
    return log_likelihood

ll_graph = build_likelihood()
grads_tf = tf.gradients(ll_graph, [coeffs_tf, noise_var_tf])
sess = tf.InteractiveSession()
coeffs_guess = np.random.randn(2, 1)
noise_guess = np.random.rand(1)
log_lik, grad_coeff, grad_noise = sess.run([ll_graph]+grads_tf, feed_dict={X_tf:X,
                                                             Y_tf:Y,
                                                             coeffs_tf:coeffs_guess,
                                                             noise_var_tf:noise_guess})
print(log_lik, '\n\n', grad_coeff, '\n\n', grad_noise)<jupyter_output>[-1202.81132182] 

 [[-152.9072189 ]
 [ 389.46568891]] 

 [ 2588.3956483]<jupyter_text>That's quite a lot of code for such a simple task! Here's the same code using GPflow<jupyter_code>import gpflow

class LinReg(gpflow.model.Model):
    
    def __init__(self, X, Y):
        gpflow.model.Model.__init__(self)
        self.coeffs = gpflow.param.Param(np.random.randn(2, 1)) # Param contains initial guess at value
        self.noise_var = gpflow.param.Param(np.random.rand(), transform=gpflow.transforms.Exp())
        self.X, self.Y = gpflow.param.DataHolder(X), gpflow.param.DataHolder(Y)
        
    def build_likelihood(self):
        err = self.Y - tf.matmul(self.X, self.coeffs)  # we can use self.coeffs as if it were a tf object
        log_likelihood = -0.5*tf.log(self.noise_var) -0.5 / self.noise_var * tf.reduce_sum(tf.square(err))
        return log_likelihood
    
m = LinReg(X, Y)
m.compile()
print(m._objective(m.get_free_state()))<jupyter_output>(array([ 676.82951118]), array([-676.35573593, -184.13194821,  165.4607482 ]))<jupyter_text>Sidenote: that that's not the recommended way to get the log likelihood out of the model. Use `m.compute_log_likelihood()` instead. That's not a huge saving in code, but the savings mount up for more complex models. Here's what happens in the above code. 1. When the Model is instantiated (`m = LinReg(X, Y)`), the constructor `__init__` is called. In this function:  - `Model.__init__` creates a general purpose model. This contains its own `Session`, a placeholder `m._free_vars` for all our parameters, and a some convenient functions for e.g. optimization.    - We define a `Param` object `self.coeffs` for the coefficients, and `self.noise_var` for the noise variance. This lets us pass the current guess at the parameter into `model.session`. The current state of each parameter is stored as `self._array`; we have initialized each parameter with a guess.   - We store the data as `DataHolders`, which allows us to pass the data into `m.session`.  2. When we manually compile the model (`m.compile`), the models calls the build_likelihod function. But, in order to let us use the parameters and dataholders as a tensorflow object (see e.g. `tf.matmul(self.X, self.coeffs)`), this is done inside a python 'context' that we call `tf_mode`. Before we can use `tf_mode`, we must first call `make_tf_array`.3. `make_tf_array` takes the tensorflow placeholder `m._free_vars` and distributes it amongst the parameters. We go through all the parameters in turn, and take a chunk of `_free_vars` of the correct length, and reshape it (and possibly transform it, see later) into a tensorflow variable that can represent the parameter in the graph. In our simple example, `_free_vars` has length three. The first two elements are transformed into a 2x1 matrix of coeffs, and the last one is exponentiated to become the noise variance. Each Param object stores its transformed, reshaped section of `_free_vars` as `self._tf_array`. Similiarly, the DataHolder objects hold placeholders into which we later feed the data as `self._tf_array`. 4. Inside the clause `with m.tf_mode()`, all of the parameters (and dataholders) are replaced with their `_tf_array` attributes. The `compile` function can then call the `build_likelhood` function which builds a representation of the likelihood as a function of `_free_vars` (via the `_tf_array` attributes). It also builds a convenient function `self._objective` which allows us to link up with scipy's `optimize` routines. 5. The `_objective` function is a wrapper around a call to the model's `_session.run`. It calls `session.run()`, requesting the negative log_likelihood and the gradients with respect to `_free_vars`. Into the feed_dict, it passes the<jupyter_code>print(m.coeffs._tf_array)
print(m.noise_var._tf_array)<jupyter_output>Tensor("Reshape_1:0", shape=(2, 1), dtype=float64)
Tensor("Reshape:0", shape=(1,), dtype=float64)
{'fraction_non_alphanumeric': 0.0922920892494929, 'fraction_numerical': 0.017354068064007213, 'mean_word_length': 4.199179847685999, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '466361', 'license': 'Apache-2.0', 'n_tokens_mistral': 2780, 'n_tokens_neox': 2609, 'n_words': 1091, 'path': 'GPflow/doc/source/notebooks/structure.ipynb', 'repo_name': 'mlilab/Mixed-Effect-Composite-RNN-Gaussian-Process', 'size': '8902'}