<jupyter_start><jupyter_text>Demo: `gpflow_monitor`In this notebook we'll demo how to use `gpflow_monitoring` for logging the optimisation of a GPflow model. The example should cover pretty much all use cases. Creating the GPflow modelWe first create the GPflow model as usual.<jupyter_code>m = gpflow.models.SVGP(X, Y, gpflow.kernels.RBF(1), gpflow.likelihoods.Gaussian(), Z=np.linspace(0, 10, 5)[:, None],
                       minibatch_size=100)
m.likelihood.variance = 0.01
m.compile()<jupyter_output><empty_output><jupyter_text>Setting up the optimisationNext we need to set up the optimisation process. `gpflow_monitor` provides classes that manage the optimsation, and perform certain logging tasks. In this example, we want to:- log certain scalar parameters in TensorBoard- log the full optimisation objective (log marginal likelihood bound) periodically, even though we optimise with minibatches- store a backup of the optimisation process periodically- log performance for a test set periodicallyBecause of the integration with TensorFlow ways of storing and logging, we will need to perform a few TensorFlow manipulations outside of GPflow as well.We start by creating the `global_step` variable. This is not strictly required by TensorFlow optimisers, but they do all have support for it. Its purpose is to track how many optimisation steps have occurred. It is useful to keep this in a TensorFlow variable as this allows it to be restored together with all the parameters of the model.<jupyter_code>global_step = tf.Variable(0, trainable=False, name="global_step")
m.enquire_session().run(global_step.initializer)<jupyter_output><empty_output><jupyter_text>Next, we create an instance of `FileWriter`, which will save the TensorBoard logs to a file. This object needs to be shared between all `gpflow_monitor.TensorBoard` objects, if they are to write to the same path.<jupyter_code>fw = tf.summary.FileWriter(os.path.join("./results/test/tensorboard/"), m.enquire_session().graph)<jupyter_output><empty_output><jupyter_text>Now the TensorFlow side is set up, we can focus on the `gpflow_monitor` part. The optimsation is taken care of by the `ManagedOptimisation` class. This will run the training loop. The `ManagedOptimisation` object will also take care of running `Task`s.Each `Task` is something that needs to be run periodically during the optimisation. The first and second parameters of all tasks are a generator returning times (either in iterations or time) of when the `Task` needs to be run. The second determines whether a number of iterations (`Trigger.ITER`), an amount of time spent optimising (`Trigger.OPTIMISATION_TIME`), or the wall-clock time (`Trigger.TOTAL_TIME`) triggers the `Task` to be run. The following `Task`s are run once in every 100 or 1000 iterations.<jupyter_code>opt_method = ManagedOptimisation(m, gpflow.train.AdamOptimizer(0.01), global_step)
opt_method.tasks += [
    PrintTimings((x * 100 for x in itertools.count()), Trigger.ITER),
    ModelTensorBoard((x * 100 for x in itertools.count()), Trigger.ITER, m, fw),
    LmlTensorBoard((x * 1000 for x in itertools.count()), Trigger.ITER, m, fw, verbose=False),
    StoreSession((x * 1000 for x in itertools.count()), Trigger.ITER, m.enquire_session(), "./results/test/checkpoint")
]<jupyter_output>INFO:tensorflow:Summary name full lml is illegal; using full_lml instead.<jupyter_text>We may also want to perfom certain tasks that do not have pre-defined `Task` classes. For example, computing the performance on a test set. Here we create such a class by extending `ModelTensorBoard` to log the testing benchmarks in addition to all the scalar parameters.<jupyter_code>class TestTensorBoard(ModelTensorBoard):
    def __init__(self, sequence, trigger: Trigger, model, file_writer, Xt, Yt):
        super().__init__(sequence, trigger, model, file_writer)
        self.Xt = Xt
        self.Yt = Yt
        self._full_test_err = tf.placeholder(gpflow.settings.tf_float, shape=())
        self._full_test_nlpp = tf.placeholder(gpflow.settings.tf_float, shape=())

        self.summary = tf.summary.merge([tf.summary.scalar("test_rmse", self._full_test_err),
                                         tf.summary.scalar("test_nlpp", self._full_test_nlpp)])

    def _event_handler(self, manager):
        minibatch_size = 100
        preds = np.vstack([m.predict_y(Xt[mb * minibatch_size:(mb + 1) * minibatch_size, :])[0]
                            for mb in range(-(-len(Xt) // minibatch_size))])
        test_err = np.mean((Yt - preds) ** 2.0)**0.5
        summary, step = m.enquire_session().run([self.summary, global_step],
                                      feed_dict={self._full_test_err: test_err,
                                                 self._full_test_nlpp: 0.0})
        self.file_writer.add_summary(summary, step)<jupyter_output><empty_output><jupyter_text>We then add it to the task list.<jupyter_code>opt_method.tasks.append(TestTensorBoard((x * 1000 for x in itertools.count()), Trigger.ITER, m, fw, Xt, Yt))<jupyter_output><empty_output><jupyter_text>Running the optimisationWe finally get to running the optimisation. The second time this is run, the session should be restored from a checkpoint created by `StoreSession`. To confirm this, we print out the first value in all TensorFlow tensors. This includes any values used by the optimiser. This is important to ensure that the optimiser starts off from _exactly_ the same state as that it left. If this is not done correctly, models may start diverging after loading.<jupyter_code>sess = m.enquire_session()
[u[1] if isinstance(u[1], numbers.Number) else u[1].flatten()[0]  for u in sorted([(v.name, sess.run(v)) for v in tf.global_variables()], key=lambda x: x[0])]
opt_method.minimize(maxiter=8000)<jupyter_output>1, 1:	2.16 optimisation iter/s	2.16 total iter/s	0.00 last iter/sFull lml: -1198200.924550 (-1.20e+06)
1000, 1000:	454.39 optimisation iter/s	325.05 total iter/s	576.21 last iter/sFull lml: -30515.580038 (-3.05e+04)
2000, 2000:	506.20 optimisation iter/s	367.32 total iter/s	570.79 last iter/sFull lml: -17402.941729 (-1.74e+04)
3000, 3000:	527.07 optimisation iter/s	384.49 total iter/s	573.41 last iter/sFull lml: -15024.233225 (-1.50e+04)<jupyter_text>Here, we print the optimised variables for comparison on the next run.<jupyter_code>[u[1] if isinstance(u[1], numbers.Number) else u[1].flatten()[0]  for u in sorted([(v.name, sess.run(v)) for v in tf.global_variables()], key=lambda x: x[0])]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09730309531106345, 'fraction_numerical': 0.032332209623046275, 'mean_word_length': 4.42109634551495, 'pattern_counts': {'":': 0, '<': 31, '<?xml version=': 0, '>': 31, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '533081', 'license': 'Apache-2.0', 'n_tokens_mistral': 2115, 'n_tokens_neox': 1949, 'n_words': 702, 'path': 'notebooks/example.ipynb', 'repo_name': 'rhaps0dy/gpflow-monitor', 'size': '6526'}