<jupyter_start><jupyter_text>Table of Contents1&nbsp;&nbsp;Callbacks as GUI events1.1&nbsp;&nbsp;Creating our own callback1.2&nbsp;&nbsp;Classes (not functions) as callbacks1.3&nbsp;&nbsp;Multiple callbacks; *args and **kwargs1.4&nbsp;&nbsp;Using callbacks to modify behavior From https://www.youtube.com/watch?v=HR0lt1hlR6U&feature=youtu.be&list=PLfYUBJiXbdtTIdtE1U8qgyxo4Jy2Y91uj&t=445<jupyter_code>import torch
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>Callbacks as GUI events<jupyter_code>import ipywidgets as widgets
def f(o): print('hi')<jupyter_output><empty_output><jupyter_text>"The button widget is used tohandle mouse clicks. The on_click method of the Button can be used to register a function to be called when the button is clicked."<jupyter_code>w = widgets.Button(description='Click me')
w
w.on_click(f)<jupyter_output><empty_output><jupyter_text>When callbacks are used this way, they are often called "events". What we did is we told w that when a click occurs, we should "call back" to the f function and run it. `f` here is a callback (a callback is a concept). It's a function. Callbacks are functions. (Callbacks are a kind of "function pointer").  Creating our own callback The event this function will call back on is "after a calculation is complete."<jupyter_code>from time import sleep
# this is a toy example that does 5 calculations.
# each calculation can be thought of as an epoch.
def train_n_sessions(n=5):
    res = 0
    for i in range(n):
        res += i*i
        sleep(1)
    return res
train_n_sessions()<jupyter_output><empty_output><jupyter_text>What I, Rory, really want to know is: how is this thing going? So, let's modify this example to take a callback!<jupyter_code># this is a toy example that does 5 calculations.
# each calculation can be thought of as an epoch.
def train_n_sessions(n=5, cb=None):
    res = 0
    for i in range(n):
        res += i*i
        sleep(1)
        if cb: cb(i) # if there's a callback, call it and pass in epoch number.
    return res
def show_progress(epoch): print(f"Finished epoch {epoch}.")
train_n_sessions()
train_n_sessions(cb=show_progress)<jupyter_output>Finished epoch 0.
Finished epoch 1.
Finished epoch 2.
Finished epoch 3.
Finished epoch 4.<jupyter_text>Awesome! My first callback :) . Tip: a call-back is also known as a "call-after" function, and is defined as any function that is passed as an argument to other code. That other code is expected to "call back" to that function at a specified time. (Callable classes can also be used as callbacks). Lambdas work.<jupyter_code>train_n_sessions(cb=lambda x: print(f"finished epoch {x}"))<jupyter_output>finished epoch 0
finished epoch 1
finished epoch 2
finished epoch 3
finished epoch 4<jupyter_text>What if we wanted to pass an argument to the callback, such as a phrase to state before the epoch? Let's try.<jupyter_code>def show_progress(phrase,epoch): print(f"{phrase} {epoch}.")
train_n_sessions(cb=lambda x: show_progress("Yip-yip!",x))<jupyter_output>Yip-yip! 0.
Yip-yip! 1.
Yip-yip! 2.
Yip-yip! 3.
Yip-yip! 4.<jupyter_text>This is a function that returns a function.<jupyter_code>def make_show_progress(phrase):
    def _inner(epoch): print(f"{phrase} {epoch}")
    return _inner
show_prog = make_show_progress("Epoch completed:")
train_n_sessions(cb=show_prog)<jupyter_output>Epoch completed: 0
Epoch completed: 1
Epoch completed: 2
Epoch completed: 3
Epoch completed: 4<jupyter_text>*Side note:* Congrats, Rory, on making your first **closure**!! A function that stores information from the external context and the information can be different every time is called a closure. Google: "A closure is a record storing a function together with an environment." From stack overflow: "A closure is a persistent scope which holds on to local variables even after the code execution has moved out of that block. Langs that support closures allow you to keep a reference to a scope. The scope object and all of its local variables are tied to the function and will persist as long as that function persists. This gives function portability."The implementation of a closure often looks like a function `f` that returns a function `g`, s.t. the function g is defined with variables that are local to f. Here's what that looks like:`def f():    local_var = 1    def g(x): return x + local_var    return g`    Here's a silly example:`def make_greeting():    joke_of_the_day = get_joke()    def greeting(name):        print(f"Hi {name}! {joke_of_the_day}.")    return greeting`    In a lang that doesn't support closures, joke_of_the_day would be garbage collected and tossed out when make_greeting is exited. But since python supports closures, it persists because the variable scope is created when the function is first declared and persists as long as the function does.<jupyter_code>from functools import partial<jupyter_output><empty_output><jupyter_text>Because it's so common to want to turn a function of many params into a function of one param, python has a builtin for it called partial. This is great for fixing multiple args in a function.<jupyter_code>train_two_sessions = partial(train_n_sessions,n=2)
train_two_sessions(cb=show_prog)<jupyter_output>Epoch completed: 0
Epoch completed: 1<jupyter_text>It apparantly works by setting defaults for the function's args; you can still pass in a value for n in train_two_sessions.<jupyter_code>train_two_sessions(n=1,cb=show_prog)<jupyter_output>Epoch completed: 0<jupyter_text>Classes (not functions) as callbacks Pretty much any place you can use a closure, you can use a class. The state can be stored in `__init__`, and `__call__` will hold the callable. (This is called a "functor" in other langs, or a "callable" in python).<jupyter_code>class ProgressShowingCallback():
    def __init__(self, phrase='Finished epoch'): self.phrase = phrase
    def __call__(self, epoch): print(f"{self.phrase} {epoch}")
cb_cls = ProgressShowingCallback()
cb_cls(1)
train_n_sessions(cb=cb_cls)<jupyter_output>Finished epoch 0
Finished epoch 1
Finished epoch 2
Finished epoch 3
Finished epoch 4<jupyter_text>Multiple callbacks; `*args` and `**kwargs`<jupyter_code>def f(*args, **kwargs): print(f"args: {args} | kwargs: {kwargs}")
f(1,2,3,'rice is nice',x=42,y=69,z='blink182')<jupyter_output>args: (1, 2, 3, 'rice is nice') | kwargs: {'x': 42, 'y': 69, 'z': 'blink182'}<jupyter_text>PyTorch's `nn.Sequential` takes in argumetns as `*args`. A common way to use this is if you want to wrap an object to create a new object, you can define the new obj's args, and then use kwargs to pass off all the stuff you'd normally pass on to the wrapped object. Note that the new object will NOT show the old object's args when you tab-search or tab-complete, so the full arg list will be hidden from end users. Basically, the end user will have to already know what the wrapped object's args are to use the new object. Callbacks are a place where you really want to use kwargs. Here I have two callbacks: cb.before_epoch and cb.after_epoch. I can't use cb(i) anymore; I have to use these methods.<jupyter_code>def do_calc(x): sleep(1); return x*x
def train_n_sessions(n=5, cb=None):
    res = 0
    for i in range(n):
        # Before epoch
        if cb: cb.before_epoch(i)
        calc = do_calc(i)
        res += calc
        # After epoch
        if cb: cb.after_epoch(i, val=res)
    return res
class PrintStepCallback():
    def __init__(self): pass
    def before_epoch(self, *args, **kwargs): print(f"Epoch starting")
    def after_epoch (self, *args, **kwargs): print(f"Epoch finished")
cb_cls = PrintStepCallback()
train_n_sessions(cb=cb_cls)<jupyter_output>Epoch starting
Epoch finished
Epoch starting
Epoch finished
Epoch starting
Epoch finished
Epoch starting
Epoch finished
Epoch starting
Epoch finished<jupyter_text>In the above, we used args and kwargs to create functions (methods) that don't care about one or more parameters; they're more flexible. "If you put in both positional and keyword arguments, it will always work everywhere."Let's actually use the values passed now.<jupyter_code>class PrintStepCallback():
    def __init__(self): pass
    def before_epoch(self, epoch, **kwargs): print(f"Starting epoch {epoch}")
    def after_epoch (self, epoch, val, **kwargs): print(f"Finished epoch {epoch} at {val}")
cb_cls = PrintStepCallback()
train_n_sessions(cb=cb_cls)<jupyter_output>Starting epoch 0
Finished epoch 0 at 0
Starting epoch 1
Finished epoch 1 at 1
Starting epoch 2
Finished epoch 2 at 5
Starting epoch 3
Finished epoch 3 at 14
Starting epoch 4
Finished epoch 4 at 30<jupyter_text>In the above, kwargs was kept to make the code more future proof. We may later want to pass in more vars.  Using callbacks to modify behavior We can check if a callback class has certain callbacks and only do those callbacks if it does.<jupyter_code>def train_n_sessions(n=5, cb=None):
    res = 0
    for i in range(n):
        # Before epoch
        if cb and hasattr(cb,'before_epoch'): cb.before_epoch(i)
        calc = do_calc(i)
        res += calc
        # After epoch
        if cb and hasattr(cb,'after_epoch'):
            if cb.after_epoch(i, val=res):
                print('Stopping early')
                break
    return res
class PrintAfterCallback():
    def after_epoch(self, epoch, val):
        print(f"After epoch: {epoch} | val: {val}")
        if val>10: return True
cb_cls = PrintAfterCallback()
train_n_sessions(cb=cb_cls)<jupyter_output>After epoch: 0 | val: 0
After epoch: 1 | val: 1
After epoch: 2 | val: 5
After epoch: 3 | val: 14
Stopping early<jupyter_text>Let's change the way the calculation is being done! We'll do this by making train_n_sessions into a class and the value it calculates an attr of it.<jupyter_code>class TrainSessions():
    def __init__(self, cb=None, epochs=5):
        self.cb, self.res, self.epochs = cb, 0, epochs
        
    def callback(self, cb_name, *args):      # Usage: callback('hook_name', *args). Ex: callback('after_epoch', i)
        if not self.cb: return               # Is self.cb is still None? Do nothing.
        cb = getattr(self.cb, cb_name, None) # Look in cb's attrs for 'hook_name'; should be a method of the cb.
        if cb: return cb(self, *args)        # If found: cb.hook_name(self, *args).
        
    def do_calc(self, x):
        sleep(1)
        return x*x
    
    def do_epochs(self):
        for i in range(self.epochs):
            self.callback('before_epoch', i)
            self.res += self.do_calc(i)
            if self.callback('after_epoch', i):
                print('Stopping early')
                break
class ModifyingCallback():
    def after_epoch(self, trainer, epoch):
        print(f"After {epoch}: {trainer.res}")
        if trainer.res>10: return True
        if trainer.res<3: trainer.res = trainer.res*2<jupyter_output><empty_output><jupyter_text>The above is a callback that has a hook into after_epoch. after_epoch is a special name. All of the hooks have special names; if I defined a method called after_sunset, it would have nothing to interface into.<jupyter_code>trainer = TrainSessions(ModifyingCallback())
trainer.do_epochs()
trainer.res<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08830081083489263, 'fraction_numerical': 0.01078143098993139, 'mean_word_length': 4.411764705882353, 'pattern_counts': {'":': 0, '<': 63, '<?xml version=': 0, '>': 64, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '218769', 'license': 'Apache-2.0', 'n_tokens_mistral': 3582, 'n_tokens_neox': 3485, 'n_words': 1366, 'path': '_20201015_callback_playground.ipynb', 'repo_name': 'mccallion/rory', 'size': '11223'}