<jupyter_start><jupyter_text>Data block API foundations<jupyter_code>%load_ext autoreload
%autoreload 2

%matplotlib inline
#export
from exp.nb_07a import *<jupyter_output><empty_output><jupyter_text>Let's move beyond MNIST. ImageNet takes days to train on a single GPU. CIFAR10 contains 32x32 images, and it seems that small images have very different characteristics than larger ones. In particular, it seems that once you go beyond 96x96, things change quite a lot. So, stuff that works well on CIFAR10, does not really work on ImageNet. We would like to have a dataset that doesn't take days to train, but still is realistic enough.Still <NAME> found out that methods that worked well on 128x128 ImageNet images, still worked well on the full size images. The full ImageNet at 128x128 is still too big. J.H. created a dataset called *imaginette* that is a subset of ImageNet containing 10 out of the 1000 classes. There are multiple versions: full size, 320px, 160px. The classes are very different to each other, and this dataset is supposed to be easy. You should be able to train for just an epoch or two and have an idea of the performance.The *Imagewoof* is, conversely, a dataset of dog breeds, and it's supposed to be hard. It is available in the same formats.There is a leaderboard for either dataset. J.H. says that working on these two datasets has given him more insights on CV than (as usual) anything else he's ever done. He says that putting your own version of a toy problem can be very helpful having a better understanding of the domain you are working on.[Jump_to lesson 11 video](https://course.fast.ai/videos/?lesson=11&t=600)<jupyter_code>datasets.URLs.IMAGENETTE_160<jupyter_output><empty_output><jupyter_text>Image ItemList Previously we were reading in to RAM the whole MNIST dataset at once, loading it as a pickle file. We can't do that for datasets larger than our RAM capacity, so instead we leave the images on disk and just grab the ones we need for each mini-batch as we use them. In so doing we will build a `DataBlock` API. By the end of this notebook you should be able to build your own `DataBlock` API or modify fastai's.Let's use the [imagenette dataset](https://github.com/fastai/imagenette/blob/master/README.md) and build the data blocks we need along the way.  Get images<jupyter_code>path = datasets.untar_data(datasets.URLs.IMAGENETTE_160)
path<jupyter_output><empty_output><jupyter_text>In general, if we want to read files of a given format we need to rely on a library that supports such format. In the case of images we rely on the Python Image Library (PIL).To be able to look at what's inside a directory from a notebook, we add the `.ls` method to `Path` with a monkey-patch.<jupyter_code># export
import PIL
import os
import mimetypes
Path.ls = lambda x: list(x.iterdir())
path.ls()<jupyter_output><empty_output><jupyter_text>The command below will show the 10 categories (the names are not terribly informative).<jupyter_code>(path/'val').ls()<jupyter_output><empty_output><jupyter_text>Let's have a look inside a class folder (the first class is tench):Interesting observation from J.H.: "they are pretty much always held up by middle aged men, or in nets".<jupyter_code>path_tench = path/'val'/'n01440764'
img_fn = path_tench.ls()[0]
img_fn
img = PIL.Image.open(img_fn)
img
plt.imshow(img)
import numpy
imga = numpy.array(img)
imga.shape
imga[:10, :10, 0]<jupyter_output><empty_output><jupyter_text>Just in case there are other files in the directory (models, texts...) we want to keep only the images. Let's not write it out by hand, but instead use what's already on our computer (the MIME types database).<jupyter_code>#export
image_extensions = set(
    k for k, v in mimetypes.types_map.items() if v.startswith('image/'))
' '.join(image_extensions)
#export
def setify(o):
    return o if isinstance(o, set) else set(listify(o))
test_eq(setify('aa'), {'aa'})
test_eq(setify(['aa', 1]), {'aa', 1})
test_eq(setify(None), set())
test_eq(setify(1), {1})
test_eq(setify({1}), {1})<jupyter_output><empty_output><jupyter_text>Now let's walk through the directories and grab all the images. The first private function grabs all the images inside a given directory and the second one walks (potentially recursively) through all the folder in `path`. [Jump_to lesson 11 video](https://course.fast.ai/videos/?lesson=11&t=1325)<jupyter_code># export
def _get_files(p, fs, extensions=None):
    p = Path(p)
    res = [p/f for f in fs if not f.startswith('.')
           and ((not extensions) or f'.{f.split(".")[-1].lower()}' in extensions)]
    return res<jupyter_output><empty_output><jupyter_text>`os.scandir()` takes a path and returns the content of that path. You can learn more [here](https://docs.python.org/3/library/os.htmlos.scandir). `os.scandir()` is very fast, as it's a thin wrapper around a C function.<jupyter_code>t = [o.name for o in os.scandir(path_tench)]
t = _get_files(path, t, extensions=image_extensions)
t[:3]<jupyter_output><empty_output><jupyter_text>`os.walk()` generates the files in a directory tree by walking the tree either top-down or bottom-up. You can learn more [here](https://docs.python.org/3/library/os.htmlos.walk). `os.walk()` internally relies on `os.scandir()`.<jupyter_code>#export
def get_files(path, extensions=None, recurse=False, include=None):
    path = Path(path)
    extensions = setify(extensions)
    extensions = {e.lower() for e in extensions}
    if recurse:
        res = []
        for i, (p, d, f) in enumerate(os.walk(path)):  # returns (dirpath, dirnames, filenames)
            if include is not None and i == 0:
                d[:] = [o for o in d if o in include]
            else:
                d[:] = [o for o in d if not o.startswith('.')]
            res += _get_files(p, f, extensions)
        return res
    else:
        f = [o.name for o in os.scandir(path) if o.is_file()]
        return _get_files(path, f, extensions)
get_files(path_tench, image_extensions)[:3]<jupyter_output><empty_output><jupyter_text>We need the recurse argument when we start from `path` since the pictures are two level below in directories.<jupyter_code>get_files(path, image_extensions, recurse=True)[:3]
all_fns = get_files(path, image_extensions, recurse=True)
len(all_fns)<jupyter_output><empty_output><jupyter_text>Imagenet is 100 times bigger than imagenette, so we need this to be fast.<jupyter_code>%timeit -n 10 get_files(path, image_extensions, recurse=True)<jupyter_output>49.5 ms ± 973 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)<jupyter_text>Prepare for modelingThis is the list of things that, eventually, form a `DataBlock`. What we need to do:- Get files- Split validation set  - random%, folder name, csv, ...- Label:   - folder name, file name/re, csv, ...- Transform per image (optional)- Transform to tensor- DataLoader- Transform per batch (optional)- DataBunch- Add test set (optional) [Jump_to lesson 11 video](https://course.fast.ai/videos/?lesson=11&t=1728)  Get files We use the `ListContainer` class from notebook 06 to store our objects in an `ItemList`. The `get` method will need to be subclassed to explain how to access an element (open an image for instance), then the private `_get` method can allow us to apply any additional transform to it.`compose` is based on functional programming concepts and what it does is to take an object `x` and a list of functions and  and in turn transform `x` by applying the functions in the list and assign the result to `x` itself. By doing this, we are composing multiple transformations.```pythondef compose(x, funcs, *args, order_key='_order', **kwargs):    def key(o):        return getattr(o, order_key, 0)    for f in sorted(listify(funcs), key=key):        x = f(x, **kwargs)    return x````compose` has in addition an optional ordering key and some keyword arguments. Python contains a number of functions to implement functional programming concepts. You can learn more [here](https://docs.python.org/3.8/library/functools.html).In our case, `compose` is used in `get(self, i)` and applies the transformation functions in `tfms` to the item `i`.`new` will be used in conjunction with `__getitem__` (that works for one index or a list of indices) to create training and validation set from a single stream when we split the data.`ImageList` takes a list of files and passes them to the class constructor `cls` which expects a list of items `items`.<jupyter_code>#export
def compose(x, funcs, *args, order_key='_order', **kwargs):
    def key(o):
        return getattr(o, order_key, 0)
    for f in sorted(listify(funcs), key=key):
        x = f(x, **kwargs)
    return x


class ItemList(ListContainer):
    def __init__(self, items, path='.', tfms=None):
        super().__init__(items)
        self.path = Path(path)
        self.tfms = tfms

    def __repr__(self):
        return f'{super().__repr__()}\nPath: {self.path}'

    def new(self, items, cls=None):
        if cls is None:
            cls = self.__class__
        return cls(items, self.path, tfms=self.tfms)

    def get(self, i):
        return i

    def _get(self, i):
        return compose(self.get(i), self.tfms)

    def __getitem__(self, idx):
        res = super().__getitem__(idx)
        if isinstance(res, list):
            return [self._get(o) for o in res]
        return self._get(res)


class ImageList(ItemList):
    @classmethod
    def from_files(cls, path, extensions=None, recurse=True, include=None, **kwargs):
        if extensions is None:
            extensions = image_extensions
        return cls(
            get_files(path, extensions, recurse=recurse, include=include),
            path, **kwargs)

    def get(self, fn): return PIL.Image.open(fn)<jupyter_output><empty_output><jupyter_text>One frequent problem is that sometimes there are black and white images mixed with the RGB ones. When PIL reads a black and white image, it returns a rank 2 tensor (just x and y, no channels), and we cannot stack it into a mini-batch. This is why we are adding the `MakeRGB` class below. We could, instead, just use a function, like `make_rgb()` below. Both approaches are fine.Transforms aren't only used for data augmentation. To allow total flexibility, `ImageList` returns the raw PIL image. The first thing is to convert it to 'RGB' (or something else).Transforms only need to be functions that take an element of the `ItemList` and transform it. If they need state, they can be defined as a class. Also, having them as a class allows to define an `_order` attribute (default 0) that is used to sort the transforms.<jupyter_code>#export
class Transform():
    _order = 0


class MakeRGB(Transform):
    def __call__(self, item):
        return item.convert('RGB')


def make_rgb(item):
    return item.convert('RGB')
il = ImageList.from_files(path, tfms=make_rgb)
il
img = il[0]; img<jupyter_output><empty_output><jupyter_text>We can also index with a range or a list of integers:<jupyter_code>il[:1]<jupyter_output><empty_output><jupyter_text>Split validation set Here, we need to split the files between those in the folder train and those in the folder val.<jupyter_code>fn = il.items[0]
fn<jupyter_output><empty_output><jupyter_text>Since our filenames are `path` object, we can find the directory of the file with `.parent`. We need to go back two folders before since the last folders are the class names.<jupyter_code>fn.parent.parent.name<jupyter_output><empty_output><jupyter_text>[Jump_to lesson 11 video](https://course.fast.ai/videos/?lesson=11&t=2175)The function `grandparent_splitter()` returns a boolean value that equals `True` if we are working with the validation set, `False` if it's the training set, and `None` otherwise.<jupyter_code>#export
def grandparent_splitter(fn, valid_name='valid', train_name='train'):
    gp = fn.parent.parent.name
    return True if gp == valid_name else False if gp == train_name else None


def split_by_func(items, f):
    mask = [f(o) for o in items]
    # `None` values will be filtered out
    f = [o for o, m in zip(items, mask) if m == False]
    t = [o for o, m in zip(items, mask) if m == True]
    return f, t
splitter = partial(grandparent_splitter, valid_name='val')
%time train,valid = split_by_func(il, splitter)
len(train), len(valid)<jupyter_output><empty_output><jupyter_text>Now that we can split our data, let's create the class that will contain it. It just needs two `ItemList` to be initialized, and we create a shortcut to all the unknown attributes by trying to grab them in the `train` `ItemList`.The `__getattr__()` method makes it so that if we look for an attributes it doesn't know about it will grab it from the training set.`il.new` uses the `new()` method in the `ItemList`, shown below.```pythondef new(self, items, cls=None):    if cls is None:        cls = self.__class__    return cls(items, self.path, tfms=self.tfms)````new()` is going to create a new `ItemList` with the same path, transforms etc, but with those `items`.<jupyter_code>#export
class SplitData():
    def __init__(self, train, valid):
        self.train = train
        self.valid = valid

    def __getattr__(self, k):
        return getattr(self.train, k)
    # This is needed if we want to pickle SplitData and be able to load it back without recursion errors

    def __setstate__(self, data: Any):
        self.__dict__.update(data)

    @classmethod
    def split_by_func(cls, il, f):
        lists = map(il.new, split_by_func(il.items, f))
        return cls(*lists)

    def __repr__(self):
        return f'{self.__class__.__name__}\nTrain: {self.train}\nValid: {self.valid}\n'
sd = SplitData.split_by_func(il, splitter)
sd<jupyter_output><empty_output><jupyter_text>Labeling Labeling has to be done *after* splitting, because it uses *training* set information to apply to the *validation* set, using a *Processor*.A *Processor* is a transformation that is applied to all the inputs once at initialization, with some *state* computed on the training set that is then applied without modification on the validation set (and maybe the test set or at inference time on a single item). For instance, it could be **processing texts** to **tokenize**, then **numericalize** them. In that case we want the validation set to be numericalized with exactly the same vocabulary as the training set.Another example is in **tabular data**, where we **fill missing values** with (for instance) the median computed on the training set. That statistic is stored in the inner state of the *Processor* and applied on the validation set.In our case, we want to **convert label strings to numbers** in a consistent and reproducible way. So we create a list of possible labels in the training set, and then convert our labels to numbers based on this *vocab*. [Jump_to lesson 11 video](https://course.fast.ai/videos/?lesson=11&t=2368)<jupyter_code>#export
from collections import OrderedDict


def uniqueify(x, sort=False):
    res = list(OrderedDict.fromkeys(x).keys())
    if sort:
        res.sort()
    return res<jupyter_output><empty_output><jupyter_text>First, let's define the processor. We also define a `ProcessedItemList` with an `obj` method that can get the unprocessed items: for instance a processed label will be an index between 0 and the number of classes - 1, the corresponding `obj` will be the name of the class. The first one is needed by the model for the training, but the second one is better for displaying the objects.<jupyter_code># export
class Processor():
    def process(self, items):
        return items


class CategoryProcessor(Processor):
    def __init__(self):
        self.vocab = None

    def __call__(self, items):
        # The vocab is defined on the first use.
        if self.vocab is None:
            self.vocab = uniqueify(items)
            self.otoi = {v: k for k, v in enumerate(self.vocab)}
        return [self.proc1(o) for o in items]

    def proc1(self, item):
        return self.otoi[item]

    def deprocess(self, idxs):
        assert self.vocab is not None
        return [self.deproc1(idx) for idx in idxs]

    def deproc1(self, idx):
        return self.vocab[idx]<jupyter_output><empty_output><jupyter_text>Here we label according to the folders of the images, so simply `fn.parent.name`. We label the training set first with a newly created `CategoryProcessor` so that it computes its inner `vocab` on that set. Then we label the validation set using the same processor, which means it uses the same `vocab`. The end result is another `SplitData` object.<jupyter_code>#export
def parent_labeler(fn):
    return fn.parent.name


def _label_by_func(ds, f, cls=ItemList):
    return cls([f(o) for o in ds.items], path=ds.path)

# This is a slightly different from what was seen during the lesson,
#   we'll discuss the changes in lesson 11


class LabeledData():
    def process(self, il, proc):
        return il.new(compose(il.items, proc))

    def __init__(self, x, y, proc_x=None, proc_y=None):
        self.x = self.process(x, proc_x)
        self.y = self.process(y, proc_y)
        self.proc_x = proc_x
        self.proc_y = proc_y

    def __repr__(self):
        return f'{self.__class__.__name__}\nx: {self.x}\ny: {self.y}\n'

    def __getitem__(self, idx):
        return self.x[idx], self.y[idx]

    def __len__(self):
        return len(self.x)

    def x_obj(self, idx):
        return self.obj(self.x, idx, self.proc_x)

    def y_obj(self, idx):
        return self.obj(self.y, idx, self.proc_y)

    def obj(self, items, idx, procs):
        isint = isinstance(idx, int) or (isinstance(
            idx, torch.LongTensor) and not idx.ndim)
        item = items[idx]
        for proc in reversed(listify(procs)):
            item = proc.deproc1(item) if isint else proc.deprocess(item)
        return item

    @classmethod
    def label_by_func(cls, il, f, proc_x=None, proc_y=None):
        return cls(il, _label_by_func(il, f), proc_x=proc_x, proc_y=proc_y)


def label_by_func(sd, f, proc_x=None, proc_y=None):
    train = LabeledData.label_by_func(
        sd.train, f, proc_x=proc_x, proc_y=proc_y)
    valid = LabeledData.label_by_func(
        sd.valid, f, proc_x=proc_x, proc_y=proc_y)
    return SplitData(train, valid)
ll = label_by_func(sd, parent_labeler, proc_y=CategoryProcessor())
assert ll.train.proc_y is ll.valid.proc_y
ll.train.y
ll.train.y.items[0], ll.train.y_obj(0), ll.train.y_obj(slice(2))
ll<jupyter_output><empty_output><jupyter_text>Transform to tensorWe still cannot train a model because we have PIL objects, not tensors. [Jump_to lesson 11 video](https://course.fast.ai/videos/?lesson=11&t=3044)<jupyter_code>ll.train[0]
ll.train[0][0]<jupyter_output><empty_output><jupyter_text>To be able to put all our images in a batch, we need them to have all the same size. We can do this easily in PIL. Apparently it's not a great way to do it, but it's a start.<jupyter_code>ll.train[0][0].resize((128,128))<jupyter_output><empty_output><jupyter_text>A better way is to use a transform. It has to be after all the other transforms we have seen so far.The first transform resizes to a given size, then we convert the image to a byte tensor before converting it to float and dividing by 255. This has to happen after resizing, and the conversion to float tensors must happen after the conversion to byte tensors. Note that we can just add state to objects with simple assignments, as is the case of `to_byte_tensor.order = 20`.We will investigate data augmentation transforms at length in notebook 10.Pillow puts the channel last, while PyTorch assumes that the channel comes first. This is why we have a `.permute(2, 0, 1)`.<jupyter_code>#export
class ResizeFixed(Transform):
    _order = 10

    def __init__(self, size):
        if isinstance(size, int):
            size = (size, size)
        self.size = size

    def __call__(self, item):
        return item.resize(self.size, PIL.Image.BILINEAR)


def to_byte_tensor(item):
    res = torch.ByteTensor(torch.ByteStorage.from_buffer(item.tobytes()))
    w, h = item.size
    return res.view(h, w, -1).permute(2, 0, 1)


to_byte_tensor._order = 20


def to_float_tensor(item):
    return item.float().div_(255.)


to_float_tensor._order = 30
tfms = [make_rgb, ResizeFixed(128), to_byte_tensor, to_float_tensor]

il = ImageList.from_files(path, tfms=tfms)
sd = SplitData.split_by_func(il, splitter)
ll = label_by_func(sd, parent_labeler, proc_y=CategoryProcessor())<jupyter_output><empty_output><jupyter_text>Here is a little convenience function to show an image from the corresponding tensor.<jupyter_code>#export
def show_image(im, figsize=(3, 3)):
    plt.figure(figsize=figsize)
    plt.axis('off')
    plt.imshow(im.permute(1, 2, 0))
x, y = ll.train[0]
x.shape
show_image(x)<jupyter_output><empty_output><jupyter_text>Modeling  DataBunch Now we are ready to put our datasets together in a `DataBunch`. [Jump_to lesson 11 video](https://course.fast.ai/videos/?lesson=11&t=3226)<jupyter_code>bs = 64
train_dl, valid_dl = get_dls(ll.train, ll.valid, bs, num_workers=4)
x, y = next(iter(train_dl))
x.shape<jupyter_output><empty_output><jupyter_text>We can still see the images in a batch and get the corresponding classes.<jupyter_code>show_image(x[0])
ll.train.proc_y.vocab[y[0]]
y<jupyter_output><empty_output><jupyter_text>We change a little bit our `DataBunch` to add a few attributes: `c_in` (for channel in) and `c_out` (for channel out) instead of just `c`. This will help when we need to build our model.<jupyter_code>#export
class DataBunch():
    def __init__(self, train_dl, valid_dl, c_in=None, c_out=None):
        self.train_dl = train_dl
        self.valid_dl = valid_dl
        self.c_in = c_in
        self.c_out = c_out

    @property
    def train_ds(self):
        return self.train_dl.dataset

    @property
    def valid_ds(self):
        return self.valid_dl.dataset<jupyter_output><empty_output><jupyter_text>Then we define a function that goes directly from the `SplitData` to a `DataBunch`. **Question** shall we include this directly into the `SplitData` class. This is a "monkey patch", according to J.H.<jupyter_code>#export
def databunchify(sd, bs, c_in=None, c_out=None, **kwargs):
    dls = get_dls(sd.train, sd.valid, bs, **kwargs)
    return DataBunch(*dls, c_in=c_in, c_out=c_out)


SplitData.to_databunch = databunchify<jupyter_output><empty_output><jupyter_text>This gives us the full summary on how to grab our data and put it in a `DataBunch`:<jupyter_code>path = datasets.untar_data(datasets.URLs.IMAGENETTE_160)
tfms = [make_rgb, ResizeFixed(128), to_byte_tensor, to_float_tensor]

il = ImageList.from_files(path, tfms=tfms)
sd = SplitData.split_by_func(il, partial(
    grandparent_splitter, valid_name='val'))
ll = label_by_func(sd, parent_labeler, proc_y=CategoryProcessor())
data = ll.to_databunch(bs, c_in=3, c_out=10, num_workers=4)<jupyter_output><empty_output><jupyter_text>Model [Jump_to lesson 11 video](https://course.fast.ai/videos/?lesson=11&t=3360)<jupyter_code>cbfs = [partial(AvgStatsCallback, accuracy),
        CudaCallback]<jupyter_output><empty_output><jupyter_text>We will normalize with the statistics from a batch.<jupyter_code>m, s = x.mean((0, 2, 3)).cuda(), x.std((0, 2, 3)).cuda()
m, s
#export
def normalize_chan(x, mean, std):
    return (x-mean[..., None, None]) / std[..., None, None]


_m = tensor([0.47, 0.48, 0.45])
_s = tensor([0.29, 0.28, 0.30])

norm_imagenette = partial(normalize_chan, mean=_m.cuda(), std=_s.cuda())
cbfs.append(partial(BatchTransformXCallback, norm_imagenette))
nfs = [64, 64, 128, 256]<jupyter_output><empty_output><jupyter_text>We build our model using [Bag of Tricks for Image Classification with Convolutional Neural Networks](https://arxiv.org/abs/1812.01187), in particular: we don't use a big conv 7x7 at first but three 3x3 convs, and don't go directly from 3 channels to 64 but progressively add those.`l1 = data.c_in` reads the number of input channels automatically. If we change the type of images from 3 to 1 (or 4 or whatever) channels, we don't have to change anything.If the input has `c_in` channels and a kernel of size 3, then we have 9 x `c_in` numbers coming in, and we want to have *fewer* numbers going out. The function `prev_pow_2` takes a number `x` and looks for the closest power of two that is smaller or equal to it. Then, for the next two layers, we multiply by 2. In the `Bag of Tricks` paper there is an analysis of why 3x3 gives "more bang for your buck". In the old days we used to use 5x5 and 7x7 kernels, but we now know that this is not in general a good idea.The ideas in the bagk of tricks are in turn based on the [Zeiler and Fergus 2013 paper](https://arxiv.org/abs/1311.2901).<jupyter_code>#export
import math


def prev_pow_2(x):
    return 2**math.floor(math.log2(x))


def get_cnn_layers(data, nfs, layer, **kwargs):
    def f(ni, nf, stride=2):
        return layer(ni, nf, 3, stride=stride, **kwargs)
    l1 = data.c_in
    l2 = prev_pow_2(l1 * 3 * 3)
    layers = [f(l1, l2, stride=1),
              f(l2, l2 * 2, stride=2),
              f(l2 * 2, l2 * 4, stride=2)]
    nfs = [l2 * 4] + nfs
    layers += [f(nfs[i], nfs[i + 1]) for i in range(len(nfs) - 1)]
    layers += [nn.AdaptiveAvgPool2d(1), Lambda(flatten),
               nn.Linear(nfs[-1], data.c_out)]
    return layers


def get_cnn_model(data, nfs, layer, **kwargs):
    return nn.Sequential(*get_cnn_layers(data, nfs, layer, **kwargs))


def get_learn_run(nfs, data, lr, layer, cbs=None, opt_func=None, **kwargs):
    model = get_cnn_model(data, nfs, layer, **kwargs)
    init_cnn(model)
    return get_runner(model, data, lr=lr, cbs=cbs, opt_func=opt_func)
sched = combine_scheds([0.3, 0.7], cos_1cycle_anneal(0.1, 0.3, 0.05))
learn, run = get_learn_run(nfs, data, 0.2, conv_layer, cbs=cbfs+[
    partial(ParamScheduler, 'lr', sched)
])<jupyter_output><empty_output><jupyter_text>And we can train the model:<jupyter_code>%time run.fit(5, learn)<jupyter_output>train: [1.7745269926866618, tensor(0.3901, device='cuda:0')]
valid: [1.5689654657643313, tensor(0.4810, device='cuda:0')]
train: [1.4052108360175308, tensor(0.5297, device='cuda:0')]
valid: [1.3637435310509554, tensor(0.5531, device='cuda:0')]
train: [1.052731796685236, tensor(0.6464, device='cuda:0')]
valid: [1.1926710539410827, tensor(0.6018, device='cuda:0')]
train: [0.6749605208443341, tensor(0.7875, device='cuda:0')]
valid: [1.1280363007563694, tensor(0.6438, device='cuda:0')]
train: [0.3776792300038283, tensor(0.8995, device='cuda:0')]
valid: [1.1151105941480892, tensor(0.6510, device='cuda:0')]
CPU times: user 11.1 s, sys: 3.59 s, total: 14.7 s
Wall time: 21.8 s<jupyter_text>It is very often to break things when experimenting with these tools, and the only way to understand what's going on is to see what the network contains. This is why we define the `model_summary()` function. This insights and telemetry tools are very important for debugging.Let's have a look at our model using Hooks. We print the layers and the shapes of their outputs.<jupyter_code>#export
def model_summary(run, learn, data, find_all=False):
    xb, yb = get_batch(data.valid_dl, run)
    # Model may not be on the GPU yet
    device = next(learn.model.parameters()).device
    xb, yb = xb.to(device), yb.to(device)
    mods = find_modules(
        learn.model, is_lin_layer) if find_all else learn.model.children()

    def f(hook, mod, inp, out):
        return print(f"{mod}\n{out.shape}\n")
    with Hooks(mods, f) as hooks:
        learn.model(xb)
# This must be called after training the model, not before as in the video.
# If you do as in the original notebook you get the following
# AttributeError: 'Runner' object has no attribute 'in_train'
model_summary(run, learn, data)<jupyter_output>Sequential(
  (0): Conv2d(3, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
  (1): GeneralRelu()
  (2): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
torch.Size([128, 16, 128, 128])

Sequential(
  (0): Conv2d(16, 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
  (1): GeneralRelu()
  (2): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
torch.Size([128, 32, 64, 64])

Sequential(
  (0): Conv2d(32, 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
  (1): GeneralRelu()
  (2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
torch.Size([128, 64, 32, 32])

Sequential(
  (0): Conv2d(64, 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
  (1): GeneralRelu()
  (2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
torch.Size([128, 64, 16, 16])

Sequential(
  (0): Conv2d(64, 64, kernel_size=([...]<jupyter_text>The [leaderboard](https://github.com/fastai/imagenette/blob/master/README.md) as this notebook is written has ~85% accuracy for 5 epochs at 128px size, so we're definitely on the right track!  Export<jupyter_code>!python notebook2script.py 08_data_block.ipynb<jupyter_output>Converted 08_data_block.ipynb to exp/nb_08.py
{'fraction_non_alphanumeric': 0.10947325018039378, 'fraction_numerical': 0.025392571212589768, 'mean_word_length': 4.347051258497152, 'pattern_counts': {'":': 0, '<': 146, '<?xml version=': 0, '>': 146, 'https://': 15, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '37', 'dir': 'jupyter-structured-clean-dedup', 'id': '212279', 'license': 'Apache-2.0', 'n_tokens_mistral': 10103, 'n_tokens_neox': 9480, 'n_words': 3377, 'path': 'nbs/dl2/08_data_block.ipynb', 'repo_name': 'gdario/course-v3', 'size': '29111'}