<jupyter_start><jupyter_text>Image segmentation with CamVid<jupyter_code>%reload_ext autoreload
%autoreload 2
%matplotlib inline
from fastai.vision import *
from fastai.callbacks.hooks import *
from fastai.utils.mem import *
path = untar_data(URLs.CAMVID)
path.ls()
codes = np.loadtxt(path/'codes.txt', dtype=str); codes, len(codes)
path_lbl = path/'labels' # These are masks. They colour code the actual images.
path_img = path/'images'
# A little function that basically took the filename and added the _P and put it in the path "path_lbl".
get_y_fn = lambda x: path_lbl/f'{x.stem}_P{x.suffix}'
fnames = get_image_files(path_img)
img_f = fnames[0] # Pick the first image arbitrarily
mask = open_mask(get_y_fn(img_f)) 
src_size = np.array(mask.shape[1:])
src_size<jupyter_output><empty_output><jupyter_text>Understanding What's Going On with Individual Examples You can skip this section if you know what's up.<jupyter_code>fnames = get_image_files(path_img)
fnames[:3]
# These are masks
lbl_names = get_image_files(path_lbl)
lbl_names[:3]
img_f = fnames[0] # Pick the first image arbitrarily
img = open_image(img_f)
img.show(figsize=(5,5))
# Load the corresponding mask of the first image. Same time we ensure that the fn is implemented properly

mask = open_mask(get_y_fn(img_f)) 
print(mask.data)
mask.show(figsize=(5,5), alpha=1)<jupyter_output><empty_output><jupyter_text>TY - Since the above is a mask, we use `open_mask` rather than `open_image` for normal images. Had we use `open_image` for a mask, we would get a very dark image. You can see for yourself by visually inspecting the masks through your explorer. TY - The 720x960 is the dimension of the mask. Note all figures are in integers. You can think of each individual integer as a pixel. However the number is not greyscale value. It instead varies from 0 to 31, because in `codes.txt`, we noted 32 distinct categories. Each integer represents a particular component in the image (like 'Building' or Wall'). fastai's `open_mask()` natively converts each distinct integer as a unique colour (colour code) for easy viewing. An ordinary image viewer that does not know this is a mask, would treat the integers as greyscale values.  Datasets<jupyter_code>free = gpu_mem_get_free_no_cache()
# the max size of bs depends on the available GPU RAM
if free > 8200: bs=8
else:           bs=4
print(f"using bs={bs}, have {free}MB of GPU RAM free")<jupyter_output>using bs=4, have 5991MB of GPU RAM free<jupyter_text>Every pixel is a classifier. We need to keep batch size small because that's a lot of pixels for our GPU to manage at a time. The people that created this dataset actually gave us a list of file names (valid.txt) that are meant to be in your validation set and they are non-contiguous parts of the video. So here's how you can split your validation and training using a file name file. In this case I don't do it randomly because the pictures they've given us are frames from videos. If I did them randomly I would be having two frames next to each other: one in the validation set, one in the training set. That would be far too easy and treating.<jupyter_code>src = (SegmentationItemList.from_folder(path_img)
       .split_by_fname_file('../valid.txt')
       .label_from_func(get_y_fn, classes=codes))
data = (src.transform(get_transforms(), 
                      size=src_size//2,
                      tfm_y=True)  # Transformation: If flip a particular image, the trfm must also apply to Y (the target mask)
        .databunch(bs=bs)
        .normalize(imagenet_stats))
data.show_batch(2, figsize=(10,7))
data.show_batch(2, figsize=(10,7), ds_type=DatasetType.Valid)<jupyter_output><empty_output><jupyter_text>Model  Accuracy FunctionAccuracy for pixel-wise segmentation is basically `correctly classified pixels / total number of pixels`.You can imagine each pixel was a separate object you're classifying, it's exactly the same accuracy. So you actually can just pass in accuracy as your metric, but in this case, we actually don't. The reason for creating a new metric called `acc_camvid` is that in the CamVid paper, they say when you're reporting accuracy, you should remove the void pixels. So all metrics take the actual output of the neural net (i.e. that's the input to the metric) and the target (i.e. the labels we are trying to predict).<jupyter_code>name2id = {v:k for k,v in enumerate(codes)}
void_code = name2id['Void']

def acc_camvid(input, target):
    target = target.squeeze(1)
    mask = target != void_code
    return (input.argmax(dim=1)[mask]==target[mask]).float().mean()<jupyter_output><empty_output><jupyter_text>The above function `acc_camvid` seems to work in mysterious ways. Let me try to elucidate.`input` and `target` are both tensors (arrays). They both represent one image at a time - `acc_camvid` is called for every validation check on an image in the validatation dataset. We squeeze `target`, which changes it from `1 x 720 x 960` to `720 x 960`. (Assuming we did not resize)`input` tensor is the predicted tensor of a given image. Its dimensions are `720 x 960 x 32` (I might get the axis ordering wrong). 32 represents the 32 different categories. Remember that each pixel in the predicted tensor is a probability distribution spread over 32 categories. This is why we will use `argmax` to extract the index for which the probability (out of 32 categories) is the maximum. This index as returned by `argmax` will just so happen to refer to the notation used to indicate which category each pixel it belong in. `argmax` as a result collapses the 32-wide dimension into a single integer, and now the `input` tensor has the same dimension as the squeezed `target`. We can do a pixel-wise boolean comparision to return a boolean tensor. This boolean tensor will consists of mostly 1s if our `input` tensor (i.e. the prediction) is highly similar to its taget, or be mostly 0s if it's poor. We find the mean value; number of 1s divided by total no. of pixels, to get a result accuracy resulting value between 0 to 1.As for the non-void, this is hard to piece out. Like, if the predicted image is entirely void, I don't know what is the resulting computation of accuracy.<jupyter_code>metrics=acc_camvid
# metrics=accuracy
wd=1e-2 # Weight decay
learn = unet_learner(data, models.resnet34, metrics=metrics, wd=wd)
lr_find(learn)
learn.recorder.plot()
lr=3e-3
learn.fit_one_cycle(10, slice(lr), pct_start=0.9)

# pct_start is the percentage of overall iterations where the LR is increasing
# Recall one epoch can have many iterations (unless your batch size is equal to entire data set -> you then have one iteration/epoch)
# So, given the default of 0.3, means LR is going up for 30% of your iterations and then decreasing over the last 70%.<jupyter_output><empty_output><jupyter_text>TY - `pct-start` seems weird, but I talk about it below.<jupyter_code>learn.save('stage-1')
learn.load('stage-1');
learn.show_results(rows=3, figsize=(8,9))
learn.unfreeze()
lrs = slice(lr/400,lr/4)
learn.fit_one_cycle(12, lrs, pct_start=0.8)
learn.save('stage-2');<jupyter_output><empty_output><jupyter_text>Go big Further training on the same images, but bigger. Progressive resizing. Note that we got 0.936091. You may have to restart your kernel and come back to this stage if you run out of memory, and may also need to decrease `bs`.We are stepping up on our learning, by training on a bigger picture. Where `size = src_size` (720\*960). Previously, we set `size = src_size//2`.We have to destroy our `learn` and make a new one. Because there is a change in: (1) The dimensions of our image, and (2) Batch size, which become smaller as it's now more computationally heavy on our GPU.Destroying our `learn` does not mean we starting from scratch. Actually once we re-initialised our new learning model, we can reload our original parameters. Of course, since the input layer is changed, I honestly don't know how the previous parameters can be mapped to this new setup.<jupyter_code>learn.destroy()

free = gpu_mem_get_free_no_cache()
# the max size of bs depends on the available GPU RAM
if free > 8200: bs=3
else:           bs=1
print(f"using bs={bs}, have {free}MB of GPU RAM free")
data = (src.transform(get_transforms(), 
                      size = src_size, 
                      tfm_y=True)
        .databunch(bs=bs)
        .normalize(imagenet_stats))
learn = unet_learner(data, models.resnet34, metrics=metrics, wd=wd)
learn.load('stage-2');
lr_find(learn)
learn.recorder.plot()
lr=1e-3
learn.fit_one_cycle(10, slice(lr), pct_start=0.8)
learn.save('stage-1-big')
learn.load('stage-1-big');
learn.unfreeze()
lr=1e-3 # From Stage-1
lrs = slice(1e-6,lr/10)
learn.fit_one_cycle(10, lrs)

# Note that we got 0.936091.
learn.save('stage-2-big')
learn.load('stage-2-big');
learn.show_results(rows=3, figsize=(10,10))
learn.recorder.plot_losses()
learn.recorder.plot_lr()<jupyter_output><empty_output><jupyter_text>Let's try FP16 I don't think it works.<jupyter_code>free = gpu_mem_get_free_no_cache()
# the max size of bs depends on the available GPU RAM
if free > 8200: bs=3
else:           bs=1
print(f"using bs={bs}, have {free}MB of GPU RAM free")
src = (SegmentationItemList.from_folder(path_img)
       .split_by_fname_file('../valid.txt')
       .label_from_func(get_y_fn, classes=codes))

data = (src.transform(get_transforms(), 
                      size = src_size, 
                      tfm_y=True)
        .databunch(bs=bs)
        .normalize(imagenet_stats))

learn.destroy()
learn = unet_learner(data, models.resnet34, metrics=metrics, wd=wd).to_fp16()
learn.load('stage-2');
lr_find(learn)
learn.recorder.plot()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08894404798841658, 'fraction_numerical': 0.01871961940221326, 'mean_word_length': 4.551090700344432, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 40, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '569159', 'license': 'Apache-2.0', 'n_tokens_mistral': 3098, 'n_tokens_neox': 2934, 'n_words': 1210, 'path': 'nbs/dl1/lesson3-camvid.ipynb', 'repo_name': 'ktingyew/course-v3', 'size': '9669'}