<jupyter_start><jupyter_text>Multi-output classification for captcha recognition using fastai> Implementing a multi-output model to recognize easy captcha images using the fastai library- toc: true - badges: true- comments: true- categories: [machine-learning, deep-learning, multi-output-model, classification, dataloader, fastai, vision]- author: <NAME>- image: images/captcha.png  TLDRSpecial thanks to my dear friends [Julius](https://github.com/jul1u5) and [Laurynas](https://github.com/laurislopata) who joined me to work on this project.We worked on and off for more than a month on this project, and I must say that it was not an easy, but definitely an extremely rewarding experience considering how much I've learnt. It was also a good chance to hone my problem solving skills and endurance, as we switched approaches (and even libraries!) at least three times, spent substantial amounts of time reading through various resources and codebases and trying to adapt the codes to our purpose through trial and error.**The key takeaways are:**1. **Do not ever use old captchas** and encourage others not to, as there are too many ways to break them nowadays.2. Fastai has a **great mid-level API** that allows for customization for almost any use, but getting the hang of it might be tricky. Here are some resources that tremendously helped us:    - Looking at fastai [docs and tutorials](https://docs.fast.ai/) and reading through source code    - [A walk with fastai2](https://www.youtube.com/playlist?list=PLFDkaGxp5BXDvj3oHoKDgEcH73Aze-eET) playlist that has excellent material on using the fastai mid-level API    - [fastai v2 walk-thru](https://www.youtube.com/playlist?list=PLfYUBJiXbdtSWRCYUHh-ThVCC39bp5yiq) playlist on YouTube where <NAME> talks how and why the library is built the way it is3. **Perseverance is key** - great ideas and solutions usually do not come overnight, but breaking the problem into smaller pieces and continously improving on each of those is a good way to go.  IntroductionThe idea is to use deep learning to recognise images from [this](https://github.com/PacktPublishing/Machine-Learning-for-Cybersecurity-Cookbook/blob/master/Chapter05/CAPTCHA%20Breaker/captcha_images.7z) captcha image dataset. The images look like the following:![](my_icons/captcha.png "Example captcha image")Our final approach was the following:1. Create a data loader that would load in an image and a length four tensor that would represent the encoded symbol at each position in the captcha2. Create a multi-output model that would output a 128 length tensor and then reshape it to a 32x4 tensor    - The output is 32x4 because there are 32 possible letters (the dataset that we used had no `0`, `1`, `O` or `I`) for each position, i.e. model has four outputs, each predicting one of the 32 symbols for each position in the captcha3. Use `CrossEntropyLoss` with `reduction="sum"` which will calculate the losses for each of the four outputs and sum them up. This way, we can easily build a model to predict multiple things at the same timeEverything looked rather easy at first, however, it took us quite some time to come up with an elegant and easy to follow implementation. Continue reading for details!  ImportsToggle cells below if you want to see what imports are being made.<jupyter_code>#hide
# Uncomment these and run once if you are using colab
# !pip install -qq torchtext==0.8.1
# !pip install -Uqq fastbook
#hide
%load_ext autoreload
%autoreload 2

%matplotlib inline
#collapse-hide
from fastai.vision.all import *
import PIL<jupyter_output><empty_output><jupyter_text>Data<jupyter_code>#hide
# Run this once to download the data
# !wget -q --no-check-certificate --content-disposition https://github.com/PacktPublishing/Machine-Learning-for-Cybersecurity-Cookbook/blob/master/Chapter05/CAPTCHA%20Breaker/captcha_images.7z?raw=true
#hide
# Run this once to unzip the data
# !7z x captcha_images.7z
#hide
# Check that everything is as it should be
!ls captcha_images | wc<jupyter_output>9955    9955   89595<jupyter_text>Use fastai `get_image_files` utility function to get a list of paths to captcha files:<jupyter_code>PATH = Path("captcha_images")
files = get_image_files(PATH)
files[0]<jupyter_output><empty_output><jupyter_text>We separately checked that all the images are 24x72x3.Create python dictionaries that will be used to map symbols to integers and vice versa:<jupyter_code># Find all the unique labels
ld = set()
for f in files:
    for l in f.stem:
        ld.add(l)

label_mapper = "".join(sorted(ld))
l2i = { label_mapper[i]: i for i in range(len(label_mapper)) } # labels to int
i2l = { v: k for k, v in l2i.items() } # int to labels<jupyter_output><empty_output><jupyter_text>Dataset This custom thing that we need is a fastai `Transform` that knows how to deal with a multi-output label. Reading through fastai source code we've managed to find that a convenient way to do this is to create a class that inherits from `DisplayedTransform`. The attributes `l2i` and `i2l` are used for label to int mapping. The `encodes` method is how to tell the library how something should be encoded, here we just take a string type label and turn it into a `TensorCategory` which is nothing more but a length four tensor with the name given for it to have a more human-readable semantic meaning. Similarly, the `decodes` method takes an encoded label and reverses it back to a string label which is wrapped in the `Category` class that is intended for categories that will be shown somewhere, e.g. in plots.<jupyter_code>class MultiOutputCategorize(DisplayedTransform):
    def __init__(self, l2i=l2i, i2l=i2l):
        self.l2i = l2i
        self.i2l = i2l

    def encodes(self, string_label):
        return TensorCategory([self.l2i[l] for l in string_label])

    def decodes(self, encoded_label):
        return Category("".join([self.i2l[v.item()] for v in encoded_label]))<jupyter_output><empty_output><jupyter_text>We can now use our new transform to create a `DataBlock`, the cornerstone of fastai's mid-level API that is very flexible and can be customized for various purposes (a good example is [this](https://docs.fast.ai/tutorial.siamese.html) where it is used to implement a siamese network).Important thing to notice is how we wrap our `MultiOutputCategorize` in a `TransformBlock`.We also use augmentations to have more variety in our data while training the model. `CropPad` is used to pad image to a square (remember that all images were 24x72). All the other transform are pretty self explanatory, but make sure you read about them in the fastai docs if you feel like you need to.<jupyter_code>captcha_data_block = DataBlock(
    blocks=(ImageBlock, TransformBlock(type_tfms=MultiOutputCategorize())),
    splitter=RandomSplitter(valid_pct=0.2, seed=42),
    get_y=lambda x: x.stem,
    item_tfms=[CropPad(size=72, pad_mode=PadMode.Border), Resize(64)],
    batch_tfms=[*aug_transforms(do_flip=False, max_rotate=5.0)],
)<jupyter_output><empty_output><jupyter_text>Now create a dataloader, we use batch size 128 since experiments showed that it was the optimal one:<jupyter_code>dls = captcha_data_block.dataloaders(files, bs=128, num_workers=4, shuffle=True)
dls = dls.cuda()<jupyter_output><empty_output><jupyter_text>Another important thing to note is how fastai handles types behind the scenes - even though the images and labels were transformed and their actual types changed, the library manages to bubble up the correct initial types if needed. One of the authors of the library <NAME> mentions in his code walk-thrus on YouTube that such an approach was borrowed from other programming languages that he used.<jupyter_code>dls._types<jupyter_output><empty_output><jupyter_text>Let's just one batch to make sure everything is alright:<jupyter_code>dls.show_batch()<jupyter_output><empty_output><jupyter_text>Also check that our custom transform is working as it should be:<jupyter_code>b = dls.one_batch()
b[1][0]<jupyter_output><empty_output><jupyter_text>Model It is now time to build our multi-output model. Let's begin by creating the final layer that will just take the outputs and will reshape them to 32x4 (ignoring batch dimension)<jupyter_code>N_OUTPUTS = 4

class ReshapeLayer(nn.Module):
    def __init__(self, n_classes=len(l2i), n_outputs=N_OUTPUTS):
        super().__init__()
        self.n_classes = n_classes
        self.n_outputs = n_outputs
    
    def forward(self, out):
        out = out.reshape(-1, self.n_classes, self.n_outputs)

        return out<jupyter_output><empty_output><jupyter_text>Next, we use fastai's `create_body` method to get a `resnet18` model backbone pretrained on ImageNet data. We then use the `create_head` method to add a head mainly consisting of fully-connected layers to map the outputs from the convolutional backbone to a `32*4=128` length vector. The weird parameter `body[-1][-1].bn2.num_features` is actually equal to `512` and is just the number of outputs of the convolutional backbone. We found that the model does not overfit so we turn off dropout by setting `ps=0.0`:<jupyter_code>body = create_body(resnet18, cut=-2, pretrained=True)
head = create_head(body[-1][-1].bn2.num_features, N_OUTPUTS * len(l2i), ps=0.0)<jupyter_output><empty_output><jupyter_text>Combine the `body`, the `head` and the `ReshapeLayer` into a final model:<jupyter_code>model = nn.Sequential(
    body,
    head,
    ReshapeLayer(n_classes=len(l2i), n_outputs=N_OUTPUTS),
    # output will be [bs, num_classes=32, n_outputs=4]
)
model = model.cuda()<jupyter_output><empty_output><jupyter_text>This last line that casts to model to the GPU might be redundant, but we haven't checked that, so for now it is better to be safe than sorry.Finally create a function that will tell how the model's parameters have to be split when we'll use transfer learning (e.g. which parts should be frozen or not):<jupyter_code>def model_splitter(model):
    return [params(model[0]), params(model[1:])]<jupyter_output><empty_output><jupyter_text>Training We have come up to training the model.First we need some custom functions that will track our metrics. The first one calculates overall accuracy (fraction of images were the model gets **all** four outputs right) and the second one calculates the fraction of individual outputs that the model got right. For example, for captcha "ABCD", if the model guesses "ABCA", then the overall accuracy will be 0, but individual accuracy will be 75% (3 out of 4 letters guessed correctly):<jupyter_code>def multi_output_accuracy(inp, targ):
    labels_pred = torch.softmax(inp, dim=1)
    labels_pred = torch.argmax(labels_pred, dim=1)

    return ((labels_pred == targ).all(dim=1)).sum() / inp.size(0)
def multi_output_individual_accuracy(inp, targ):
    labels_pred = torch.softmax(inp, dim=1)
    labels_pred = torch.argmax(labels_pred, dim=1)

    return (labels_pred == targ).sum() / (inp.size(0) * targ.size(1))<jupyter_output><empty_output><jupyter_text>Next, we create a custom `MultiOutputCrossEntropyLoss` which is an adaption of fastai's `CrossEntropyLossFlat` for our multi-output needs.Figuring out the details of this one took a substantial amount of time, but in the end we figured out what each of the parameters do:- setting `flatten=False` turns off the default output flattening which is what we want- `reduction="sum"` makes the loss functions sum up the losses of individual outputs instead of averaging them- `axis=1` tells the loss function that softmax and argmax should operate on dim=1, in our case this means that we go from shape [bs, num_classes=32, n_output=4] to shape [bs, n_output=4] after those activations are applied which is exactly what we wantFinally, we override the `__call__` method because we noticed that the output gets weirdly transposed internally, so adding an additional transpose here reverses it and makes things work the way they should be:<jupyter_code>class MultiOutputCrossEntropyLoss(CrossEntropyLossFlat):
    def __init__(self, *args, **kwargs):
        super().__init__(flatten=False, reduction="sum", axis=1, *args, **kwargs)
    
    def __call__(self, inp, targ, **kwargs):
        # fastai does an extra transpose inside, but we do need it, so this reverses it
        inp = inp.transpose(1, 2)
        return super().__call__(inp, targ, **kwargs)<jupyter_output><empty_output><jupyter_text>Now assemble all the bits from above to create a fastai learner:<jupyter_code>learn = Learner(dls, model, loss_func=MultiOutputCrossEntropyLoss(), metrics=[multi_output_accuracy, multi_output_individual_accuracy], splitter=model_splitter)
#hide
# Could try to use `learn.fine_tune`<jupyter_output><empty_output><jupyter_text>We want to start by only training the head for a little bit, so let's freeze the convolutional backbone:<jupyter_code>learn.freeze()<jupyter_output><empty_output><jupyter_text>Look for a suitable learning rate:<jupyter_code>learn.lr_find()<jupyter_output><empty_output><jupyter_text>Train the head for 2 epochs with the found learning rate:<jupyter_code>learn.fit_one_cycle(2, 5e-3)<jupyter_output><empty_output><jupyter_text>Unfreeze the convolutional backbone:<jupyter_code>learn.unfreeze()<jupyter_output><empty_output><jupyter_text>Look for a good learning rate and then train the whole model for 3 epochs:<jupyter_code>learn.lr_find()
learn.fit_one_cycle(3, 5e-3)<jupyter_output><empty_output><jupyter_text>Look for a learning rate again (we will find it is a much smaller one) and train the model for one more epoch to get those extra bits of accuracy:<jupyter_code>learn.lr_find()
learn.fit_one_cycle(1, 1e-4)<jupyter_output><empty_output><jupyter_text>In just 6 epochs, transfer learning allowed us to reach we managed toan accuracy of 99.7% percent. Amazing!<jupyter_code>#hide
learn.save("captcha-model")<jupyter_output><empty_output><jupyter_text>Testing  Check validation set results Check the model's performance on a batch of data:<jupyter_code>learn.show_results()<jupyter_output><empty_output><jupyter_text>Get failed captchas Get predictions:<jupyter_code>inputs, _, targets, decoded_preds = learn.get_preds(with_input=True, with_decoded=True)
inputs.size(), targets.size(), decoded_preds.size()<jupyter_output><empty_output><jupyter_text>Indices of failed captchas are:<jupyter_code>failed_idxs = (~(decoded_preds == targets).all(dim=1)).nonzero().view(-1)<jupyter_output><empty_output><jupyter_text>Show those captchas to check why the model failed:<jupyter_code>dls.show_results(b=(inputs[failed_idxs], targets[failed_idxs]), out=decoded_preds[failed_idxs])<jupyter_output><empty_output><jupyter_text>We can see that the model does some slight errors, but it only fails on 4 captchas out of a validation set of 1991 images which is an amazing result. I hope that it is now obvious that using captchas to protect a website from bots is not a good idea.Check the model's confidence when it makes errors:<jupyter_code>for idx in failed_idxs.detach().cpu().numpy().flatten():
    decoded_pred, pred, model_output = learn.predict(dls.valid_ds[idx][0])
    probs = torch.softmax(model_output, dim=0)[pred].diag()
    print(f"Actual: {''.join([i2l[v.item()] for v in dls.valid_ds[idx][1]])}, predicted: {decoded_pred}, probs: {probs}")<jupyter_output><empty_output><jupyter_text>> Note: In each case the probabilities for the failed letters were **lower**.<jupyter_code>#hide
# Check a random image
idx = np.random.randint(len(files))
pred = learn.predict(files[idx])[0]
files[idx].stem, pred<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08509680324178298, 'fraction_numerical': 0.012285328359168972, 'mean_word_length': 5.571428571428571, 'pattern_counts': {'":': 0, '<': 123, '<?xml version=': 0, '>': 125, 'https://': 8, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '30', 'dir': 'jupyter-structured-clean-dedup', 'id': '433156', 'license': 'Apache-2.0', 'n_tokens_mistral': 4891, 'n_tokens_neox': 4684, 'n_words': 1782, 'path': '_notebooks/2021-03-31-captcha-fastai.ipynb', 'repo_name': 'AugustasMacijauskas/personal-website', 'size': '15575'}