<jupyter_start><jupyter_text>Hide your messy video background using neural nets, Part 1> "Use fastai to separate a person in a video from the background. Start from scratch with creating your own dataset."- toc: true- branch: master- badges: true- comments: false- categories: [fastai, privacy, getting started]- image: images/articles/2021-backgroundblur-1/thumbnail-yellow.png- hide: false &nbsp;    Your browser does not support the video tag. This tutorial teaches you to train a neural network to distinguish the person in a webcam shot from the background. Foreground/background detection can be useful in many applications. Examples are adding bokeh to photos (portrait mode of smartphones) or automatic removal of the background in photo editing.I did this as a little project inspired by OpenMined's [privacy course](https://deeplearning.berlin/online%20courses/privacy/the%20private%20ai%20series/openmined/2021/01/08/Our-Privacy-Opportunity-Part-1.html). Blurring the background during a Zoom call is an example of [output privacy](https://deeplearning.berlin/online%20courses/privacy/the%20private%20ai%20series/openmined/2021/01/19/Our-Privacy-Opportunity-Part-3.htmlOutput-Privacy). It refers to the concept that you often want to share some information (that you are present in a video call, how you look, ...) but not additional information (your location, the stuff in your living room, the medication on the shelf behind you).We begin with collecting our own data, annotating images, and end up with a useful model. In this blog post, I'll guide you through the process. We'll be using a webcam and SuperAnnotate to create our dataset. Then, we'll use fastai to train a neural network (specifically, a U-Net) to recognize a person in a frame. I assume that you know the basics of fastai. The training process is straightforward however, so it's a good example of how the library works even if you never used it.  Creating Your Own DatasetTo create your own dataset, just use your webcam to snap a few photos of you. I took around **40 photos** in slightly different poses and hope that's enough!The next step is annotation. I recently discovered [SuperAnnotate](https://superannotate.com/) which promises to be the "fastest annotation platform". Let's give it a try.The application runs in the browser and comes with a rather hefty price tag, starting from 62$ per month. They offer a 14-day trial however, and also a free desktop software. The desktop version is missing some features but can be used without any limitations.1\. Create a `Pixel` Project:![Create pixel project](../images/articles/2021-backgroundblur-1/pixelproject.png)2\. Add a single class "Person":![Add a class](../images/articles/2021-backgroundblur-1/class.png)3\. You can leave "Contributers" empty and click `Create`.4\. Now you can `Upload` your photos.5\. Select the outlines of your body with the Polygon tool (key binding `Z`)6\. Hit `Space` to create an instance and choose the Person class 7\. Click `Complete` in the lower right.8\. Repeat for all images.9\. Return to the home page and select `Download`. Click `Prepare for Download`, select `Completed` images and click `Prepare`. Download the resulting zip file.One feature I found extremely helpful was the auto-segmentation feature (you can see it in action with the green lines in the screenshot). To activate it, select all images on the home page and select the `Smart Segmentation` button to the right:![Use smart select](../images/articles/2021-backgroundblur-1/home.png)The detection of connected areas works very well and makes exact annotations easy. Some manual corrections are quickly done using the brush tool. This feature is unfortunately only available in the browser version.![SuperAnnotate Screenshot](../images/articles/2021-backgroundblur-1/superannotate.png)  Preparing DataI ran this notebook on [Paperspace Gradient](https://gradient.paperspace.com/), a cloud platform that offers a ready-to-use fastai environment and free GPU options. You can also run it on Google Colab, which is entirely free.<jupyter_code>#hide
!pip install fastai==2.2.5 -q
from fastai.vision.all import *
from zipfile import ZipFile
path = Path('')<jupyter_output><empty_output><jupyter_text>I uploaded the zip files with the training images and the annotations to Paperspace<jupyter_code>with ZipFile('training.zip') as file:
    file.extractall()
with ZipFile('SuperAnnotate.zip') as file:
    file.extractall('masks')<jupyter_output><empty_output><jupyter_text>Collect all training images using the fastai function `get_image_files` and look at filename format:<jupyter_code>images = get_image_files(path/'training')
images[0]<jupyter_output><empty_output><jupyter_text>How many images do we have?<jupyter_code>len(images)<jupyter_output><empty_output><jupyter_text>Get the annotation files, look at filename format:<jupyter_code>masks = get_image_files(path/'masks')
masks[0]
len(masks)<jupyter_output><empty_output><jupyter_text>We have 41 training images as well as 41 matching annotations. The annotated files have the same name as the training images with the extension of `___save.png` A function to get the training image for every mask:<jupyter_code>def get_y(fn):
    return path/'masks'/(fn.name+'___save.png')<jupyter_output><empty_output><jupyter_text>Let's look at the images.<jupyter_code>img = PILImage.create(images[2])
img.show();<jupyter_output><empty_output><jupyter_text>Looks as expected. What about the annotations?<jupyter_code>annotation = PILMask.create(get_y(images[2]))
annotation.show();<jupyter_output><empty_output><jupyter_text>This is not the kind of mask we wanted. We just want to classify whether a pixel is `person` or `background`. The more fine-grained segmentation here is an artifact of SuperAnnotate's auto-segmentation feature. Which is extremely helpful to quickly annotate images, but it creates more categories than we want:<jupyter_code>np.unique(annotation)<jupyter_output><empty_output><jupyter_text>These are too many categories, we need to convert this mask into a simple `0` or `1` mask.<jupyter_code>def create_mask(fn):
    maskfile = get_y(fn)
    mask = np.array(PILMask.create(maskfile))
    # Turn every number larger than 0 into a 1
    mask[mask > 0] = 1
    return PILMask.create(mask)
mask = create_mask(fnames[0])
np.unique(mask)<jupyter_output><empty_output><jupyter_text>Now the masks are looking good.  Loading the DataWe use fastai's `DataBlock` API to quickly collect our data. For details, please check [chapter 2](https://github.com/fastai/fastbook/blob/master/02_production.ipynb) of the fabulous book from <NAME> & <NAME>.- Use ImageBlock and MaskBlock to define the task- Collect all training images with `get_image_files`- Set aside 20% of images as the validation set- Define the masks with our `create_mask` function- Resize frames to a managable size of 640x360- Normalize according to the Imagenet stats<jupyter_code>codes = ['Background', 'Person']
binary = DataBlock(blocks=(ImageBlock, MaskBlock(codes)),
                   get_items=get_image_files,
                   splitter=RandomSplitter(0.2),
                   get_y=create_mask,
                   item_tfms=Resize((360, 640)),
                   batch_tfms=[Normalize.from_stats(*imagenet_stats)])<jupyter_output><empty_output><jupyter_text>Create `dataloaders` with all files in the `training` folder. The batch size is only 2 in order to avoid running out of GPU RAM, since our model will be quite heavy.<jupyter_code>dls = binary.dataloaders(path/'training', bs=2)<jupyter_output><empty_output><jupyter_text>Look at some example pictures. We set `vmin` and `vmax` to match our 2 classes.<jupyter_code>dls.show_batch(cmap='Blues', vmin=0, vmax=1)<jupyter_output><empty_output><jupyter_text>Looks good! Let's train.  Training a Model We treat this as a semantic segmentation problem, where for every pixel the neural net determines whether it belongs to a person or to the background. fastai has a convenient function to create a model with the U-Net architecture, using a pretrained backbone. For details, please check out [chapter 15](https://github.com/fastai/fastbook/blob/master/15_arch_details.ipynb) of the fastai book.<jupyter_code>learn = unet_learner(dls, resnet18)
learn.fine_tune(5)<jupyter_output><empty_output><jupyter_text>Our validation loss decreased nicely after only a few seconds of training. Let's see if our tiny dataset is actually sufficient!<jupyter_code>learn.show_results(cmap='Blues', vmin=0, vmax=1, figsize=(14,6))<jupyter_output><empty_output><jupyter_text>Looks pretty good! Let's export our model.<jupyter_code>learn.export('unet-resnet18-person-background.pkl')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08589449541284404, 'fraction_numerical': 0.015022935779816515, 'mean_word_length': 5.37966349670812, 'pattern_counts': {'":': 1, '<': 67, '<?xml version=': 0, '>': 69, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '603943', 'license': 'MIT', 'n_tokens_mistral': 2613, 'n_tokens_neox': 2454, 'n_words': 977, 'path': '2021-02-09-Background-Blur-Part-1.ipynb', 'repo_name': 'JohannesStutz/background-detection', 'size': '8735'}