<jupyter_start><jupyter_text>Baby boy/girl classifier model preparation*based on: <NAME> and <NAME>. Inspired by [<NAME>](https://www.pyimagesearch.com/2017/12/04/how-to-create-a-deep-learning-dataset-using-google-images/)**by: <NAME>*Notebook execution and model training is made in Google Colab<jupyter_code>from fastai.vision import *
from pathlib import Path
# Check if running in Google Colab and save it to bool variable
try:
    import google.colab
    IS_COLAB = True
except:
    IS_COLAB = False
print("Is Colab:", IS_COLAB)<jupyter_output>Is Colab: False<jupyter_text>Get a list of URLs  How to get a dataset from Google Images Go to [Google Images](http://images.google.com) and search for the images you are interested in. The more specific you are in your Google Search, the better the results and the less manual pruning you will have to do.Scroll down until you've seen all the images you want to download, or until you see a button that says 'Show more results'. All the images you scrolled past are now available to download. To get more, click on the button, and continue scrolling. The maximum number of images Google Images shows is 700.It is a good idea to put things you want to exclude into the search query, for instance if you are searching for the Eurasian wolf, "canis lupus lupus", it might be a good idea to exclude other variants:    "canis lupus lupus" -dog -arctos -familiaris -baileyi -occidentalisYou can also limit your results to show only photos by clicking on Tools and selecting Photos from the Type dropdown.  How to download image URLs Now you must run some Javascript code in your browser which will save the URLs of all the images you want for you dataset.Press CtrlShiftJ in Windows/Linux and CmdOptJ in Mac, and a small window the javascript 'Console' will appear. That is where you will paste the JavaScript commands.You will need to get the urls of each of the images. Before running the following commands, you may want to disable ad blocking extensions (uBlock, AdBlockPlus etc.) in Chrome. Otherwise the window.open() command doesn't work. Then you can run the following commands:```javascripturls = Array.from(document.querySelectorAll('.rg_di .rg_meta')).map(el=>JSON.parse(el.textContent).ou);window.open('data:text/csv;charset=utf-8,' + escape(urls.join('\n')));```  What to do with babies For this particular application (baby boy/girl classifier) you can just search for "baby boys" and "baby girls". Then run the script mentioned above and save the URLs in "boys_urls.csv" and "girls_urls.csv".  Download images fast.ai has a function that allows you to do just that. You just have to specify the urls filename as well as the destination folder and this function will download and save all images that can be opened. If they have some problem in being opened, they will not be saved.Let's download our images! Notice you can choose a maximum number of images to be downloaded. In this case we will not download all the urls.<jupyter_code>class_boy = 'boys'
class_girl = 'girls'
classes = [class_boy, class_girl]
path = Path('./data')
path.mkdir(parents=True, exist_ok=True)

def download_dataset(is_colab):
    if is_colab:
        from google.colab import drive
        import shutil
        import zipfile

        # You'll be asked to sign in Google Account and copy-paste a code here. Do it.
        drive.mount('/content/gdrive')
        # Copy this model from Google Drive after export and manually put it in the "ai_models" folder in the repository
        # If there'll be an error during downloading the model - share it with some other Google account and download
        # from this 2nd account - it should work fine.
        zip_remote_path = Path('/content/gdrive/My Drive/Colab/boyorgirl/train.zip')
        shutil.copy(str(zip_remote_path), str(path))
        zip_local_path = path/'train.zip'
        with zipfile.ZipFile(zip_local_path, 'r') as zip_ref:
            zip_ref.extractall(path)
        print("train folder contents:", (path/'train').ls())
    else:
        data_sources = [
          ('./boys_urls.csv', path/'train'/class_boy),
          ('./girls_urls.csv', path/'train'/class_girl)
        ]

        # Download the images listed in URL's files
        for urls_path, dest_path in data_sources:
            dest = Path(dest_path)
            dest.mkdir(parents=True, exist_ok=True)
            download_images(urls_path, dest, max_pics=800)
            # If you have problems download, try the code below with `max_workers=0` to see exceptions:
            # download_images(urls_path, dest, max_pics=20, max_workers=0)

        # Then we can remove any images that can't be opened:
        for _, dest_path in data_sources:
            verify_images(dest_path, delete=True, max_size=800)

# If running from colab - zip your train set (train folder) and put it to "Colab/boyorgirl/train.zip" in your Google Drive
download_dataset(IS_COLAB)<jupyter_output><empty_output><jupyter_text>Cleaning the data Now it's a good moment to review the downloaded images and clean them. There will be some non-relevant images - photos of adults, photos of the baby clothes without the babies etc. Just review the images and remove non-relevant ones. For 2x400 images it'll take just 10-20 minutes in total.There's also another way to clean the data - use the `fastiai.widgets.ImageCleaner`. It's used after you've trained your model. Even if you plan to use `ImageCleaner` later - it still makes sense to review the dataset briefly by yourself at the beginning.  Load the data<jupyter_code>np.random.seed(42)
data = ImageDataBunch.from_folder(path, train='train', valid_pct=0.2,
        ds_tfms=get_transforms(), size=224, num_workers=4).normalize(imagenet_stats)<jupyter_output><empty_output><jupyter_text>Good! Let's take a look at some of our pictures then.<jupyter_code># Check if all the classes were correctly read
print(data.classes)
print(data.classes == classes)
data.show_batch(rows=3, figsize=(7,8), ds_type=DatasetType.Train)
data.show_batch(rows=3, figsize=(7,8), ds_type=DatasetType.Valid)
print('Train set size: {}. Validation set size: {}'.format(len(data.train_ds), len(data.valid_ds)))<jupyter_output>Train set size: 592. Validation set size: 147<jupyter_text>Train model<jupyter_code>learn = cnn_learner(data, models.resnet50, metrics=error_rate)
learn.fit_one_cycle(4)
learn.save('stage-1')
learn.unfreeze()
learn.lr_find()
# If the plot is not showing try to give a start and end learning rate
# learn.lr_find(start_lr=1e-5, end_lr=1e-1)
learn.recorder.plot()
learn.fit_one_cycle(2, max_lr=slice(1e-5,1e-3))
learn.save('stage-2')<jupyter_output><empty_output><jupyter_text>Interpretation<jupyter_code>learn.load('stage-2');
interp = ClassificationInterpretation.from_learner(learn)
interp.plot_confusion_matrix()<jupyter_output><empty_output><jupyter_text>Putting your model in production First thing first, let's export the content of our `Learner` object for production. Below are 2 variants of export - for local environment and for colab environment:<jupyter_code># Use this cell to export model from local environment within the repository
def get_export_path(is_colab):
    if is_colab:
        from google.colab import drive
        
        # You'll be asked to sign in Google Account and copy-paste a code here. Do it.
        # force_remount=True is needed to write model if it was deleted from Google Drive, but remains in Colab local file system
        drive.mount('/content/gdrive', force_remount=True)
        # Copy this model from Google Drive after export and manually put it in the "ai_models" folder in the repository
        # If there'll be an error during downloading the model - share it with some other Google account and download
        # from this 2nd account - it should work fine.
        return Path('/content/gdrive/My Drive/Colab/boyorgirl/ai_models/export.pkl')
    else:
        # Used in case when notebook is run from the repository, but not in the Colab
        return Path('../backend/ai_models/export.pkl')

# In case of Colab - model will be exported to 'Colab/boyorgirl/ai_models/export.pkl'. Download and save it in your repository manually
#  in the 'ai_models' folder
export_path = get_export_path(IS_COLAB)
# ensure folder exists
export_path.parents[0].mkdir(parents=True, exist_ok=True)
# absolute path is passed as learn object attaches relative path to it's data folder rather than to notebook folder
learn.export(export_path.absolute())
print("Export folder contents:", export_path.parents[0].ls())<jupyter_output>Export folder contents: [PosixPath('../backend/ai_models/.DS_Store'), PosixPath('../backend/ai_models/export.pkl')]<jupyter_text>This will create a file named 'export.pkl' in the given directory. This exported model contains everything we need to deploy our model (the model, the weights but also some metadata like the classes or the transforms/normalization used).<jupyter_code><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08283166109253066, 'fraction_numerical': 0.007803790412486065, 'mean_word_length': 4.270857814336075, 'pattern_counts': {'":': 0, '<': 34, '<?xml version=': 0, '>': 35, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 20, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '9963', 'license': 'MIT', 'n_tokens_mistral': 2644, 'n_tokens_neox': 2543, 'n_words': 1033, 'path': 'notebooks/babies.ipynb', 'repo_name': 'artoby/boyorgirl', 'size': '9006'}