<jupyter_start><jupyter_text>Transfer Learning with skorch In this tutorial, you will learn how to train a neutral network using transfer learning with the `skorch` API. Transfer learning uses a pretrained model to initialize a network. This tutorial converts the pure PyTorch approach described in [PyTorch's Transfer Learning Tutorial](https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html) to `skorch`.We will be using `torchvision` for this tutorial. Instructions on how to install `torchvision` for your platform can be found at https://pytorch.org.<jupyter_code>import os
from urllib import request
from zipfile import ZipFile

import torch
import torch.nn as nn
import torch.optim as optim
import numpy as np
from torchvision import datasets, models, transforms

from skorch import NeuralNetClassifier
from skorch.callbacks import LRScheduler, Checkpoint
from skorch.helper import filtered_optimizer
from skorch.helper import filter_requires_grad
from skorch.helper import predefined_split

torch.manual_seed(360);<jupyter_output><empty_output><jupyter_text>Preparations Before we begin, lets download the data needed for this tutorial:<jupyter_code>def download_and_extract_data(dataset_dir='datasets'):
    data_zip = os.path.join(dataset_dir, 'hymenoptera_data.zip')
    data_path = os.path.join(dataset_dir, 'hymenoptera_data')
    url = "https://download.pytorch.org/tutorial/hymenoptera_data.zip"

    if not os.path.exists(data_path):
        if not os.path.exists(data_zip):
            print("Starting to download data...")
            data = request.urlopen(url, timeout=15).read()
            with open(data_path, 'wb') as f:
                f.write(data)

        print("Starting to extract data...")
        with ZipFile(data_zip, 'r') as zip_f:
            zip_f.extractall(dataset_dir)
        
    print("Data has been downloaded and extracted to {}.".format(dataset_dir))
    
download_and_extract_data()<jupyter_output>Data has been downloaded and extracted to datasets.<jupyter_text>The Problem We are going to train a neutral network to classify **ants** and **bees**. The dataset consist of 120 training images and 75 validiation images for each class. First we create the training and validiation datasets:<jupyter_code>data_dir = 'datasets/hymenoptera_data'
train_transforms = transforms.Compose([
    transforms.RandomResizedCrop(224),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406], 
                         [0.229, 0.224, 0.225])
])
val_transforms = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406], 
                         [0.229, 0.224, 0.225])
])

train_ds = datasets.ImageFolder(
    os.path.join(data_dir, 'train'), train_transforms)
val_ds = datasets.ImageFolder(
    os.path.join(data_dir, 'val'), val_transforms)<jupyter_output><empty_output><jupyter_text>The train dataset includes data augmentation techniques such as cropping to size 224 and horizontal flips.The train and validiation datasets are normalized with mean: `[0.485, 0.456, 0.406]`, and standard deviation: `[0.229, 0.224, 0.225]`. These values are the means and standard deviations of the ImageNet images. We used these values because the pretrained model was trained on ImageNet.  Loading pretrained model We use a pretrained `ResNet18` neutral network model with its final layer replaced with a fully connected layer:<jupyter_code>model_ft = models.resnet18(pretrained=True)
num_ftrs = model_ft.fc.in_features
model_ft.fc = nn.Linear(num_ftrs, 2)<jupyter_output><empty_output><jupyter_text>Since we are training a binary classifier, the output of the final fully connected layer has size 2. Next, we freeze all layers except the final layer by setting `requires_grad` to False:<jupyter_code>for name, param in model_ft.named_parameters():
    if not name.startswith('fc'):
        param.requires_grad_(False)<jupyter_output><empty_output><jupyter_text>Using skorch's API In this section, we will create a `skorch.NeuralNetClassifier` to solve our classification problem.   Callbacks First, we create two callbacks:<jupyter_code>lrscheduler = LRScheduler(
    policy='StepLR', step_size=7, gamma=0.1)

checkpoint = Checkpoint(
    f_params='best_model.pt', monitor='valid_acc_best')

callbacks = [lrscheduler, checkpoint]<jupyter_output><empty_output><jupyter_text>The `LRScheduler` callback defines a learning rate scheduler that uses `torch.optim.lr_scheduler.StepLR` to scale learning rates by `gamma=0.1` every 7 steps. The `Checkpoint` callback saves the best model by by monitoring the validation accuracy.  Filtered optimizer Since we froze some layers in our `Resnet18` neutral network, we need to configure our optimizer to only update gradients in our final fully connected layer. Luckily, `skorch` provides two functions that make this simple:<jupyter_code>optimizer = filtered_optimizer(
    optim.SGD, filter_requires_grad
)<jupyter_output><empty_output><jupyter_text>This function does not do any processing and returns the two datasets.   skorch.NeutralNetClassifier With all the preparations out of the way, we can now define our `NeutralNetClassifier`:<jupyter_code>net = NeuralNetClassifier(
    model_ft, 
    criterion=nn.CrossEntropyLoss,
    lr=0.001,
    batch_size=4,
    max_epochs=25,
    optimizer=optimizer,
    optimizer__momentum=0.9,
    iterator_train__shuffle=True,
    iterator_train__num_workers=4,
    iterator_valid__shuffle=True,
    iterator_valid__num_workers=4,
    train_split=predefined_split(val_ds),
    callbacks=callbacks,
    device='cuda' # uncomment to train on gpu
)<jupyter_output><empty_output><jupyter_text>That is quite a few parameters! Lets walk through each one:1. `model_ft`: Our `ResNet18` neutral network2. `criterion=nn.CrossEntropyLoss`: loss function3. `lr`: Initial learning rate4. `batch_size`: Size of a batch5. `max_epochs`: Number of epochs to train6. `optimizer`: Our filtered optimizer7. `optimizer__momentum`: The initial momentum8. `iterator_{train,valid}__{shuffle,num_workers}`: Parameters that are passed to the dataloader.9. `train_split`: A wrapper around `val_ds` to use our validation dataset.10. `callbacks`: Our callbacks 11. `device`: Set to `cuda` to train on gpu.Now we are ready to train our neutral network:<jupyter_code>net.fit(train_ds, y=None);<jupyter_output>epoch    train_loss    valid_acc    valid_loss    cp     dur
-------  ------------  -----------  ------------  ----  ------
      1        [36m0.8333[0m       [32m0.9346[0m        [35m0.2150[0m     +  1.4656
      2        [36m0.5203[0m       0.9150        0.2439        1.2813
      3        [36m0.4469[0m       0.8693        0.3494        1.2942
      4        0.4665       [32m0.9542[0m        [35m0.1949[0m     +  1.2363
      5        [36m0.3884[0m       0.9412        0.1962        1.2834
      6        [36m0.3807[0m       0.9412        [35m0.1923[0m        1.3064
      7        [36m0.3292[0m       0.9412        [35m0.1876[0m        1.3428
      8        [36m0.2864[0m       0.9412        0.1961        1.3132
      9        0.4199       0.9346        0.1987        1.2935
     10        0.4462       0.9412        0.2054        1.2886
     11        0.2971       0.9412        0.1952        1.4172
     12        0.3474       0.9412        0.2092        1.3306
 [...]
{'fraction_non_alphanumeric': 0.09673956795921106, 'fraction_numerical': 0.057426539648463705, 'mean_word_length': 3.4930681133212778, 'pattern_counts': {'":': 0, '<': 35, '<?xml version=': 0, '>': 35, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '181468', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 2599, 'n_tokens_neox': 2318, 'n_words': 673, 'path': 'notebooks/Transfer_Learning.ipynb', 'repo_name': 'spott/skorch', 'size': '7453'}