<jupyter_start><jupyter_text>This is a follow-along with [How to write custom omdels with fast.ai](https://github.com/yanneta/pytorch-tutorials/blob/master/modified_VGG.ipynb)  How to Write Custom Models with Fast.AIExample on how to modify [fastai](https://github.com/fastai/fastai) to use a custom pretrained network.<jupyter_code>%reload_ext autoreload
%autoreload 2
%matplotlib inline
from fastai.transforms import *
from fastai.conv_learner import *
from fastai.model import *
from fastai.dataset import *
from fastai.sgdr import *
from fastai.plots import *
import pandas as pd
import numpy as np
path = 'data/gloc/'
model_path = path + 'results/'<jupyter_output><empty_output><jupyter_text>Custom ConvnetBuilderThis ConvnetBuilderVGG is similar to [ConvnetBuilder](https://github.com/fastai/fastai/blob/master/fastai/conv_learner.py). It's modifying VGG16. The default vgg16 in FastAI 'cuts' all the fully connected layers. In this example we keep all layers but the last one. There's a very small dataset so the intention is to train the last 2 fully connected layers.<jupyter_code># to override the fastai vgg16 function
from torchvision.models import vgg16

# Creates a ConvnetBuilder with all pretrained layers from vgg16 but the last fully connected layer
class ConvnetBuilderVGG():
    """
    Class representing a convolutional network.
    Arguments:
        c (int): size of the last layer
        is_multi (bool): is multilabel classification
        is_reg (bool): is a regression
        ps (float): dropout parameter for last layer
    """
    
    def __init__(self, c, is_multi, is_reg, ps=None):
        self.c, self.is_multi, self.is_reg = c, is_multi, is_reg
        self.ps = ps or 0.5
        vgg = vgg16(True) # NOTE: okay so I need to study how PyTorch does this
        self.lr_cut = 30
        layers = children(vgg.features)
        layers += [Flatten()] + children(vgg.classifier)[:5]
        #self.nf = 4096
        # here top model is everything but the last layer
        self.top_model = nn.Sequential(*layers) # NOTE: I need to find out what the fn(*arg) syntax is
        
        fc_layers = self.create_fc_layer(4096, c, p=None)
        self.n_fc = len(fc_layers)
        self.fc_model = to_gpu(nn.Sequential(*fc_layers))
        apply_init(self.fc_model, kaiming_normal)
        self.model = to_gpu(nn.Sequential(*(layers+fc_layers)))
        
    def create_fc_layer(self, ni, nf, p, actn=None):
        res=[]
        if p: res.append(nn.Dropout(p=p))
        res.append(nn.Linear(in_features=ni, out_features=nf))
        if actn: res.append(actn())
        return res

    @property # NOTE: I also need to learn Python Static Method syntax --> https://stackoverflow.com/questions/400739/what-does-asterisk-mean-in-python
    def name(self): return "vgg16"
    
    def get_layer_groups(self, do_fc=False):
        if do_fc:
            m, idxs = self.fc_model, []
        else:
            m, idxs = self.model, [self.lr_cut, -self.n_fc]
        lgs = list(split_by_idxs(children(m), idxs))
        return lgs
bs=32; sz=224
f_model = vgg16
n = 7637
val_idxs = get_cv_idxs(n, 0, val_pct=0.2)
tfms = tfms_from_model(f_model, sz) # NOTE: how would it know, if this is a custom/PyTorch model?
data = ImageClassifierData.from_csv(path, 'train', f'{path}train.csv', bs, tfms, 
                                    val_idxs=val_idxs, continuous=True)
# note precompute=False
models = ConvnetBuilderVGG(data.c, data.is_multi, data.is_reg)
models.model
class ConvLearnerVGG(ConvLearner):
    
    # rewriting pretrained
    @classmethod
    def pretrained(cls, data, ps=None, **kwargs):
        models = ConvnetBuilderVGG(data.c, data.is_multi, data.is_reg, ps=ps)
        return cls(data, models, **kwargs)
    
    # redefining freeze to freeze everything but last layer
    def freeze(self):
        layers = children(self.model)
        n = len(layers)
        for λ in layers:
            λ.trainable=False
            for p in λ.parameters(): p.requires_grad=False
        λ = layers[n-1]
        λ.trainable=True
        for p in λ.parameters(): p.requires_grad=True
    
    def unfreeze_prev_layer(self):
        layers = children(self.model)
        λ = layers[35]
        λ.trainable=True
        for p in λ.parameters(): p.requires_grad=True
bs = 32; sz = 224
f_model = vgg16
n = 7637
val_idxs = get_cv_idxs(n, 0, val_pct=0.2)
tfms = tfms_from_model(f_model, sz)
data = ImageClassifierData.from_csv(path, 'train', f'{path}train.csv', bs, tfms, 
                                    val_idxs=val_idxs, continuous=True)
learn = ConvLearnerVGG.pretrained(data, ps=0.0, precompute=False)<jupyter_output><empty_output><jupyter_text>NOTE: this is on my MacBook w/o a GPU.<jupyter_code>m = learn.models.model
trainable_params_(m)
learn.unfreeze_prev_layer()
trainable_params_(m)<jupyter_output><empty_output><jupyter_text>So, the numbers in the top layers are different from [Yannet's](https://github.com/yanneta/pytorch-tutorials/blob/master/modified_VGG.ipynb), but the Conv layer pars are exactly the same. Pretty sure this is because when you finetune a model, the FC layer/s you stack ontop are randomly initialized, but the Conv layers coming from a pretrained network (PyTorch VGG 16), are necessarily exactly the same, given you got the same pretrained weights.  Cross Validation<jupyter_code>bs=32; sz=224
n = 7637
transforms_basic = [RandomRotateXY(10), RandomDihedralXY()]
transforms_basic = [RandomRotateXY(10)]<jupyter_output><empty_output><jupyter_text>Here's code to do cross-validation:<jupyter_code>def get_model_i(i=0):
    val_idxs = get_cv_idxs(n, i, val_pct=0.1)
    tfms = tfms_from_model(f_model, sz, aug_tfms=transforms_basic, max_zoom=1.05)
    data = ImageClassifierData.from_csv(path, 'train', f'{path}train.csv', bs, tfms, 
                                        val_idxs=val_idxs, suffix='.jpg', continuous=True)
    learn = ConvLearnerVGG.pretrained(data, ps=0.0, precompute=False)
    return learn<jupyter_output><empty_output><jupyter_text>How to train is a W.I.P. --- *great..*<jupyter_code>def fit_and_predict(learn):
    learn.fit(1e-3, 3)
    learn.fit(1e-4, 4)
    print("unfreezing")
    learn.unfreeze_prev_layer()
    #learn.fit(1e-5, 3, cycle_len=1, cycle_mult=2)
    learn.fit(1e-5, 3)
    return learn.TTA()<jupyter_output><empty_output><jupyter_text>Predictions<jupyter_code>preds = []
for i in range(11):
    print("iteration ", i)
    learn = get_model_i(i)
    preds.append(fit_and_predict(learn))<jupyter_output><empty_output><jupyter_text>This'll actually take forever on an i5 CPU, but it does start training so, that's good.<jupyter_code>def reshape_preds(preds):
    predictions = [preds[i][0] for i in range(11)]
    y = [preds[i][1] for i in range(11)]
    pp = np.vstack(predictions)
    yy = np.vstack(y)
    print(yy.shape)
    pp = np.maximum(pp, 0.0)
    err = np.abs(pp - yy).mean()
    print("err", err)
reshape_preds(preds)<jupyter_output><empty_output><jupyter_text>---Alrighty, that gave me some idea of how a custom model is added to FastAI. Getting length of dataset. Dataset is my custom-built set for my G-LOC-Detector.<jupyter_code>df = pd.read_csv(path + 'train.csv')
len(df['id'])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11008031016338965, 'fraction_numerical': 0.01606203267792855, 'mean_word_length': 3.597708465945258, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 38, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '595623', 'license': 'MIT', 'n_tokens_mistral': 2592, 'n_tokens_neox': 2480, 'n_words': 638, 'path': 'FADL1/custom_model.ipynb', 'repo_name': 'WNoxchi/Kawkasos', 'size': '7222'}