<jupyter_start><jupyter_text>Kaggle -- blindness https://www.kaggle.com/c/aptos2019-blindness-detection/overview We import all the necessary packages. We are going to work with the [fastai V1 library](http://www.fast.ai/2018/10/02/fastai-ai/) which sits on top of [Pytorch 1.0](https://hackernoon.com/pytorch-1-0-468332ba5163). The fastai library provides many useful functions that enable us to quickly and easily build neural networks and train our models.<jupyter_code>from fastai.script import *
from fastai.vision import *
from fastai.callbacks import *
from fastai.distributed import *
from fastai.callbacks.tracker import *
torch.backends.cudnn.benchmark = True
import time
from fastprogress import fastprogress
from fastai.general_optimizer import *<jupyter_output><empty_output><jupyter_text>If you're using a computer with an unusually small GPU, you may get an out of memory error when running this notebook. If this happens, click Kernel->Restart, uncomment the 2nd line below to use a smaller *batch size* (you'll learn all about what this means during the course), and try again.<jupyter_code>bs = 64
# bs = 16   # uncomment this line if you run out of memory even after clicking Kernel->Restart<jupyter_output><empty_output><jupyter_text>Get data from kaggle seedling playground<jupyter_code>!kaggle_setup.txt # setup kaggle account
!kaggle competitions download -c aptos2019-blindness-detection<jupyter_output>Downloading train_images.zip to /home/jupyter/tutorials/fastai/course-v3/nbs/dl1/blind
100%|█████████████████████████████████████▉| 7.99G/8.01G [01:55<00:00, 59.8MB/s]
100%|██████████████████████████████████████| 8.01G/8.01G [01:55<00:00, 74.4MB/s]
Downloading test_images.zip to /home/jupyter/tutorials/fastai/course-v3/nbs/dl1/blind
 98%|██████████████████████████████████████▎| 1.48G/1.50G [00:15<00:00, 142MB/s]
100%|███████████████████████████████████████| 1.50G/1.50G [00:15<00:00, 106MB/s]
Downloading train.csv to /home/jupyter/tutorials/fastai/course-v3/nbs/dl1/blind
  0%|                                               | 0.00/53.7k [00:00<?, ?B/s]
100%|██████████████████████████████████████| 53.7k/53.7k [00:00<00:00, 40.1MB/s]
Downloading test.csv to /home/jupyter/tutorials/fastai/course-v3/nbs/dl1/blind
  0%|                                               | 0.00/24.5k [00:00<?, ?B/s]
100%|██████████████████████████████████████| 24.5k/24.5k [00:00<00:00, 30.4MB/s]
Downloading sample_s[...]<jupyter_text>Looking at the data<jupyter_code>path = Path('/home/jupyter/tutorials/fastai/course-v3/nbs/dl1/blind/')
path.ls()
path_img = path/'train'<jupyter_output><empty_output><jupyter_text>The first thing we do when we approach a problem is to take a look at the data. We _always_ need to understand very well what the problem is and what the data looks like before we can figure out how to solve it. Taking a look at the data means understanding how the data directories are structured, what the labels are and what some sample images look like.<jupyter_code>fnames_train = get_image_files(path_img)
plt.imshow(plt.imread(str(fnames_train[20])))<jupyter_output><empty_output><jupyter_text>use Databunch api<jupyter_code>train_df = pd.read_csv('train.csv')
train_df.head()
tfms = get_transforms(do_flip=True)
data = (ImageList.from_df(train_df, path_img, suffix='.png')
        .split_by_rand_pct(0.2)
        .label_from_df()
        .transform(tfms, size=229)
        .databunch()
        .normalize(imagenet_stats))<jupyter_output><empty_output><jupyter_text>Training: mixup and label smoothing (https://forums.fast.ai/t/implement-bag-of-tricks-for-image-classification-with-convolutional-neural-networks/41105)<jupyter_code>learn = (Learner(data, models.xresnet34(),
             metrics=error_rate, wd=1e-3,
             bn_wd=False, true_wd=True, loss_func = LabelSmoothingCrossEntropy())
         .mixup(alpha=0.2))
learn.lr_find()
learn.recorder.plot()
learn.fit_one_cycle(5, 1e-1)
learn.recorder.plot_losses()
interp = ClassificationInterpretation.from_learner(learn)

losses,idxs = interp.top_losses()


interp.plot_confusion_matrix(figsize=(8,8), dpi=60)<jupyter_output><empty_output><jupyter_text>it's not very successful  Unfreezing, fine-tuning, and learning rates Since our model is working as we expect it to, we will *unfreeze* our model and train some more.<jupyter_code>learn.unfreeze()
learn.fit_one_cycle(1, 1e-3)<jupyter_output><empty_output><jupyter_text>Add lr scheduler<jupyter_code>lr = 1e-1
epochs = 5
b_its = len(data.train_dl)
bs_rat = bs/256
lr *= bs_rat

ph1 = (TrainingPhase(epochs*0.5*b_its)
            .schedule_hp('lr', (lr/20,lr), anneal=annealing_cos)
            .schedule_hp('eps', (1e-4,1e-7), anneal=annealing_cos)
            )
ph2 = (TrainingPhase(epochs*0.5*b_its)
            .schedule_hp('lr', (lr,lr/1e5), anneal=annealing_cos)
            .schedule_hp('eps', (1e-7,1e-7), anneal=annealing_cos)
            )

learn = (Learner(data, models.xresnet34(),
             metrics=error_rate, wd=1e-3,
             bn_wd=False, true_wd=True, loss_func = LabelSmoothingCrossEntropy())
         .mixup(alpha=0.2))

gs = GeneralScheduler(learn, (ph1,ph2))
learn.fit(epochs, lr=1, callbacks=gs)

learn.unfreeze()
learn.fit(epochs, lr=0.01, callbacks=gs)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.16245006657789615, 'fraction_numerical': 0.0477458626593114, 'mean_word_length': 4.617521367521367, 'pattern_counts': {'":': 0, '<': 44, '<?xml version=': 0, '>': 38, 'https://': 3, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '588881', 'license': 'Apache-2.0', 'n_tokens_mistral': 2249, 'n_tokens_neox': 1972, 'n_words': 395, 'path': 'nbs/dl1/blind/blind-part4.ipynb', 'repo_name': 'ohliumliu/fastai_v3', 'size': '5257'}