<jupyter_start><jupyter_text>Enter State Farm<jupyter_code>import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
%matplotlib inline
from __future__ import print_function, division
path = "data/state/"
#path = "data/state/sample/"
import imp
import utils
from utils import *
from IPython.display import FileLink
batch_size= 16<jupyter_output><empty_output><jupyter_text>Setup batches<jupyter_code>batches = get_batches(path+'train', batch_size=batch_size)
val_batches = get_batches(path+'valid', batch_size=batch_size*2, shuffle=False)
(val_classes, trn_classes, val_labels, trn_labels, 
    val_filenames, filenames, test_filenames) = get_classes(path)<jupyter_output>Found 20924 images belonging to 10 classes.
Found 1500 images belonging to 10 classes.
Found 79726 images belonging to 1 classes.<jupyter_text>Rather than using batches, we could just import all the data into an array to save some processing time. (In most examples I'm using the batches, however - just because that's how I happened to start out.)与其使用批处理，不如将所有数据导入到数组中，以节省一些处理时间。(不过，在大多数例子中，我使用批次——只是因为我恰好就是这样开始的。)<jupyter_code>trn = get_data(path+'train')
val = get_data(path+'valid')
save_array(path+'results/val.dat', val)
save_array(path+'results/trn.dat', trn)
val = load_array(path+'results/val.dat')
trn = load_array(path+'results/trn.dat')<jupyter_output><empty_output><jupyter_text>Re-run sample experiments on full dataset We should find that everything that worked on the sample (see statefarm-sample.ipynb), works on the full dataset too. Only better! Because now we have more data. So let's see how they go - the models in this section are exact copies of the sample notebook models.  单个卷积层<jupyter_code>def conv1(batches):
    model = Sequential([
            BatchNormalization(axis=1, input_shape=(3,224,224)),
            Conv2D(filters=32,kernel_size=(3,3), activation='relu'),
            BatchNormalization(axis=1),
            MaxPooling2D((3,3)),
            Conv2D(filters=64,kernel_size=(3,3), activation='relu'),
            BatchNormalization(axis=1),
            MaxPooling2D((3,3)),
            Flatten(),
            Dense(200, activation='relu'),
            BatchNormalization(),
            Dense(10, activation='softmax')
        ])

    model.compile(Adam(lr=1e-4), loss='categorical_crossentropy', metrics=['accuracy'])
    #model.fit_generator(batches, epochs=2, validation_data=val_batches)
    model.fit_generator(batches, steps_per_epoch=batches.n // batch_size, epochs=2, validation_data=val_batches, 
                    validation_steps=val_batches.n // batch_size)
    model.optimizer.lr = 0.001
    model.fit_generator(batches, steps_per_epoch=batches.n // batch_size, epochs=4, validation_data=val_batches, 
                    validation_steps=val_batches.n // batch_size)
    return model
batch_size
batches.n //batch_size
len(batches)
batches.n
model = conv1(batches)<jupyter_output>Epoch 1/2
326/326 [==============================] - 114s 350ms/step - loss: 0.2292 - acc: 0.9405 - val_loss: 0.0528 - val_acc: 0.9897
Epoch 2/2
326/326 [==============================] - 108s 332ms/step - loss: 0.0139 - acc: 0.9984 - val_loss: 0.0199 - val_acc: 0.9959
Epoch 1/4
326/326 [==============================] - 111s 341ms/step - loss: 0.0057 - acc: 0.9993 - val_loss: 0.0188 - val_acc: 0.9945
Epoch 2/4
326/326 [==============================] - 109s 335ms/step - loss: 0.0051 - acc: 0.9994 - val_loss: 0.0107 - val_acc: 0.9969
Epoch 3/4
326/326 [==============================] - 109s 334ms/step - loss: 0.0015 - acc: 1.0000 - val_loss: 0.0072 - val_acc: 0.9976
Epoch 4/4
326/326 [==============================] - 109s 335ms/step - loss: 0.0015 - acc: 0.9999 - val_loss: 0.0074 - val_acc: 0.9976<jupyter_text>Interestingly, with no regularization or augmentation we're getting some reasonable results from our simple convolutional model. So with augmentation, we hopefully will see some very good results.有趣的是，由于没有正规化或增强，我们从简单的卷积模型得到了一些合理的结果。因此，通过增加，我们希望能看到一些很好的结果。  数据增广<jupyter_code>gen_t = image.ImageDataGenerator(rotation_range=15, height_shift_range=0.05, 
                shear_range=0.1, channel_shift_range=20, width_shift_range=0.1)
batches = get_batches(path+'train', gen_t, batch_size=batch_size)
model = conv1(batches)
1e-4
# 在之前的模型上，变更学习率为0.00001
model.optimizer.lr = 0.0001
# model.fit_generator(batches, steps_per_epoch=batches.n // batch_size, epochs=15, validation_data=val_batches, 
#                     validation_steps=val_batches.n // batch_size)
model.fit_generator(batches, epochs=15, validation_data=val_batches)<jupyter_output>Epoch 1/15
327/327 [==============================] - 318s 974ms/step - loss: 0.2005 - acc: 0.9471 - val_loss: 0.1389 - val_acc: 0.9627
Epoch 2/15
327/327 [==============================] - 306s 935ms/step - loss: 0.1746 - acc: 0.9542 - val_loss: 0.1177 - val_acc: 0.9707
Epoch 3/15
327/327 [==============================] - 310s 948ms/step - loss: 0.1539 - acc: 0.9597 - val_loss: 0.0989 - val_acc: 0.9727
Epoch 4/15
327/327 [==============================] - 316s 967ms/step - loss: 0.1409 - acc: 0.9622 - val_loss: 0.1140 - val_acc: 0.9707
Epoch 5/15
327/327 [==============================] - 314s 960ms/step - loss: 0.1291 - acc: 0.9654 - val_loss: 0.1210 - val_acc: 0.9640
Epoch 6/15
327/327 [==============================] - 311s 950ms/step - loss: 0.1214 - acc: 0.9667 - val_loss: 0.1093 - val_acc: 0.9687
Epoch 7/15
327/327 [==============================] - 309s 946ms/step - loss: 0.1086 - acc: 0.9716 - val_loss: 0.0873 - val_acc: 0.9800
Epoch 8/15
327/327 [============================[...]<jupyter_text>I'm shocked by *how* good these results are! We're regularly seeing 75-80% accuracy on the validation set, which puts us into the top third or better of the competition. With such a simple model and no dropout or semi-supervised learning, this really speaks to the power of this approach to data augmentation.我对这些结果如此之好感到震惊!我们经常在验证集上看到75-80%的准确率，这使我们进入了竞争中排名前三或更高的位置。有了这样一个简单的模型，没有dropout或半监督学习，这就证明了这种方法对数据增强的作用。  Four conv/pooling pairs + dropout 显卡不行不能完成 Unfortunately, the results are still very unstable - the validation accuracy jumps from epoch to epoch. Perhaps a deeper model with some dropout would help.由于显卡太烂，这个部分无法训练完成<jupyter_code>gen_t = image.ImageDataGenerator(rotation_range=15, height_shift_range=0.05, 
                shear_range=0.1, channel_shift_range=20, width_shift_range=0.1)
batches = get_batches(path+'train', gen_t, batch_size=batch_size)
model = Sequential([
        BatchNormalization(axis=1, input_shape=(3,224,224)),
        Conv2D(32,3, activation='relu'),
        BatchNormalization(axis=1),
        MaxPooling2D(),
        Conv2D(64,3, activation='relu'),
        BatchNormalization(axis=1),
        MaxPooling2D(),
        Conv2D(128,3, activation='relu'),
        BatchNormalization(axis=1),
        MaxPooling2D(),
        Flatten(),
        Dense(200, activation='relu'),
        BatchNormalization(),
        Dropout(0.5),
        Dense(200, activation='relu'),
        BatchNormalization(),
        Dropout(0.5),
        Dense(10, activation='softmax')
    ])
model.summary()
10e-5
10e-4
model.compile(Adam(lr=10e-5), loss='categorical_crossentropy', metrics=['accuracy'])
model.fit_generator(batches, steps_per_epoch=batches.n // batch_size, epochs=2, validation_data=val_batches, 
                    validation_steps=val_batches.n // batch_size)
model.optimizer.lr=0.001 # 提高学习率
model.fit_generator(batches, steps_per_epoch=batches.n // batch_size, epochs=10, validation_data=val_batches, 
                    validation_steps=val_batches.n // batch_size)
model.optimizer.lr=0.00001 # 降低学习率
model.fit_generator(batches, steps_per_epoch=batches.n // batch_size, epochs=10, validation_data=val_batches, 
                    validation_steps=val_batches.n // batch_size)<jupyter_output>Epoch 1/10
326/326 [==============================] - 316s 969ms/step - loss: 0.2465 - acc: 0.9202 - val_loss: 0.0839 - val_acc: 0.9783
Epoch 2/10
326/326 [==============================] - 318s 974ms/step - loss: 0.2336 - acc: 0.9263 - val_loss: 0.0689 - val_acc: 0.9818
Epoch 3/10
326/326 [==============================] - 313s 960ms/step - loss: 0.2202 - acc: 0.9301 - val_loss: 0.0620 - val_acc: 0.9862
Epoch 4/10
326/326 [==============================] - 317s 972ms/step - loss: 0.2012 - acc: 0.9347 - val_loss: 0.0554 - val_acc: 0.9859
Epoch 5/10
326/326 [==============================] - 315s 965ms/step - loss: 0.1916 - acc: 0.9400 - val_loss: 0.0493 - val_acc: 0.9873
Epoch 6/10
326/326 [==============================] - 319s 977ms/step - loss: 0.1771 - acc: 0.9450 - val_loss: 0.0544 - val_acc: 0.9883
Epoch 7/10
326/326 [==============================] - 315s 965ms/step - loss: 0.1744 - acc: 0.9443 - val_loss: 0.0587 - val_acc: 0.9852
Epoch 8/10
326/326 [============================[...]<jupyter_text>This is looking quite a bit better - the accuracy is similar, but the stability is higher. There's still some way to go however...  Imagenet conv features 有报错暂时无法尝试这里 Since we have so little data, and it is similar to imagenet images (full color photos), using pre-trained VGG weights is likely to be helpful - in fact it seems likely that we won't need to fine-tune the convolutional layer weights much, if at all. So we can pre-compute the output of the last convolutional layer, as we did in lesson 3 when we experimented with dropout. (However this means that we can't use full data augmentation, since we can't pre-compute something that changes every image.)由于我们的数据如此之少，而且它与imagenet图像(全彩照片)相似，使用预先训练的VGG权重很可能是有用的——事实上，我们似乎不需要调整卷积层的权重，如果有的话。所以我们可以预先计算最后一个卷积层的输出，就像我们在第3课中做的那样，当我们做dropout实验的时候。(然而，这意味着我们不能使用全数据增强，因为我们不能预先计算改变每一张图像的东西。)<jupyter_code>vgg = Vgg16()
model = vgg.model
model.summary()
# 寻找最后一个卷积层

for i,layer in enumerate(model.layers):
    if type(layer) is Conv2D:
        print(i,type(layer))
[i for i,layer in enumerate(model.layers) if type(layer) is Conv2D]
last_conv_idx = [i for i,layer in enumerate(model.layers) if type(layer) is Conv2D][-1]
last_conv_idx
conv_layers = model.layers[:last_conv_idx+1]
# 建立顺序模型
conv_model = Sequential(conv_layers)
batch_size = 16
# batches shuffle must be set to False when pre-computing features
# 在预训练模型中，获取批量数据不能打乱
batches = get_batches(path+'train', batch_size=batch_size, shuffle=False)
(val_classes, trn_classes, val_labels, trn_labels, 
    val_filenames, filenames, test_filenames) = get_classes(path)
conv_feat = conv_model.predict_generator(batches,steps=batches.n //batch_size,verbose=1) # 训练

conv_val_feat = conv_model.predict_generator(val_batches,steps=val_batches.n//batch_size) # 验证
conv_test_feat = conv_model.predict_generator(test_batches,steps=test_batches.n //batch_size) # 测试
save_array(path+'results/conv_val_feat.dat', conv_val_feat)
save_array(path+'results/conv_test_feat.dat', conv_test_feat)
save_array(path+'results/conv_feat.dat', conv_feat)
conv_feat = load_array(path+'results/conv_feat.dat')
conv_val_feat = load_array(path+'results/conv_val_feat.dat')
conv_val_feat.shape<jupyter_output><empty_output><jupyter_text>Batchnorm dense layers on pretrained conv layers 有问题 Since we've pre-computed the output of the last convolutional layer, we need to create a network that takes that as input, and predicts our 10 classes. Let's try using a simplified version of VGG's dense layers.<jupyter_code>def get_bn_layers(p):
    return [
        MaxPooling2D(input_shape=conv_layers[-1].output_shape[1:]),
        Flatten(),
        Dropout(p/2),
        Dense(128, activation='relu'),
        BatchNormalization(),
        Dropout(p/2),
        Dense(128, activation='relu'),
        BatchNormalization(),
        Dropout(p),
        Dense(10, activation='softmax')
        ]
p=0.8
bn_model = Sequential(get_bn_layers(p))
bn_model.compile(Adam(lr=0.001), loss='categorical_crossentropy', metrics=['accuracy'])
bn_model.fit(conv_feat, trn_labels, batch_size=batch_size, nb_epoch=1, 
             validation_data=(conv_val_feat, val_labels))
bn_model.optimizer.lr=0.01
bn_model.fit(conv_feat, trn_labels, batch_size=batch_size, nb_epoch=2, 
             validation_data=(conv_val_feat, val_labels))
bn_model.save_weights(path+'models/conv8.h5')<jupyter_output><empty_output><jupyter_text>Looking good! Let's try pre-computing 5 epochs worth of augmented data, so we can experiment with combining dropout and augmentation on the pre-trained model.  Pre-computed data augmentation + dropout We'll use our usual data augmentation parameters:<jupyter_code>gen_t = image.ImageDataGenerator(rotation_range=15, height_shift_range=0.05, 
                shear_range=0.1, channel_shift_range=20, width_shift_range=0.1)
da_batches = get_batches(path+'train', gen_t, batch_size=batch_size, shuffle=False)<jupyter_output>Found 20924 images belonging to 10 classes.<jupyter_text>We use those to create a dataset of convolutional features 5x bigger than the training set.<jupyter_code>da_conv_feat = conv_model.predict_generator(da_batches, da_batches.nb_sample*5)
save_array(path+'results/da_conv_feat2.dat', da_conv_feat)
da_conv_feat = load_array(path+'results/da_conv_feat2.dat')<jupyter_output><empty_output><jupyter_text>Let's include the real training data as well in its non-augmented form.<jupyter_code>da_conv_feat = np.concatenate([da_conv_feat, conv_feat])<jupyter_output><empty_output><jupyter_text>Since we've now got a dataset 6x bigger than before, we'll need to copy our labels 6 times too.<jupyter_code>da_trn_labels = np.concatenate([trn_labels]*6)<jupyter_output><empty_output><jupyter_text>Based on some experiments the previous model works well, with bigger dense layers.<jupyter_code>def get_bn_da_layers(p):
    return [
        MaxPooling2D(input_shape=conv_layers[-1].output_shape[1:]),
        Flatten(),
        Dropout(p),
        Dense(256, activation='relu'),
        BatchNormalization(),
        Dropout(p),
        Dense(256, activation='relu'),
        BatchNormalization(),
        Dropout(p),
        Dense(10, activation='softmax')
        ]
p=0.8
bn_model = Sequential(get_bn_da_layers(p))
bn_model.compile(Adam(lr=0.001), loss='categorical_crossentropy', metrics=['accuracy'])<jupyter_output><empty_output><jupyter_text>Now we can train the model as usual, with pre-computed augmented data.<jupyter_code>bn_model.fit(da_conv_feat, da_trn_labels, batch_size=batch_size, nb_epoch=1, 
             validation_data=(conv_val_feat, val_labels))
bn_model.optimizer.lr=0.01
bn_model.fit(da_conv_feat, da_trn_labels, batch_size=batch_size, nb_epoch=4, 
             validation_data=(conv_val_feat, val_labels))
bn_model.optimizer.lr=0.0001
bn_model.fit(da_conv_feat, da_trn_labels, batch_size=batch_size, nb_epoch=4, 
             validation_data=(conv_val_feat, val_labels))<jupyter_output><empty_output><jupyter_text>Looks good - let's save those weights.<jupyter_code>bn_model.save_weights(path+'models/da_conv8_1.h5')<jupyter_output><empty_output><jupyter_text>Pseudo labeling We're going to try using a combination of [pseudo labeling](http://deeplearning.net/wp-content/uploads/2013/03/pseudo_label_final.pdf) and [knowledge distillation](https://arxiv.org/abs/1503.02531) to allow us to use unlabeled data (i.e. do semi-supervised learning). For our initial experiment we'll use the validation set as the unlabeled data, so that we can see that it is working without using the test set. At a later date we'll try using the test set. To do this, we simply calculate the predictions of our model...<jupyter_code>val_pseudo = bn_model.predict(conv_val_feat, batch_size=batch_size)<jupyter_output><empty_output><jupyter_text>...concatenate them with our training labels...<jupyter_code>comb_pseudo = np.concatenate([da_trn_labels, val_pseudo])
comb_feat = np.concatenate([da_conv_feat, conv_val_feat])<jupyter_output><empty_output><jupyter_text>...and fine-tune our model using that data.<jupyter_code>bn_model.load_weights(path+'models/da_conv8_1.h5')
bn_model.fit(comb_feat, comb_pseudo, batch_size=batch_size, nb_epoch=1, 
             validation_data=(conv_val_feat, val_labels))
bn_model.fit(comb_feat, comb_pseudo, batch_size=batch_size, nb_epoch=4, 
             validation_data=(conv_val_feat, val_labels))
bn_model.optimizer.lr=0.00001
bn_model.fit(comb_feat, comb_pseudo, batch_size=batch_size, nb_epoch=4, 
             validation_data=(conv_val_feat, val_labels))<jupyter_output><empty_output><jupyter_text>That's a distinct improvement - even although the validation set isn't very big. This looks encouraging for when we try this on the test set.<jupyter_code>bn_model.save_weights(path+'models/bn-ps8.h5')<jupyter_output><empty_output><jupyter_text>Submit We'll find a good clipping amount using the validation set, prior to submitting.<jupyter_code>def do_clip(arr, mx): return np.clip(arr, (1-mx)/9, mx)
keras.metrics.categorical_crossentropy(val_labels, do_clip(val_preds, 0.93)).eval()
conv_test_feat = load_array(path+'results/conv_test_feat.dat')
preds = bn_model.predict(conv_test_feat, batch_size=batch_size*2)
subm = do_clip(preds,0.93)
subm_name = path+'results/subm.gz'
classes = sorted(batches.class_indices, key=batches.class_indices.get)
submission = pd.DataFrame(subm, columns=classes)
submission.insert(0, 'img', [a[4:] for a in test_filenames])
submission.head()
submission.to_csv(subm_name, index=False, compression='gzip')
FileLink(subm_name)<jupyter_output><empty_output><jupyter_text>This gets 0.534 on the leaderboard.  The "things that didn't really work" section You can safely ignore everything from here on, because they didn't really help.  Finetune some conv layers too<jupyter_code>for l in get_bn_layers(p): conv_model.add(l)
for l1,l2 in zip(bn_model.layers, conv_model.layers[last_conv_idx+1:]):
    l2.set_weights(l1.get_weights())
for l in conv_model.layers: l.trainable =False
for l in conv_model.layers[last_conv_idx+1:]: l.trainable =True
comb = np.concatenate([trn, val])
gen_t = image.ImageDataGenerator(rotation_range=8, height_shift_range=0.04, 
                shear_range=0.03, channel_shift_range=10, width_shift_range=0.08)
batches = gen_t.flow(comb, comb_pseudo, batch_size=batch_size)
val_batches = get_batches(path+'valid', batch_size=batch_size*2, shuffle=False)
conv_model.compile(Adam(lr=0.00001), loss='categorical_crossentropy', metrics=['accuracy'])
conv_model.fit_generator(batches, batches.N, nb_epoch=1, validation_data=val_batches, 
                 nb_val_samples=val_batches.N)
conv_model.optimizer.lr = 0.0001
conv_model.fit_generator(batches, batches.N, nb_epoch=3, validation_data=val_batches, 
                 nb_val_samples=val_batches.N)
for l in conv_model.layers[16:]: l.trainable =True
conv_model.optimizer.lr = 0.00001
conv_model.fit_generator(batches, batches.N, nb_epoch=8, validation_data=val_batches, 
                 nb_val_samples=val_batches.N)
conv_model.save_weights(path+'models/conv8_ps.h5')
conv_model.load_weights(path+'models/conv8_da.h5')
val_pseudo = conv_model.predict(val, batch_size=batch_size*2)
save_array(path+'models/pseudo8_da.dat', val_pseudo)<jupyter_output><empty_output><jupyter_text>Ensembling<jupyter_code>drivers_ds = pd.read_csv(path+'driver_imgs_list.csv')
drivers_ds.head()
img2driver = drivers_ds.set_index('img')['subject'].to_dict()
driver2imgs = {k: g["img"].tolist() 
               for k,g in drivers_ds[['subject', 'img']].groupby("subject")}
def get_idx(driver_list):
    return [i for i,f in enumerate(filenames) if img2driver[f[3:]] in driver_list]
drivers = driver2imgs.keys()
rnd_drivers = np.random.permutation(drivers)
ds1 = rnd_drivers[:len(rnd_drivers)//2]
ds2 = rnd_drivers[len(rnd_drivers)//2:]
models=[fit_conv([d]) for d in drivers]
models=[m for m in models if m is not None]
all_preds = np.stack([m.predict(conv_test_feat, batch_size=128) for m in models])
avg_preds = all_preds.mean(axis=0)
avg_preds = avg_preds/np.expand_dims(avg_preds.sum(axis=1), 1)
keras.metrics.categorical_crossentropy(val_labels, np.clip(avg_val_preds,0.01,0.99)).eval()
keras.metrics.categorical_accuracy(val_labels, np.clip(avg_val_preds,0.01,0.99)).eval()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14502369668246445, 'fraction_numerical': 0.055026191070092295, 'mean_word_length': 5.185128046899105, 'pattern_counts': {'":': 0, '<': 84, '<?xml version=': 0, '>': 84, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '22', 'dir': 'jupyter-structured-clean-dedup', 'id': '194786', 'license': 'Apache-2.0', 'n_tokens_mistral': 8031, 'n_tokens_neox': 7320, 'n_words': 1471, 'path': 'deeplearning1/nbs/statefarm.ipynb', 'repo_name': 'mousechen/courses', 'size': '20045'}