<jupyter_start><jupyter_text>Fisheries competition 大自然保护渔业监测 In this notebook we're going to investigate a range of different architectures for the [Kaggle fisheries competition](https://www.kaggle.com/c/the-nature-conservancy-fisheries-monitoring/).  The video states that vgg.py and ``vgg_ft()`` from utils.py have been updated to include VGG with batch normalization, but this is not the case.  We've instead created a new file [vgg_bn.py](https://github.com/fastai/courses/blob/master/deeplearning1/nbs/vgg16bn.py) and an additional method ``vgg_ft_bn()`` (which is already in utils.py) which we use in this notebook.<jupyter_code>%matplotlib inline
import imp
import utils
imp.reload(utils)
from utils import *
from __future__ import division, print_function
#path = "data/fish/sample/"
path = "data/fish/"
batch_size=64
batches = get_batches(path+'train', batch_size=batch_size)
val_batches = get_batches(path+'valid', batch_size=batch_size*2, shuffle=False)
(val_classes, trn_classes, val_labels, trn_labels, 
    val_filenames, filenames, test_filenames) = get_classes(path)
test2_batches = get_batches(path+'test2', shuffle=False, batch_size=1)<jupyter_output>Found 12153 images belonging to 1 classes.<jupyter_text>Sometimes it's helpful to have just the filenames, without the path.<jupyter_code>raw_filenames = [f.split('/')[-1] for f in filenames]
raw_test_filenames = [f.split('/')[-1] for f in test_filenames]
raw_val_filenames = [f.split('/')[-1] for f in val_filenames]
raw_test2_filenames = [f.split('/')[-1] for f in test2_batches.filenames]
raw_test2_filenames
len(raw_test2_filenames)<jupyter_output><empty_output><jupyter_text>建立文件夹 **整个流程可以作为，视觉处理的通用流程** We create the validation and sample sets in the usual way.<jupyter_code>%pwd
%cd data/fish
%cd train
%mkdir ../valid
g = glob('*') # 匹配当前路径下所有的文件夹名字
for d in g:
    os.mkdir('../valid/'+d) # 为valid建立各个类别的文件夹
g = glob('*/*.jpg') # 匹配train目录下所有的图片
shuffle = np.random.permutation(g) # 打乱
for i in range(500):# 随机500个图片分到valid集
    os.rename(shuffle[i],'../valid/'+shuffle[i])
os.mkdir('../sample')
os.mkdir('../sample/train')
os.mkdir('../sample/valid')
# 建立sample文件夹目录，sample文件夹的文件小，训练速度快能够迅速验证想法/模型是否正确
g = glob('*')
for d in g:
    os.mkdir('../sample/train/'+d)
    os.mkdir('../sample/valid/'+d)
# 复制文件到sample文件夹
from shutil import copyfile

g = glob('*/*jpg')
shuffle= np.random.permutation(g)
for i in range(400): 
    copyfile(shuffle[i], '../sample/train/' + shuffle[i])
%cd ../valid
g = glob('*/*.jpg')
shuf = np.random.permutation(g)
for i in range(200): 
    copyfile(shuf[i], '../sample/valid/' + shuf[i])

%cd ..
os.mkdir('results')
os.mkdir('sample/results')
%cd ../..<jupyter_output><empty_output><jupyter_text>**补充处理：**<jupyter_code>%pwd
% cd test2
g = glob('*/*.jpg')
for i in g:
    x = i.split('\\')[1]
    print(x)
    os.rename(i,'unknown/test_stg2/')
%pwd
%cd ../..
%cd ..<jupyter_output>D:\ML\courses\deeplearning1\nbs<jupyter_text>基本的VGG16模型 We start with our usual VGG approach.  We will be using VGG with batch normalization.  We explained how to add batch normalization to VGG in the [imagenet_batchnorm notebook](https://github.com/fastai/courses/blob/master/deeplearning1/nbs/imagenet_batchnorm.ipynb).  VGG with batch normalization is implemented in [vgg_bn.py](https://github.com/fastai/courses/blob/master/deeplearning1/nbs/vgg16bn.py), and there is a version of ``vgg_ft`` (our fine tuning function) with batch norm called ``vgg_ft_bn`` in [utils.py](https://github.com/fastai/courses/blob/master/deeplearning1/nbs/utils.py).  初始化模型 首先，我们建立一个经过微调的VGG模型作为开始。<jupyter_code>import myvgg16bn
imp.reload(myvgg16bn)
from myvgg16bn import Vgg16BN
model = vgg_ft_bn(8) # 就8个类型，其实就是把Dense(8,activation="softmax")
# 不同取数据的方式而已，可以用get_data(),也可以用get_batches()
trn = get_data(path+'train')
val = get_data(path+'valid')
test = get_data(path+'test')
test2 = get_data(path+'test2')
save_array(path+'results/trn.dat', trn)
save_array(path+'results/val.dat', val)
save_array(path+'results/test.dat', test)
save_array(path+'results/test2.dat', test2)
trn = load_array(path+'results/trn.dat')
val = load_array(path+'results/val.dat')
test = load_array(path+'results/test.dat')
test = load_array(path+'results/test2.dat')
gen = image.ImageDataGenerator()
1e-3
model.compile(optimizer=Adam(1e-3),loss='categorical_crossentropy', metrics=['accuracy']) # 多分类交叉熵损失
model.summary()
trn.shape
val.shape
trn_labels.shape
model.fit(trn, trn_labels, batch_size=batch_size, epochs=10, validation_data=(val, val_labels))
model.save_weights(path+'results/ft1.h5')<jupyter_output><empty_output><jupyter_text>**效果很差**  预计算卷积层的输出 We pre-compute the output of the last convolution layer of VGG, since we're unlikely to need to fine-tune those layers. (All following analysis will be done on just the pre-computed convolutional features.)我们将计算VGG模型最后一层卷积的输出，通常我们不会去调整卷积层（接下来的分析将只对预计算的卷积特性进行）<jupyter_code>model.load_weights(path+'results/ft1.h5')
# 找出最后一个卷积层，并且分开卷积层和全连接层
for l,layer in enumerate(model.layers):
    if(type(layer) is Conv2D):
        print(l,layer,type(layer))
        
layer_index = [l for l, layer in enumerate(model.layers) if (type(layer)) is Conv2D]
print(layer_index)
print(layer_index[-1])
conv_layers = model.layers[:layer_index[-1]+1]
fc_layers = model.layers[layer_index[-1]+1]
# def split_at(model, layer_type):
#     layers = model.layers
#     layer_idx = [index for index,layer in enumerate(layers)
#                  if type(layer) is layer_type][-1]
#     return layers[:layer_idx+1], layers[layer_idx+1:]
conv_layers,fc_layers = split_at(model, Conv2D)
conv_model = Sequential(conv_layers)<jupyter_output><empty_output><jupyter_text>直接用卷积层建立模型，输出预测，作为全连接层的输入，也叫预计算<jupyter_code>conv_feat = conv_model.predict(trn,)
conv_val_feat = conv_model.predict(val)
conv_test_feat = conv_model.predict(test)
conv_test2_feat = conv_model.predict(test2)
save_array(path+'results/conv_val_feat.dat', conv_val_feat)
save_array(path+'results/conv_feat.dat', conv_feat)
save_array(path+'results/conv_test_feat.dat', conv_test_feat)
save_array(path+'results/conv_test2_feat.dat', conv_test_feat)
conv_feat = load_array(path+'results/conv_feat.dat')
conv_val_feat = load_array(path+'results/conv_val_feat.dat')
conv_test_feat = load_array(path+'results/conv_test_feat.dat')
conv_val_feat.shape
conv_test_feat.shape
conv_test2_feat.shape
conv_feat.shape<jupyter_output><empty_output><jupyter_text>训练模型 We can now create our first baseline model - a simple 3-layer FC net.现在建立我们的第一个基线模型 - 一个简单的3层全联接网络<jupyter_code>conv_layers[-1].output_shape[1:]
def get_bn_layers(p):
    return [
        MaxPooling2D(input_shape=conv_layers[-1].output_shape[1:]), # 序列模型第一个必须指定input_shape
        BatchNormalization(axis=1),
        Dropout(p/4),
        Flatten(),
        Dense(512, activation='relu'),
        BatchNormalization(),
        Dropout(p),
        Dense(512, activation='relu'),
        BatchNormalization(),
        Dropout(p/2),
        Dense(8, activation='softmax')
    ]
p=0.6
bn_model = Sequential(get_bn_layers(p))
bn_model.compile(Adam(lr=0.001), loss='categorical_crossentropy', metrics=['accuracy'])
bn_model.fit(conv_feat, trn_labels, batch_size=batch_size, epochs=30, 
             validation_data=(conv_val_feat, val_labels))
bn_model.optimizer.lr = 1e-4
bn_model.fit(conv_feat, trn_labels, batch_size=batch_size, epochs=10, 
             validation_data=(conv_val_feat, val_labels))
bn_model.save_weights(path+'models/conv_512_6.h5')
bn_model.evaluate(conv_val_feat, val_labels) # 在验证集上估计效果
bn_model.load_weights(path+'models/conv_512_6.h5')<jupyter_output><empty_output><jupyter_text>直接预测提交 由于要提交2份测试数据的预测，我开始的做法是分别预测生成csv然后手动粘贴复制成一个csv的，能不能找到一个办法一起预测呢<jupyter_code>def do_clip(arr, mx): 
    return np.clip(arr, (1-mx)/7, mx)
conv_test_feat.shape
conv_test2_feat.shape
preds = bn_model.predict(conv_test_feat,verbose=1)
preds.shape
preds2 = bn_model.predict(conv_test2_feat,verbose=1)
preds2.shape
subm = do_clip(preds,0.82)
subm.shape
subm_name = path+'results/subm_bb.gz'
# classes = sorted(batches.class_indices, key=batches.class_indices.get)
classes = ['ALB', 'BET', 'DOL', 'LAG', 'NoF', 'OTHER', 'SHARK', 'YFT']
submission = pd.DataFrame(data = subm, columns=classes)
submission.insert(0, 'image', [a[8:] for a in raw_test_filenames])
submission.to_csv(subm_name, index=False, compression='gzip')
subm_name = path+'results/subm_bb_2.gz'
subm = do_clip(preds2,0.82)
subm.shape
submission = pd.DataFrame(data = subm, columns=classes)
submission.insert(0, 'image', [a for a in raw_test2_filenames])
submission.to_csv(subm_name, index=False, compression='gzip')
FileLink(subm_name)<jupyter_output><empty_output><jupyter_text>Multi-input 这些图像有不同的尺寸，很可能代表它们来自的船（因为不同的船会使用不同的相机）。也许这造成了一些数据泄露，我们可以利用这一点来获得一个更好的“Kaggle”排行榜的位置？为了找到答案，首先我们为每个图像创建文件大小的数组：<jupyter_code>path
# 获得所有图片的大小，用PIL的Image.open方法

sizes = [PIL.Image.open(path+'train/'+f).size for f in filenames]

id2size = list(set(sizes))
size2id = {o:i for i,o in enumerate(id2size)}
from collections import Counter
Counter(sizes)
len(id2size)<jupyter_output><empty_output><jupyter_text>Then we one-hot encode them (since we want to treat them as categorical) and normalize the data.把图片大小视为一种类型，采用one-hot编码：<jupyter_code>test_filenames
# 训练数据
trn_sizes_orig = to_categorical([size2id[o] for o in sizes],len(id2size))
# 验证数据，图片大小
raw_val_sizes = [PIL.Image.open(path+'valid/'+f).size for f in val_filenames]
val_sizes = to_categorical([size2id[o] for o in raw_val_sizes], len(id2size))
# 测试数据
raw_test_sizes = [PIL.Image.open(path+'test/'+f).size for f in test_filenames]
test_sizes =  to_categorical([size2id[o] for o in raw_test_sizes], len(id2size))
# 原值-平均值 / 标准差 这是规范化的做法
trn_sizes = trn_sizes_orig-trn_sizes_orig.mean(axis=0)/trn_sizes_orig.std(axis=0)
val_sizes = val_sizes-trn_sizes_orig.mean(axis=0)/trn_sizes_orig.std(axis=0)
test_sizes = test_sizes-trn_sizes_orig.mean(axis=0)/trn_sizes_orig.std(axis=0)
trn_sizes
test_sizes<jupyter_output><empty_output><jupyter_text>To use this additional "meta-data", we create a model with multiple input layers - `sz_inp` will be our input for the size information.为了使用这个额外的“元数据”，我们创建了一个具有多个输入层的模型——“sz_inp”将是我们对图片大小信息的输入。<jupyter_code>p=0.6
inp = Input(conv_layers[-1].output_shape[1:])
sz_inp = Input((len(id2size),))
bn_inp = BatchNormalization()(sz_inp)

x = MaxPooling2D()(inp)
x = BatchNormalization(axis=1)(x)
x = Dropout(p/4)(x)
x = Flatten()(x)
x = Dense(512, activation='relu')(x)
x = BatchNormalization()(x)
x = Dropout(p)(x)
x = Dense(512, activation='relu')(x)
x = BatchNormalization()(x)
x = Dropout(p/2)(x)
x = keras.layers.concatenate([x,bn_inp])
x = Dense(8, activation='softmax')(x)<jupyter_output><empty_output><jupyter_text>When we compile the model, we have to specify all the input layers in an array.当我们编译模型时，我们必须指定一个数组中的所有输入层。<jupyter_code>model = Model([inp, sz_inp], x)
model.compile(Adam(lr=0.001), loss='categorical_crossentropy', metrics=['accuracy'])<jupyter_output><empty_output><jupyter_text>And when we train the model, we have to provide all the input layers' data in an array.当我们训练模型时，我们必须指定一个数组中的所有输入层。<jupyter_code>model.fit([conv_feat, trn_sizes], trn_labels, batch_size=batch_size, epochs=30, 
             validation_data=([conv_val_feat, val_sizes], val_labels))<jupyter_output><empty_output><jupyter_text>比较以下`bn_model`模型的效果：<jupyter_code>bn_model.optimizer.lr = 1e-4
bn_model.fit(conv_feat, trn_labels, batch_size=batch_size, epochs=8, 
             validation_data=(conv_val_feat, val_labels))<jupyter_output><empty_output><jupyter_text>The model did not show an improvement by using the leakage, other than in the early epochs. This is most likely because the information about what boat the picture came from is readily identified from the image itself, so the meta-data turned out not to add any additional information.除了在早期阶段，该模型没有显示出使用泄漏的改进。这很有可能是因为图片本身的信息很容易从图像本身中识别出来，所以元数据并没有添加任何额外的信息。  边界盒 & 多输出  导入/可视化 边界盒 A kaggle user has created bounding box annotations for each fish in each training set image. You can download them [from here](https://www.kaggle.com/c/the-nature-conservancy-fisheries-monitoring/forums/t/25902/complete-bounding-box-annotation). We will see if we can utilize this additional information. First, we'll load in the data, and keep just the largest bounding box for each image.一个kaggle用户在每个训练集图像中为每个鱼创建了边界框注释。你可以从这里下载。我们将看看是否能利用这些额外的信息。首先，我们将载入数据，并为每个图像保留最大的边界框。<jupyter_code>import ujson as json
anno_classes = ['alb', 'bet', 'dol', 'lag', 'other', 'shark', 'yft']
# 定义下载边界盒子的函数
def get_annotations():
    annot_urls = {
        '5458/bet_labels.json': 'bd20591439b650f44b36b72a98d3ce27',
        '5459/shark_labels.json': '94b1b3110ca58ff4788fb659eda7da90',
        '5460/dol_labels.json': '91a25d29a29b7e8b8d7a8770355993de',
        '5461/yft_labels.json': '9ef63caad8f076457d48a21986d81ddc',
        '5462/alb_labels.json': '731c74d347748b5272042f0661dad37c',
        '5463/lag_labels.json': '92d75d9218c3333ac31d74125f2b380a'
    }
    cache_subdir = os.path.abspath(os.path.join(path, 'annos'))
    url_prefix = 'https://kaggle2.blob.core.windows.net/forum-message-attachments/147157/'
    
    if not os.path.exists(cache_subdir):
        os.makedirs(cache_subdir)
    
    for url_suffix, md5_hash in annot_urls.items():
        fname = url_suffix.rsplit('/', 1)[-1]
        get_file(fname, url_prefix + url_suffix, cache_subdir=cache_subdir, md5_hash=md5_hash)
get_annotations()
bb_json = {}
for c in anno_classes:
    if c == 'other': continue # no annotation file for "other" class
    j = json.load(open('{}annos/{}_labels.json'.format(path, c), 'r'))
    for l in j:
        if 'annotations' in l.keys() and len(l['annotations'])>0:
            bb_json[l['filename'].split('/')[-1]] = sorted(
                l['annotations'], key=lambda x: x['height']*x['width'])[-1]
bb_json['img_04908.jpg']
file2idx = {o:i for i,o in enumerate(raw_filenames)}
val_file2idx = {o:i for i,o in enumerate(raw_val_filenames)}<jupyter_output><empty_output><jupyter_text>For any images that have no annotations, we'll create an empty bounding box.<jupyter_code>empty_bbox = {'height': 0., 'width': 0., 'x': 0., 'y': 0.}
for f in raw_filenames:
    if not f in bb_json.keys(): bb_json[f] = empty_bbox
for f in raw_val_filenames:
    if not f in bb_json.keys(): bb_json[f] = empty_bbox<jupyter_output><empty_output><jupyter_text>Finally, we convert the dictionary into an array, and convert the coordinates to our resized 224x224 images.<jupyter_code>bb_params = ['height', 'width', 'x', 'y']
def convert_bb(bb, size):
    bb = [bb[p] for p in bb_params]
    conv_x = (224. / size[0])
    conv_y = (224. / size[1])
    bb[0] = bb[0]*conv_y
    bb[1] = bb[1]*conv_x
    bb[2] = max(bb[2]*conv_x, 0)
    bb[3] = max(bb[3]*conv_y, 0)
    return bb
trn_bbox = np.stack([convert_bb(bb_json[f], s) for f,s in zip(raw_filenames, sizes)], 
                   ).astype(np.float32)
val_bbox = np.stack([convert_bb(bb_json[f], s) 
                   for f,s in zip(raw_val_filenames, raw_val_sizes)]).astype(np.float32)<jupyter_output><empty_output><jupyter_text>Now we can check our work by drawing one of the annotations.<jupyter_code>def create_rect(bb, color='red'):
    return plt.Rectangle((bb[2], bb[3]), bb[1], bb[0], color=color, fill=False, lw=3)

def show_bb(i):
    bb = val_bbox[i]
    plot(val[i])
    plt.gca().add_patch(create_rect(bb))
show_bb(15)<jupyter_output><empty_output><jupyter_text>建立/训练 模型 Since we're not allowed (by the kaggle rules) to manually annotate the test set, we'll need to create a model that predicts the locations of the bounding box on each image. To do so, we create a model with multiple outputs: it will predict both the type of fish (the 'class'), and the 4 bounding box coordinates. We prefer this approach to only predicting the bounding box coordinates, since we hope that giving the model more context about what it's looking for will help it with both tasks.<jupyter_code>p=0.6
inp = Input(conv_layers[-1].output_shape[1:])
x = MaxPooling2D()(inp)
x = BatchNormalization(axis=1)(x)
x = Dropout(p/4)(x)
x = Flatten()(x)
x = Dense(512, activation='relu')(x)
x = BatchNormalization()(x)
x = Dropout(p)(x)
x = Dense(512, activation='relu')(x)
x = BatchNormalization()(x)
x = Dropout(p/2)(x)
x_bb = Dense(4, name='bb')(x) # 多输出
x_class = Dense(8, activation='softmax', name='class')(x)<jupyter_output><empty_output><jupyter_text>Since we have multiple outputs, we need to provide them to the model constructor in an array, and we also need to say what loss function to use for each. We also weight the bounding box loss function down by 1000x since the scale of the cross-entropy loss and the MSE is very different.<jupyter_code>model = Model([inp], [x_bb, x_class])
model.compile(Adam(lr=0.001), loss=['mse', 'categorical_crossentropy'], metrics=['accuracy'],
             loss_weights=[.001, 1.])
model.fit(conv_feat, [trn_bbox, trn_labels], batch_size=batch_size, epochs=3, 
             validation_data=(conv_val_feat, [val_bbox, val_labels]))
model.optimizer.lr = 1e-5
model.fit(conv_feat, [trn_bbox, trn_labels], batch_size=batch_size, epochs=20, 
             validation_data=(conv_val_feat, [val_bbox, val_labels]))<jupyter_output><empty_output><jupyter_text>Excitingly, it turned out that the classification model is much improved by giving it this additional task. Let's see how well the bounding box model did by taking a look at its output.令人兴奋的是，通过给它这个额外的任务，分类模型得到了很大的改进。让我们来看看这个边界框模型是如何通过观察它的输出来实现的。<jupyter_code>pred = model.predict(conv_val_feat[0:10])
def show_bb_pred(i):
    bb = val_bbox[i]
    bb_pred = pred[0][i]
    plt.figure(figsize=(6,6))
    plot(val[i])
    ax=plt.gca()
    ax.add_patch(create_rect(bb_pred, 'yellow'))
    ax.add_patch(create_rect(bb))<jupyter_output><empty_output><jupyter_text>The image shows that it can find fish that are tricky for us to see!<jupyter_code>show_bb_pred(6)
model.evaluate(conv_val_feat, [val_bbox, val_labels])
model.save_weights(path+'models/bn_anno.h5')
model.load_weights(path+'models/bn_anno.h5')<jupyter_output><empty_output><jupyter_text>更大的图片  建立数据 Let's see if we get better results if we use larger images. We'll use 640x360, since it's the same shape as the most common size we saw earlier (1280x720), without being too big.让我们看看如果我们使用更大的图像会得到更好的结果。我们将使用640x360，因为它的形状与我们之前看到的最常见的尺寸相同（1280x720），而不是太大。<jupyter_code>trn = get_data(path+'train', (360,640))
val = get_data(path+'valid', (360,640))<jupyter_output>Found 3277 images belonging to 8 classes.
Found 500 images belonging to 8 classes.<jupyter_text>The image shows that things are much clearer at this size.<jupyter_code>plot(trn[0])
# 最早我试图一次读入13154张图片，但是内存不够，完全读不了,目前test1只有1000张图片
test = get_data(path+'test', (360,640))
save_array(path+'results/trn_640.dat', trn)
save_array(path+'results/val_640.dat', val)
save_array(path+'results/test_640.dat', test)
trn = load_array(path+'results/trn_640.dat')
val = load_array(path+'results/val_640.dat')<jupyter_output><empty_output><jupyter_text>We can now create our VGG model - we'll need to tell it we're not using the normal 224x224 images, which also means it won't include the fully connected layers (since they don't make sense for non-default sizes). We will also remove the last max pooling layer, since we don't want to throw away information yet.<jupyter_code>vgg640 = Vgg16BN((360, 640)).model
vgg640.pop()
vgg640.input_shape, vgg640.output_shape
vgg640.compile(Adam(), 'categorical_crossentropy', metrics=['accuracy'])<jupyter_output><empty_output><jupyter_text>We can now pre-compute the output of the convolutional part of VGG.<jupyter_code>conv_val_feat = vgg640.predict(val, batch_size=32, verbose=1)
conv_trn_feat = vgg640.predict(trn, batch_size=32, verbose=1)
save_array(path+'results/conv_val_640.dat', conv_val_feat)
save_array(path+'results/conv_trn_640.dat', conv_trn_feat)
conv_test_feat = vgg640.predict(test, batch_size=32, verbose=1)
save_array(path+'results/conv_test_640.dat', conv_test_feat)
conv_val_feat = load_array(path+'results/conv_val_640.dat')
conv_trn_feat = load_array(path+'results/conv_trn_640.dat')
conv_test_feat = load_array(path+'results/conv_test_640.dat')<jupyter_output><empty_output><jupyter_text>完整的全卷积网络 Fully convolutional net (FCN) Since we're using a larger input, the output of the final convolutional layer is also larger. So we probably don't want to put a dense layer there - that would be a *lot* of parameters! Instead, let's use a fully convolutional net (FCN); this also has the benefit that they tend to generalize well, and also seems like a good fit for our problem (since the fish are a small part of the image).由于我们使用的是更大的输入，最终卷积层的输出也更大。所以我们可能不想在那里放一个稠密的层——那将会有很多参数！相反，让我们使用一个完全卷积的网络（FCN）;这也有好处，他们倾向于很好地概括，而且似乎也很适合我们的问题（因为鱼是图像的一小部分）。<jupyter_code>conv_layers,_ = split_at(vgg640, Conv2D) # 丢弃全连接层，因为参数太多了<jupyter_output><empty_output><jupyter_text>I'm not using any dropout, since I found I got better results without it.<jupyter_code>nf=128; p=0.6
def get_lrg_layers():
    return [
        BatchNormalization(axis=1, input_shape=conv_layers[-1].output_shape[1:]),
        Conv2D(nf,kernel_size=(3,3),strides=(1,1), activation='relu', padding='same'),
        BatchNormalization(axis=1),
        MaxPooling2D(),
        Convolution2D(nf,kernel_size=(3,3),strides=(1,1), activation='relu',  padding='same'),
        BatchNormalization(axis=1),
        MaxPooling2D(),
        Conv2D(nf,kernel_size=(3,3),strides=(1,1), activation='relu',  padding='same'),
        BatchNormalization(axis=1),
        MaxPooling2D((1,2)),
        Conv2D(8,kernel_size=(3,3),strides=(1,1),  padding='same'),
        Dropout(p),
        GlobalAveragePooling2D(),
        Activation('softmax')
    ]
lrg_model = Sequential(get_lrg_layers())
lrg_model.summary()
lrg_model.compile(Adam(lr=0.001), loss='categorical_crossentropy', metrics=['accuracy'])
lrg_model.fit(conv_trn_feat, trn_labels, batch_size=batch_size, epochs=2, 
             validation_data=(conv_val_feat, val_labels))
lrg_model.optimizer.lr=1e-5
lrg_model.fit(conv_trn_feat, trn_labels, batch_size=batch_size, epochs=20, 
             validation_data=(conv_val_feat, val_labels))<jupyter_output>Train on 3277 samples, validate on 500 samples
Epoch 1/20
3277/3277 [==============================] - 6s 2ms/step - loss: 0.1085 - acc: 0.9722 - val_loss: 0.1825 - val_acc: 0.9540
Epoch 2/20
3277/3277 [==============================] - 6s 2ms/step - loss: 0.0676 - acc: 0.9850 - val_loss: 0.1704 - val_acc: 0.9480
Epoch 3/20
3277/3277 [==============================] - 6s 2ms/step - loss: 0.0335 - acc: 0.9918 - val_loss: 0.1316 - val_acc: 0.9540
Epoch 4/20
3277/3277 [==============================] - 6s 2ms/step - loss: 0.0368 - acc: 0.9921 - val_loss: 0.1688 - val_acc: 0.9440
Epoch 5/20
3277/3277 [==============================] - 6s 2ms/step - loss: 0.0160 - acc: 0.9973 - val_loss: 0.1403 - val_acc: 0.9620
Epoch 6/20
3277/3277 [==============================] - 6s 2ms/step - loss: 0.0065 - acc: 0.9991 - val_loss: 0.1259 - val_acc: 0.9700
Epoch 7/20
3277/3277 [==============================] - 6s 2ms/step - loss: 0.0161 - acc: 0.9957 - val_loss: 0.1291 - val_acc: 0.9600
Epoch 8/20
3277[...]<jupyter_text>When I submitted the results of this model to Kaggle, I got the best single model results of any shown here (ranked 22nd on the leaderboard as at Dec-6-2016.)<jupyter_code>lrg_model.save_weights(path+'models/lrg_nmp.h5')
lrg_model.load_weights(path+'models/lrg_nmp.h5')
lrg_model.evaluate(conv_val_feat, val_labels)
lrg_model.predict(conv_test_feat,verbose=1)<jupyter_output>1000/1000 [==============================] - 2s 2ms/step<jupyter_text>提交<jupyter_code># 补充test2的数据 ，还是memory的问题，根本没办法，目前24G内存也读不进来┏┛墓┗┓...(((m -__-)m了
test = get_data(path+'test2', (360,640))<jupyter_output>Found 12153 images belonging to 1 classes.<jupyter_text>Another benefit of this kind of model is that the last convolutional layer has to learn to classify each part of the image (since there's only an average pooling layer after). Let's create a function that grabs the output of this layer (which is the 4th-last layer of our model).<jupyter_code>l = lrg_model.layers
conv_fn = K.function([l[0].input, K.learning_phase()], l[-4].output)
def get_cm(inp, label):
    conv = conv_fn([inp,0])[0, label]
    return scipy.misc.imresize(conv, (360,640), interp='nearest')<jupyter_output><empty_output><jupyter_text>We have to add an extra dimension to our input since the CNN expects a 'batch' (even if it's just a batch of one).<jupyter_code>inp = np.expand_dims(conv_val_feat[0], 0)
np.round(lrg_model.predict(inp)[0],2)
plt.imshow(to_plot(val[0]))
cm = get_cm(inp, 0)<jupyter_output><empty_output><jupyter_text>The heatmap shows that (at very low resolution) the model is finding the fish!<jupyter_code>plt.imshow(cm, cmap="cool")<jupyter_output><empty_output><jupyter_text>All convolutional net heatmap To create a higher resolution heatmap, we'll remove all the max pooling layers, and repeat the previous steps.<jupyter_code>def get_lrg_layers():
    return [
        BatchNormalization(axis=1, input_shape=conv_layers[-1].output_shape[1:]),
        Convolution2D(nf,3,3, activation='relu', border_mode='same'),
        BatchNormalization(axis=1),
        Convolution2D(nf,3,3, activation='relu', border_mode='same'),
        BatchNormalization(axis=1),
        Convolution2D(nf,3,3, activation='relu', border_mode='same'),
        BatchNormalization(axis=1),
        Convolution2D(8,3,3, border_mode='same'),
        GlobalAveragePooling2D(),
        Activation('softmax')
    ]
lrg_model = Sequential(get_lrg_layers())
lrg_model.summary()
lrg_model.compile(Adam(lr=0.001), loss='categorical_crossentropy', metrics=['accuracy'])
lrg_model.fit(conv_trn_feat, trn_labels, batch_size=batch_size, nb_epoch=2, 
             validation_data=(conv_val_feat, val_labels))
lrg_model.optimizer.lr=1e-5
lrg_model.fit(conv_trn_feat, trn_labels, batch_size=batch_size, nb_epoch=6, 
             validation_data=(conv_val_feat, val_labels))
lrg_model.save_weights(path+'models/lrg_0mp.h5')
lrg_model.load_weights(path+'models/lrg_0mp.h5')<jupyter_output><empty_output><jupyter_text>Create heatmap<jupyter_code>l = lrg_model.layers
conv_fn = K.function([l[0].input, K.learning_phase()], l[-3].output)
def get_cm2(inp, label):
    conv = conv_fn([inp,0])[0, label]
    return scipy.misc.imresize(conv, (360,640))
inp = np.expand_dims(conv_val_feat[0], 0)
plt.imshow(to_plot(val[0]))
cm = get_cm2(inp, 0)
cm = get_cm2(inp, 4)
plt.imshow(cm, cmap="cool")
plt.figure(figsize=(10,10))
plot(val[0])
plt.imshow(cm, cmap="cool", alpha=0.5)<jupyter_output><empty_output><jupyter_text>Inception mini-net Here's an example of how to create and use "inception blocks" - as you see, they use multiple different convolution filter sizes and concatenate the results together. We'll talk more about these next year.<jupyter_code>def conv2d_bn(x, nb_filter, nb_row, nb_col, subsample=(1, 1)):
    x = Convolution2D(nb_filter, nb_row, nb_col,
                      subsample=subsample, activation='relu', border_mode='same')(x)
    return BatchNormalization(axis=1)(x)
def incep_block(x):
    branch1x1 = conv2d_bn(x, 32, 1, 1, subsample=(2, 2))
    branch5x5 = conv2d_bn(x, 24, 1, 1)
    branch5x5 = conv2d_bn(branch5x5, 32, 5, 5, subsample=(2, 2))

    branch3x3dbl = conv2d_bn(x, 32, 1, 1)
    branch3x3dbl = conv2d_bn(branch3x3dbl, 48, 3, 3)
    branch3x3dbl = conv2d_bn(branch3x3dbl, 48, 3, 3, subsample=(2, 2))

    branch_pool = AveragePooling2D(
        (3, 3), strides=(2, 2), border_mode='same')(x)
    branch_pool = conv2d_bn(branch_pool, 16, 1, 1)
    return merge([branch1x1, branch5x5, branch3x3dbl, branch_pool],
              mode='concat', concat_axis=1)
inp = Input(vgg640.layers[-1].output_shape[1:]) 
x = BatchNormalization(axis=1)(inp)
x = incep_block(x)
x = incep_block(x)
x = incep_block(x)
x = Dropout(0.75)(x)
x = Convolution2D(8,3,3, border_mode='same')(x)
x = GlobalAveragePooling2D()(x)
outp = Activation('softmax')(x)
lrg_model = Model([inp], outp)
lrg_model.compile(Adam(lr=0.001), loss='categorical_crossentropy', metrics=['accuracy'])
lrg_model.fit(conv_trn_feat, trn_labels, batch_size=batch_size, nb_epoch=2, 
             validation_data=(conv_val_feat, val_labels))
lrg_model.optimizer.lr=1e-5
lrg_model.fit(conv_trn_feat, trn_labels, batch_size=batch_size, nb_epoch=6, 
             validation_data=(conv_val_feat, val_labels))
lrg_model.fit(conv_trn_feat, trn_labels, batch_size=batch_size, nb_epoch=10, 
             validation_data=(conv_val_feat, val_labels))
lrg_model.save_weights(path+'models/lrg_nmp.h5')
lrg_model.load_weights(path+'models/lrg_nmp.h5')<jupyter_output><empty_output><jupyter_text>Pseudo-labeling<jupyter_code>model = vgg_ft_bn(8)
model.load_weights(path+'results/ft1.h5')
conv_layers,fc_layers = split_at(model, Conv2D)
conv_feat = load_array(path+'results/conv_feat.dat')
conv_val_feat = load_array(path+'results/conv_val_feat.dat')
conv_test_feat = load_array(path+'results/conv_test_feat.dat')
p=0.6
len(id2size)
conv_layers[-1].output_shape[1:]
inp = Input(conv_layers[-1].output_shape[1:])
sz_inp = Input((len(id2size),))
bn_inp = BatchNormalization()(sz_inp)

x = MaxPooling2D()(inp)
x = BatchNormalization(axis=1)(x)
x = Dropout(p/4)(x)
x = Flatten()(x)
x = Dense(512, activation='relu')(x)
x = BatchNormalization()(x)
x = Dropout(p)(x)
x = Dense(512, activation='relu')(x)
x = BatchNormalization()(x)
x = Dropout(p/2)(x)
x = keras.layers.concatenate([x,bn_inp])
x = Dense(8, activation='softmax')(x)
conv_feat.shape
model = Model([inp, sz_inp], x)
model.compile(Adam(lr=0.001), loss='categorical_crossentropy', metrics=['accuracy'])
model.fit([conv_feat, trn_sizes], trn_labels, batch_size=batch_size, epochs=30, 
             validation_data=([conv_val_feat, val_sizes], val_labels))
model.optimizer.lr = 0.0001
model.fit([conv_feat, trn_sizes], trn_labels, batch_size=batch_size, epochs=30, 
             validation_data=([conv_val_feat, val_sizes], val_labels))
preds = model.predict([conv_test_feat, test_sizes], batch_size=batch_size*2) # 半监督学些，伪标记
preds[:1] # 8种图片大小
gen = image.ImageDataGenerator()
conv_test_feat.shape
test_batches = gen.flow(conv_test_feat, preds, batch_size=16)
val_batches = gen.flow(conv_val_feat, val_labels, batch_size=4)
batches = gen.flow(conv_feat, trn_labels, batch_size=44)
next(batches)[0][0].shape
next(batches)[0][1].shape
type(batches)
class MixIterator(object):
    def __init__(self, iters):
        self.iters = iters
        self.multi = type(iters) is list
        if self.multi:
            self.n = sum([it.n for it in self.iters])
        else:
            self.n = sum([it.n for it in self.iters])

    def reset(self):
        for it in self.iters: 
            it.reset()

    def __iter__(self):
        return self

    def next(self, *args, **kwargs):
        if self.multi:
#             for o in self.iters:
#                 for it in o:
#                     print(type(it))
            nexts = [[next(o)] for o in self.iters]
            n0 = np.concatenate([n[0][0] for n in nexts])
            n1 = np.concatenate([n[1][1] for n in nexts])
            return (n0, n1)
        else:
            nexts = [next(it) for it in self.iters]
            n0 = np.concatenate([n[0][0] for n in nexts])
            n1 = np.concatenate([n[1][0] for n in nexts])
            return (n0, n1)
mi = MixIterator([batches, test_batches, val_batches])
bn_model.fit_generator(mi, mi.n, epochs=8, validation_data=(conv_val_feat, val_labels))<jupyter_output><empty_output><jupyter_text>Submit 1<jupyter_code>def do_clip(arr, mx): 
    return np.clip(arr, (1-mx)/7, mx)
lrg_model.evaluate(conv_val_feat, val_labels, batch_size*2)
preds = lrg_model.predict(conv_test_feat, batch_size=batch_size)
preds
preds.shape
preds = preds[1]
preds
subm = do_clip(preds,0.82)
subm.shape
subm_name = path+'results/subm_bb.gz'
# classes = sorted(batches.class_indices, key=batches.class_indices.get)
classes = ['ALB', 'BET', 'DOL', 'LAG', 'NoF', 'OTHER', 'SHARK', 'YFT']
submission = pd.DataFrame(data = subm, columns=classes)
submission.insert(0, 'image', [a[8:] for a in raw_test_filenames])
submission.head()
submission.to_csv(subm_name, index=False, compression='gzip')
FileLink(subm_name)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14244872591671845, 'fraction_numerical': 0.036575512740832816, 'mean_word_length': 5.6421052631578945, 'pattern_counts': {'":': 0, '<': 151, '<?xml version=': 0, '>': 152, 'https://': 7, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '39', 'dir': 'jupyter-structured-clean-dedup', 'id': '6837', 'license': 'Apache-2.0', 'n_tokens_mistral': 13549, 'n_tokens_neox': 13074, 'n_words': 2435, 'path': 'deeplearning1/nbs/lesson7.ipynb', 'repo_name': 'mousechen/courses', 'size': '32180'}