<jupyter_start><jupyter_text>CondenseNet Goes Wrong===This notebook looks at from when CondenseNet pruning goes wrong and whether there is any chance we can recover from them.<jupyter_code>import os
import sys
import json
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from scipy.stats.stats import pearsonr<jupyter_output><empty_output><jupyter_text>Plot the GOpt procedure.In order to identify the position that CondenseNet pruning goes wrong, here we load all the training logs and plot the change in training and validation accuracy to see how it actually happens.<jupyter_code>def load_gopt_logs(out_dir):
    """ Load log files from all gopt steps. """
    n_step = len(os.listdir(out_dir)) - 1 # exclude the log.txt

    df = None
    for i in range(n_step):
        step_dir = os.path.join(out_dir, 'step_{}'.format(i + 1))
        log_file = os.path.join(step_dir, 'log.txt')
        df_ = pd.read_csv(log_file, sep='\t')
        df_ = df_.loc[:, ~df_.columns.str.contains('^Unnamed')]
        df_ = df_.loc[[df_['Valid Acc.'].idxmax()]]

        if df is None:
            df = df_
        else:
            df = pd.concat([df, df_])
    
    return df
base_dir = '/mnt/ccnas2/bdp/rz3515/train/gumi/evaluation/iccv19/final/condensenet86/cifar100/prune/G_4/gopt_v2/'
out_dirs = {
    'lr_1e-3_epochs_30': 'P0.52_MIN8_dyn_LR_1e-3_cosine_max_30',
    'epochs_1': 'P0.52_MIN4_wd_5e-3_LR_1e-4_epoch_1',
    'epochs_10': 'P0.52_MIN4_wd_5e-4_LR_1e-4_epoch_10',
    'epochs_100_MAX16_0.05': 'P0.52_MIN4_MAX16_wd_1e-4_LR_1e-4_epoch_100_acc_5e-2'
}
dfs = [load_gopt_logs(os.path.join(base_dir, d)) for d in out_dirs.values()]
labels = list(out_dirs.keys())
for i, df in enumerate(dfs):
    plt.plot(np.arange(len(df)), df['Train Acc.'], label='{} (train)'.format(labels[i]))
    plt.plot(np.arange(len(df)), df['Valid Acc.'], label='{} (valid)'.format(labels[i]))
    
plt.title('Comparison between different GOpt schedule')
plt.ylabel('Accuracy (%)')
plt.xlabel('Step')
plt.legend()<jupyter_output><empty_output><jupyter_text>From this figure, we may notice that fine-tuning for at least 10 epochs can significantly slow down the deterioration procedure, especially for the training accuracy.But for validation, no matter how long we fine-tune each step, it becomes worse and worse at a similar rate, even with 100 epochs per pruning step.<jupyter_code>labels = list(out_dirs.keys())
for i, df in enumerate(dfs):
    plt.plot(np.arange(len(df)), df['Valid Acc.'], label='{} (valid)'.format(labels[i]))
    
plt.title('Comparison between different GOpt schedule')
plt.ylabel('Accuracy (%)')
plt.xlabel('Step')
plt.legend()<jupyter_output><empty_output><jupyter_text>Let's look at the validation accuracy mainly.Before the 10-th step, the validation accuracy goes down constantly, and it keeps for a while (~20 steps), and then goes down significantly again until the end.<jupyter_code>[df.head(10)['Valid Acc.'].values for df in dfs]<jupyter_output><empty_output><jupyter_text>One question that might be interesting to look at: _can we compare the validation accuracy from pruning different layers and the estimated cost?_  Analysis: Step 1Here we run pruning for one step on every module. The pruned module is fine-tuned for one epoch only. We also record the cost value that we extract from running the pruning algorithm.<jupyter_code>dbg_dir = '/mnt/ccnas2/bdp/rz3515/train/gumi/evaluation/iccv19/final/condensenet86/cifar100/prune/gopt_v2/debug_first_step/'
n_mods = len(os.listdir(dbg_dir)) - 1
data = []

for i in range(n_mods):
    mod_dir = os.path.join(dbg_dir, 'mod_{}'.format(i))
    meta_file = os.path.join(mod_dir, 'meta_data.json')
    with open(meta_file, 'r') as f:
        meta_data = json.load(f)
    
    data.append([meta_data['mod_name'], meta_data['cost'], meta_data['val_acc'], meta_data['init_acc']])

df = pd.DataFrame(data, columns=['mod_name', 'cost', 'val_acc', 'init_acc'])
plt.scatter(df['cost'], df['val_acc'], s=50, alpha=.5, label='val_acc')
plt.scatter(df['cost'], df['init_acc'], s=50, alpha=.5, label='init_acc')

plt.xlabel('Cost')
plt.ylabel('Accuracy')
plt.legend()<jupyter_output><empty_output><jupyter_text>The figure above shows that cost is not a very good indicator of the importance of weights. Basically, even a module that has high cost, they can still be recovered to an accuracy that is similar to the smallest cost.However, we should notice that, only the sample with the smallest cost can have the same validation accuracy before and after fine-tuning, which implies that the value of cost is somehow important still (WTF?)Also, **smaller cost means less likely to loss much accuracy**.<jupyter_code>plt.plot(df['val_acc'])
plt.plot(df['init_acc'])

plt.xlabel('Layer ID')
plt.ylabel('Accuracy')
plt.legend()
plt.plot(df['cost'])
plt.xlabel('Layer ID')
plt.ylabel('Cost')
plt.legend()<jupyter_output><empty_output><jupyter_text>Is there any correlation between metrics and kernel size of convolution?<jupyter_code>plt.plot(df.loc[df['mod_name'].str.contains('conv1')]['init_acc'], label='conv1')
plt.plot(df.loc[df['mod_name'].str.contains('conv2')]['init_acc'], label='conv2')
plt.plot(df.loc[df['mod_name'].str.contains('conv1')]['val_acc'], label='conv1')
plt.plot(df.loc[df['mod_name'].str.contains('conv2')]['val_acc'], label='conv2')
plt.legend()
plt.plot(df.loc[df['mod_name'].str.contains('conv1')]['cost'], label='conv1')
plt.plot(df.loc[df['mod_name'].str.contains('conv2')]['cost'], label='conv2')
plt.xlabel('Layer ID')
plt.ylabel('Cost')
plt.legend()<jupyter_output><empty_output><jupyter_text>Now let's see how will this effect be the same on other models, or will changing a cost make an improvement?<jupyter_code>dbg_dir = '/mnt/ccnas2/bdp/rz3515/train/gumi/evaluation/iccv19/final/condensenet86/cifar100/prune/gopt_v2/debug_first_step_taylor/'
n_mods = len(os.listdir(dbg_dir)) - 2
data = []

for i in range(n_mods):
    mod_dir = os.path.join(dbg_dir, 'mod_{}'.format(i))
    meta_file = os.path.join(mod_dir, 'meta_data.json')
    with open(meta_file, 'r') as f:
        meta_data = json.load(f)
    
    data.append([meta_data['mod_name'], meta_data['cost'], meta_data['val_acc'], meta_data['init_acc']])

df = pd.DataFrame(data, columns=['mod_name', 'cost', 'val_acc', 'init_acc'])
plt.scatter(df['cost'], df['val_acc'], s=50, alpha=.5, label='val_acc')
plt.scatter(df['cost'], df['init_acc'], s=50, alpha=.5, label='init_acc')

plt.xlabel('Cost')
plt.ylabel('Accuracy')
plt.legend()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14915514592933948, 'fraction_numerical': 0.021044546850998463, 'mean_word_length': 5.427443237907206, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '154698', 'license': 'MIT', 'n_tokens_mistral': 2438, 'n_tokens_neox': 2312, 'n_words': 560, 'path': 'evaluation/iccv19/debug/CondenseNet_Goes_Wrong.ipynb', 'repo_name': 'kumasento/gconv-prune', 'size': '6510'}