<jupyter_start><jupyter_text>NoteSelect "Restart & Clear Output" from the Kernel drop down above, before commiting changes to git so that diffs are interpretable. Otherwise spurious changes like the vector graphics and cell output will be added.<jupyter_code>import os
import numpy as np
from ft_plot import fault_tolerance_unique_obj, fault_tolerance_unique_obj_ADef
from ft_plot import fault_tolerance_plot_rot_from_list_30, fault_tolerance_plot_rot_from_list<jupyter_output><empty_output><jupyter_text>Figure 1Motivating input-fault tolerance in terms of information conveyed aboutthe original label when a model is subject to attacks with different objectives.We report $I(T; Y)$ and accuracy (broken line) for the SVHN testset, as a function of the SNR in dB for a basic iterative method (BIM) with threedifferent objectives: `mis-cls.`---misclassification; `one tgt.`---atargeted attack with a specific label in place of the original,e.g., mapping all $0\rightarrow 1$, $1 \rightarrow 2$, $\dots$, and $9 \rightarrow0$; `all tgt.`---draw samples from each class uniformly and map to eachincorrect label; `noise`---additive white Gaussian noise (AWGN).Attacks a) and b) are $L_\infty$- and $L_2$-norm versions of BIM respectively.<jupyter_code>SAVE = False
PREFIX = 'npdata/pytorch/ami/'
CKPT = 'final_model_eps_50_seed_1_wd_0e+00'
#CKPT = 'final_model_eps_50_seed_1_wd_1e-02'
MODEL = PREFIX + CKPT
basefilename = MODEL.split('/')[-1].split('.')[0] # + '_' + MODEL.split('/')[1]
print(basefilename)<jupyter_output><empty_output><jupyter_text>Figure 1 a) - $L_2$ BIM<jupyter_code>ft_noise = np.load(MODEL + '_ft_noise.npy')
ft_l2_m = np.load(MODEL + '_ft_bim_l2_mcls_100k.npy') # _100k
ft_l2_t = np.load(MODEL + '_ft_bim_l2_one_tgt.npy')
ft_l2_t_all = np.load(MODEL + '_ft_bim_l2_all_tgt.npy')              
# introduced a new option 'axis_labels', choices are 'mi', 'acc', or 'both'
fault_tolerance_unique_obj([ft_noise, ft_l2_m, ft_l2_t, ft_l2_t_all], 
                           legend=False, axis_labels='both', labels=['awgn', 'mis-cls.', 'one tgt.', 'all tgt.'], 
                           save=False, min_snr=5, max_snr=65, modelname=basefilename + '_bim_l2_')<jupyter_output><empty_output><jupyter_text>$L_\infty$ BIMNo longer Figure 1 b), see Note below<jupyter_code>ft_noise = np.load(MODEL + '_ft_noise.npy')
ft_inf = np.load(MODEL + '_ft_bim_inf_mcls.npy')
ft_inf_t = np.load(MODEL + '_ft_bim_inf_one_tgt.npy')
ft_inf_t_all = np.load(MODEL + '_ft_bim_inf_all_tgt.npy')              
fault_tolerance_unique_obj([ft_noise, ft_inf, ft_inf_t, ft_inf_t_all], 
                           legend=True, axis_labels='both', labels=['awgn', 'mis-cls.', 'one tgt.', 'all tgt.'], 
                           save=False, max_snr=65, modelname=basefilename + '_bim_inf_')<jupyter_output><empty_output><jupyter_text>NoteThe above figure doesn't convey a lot of additional information once you see the L2 case. Since the story with misclassification vs targeted attacks is basically the same between L2 and Linf, suggest showing the figure below instead: A comparison between L2 and Linf (misclassification only) as per the ICLR paper. This would be in place of the existing Linf curves in the appendix. Currently, the L2 curves are out of context in the Appendix, so this clearly shows how Linf is "easier" than L2 by overlaying them on the same plot.<jupyter_code>#from ft_plot import fault_tolerance_plot_from_list
fault_tolerance_unique_obj([ft_noise, ft_l2_m, ft_inf], 
                            legend=True, save=True, 
                            labels=['awgn', r'$L_2$', r'$L_\infty$'], 
                            max_snr=65,
                            modelname=basefilename)<jupyter_output><empty_output><jupyter_text>Figure 2  Figure 2 a) - ADefThis is different (and much better) than what we included in the nips submission for review. 1. fixed bug with evaluation code that resulted in non-smooth curve2. focus on effect of weight decay, not epochs, since the former is more interesting3. show the effect of changing the attack objective like we do in Fig. 1. The targeted case here selects the least likely class for each input (the most powerful targeted attack from the perspective of an adversary), while misclassification means the DeepFool candidates (Algo. used by ADef) are the full range of classes.<jupyter_code>from ft_plot import fault_tolerance_unique_obj_ADef
path_to_data = 'npdata/pytorch/ami/final_model_eps_50_seed_1_'
ft_adef_wd0_mcls = np.load(path_to_data + 'wd_0e+00_adef_data_to_plot_fullbatch_mcls.npy')
ft_adef_wd1_mcls = np.load(path_to_data + 'wd_1e-02_adef_data_to_plot_fullbatch_mcls.npy')
ft_adef_wd0_tgt = np.load(path_to_data + 'wd_0e+00_adef_data_to_plot_fullbatch_tgt.npy')
ft_adef_wd1_tgt = np.load(path_to_data + 'wd_1e-02_adef_data_to_plot_fullbatch_tgt.npy')
fault_tolerance_unique_obj_ADef([ft_adef_wd0_mcls[:, :3], ft_adef_wd1_mcls[:, :3], ft_adef_wd0_tgt[:, :3], ft_adef_wd1_tgt[:,:3]], 
                                 legend=True, labels=[r'mis-cls.', r'mis-cls.*', r'tgt. l.l.', r'tgt. l.l.*'], save=True, 
                                 plot_name=path_to_data.split('/')[-1] + 'adef_ep50_n10k_no_acc.eps')<jupyter_output><empty_output><jupyter_text>Figure 2 b) - rotations 30 degrees<jupyter_code>rot = np.linspace(0, np.pi / 6)
PATH = "npdata/pytorch/ami/"
MODEL = "final_model_eps_50_seed_1_wd_1e-02"
ft_rot_wd = np.load(os.path.join(PATH, MODEL + "_ft_rot30_test.npy"))
MODEL = "final_model_eps_50_seed_1_wd_0e+00"
ft_rot_no_wd = np.load(os.path.join(PATH, MODEL + "_ft_rot30_test.npy"))

fault_tolerance_plot_rot_from_list_30(rot, [ft_rot_no_wd, ft_rot_wd], 
                                   legend=True, save=True, 
                                   labels=[r'$\lambda=0$', r'$\lambda=0.01$'], #r'$\lambda$ 0e-0', r'$\lambda$ 1e-2'
                                   modelname='final_model_eps_50_seed_1')
rot = np.linspace(0, np.pi)
PATH = "npdata/pytorch/rot_n2e+03/"
MODEL = "final_model_eps_250_seed_1_wd_1e-02"
ft_rot_wd = np.load(os.path.join(PATH, MODEL + "_ft_rot180_test.npy"))
MODEL = "final_model_eps_250_seed_1_wd_0e+00"
ft_rot_no_wd = np.load(os.path.join(PATH, MODEL + "_ft_rot180_test.npy"))

fault_tolerance_plot_rot_from_list(rot, [ft_rot_wd, ft_rot_no_wd], 
                                   legend=False, save=False, 
                                   labels=['wd', 'no wd'], 
                                   modelname='final_model_eps_250_seed_1')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10865874363327674, 'fraction_numerical': 0.02438647939496836, 'mean_word_length': 4.031055900621118, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '637323', 'license': 'MIT', 'n_tokens_mistral': 2448, 'n_tokens_neox': 2316, 'n_words': 538, 'path': 'svhn_generate_plots_for_paper.ipynb', 'repo_name': 'uoguelph-mlrg/nips18-secml-advex-input-fault', 'size': '6479'}