<jupyter_start><jupyter_text>A Learning Based Strategy for Athena  Introduction Athena has various strategies for making a final prediction based on the predictions of the ensemble of weak defenses. These strategies are relatively basic, and leave a large potential for optimization using a neural network. Our strategy involves running the predictions of the ensemble through a neural network trained to make a final prediction based on the predictions of the ensemble.  Detailed Rundown of Approach 1. Generate data set    1. Need to gather test and training data        - must be uniformly distributed to eliminate bias        - data must be adversarial examples from Athena        - need the proper labels for each example        - need raw predictions from each weak defense for each adversarial example2. Create Neural Network    1. Train a CNN on training data        - Input will be nx10, where n is the number of weak defenses        - Error produced by comparison with 1x10 array in the form [0,0,0,0,x,0,0,0,0,0], where x is the prediction        - Output is the predicted label for the image    2. Implement into Athena        - NN will be a specific strategy that Athena can utilize, must implement this strategy name into main athena file3. Test Neural Network    1. Run previously generated test data through NN, comparing output with true labels    2. Determine error rate of new athena strategy**Important Note:**Our implemetation of this strategy uses 15 randomly selected weak defenses. This is to reduce data collection and training time. With superior computing power, all 72 weak defenses would be uses.  1. Generate Data Set The data set was generated using the AE's found in (a); this is all of the AE's included with Athena. A subsample was generated splitting the data from each AE found in (a) into 80% training, 20% testing, using (b). Using (c), we looped through our dataset, generating the raw predictions from each weak defense defined in "active_wds" field in (d). Finally, (e) was used to generate the labels for training in a manner which can be used to create an error function for the NN as described in 2A. To transform the test data into a useable form, (f) was used.**Total training samples:** 3600**Total test samples:** 1000 **Files:**- (a): src/configs/experiment/data-mnist.json- (b): src/learning_based_strategy/split_data.py- (c): src/learning_based_strategy/collect_raws.py- (d): src/configs/experiment/athena-mnist.json- (e): src/learning_based_strategy/get_training_labels.py- (f): src/learning_based_strategy/get_test_samples.py  Relevant Code Snippets for 1  (b)<jupyter_code>"""
Created on Thu Nov 12 17:49:52 2020

@author: miles
"""
import os
from utils.file import load_from_json
from utils.data import subsampling
import numpy as np

data_configs = load_from_json('../configs/experiment/data-mnist.json')

path = 'samples/'

#get data files, only take the AE type for the filename for matching later
data_files = [os.path.join(data_configs.get('dir'), ae_file) for ae_file in data_configs.get('ae_files')]
filenames = [ae_file.split('-')[-1].replace('.npy','') for ae_file in data_configs.get('ae_files')]
#Get respective label files
label_file = os.path.join(data_configs.get('dir'), data_configs.get('label_file'))
labels = np.load(label_file)

#Subsample from each AE file
for file, filename in zip(data_files, filenames):

    data = np.load(file)
    subsampling(data, labels, 10, 0.2, path, filename)<jupyter_output><empty_output><jupyter_text>(c) from collect_raw_prediction<jupyter_code>#get samples and respective labels
    samples = glob.glob('samples/*training_samples*.npy')
    labels = glob.glob('samples/*training_labels*.npy')
    
    #sort based on type of attack
    sorted_samples = []
    sorted_labels = []
    for sample in samples:
        pref = sample.split('_training_samples')[0].split('/')[1].replace('.npy','')
        sorted_samples.append(sample)
        for label in labels:
            pref2 = label.split('_training_labels')[0].split('/')[1].replace('.npy','')
            if pref == pref2:
                sorted_labels.append(label)
    
    #load data and labels, concatenate into single numpy array for easy looping
    samples_dat = [np.load(dat) for dat in sorted_samples]
    labels_dat = [np.load(dat) for dat in sorted_labels]
    samples_dat = np.concatenate(samples_dat,axis=0)
    labels_dat = np.concatenate(labels_dat)
    samples = []
    labels = []
    #Generate raw predictions from each WD for each AE
    for i in range(0, len(labels_dat), 100):
        raw_preds = athena.predict(x=samples_dat[i], raw=True)
        samples.append(raw_preds)
        labels.append(labels_dat[i])

    #Write out raw predictions to training_data directory
    samples = np.concatenate(samples,axis=1)
    labels = np.array(labels)
    samples_file = os.path.join('training_data/', 'training.npy')
    np.save(file=samples_file,arr=samples)
    labels_file = os.path.join('training_data/','training_labels.npy')
    np.save(file=labels_file,arr=labels)<jupyter_output><empty_output><jupyter_text>(e)<jupyter_code>"""
Created on Sun Nov 15 20:08:08 2020

@author: miles
"""
import numpy as np

file = 'training_data/training_labels.npy'
labels = np.load(file)
arr = []
for label in labels:
    temp = [0,0,0,0,0,0,0,0,0,0]
    temp[label] = 1;
    arr.append(temp)
    
new_arr = np.array(arr)
np.save(file='training_data/labels2D.npy', arr=new_arr)<jupyter_output><empty_output><jupyter_text>(f)<jupyter_code>"""
Created on Sun Nov 15 17:53:22 2020

@author: miles
"""
import os
import glob
import numpy as np

samples = glob.glob('samples/*test_samples*.npy')
labels = glob.glob('samples/*test_labels*.npy')
sorted_samples = []
sorted_labels = []
for sample in samples:
    pref = sample.split('_test_samples')[0].split('/')[1].replace('.npy','')
    sorted_samples.append(sample)
    for label in labels:
        pref2 = label.split('_test_labels')[0].split('/')[1].replace('.npy','')
        if pref == pref2:
            sorted_labels.append(label)
    
samples_dat = [np.load(dat) for dat in sorted_samples]
labels_dat = [np.load(dat) for dat in sorted_labels]
samples_dat = np.concatenate(samples_dat,axis=0)
labels_dat = np.concatenate(labels_dat)
samples = []
labels = []
for i in range(0, len(labels_dat), 90):
    samples.append(samples_dat[i])
    labels.append(labels_dat[i])
labels_file = os.path.join('testing/', 'test_labels.npy')
samples_file = os.path.join('testing/', 'test_samples.npy')
labels = np.array(labels)
samples = np.array(samples)
np.save(file=labels_file,arr=labels)
np.save(file=samples_file,arr=samples)<jupyter_output><empty_output><jupyter_text>2. Create Neural Network The neural network has an input layer of shape 15x10, (15 weak defenses were used for this model chosen randomly in order to reduce training time, 10 predictions from each weak defense), two hidden layers and an output layer of 10x1. The output still generates 10 values using softmax, but picks the highest value in a postprocessing phase within Athena. The hidden layers take the input to 150x4, then back down to 150. All activations are relu, except for the final layer which is softmax. **NOTE:** This method should be optimized by using all 72 weak defenses, so the input layer should be 72x10. The network should also be trained on many more than 3600 examples. We did not have the necessary resources to do this in a reasonable amount of time**FILES:**- create NN: src/learning_based_strategy/models/nn.py- NN stored in: src/learning_based_strategy/models/learning-strategy-nn.h5<jupyter_code>#nn.py
import numpy as np
import keras
from keras.datasets import mnist
from keras import models
from keras import layers
from keras.utils import to_categorical


training_data = np.load('../training_data/training.npy')
training_labels = np.load('../training_data/labels2D.npy')

network = models.Sequential()
network.add(layers.Dense(150, activation='relu', input_shape=(15 * 10,)))
network.add(layers.Dense(600, activation='relu', input_shape=(150 * 4,)))
network.add(layers.Dense(150, activation='relu'))
network.add(layers.Dense(10, activation='softmax'))
network.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])


training_data = np.transpose(training_data, (1,0,2))
training_data = training_data.reshape((3600,15*10))

network.fit(training_data, training_labels, epochs=10, batch_size=360)
network.save('learning-strategy-nn.h5')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10652071563088512, 'fraction_numerical': 0.021774952919020717, 'mean_word_length': 4.638354346383544, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '150319', 'license': 'MIT', 'n_tokens_mistral': 2714, 'n_tokens_neox': 2563, 'n_words': 860, 'path': 'src/.ipynb_checkpoints/report_task2-checkpoint.ipynb', 'repo_name': 'MaxCorbel/project-athena', 'size': '8496'}