<jupyter_start><jupyter_text>A Demo of Using EvalDNN to evaluate a Keras Model  Import necessary packages<jupyter_code>import keras
import numpy
keras.backend.set_learning_phase(0)<jupyter_output>Using TensorFlow backend.<jupyter_text>Load A pre-trained Model for MNIST<jupyter_code>model = keras.models.load_model('demo_lenet5.h5')
model.summary()<jupyter_output>Model: "model_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_1 (InputLayer)         (None, 28, 28, 1)         0         
_________________________________________________________________
block1_conv1 (Conv2D)        (None, 28, 28, 6)         156       
_________________________________________________________________
block1_pool1 (MaxPooling2D)  (None, 14, 14, 6)         0         
_________________________________________________________________
block2_conv1 (Conv2D)        (None, 14, 14, 16)        2416      
_________________________________________________________________
block2_pool1 (MaxPooling2D)  (None, 7, 7, 16)          0         
_________________________________________________________________
flatten (Flatten)            (None, 784)               0         
___________________________________________________________[...]<jupyter_text>Load MNIST Data<jupyter_code>def mnist_test_data(num_max=None):
    (x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()
    
    x_test = x_test.reshape(x_test.shape[0], 28, 28, 1).astype('float32') / 255
    y_test = y_test.flatten().astype('int32')
    
    return x_test, y_test

x_test, y_test= mnist_test_data()
print(x_test.shape)
print(y_test.shape)<jupyter_output>(10000, 28, 28, 1)
(10000,)<jupyter_text>Wrap Model<jupyter_code>from evaldnn.models.keras import KerasModel
measure_model = KerasModel(model)<jupyter_output><empty_output><jupyter_text>First, assume we want to evaluate its Accuracy<jupyter_code># specify metrics
from evaldnn.metrics.accuracy import Accuracy
accuracy = Accuracy()
measure_model.predict(x_test[:10], y_test[:10], [accuracy.update, accuracy.report])<jupyter_output>[Accuracy] Time: 2019-12-13 01:22:15, Num: 10, topK: 1, Accuracy: 1.000000(10/10)
[Accuracy] Time: 2019-12-13 01:22:15, Num: 10, topK: 5, Accuracy: 1.000000(10/10)<jupyter_text>Second, assume we want to evaluate its Neuron Coverage<jupyter_code>from evaldnn.metrics.neuron_coverage import NeuronCoverage
neuron_coverage = NeuronCoverage()
measure_model.intermediate_layer_outputs(x_test[:10], [neuron_coverage.update, neuron_coverage.report])<jupyter_output>[NeuronCoverage] Time:2019-12-13 01:22:21, Layers: 8, Neurons: 268
[NeuronCoverage] Time:2019-12-13 01:22:21, Num: 8, Threshold: 0.000000, Neuron Coverage: 0.947761(254/268)
[NeuronCoverage] Time:2019-12-13 01:22:21, Num: 8, Threshold: 0.100000, Neuron Coverage: 0.828358(222/268)
[NeuronCoverage] Time:2019-12-13 01:22:21, Num: 8, Threshold: 0.200000, Neuron Coverage: 0.701493(188/268)
[NeuronCoverage] Time:2019-12-13 01:22:21, Num: 8, Threshold: 0.300000, Neuron Coverage: 0.608209(163/268)
[NeuronCoverage] Time:2019-12-13 01:22:21, Num: 8, Threshold: 0.400000, Neuron Coverage: 0.541045(145/268)
[NeuronCoverage] Time:2019-12-13 01:22:21, Num: 8, Threshold: 0.500000, Neuron Coverage: 0.462687(124/268)
[NeuronCoverage] Time:2019-12-13 01:22:21, Num: 8, Threshold: 0.600000, Neuron Coverage: 0.350746(94/268)
[NeuronCoverage] Time:2019-12-13 01:22:21, Num: 8, Threshold: 0.700000, Neuron Coverage: 0.257463(69/268)
[NeuronCoverage] Time:2019-12-13 01:22:21, Num: 8, Threshold: 0.800000, Neuron [...]<jupyter_text>Last, assume we want to evaluate its Robustness<jupyter_code>from evaldnn.metrics.robustness import Robustness
import foolbox
bounds = (0, 1)
robustness = Robustness(bounds)
measure_model.adversarial_attack(x_test[0:10,...],y_test[0:10], bounds,
    [robustness.update, robustness.report], attack=foolbox.attacks.FGSM, distance=foolbox.distances.Linf)<jupyter_output>[Robustness] Time: 2019-12-13 01:22:21, Total Num: 1, Correct Num: 1, Incorrect Num: 0, Success Rate: 1.0000000000(1/1), Avg Mean Squared Distance: 0.0100030613, Avg Mean Absolute Distance: 0.0762355849, Avg Linfinity: 0.1320000291, Avg L0: 469.0000000000
[Robustness] Time: 2019-12-13 01:22:21, Total Num: 2, Correct Num: 2, Incorrect Num: 0, Success Rate: 1.0000000000(2/2), Avg Mean Squared Distance: 0.0096934912, Avg Mean Absolute Distance: 0.0752021298, Avg Linfinity: 0.1300000250, Avg L0: 480.0000000000
[Robustness] Time: 2019-12-13 01:22:21, Total Num: 3, Correct Num: 3, Incorrect Num: 0, Success Rate: 1.0000000000(3/3), Avg Mean Squared Distance: 0.0116078497, Avg Mean Absolute Distance: 0.0820079868, Avg Linfinity: 0.1406666835, Avg L0: 476.3333333333
[Robustness] Time: 2019-12-13 01:22:21, Total Num: 4, Correct Num: 4, Incorrect Num: 0, Success Rate: 1.0000000000(4/4), Avg Mean Squared Distance: 0.0117722467, Avg Mean Absolute Distance: 0.0828981064, Avg Linfinity: 0.1417500153,[...]
{'fraction_non_alphanumeric': 0.13036597428288824, 'fraction_numerical': 0.15153313550939665, 'mean_word_length': 5.661396574440053, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 23, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 7, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '535991', 'license': 'MIT', 'n_tokens_mistral': 2358, 'n_tokens_neox': 1755, 'n_words': 403, 'path': 'demo/demo.ipynb', 'repo_name': 'yqtianust/EvalDNN', 'size': '5055'}