<jupyter_start><jupyter_text>Load data<jupyter_code>df = load_adult()<jupyter_output><empty_output><jupyter_text>Preprocessing<jupyter_code>X, y, target_names = scale_onehot(df, target='salary')

X_train, X_test, y_train, y_test = split_with_seed(X, y)<jupyter_output>/usr/local/anaconda3/envs/distillation/lib/python3.8/site-packages/category_encoders/utils.py:21: FutureWarning: is_categorical is deprecated and will be removed in a future version.  Use is_categorical_dtype instead
  elif pd.api.types.is_categorical(cols):
/usr/local/anaconda3/envs/distillation/lib/python3.8/site-packages/category_encoders/utils.py:21: FutureWarning: is_categorical is deprecated and will be removed in a future version.  Use is_categorical_dtype instead
  elif pd.api.types.is_categorical(cols):
/usr/local/anaconda3/envs/distillation/lib/python3.8/site-packages/category_encoders/utils.py:21: FutureWarning: is_categorical is deprecated and will be removed in a future version.  Use is_categorical_dtype instead
  elif pd.api.types.is_categorical(cols):
/usr/local/anaconda3/envs/distillation/lib/python3.8/site-packages/category_encoders/utils.py:21: FutureWarning: is_categorical is deprecated and will be removed in a future version.  Use is_categorical_dtype instead
  elif[...]<jupyter_text>Load models<jupyter_code>model_paths = sorted(glob.glob(f"{ASSETS_PATH / model_prefix}_*.tf"))
model_names = [p.split('/')[-1].split('.')[0] for p in model_paths]
model_names
# load models
models = [load_keras_classifier(name) for name in model_names]    
first = models[0]
assert len(first.predict(X_test.head(5))) == 5
first.predict_proba(X_test.head(5))[:,1], first.predict(X_test.head(5))<jupyter_output>/usr/local/anaconda3/envs/distillation/lib/python3.8/site-packages/tensorflow/python/keras/engine/sequential.py:425: UserWarning: `model.predict_proba()` is deprecated and will be removed after 2021-01-01. Please use `model.predict()` instead.
  warnings.warn('`model.predict_proba()` is deprecated and '
/usr/local/anaconda3/envs/distillation/lib/python3.8/site-packages/tensorflow/python/keras/engine/sequential.py:450: UserWarning: `model.predict_classes()` is deprecated and will be removed after 2021-01-01. Please use instead:* `np.argmax(model.predict(x), axis=-1)`,   if your model does multi-class classification   (e.g. if it uses a `softmax` last-layer activation).* `(model.predict(x) > 0.5).astype("int32")`,   if your model does binary classification   (e.g. if it uses a `sigmoid` last-layer activation).
  warnings.warn('`model.predict_classes()` is deprecated and '<jupyter_text>Evaluate ensemble predictions<jupyter_code>@dataclass
class TrainedKerasEnsemble():
    keras_estimators:List[KerasClassifier]
    final_estimator:BaseEstimator
    
    def predict_proba(self, X):
        individual_preds = [model.predict_proba(X) for model in self.keras_estimators]
        individual_preds = np.stack(individual_preds, axis=1)
        
        return self.final_estimator.predict_proba(individual_preds)
    
    def predict(self, X):
        return (self.predict_proba(X) > .5).astype(int)
    
    
ensemble = TrainedKerasEnsemble(models, UnbiasedAverage())
ensemble.predict_proba(X_train.head(5))
evaluate_model(X_train, X_test, y_train, y_test, ensemble, f"{model_prefix}_ensemble", save_to_disk=save_to_disk, target_names=target_names)<jupyter_output>/usr/local/anaconda3/envs/distillation/lib/python3.8/site-packages/tensorflow/python/keras/engine/sequential.py:425: UserWarning: `model.predict_proba()` is deprecated and will be removed after 2021-01-01. Please use `model.predict()` instead.
  warnings.warn('`model.predict_proba()` is deprecated and '
/usr/local/anaconda3/envs/distillation/lib/python3.8/site-packages/tensorflow/python/keras/engine/sequential.py:425: UserWarning: `model.predict_proba()` is deprecated and will be removed after 2021-01-01. Please use `model.predict()` instead.
  warnings.warn('`model.predict_proba()` is deprecated and '
/usr/local/anaconda3/envs/distillation/lib/python3.8/site-packages/tensorflow/python/keras/engine/sequential.py:425: UserWarning: `model.predict_proba()` is deprecated and will be removed after 2021-01-01. Please use `model.predict()` instead.
  warnings.warn('`model.predict_proba()` is deprecated and '
/usr/local/anaconda3/envs/distillation/lib/python3.8/site-packages/tensorflow/python/[...]<jupyter_text>Model distillation: train on ensemble output<jupyter_code># Create a single NN, identical to the ones in the ensemble
model = KerasClassifier(
    build_fn=create_nn,
    **train_params,
    verbose=1
)

y_train_pred_ensemble = ensemble.predict_proba(X_train)
y_train_pred_ensemble[:5]
# train the NN on the output of the ensemble
model.fit(X_train, y_train_pred_ensemble)<jupyter_output>Epoch 1/50
163/163 [==============================] - 1s 3ms/step - loss: 2.3592 - accuracy: 0.8805 - val_loss: 0.5102 - val_accuracy: 0.8862
Epoch 2/50
163/163 [==============================] - 0s 2ms/step - loss: 0.3885 - accuracy: 0.9428 - val_loss: 0.4433 - val_accuracy: 0.8912
Epoch 3/50
163/163 [==============================] - 0s 2ms/step - loss: 0.3809 - accuracy: 0.9462 - val_loss: 0.3854 - val_accuracy: 0.9109
Epoch 4/50
163/163 [==============================] - 0s 2ms/step - loss: 0.3707 - accuracy: 0.9532 - val_loss: 0.3764 - val_accuracy: 0.9347
Epoch 5/50
163/163 [==============================] - 0s 2ms/step - loss: 0.3645 - accuracy: 0.9527 - val_loss: 0.4108 - val_accuracy: 0.8940
Epoch 6/50
163/163 [==============================] - 0s 2ms/step - loss: 0.3651 - accuracy: 0.9550 - val_loss: 0.3905 - val_accuracy: 0.9042
Epoch 7/50
163/163 [==============================] - 0s 2ms/step - loss: 0.3603 - accuracy: 0.9556 - val_loss: 0.3794 - val_accuracy: 0.9081
Epoch [...]<jupyter_text>Evaluate distilled model<jupyter_code>evaluate_model(X_train, X_test, y_train, y_test, model, f'{model_prefix}_{random_seed}_distilled', save_to_disk=save_to_disk, target_names=target_names)<jupyter_output>91/204 [============>.................] - ETA: 0s
{'fraction_non_alphanumeric': 0.16121031746031747, 'fraction_numerical': 0.055224867724867725, 'mean_word_length': 5.360672975814932, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 23, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '309565', 'license': 'Apache-2.0', 'n_tokens_mistral': 2360, 'n_tokens_neox': 2104, 'n_words': 410, 'path': 'nbs/06_NN_ensemble.ipynb', 'repo_name': 'spiaz/knowledge-distillation', 'size': '6048'}