<jupyter_start><jupyter_text>Basic classifier that uses Distilbert model to predict if the payload is an SQL injection Distilbert is a smaller and faster version of BERT ( Bidirectional Encoder Representations from Transformers) that is 40% lighter while retaining 97% of BERT's language understanding ability. More on Distilbert at [HuggingFace](https://huggingface.co/docs/transformers/model_doc/distilbert)  To begin, let's install and import some packages<jupyter_code>!pip3 install scikit-learn>=1.0.0
!pip3 install ktrain matplotlib tensorflow numpy
import matplotlib
import os
import numpy as np
%reload_ext autoreload
%autoreload 2
%matplotlib inline
os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID";
os.environ["CUDA_VISIBLE_DEVICES"]="0";<jupyter_output>[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv[0m
[33mWARNING: You are using pip version 21.3.1; however, version 22.0.4 is available.
You should consider upgrading via the '/usr/local/bin/python3.8 -m pip install --upgrade pip' command.[0m
Requirement already satisfied: ktrain in /usr/local/lib/python3.8/site-packages (0.31.0)
Requirement already satisfied: matplotlib in /usr/local/lib/python3.8/site-packages (3.5.0)
Requirement already satisfied: tensorflow in /usr/local/lib/python3.8/site-packages (2.6.2)
Requirement already satisfied: numpy in /usr/local/lib/python3.8/site-packages (1.19.5)
Requirement already satisfied: keras-bert>=0.86.0 in /usr/local/lib/python3.8/site-packages (from ktrain) (0.89.0)
Requirement already satisfied: chardet in /usr/local/lib/python3.8/site-packages (from ktrain) (4.0.[...]<jupyter_text>Some more imports...We are using ktrain wrapper to simplify model operations and take advantage of some cool stuff like simplified data set preprocessing, learning rate finding and "autofit" that ensures the model is not overfit. More details on ktrain here: [ktrain on GitHub](https://github.com/amaiya/ktrain)<jupyter_code>import ktrain
from ktrain import text<jupyter_output><empty_output><jupyter_text>Let's print the list of available text classifiers in ktrain. There are relatively simple models like fasttext or bigru that have only 7-10 layers, as well as some more sophisticated deep models like BERT<jupyter_code>text.print_text_classifiers()<jupyter_output>fasttext: a fastText-like model [http://arxiv.org/pdf/1607.01759.pdf]
logreg: logistic regression using a trainable Embedding layer
nbsvm: NBSVM model [http://www.aclweb.org/anthology/P12-2018]
bigru: Bidirectional GRU with pretrained fasttext word vectors [https://fasttext.cc/docs/en/crawl-vectors.html]
standard_gru: simple 2-layer GRU with randomly initialized embeddings
bert: Bidirectional Encoder Representations from Transformers (BERT) from keras_bert [https://arxiv.org/abs/1810.04805]
distilbert: distilled, smaller, and faster BERT from Hugging Face transformers [https://arxiv.org/abs/1910.01108]<jupyter_text>Here, we will load our data set.  We have a CSV file ``trainlist_22k.csv`` that contains a list of HTTP paths that are labeled according to their association with cross-site scripting (xss) and sql injection (sqli).  There is also a "regular" traffic that belongs to a "benign" class. Data set load is performed using the ```texts_from_csv``` method, which assumes the label_columns are already one-hot-encoded in the spreadsheet. Since *val_filepath* is None, 10% of the data will automatically be used as a validation set. In our set we have: 1 feature (payload), 1 label (type) that contains 3 classes: - xss - sqli - benign We will be using Distilbert model so preprocessing mode is set to ``Distilbert``<jupyter_code>DATA_PATH = 'dataset_new.csv'
NUM_WORDS = 500
MAXLEN = 200
trn, val, preproc = text.texts_from_csv(DATA_PATH,
                      'payload',
                      label_columns = ["type"],
                      val_filepath=None, # if None, 10% of data will be used for validation
                      max_features=NUM_WORDS, maxlen=MAXLEN,
                      ngram_range=1,
                      preprocess_mode='standard')<jupyter_output>detected encoding: UTF-8-SIG (if wrong, set manually)
['attack', 'normal']
        attack  normal
258309     0.0     1.0
733248     0.0     1.0
228211     0.0     1.0
340476     0.0     1.0
359807     0.0     1.0
['attack', 'normal']
         attack  normal
1006686     0.0     1.0
663315      0.0     1.0
379128      0.0     1.0
882965      0.0     1.0
296854      0.0     1.0
language: en
Word Counts: 568568
Nrows: 943312
943312 train sequences
train sequence lengths:
	mean : 1
	95percentile : 5
	99percentile : 13
x_train shape: (943312,200)
y_train shape: (943312, 2)
Is Multi-Label? False
104813 test sequences
test sequence lengths:
	mean : 1
	95percentile : 5
	99percentile : 13
x_test shape: (104813,200)
y_test shape: (104813, 2)<jupyter_text>Let's load the learner instance that uses ```Distilbert``` model. We will retain the model structure unchanged<jupyter_code>from tensorflow.keras import Sequential
from tensorflow.keras.layers import Dense, GaussianDropout, Flatten, LSTM, Embedding, GRU, SimpleRNN

def get_model():
    model = text.text_classifier('fasttext', (trn), 
                             preproc=preproc)
    #model.add(Dense(3, activation='sigmoid'))
    #model.add(GaussianDropout1D(0.2))
    #model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
    return model
nbsvm_model = get_model()
new_model = Sequential()
new_model.add(nbsvm_model)
new_model.add(Embedding(input_dim=500, output_dim=64))
new_model.add(GRU(256, return_sequences=True))
new_model.add(LSTM(64))
#new_model.add(GRU(256, return_sequences=True))
#new_model.add(SimpleRNN(128))
#new_model.add(concatenate([text_features]))
new_model.add(Dense(16, activation='sigmoid'))
new_model.add(GaussianDropout(0.2))
new_model.add(Dense(2, activation='sigmoid'))
new_model.add(Flatten())
#new_model=nbsvm_model
new_model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
learner = ktrain.get_learner(new_model, train_data=(trn), val_data=(val))<jupyter_output>Is Multi-Label? False
compiling word ID features...
maxlen is 200
done.<jupyter_text>Here is what our model looks like. It has a number of layers that are pre-trained therefore allowing us to leverage transfer learning. Source code for modeling_tf_distilbert can be found at [HuggingFace Transformers](https://huggingface.co/transformers/v2.3.0/_modules/transformers/modeling_tf_distilbert.html)<jupyter_code>learner.print_layers()<jupyter_output>0 (trainable=True) : <keras.engine.sequential.Sequential object at 0x7fb63bc9d2b0>
1 (trainable=True) : <keras.layers.embeddings.Embedding object at 0x7fb67da726d0>
2 (trainable=True) : <keras.layers.recurrent_v2.GRU object at 0x7fb62880baf0>
3 (trainable=True) : <keras.layers.recurrent_v2.LSTM object at 0x7fb6226eadf0>
4 (trainable=True) : <keras.layers.core.Dense object at 0x7fb622849280>
5 (trainable=True) : <keras.layers.noise.GaussianDropout object at 0x7fb653b8c0a0>
6 (trainable=True) : <keras.layers.core.Dense object at 0x7fb622849430>
7 (trainable=True) : <keras.layers.core.Flatten object at 0x7fb66049fe50><jupyter_text>We need to ensure that majority of existing pre-trained layers are not re-trained so we are freezing those with the following command:<jupyter_code>learner.freeze(1)<jupyter_output><empty_output><jupyter_text>The optimal learning rate for this model can be found using the **lr_find** function however it will take at least **20 minutes!** on this VM that uses CPU only. ( Optimal rate was found to be 3e-5 and therefore there is no need to spend time on this now). **If you still want to proceed**, uncomment the command and run the cell below<jupyter_code>learner.lr_find(show_plot=True, max_epochs=10)<jupyter_output>simulating training for different learning rates... this may take a few moments...
Epoch 1/10
29479/29479 [==============================] - 166s 6ms/step - loss: 0.7524 - accuracy: 0.3700
Epoch 2/10
29479/29479 [==============================] - 157s 5ms/step - loss: 0.3869 - accuracy: 0.9174
Epoch 3/10
29479/29479 [==============================] - 159s 5ms/step - loss: 0.2107 - accuracy: 0.9541
Epoch 4/10
29479/29479 [==============================] - 160s 5ms/step - loss: 0.1927 - accuracy: 0.9541
Epoch 5/10
29479/29479 [==============================] - 159s 5ms/step - loss: 0.1886 - accuracy: 0.9541
Epoch 6/10
29479/29479 [==============================] - 159s 5ms/step - loss: 0.1872 - accuracy: 0.9541
Epoch 7/10
29479/29479 [==============================] - 156s 5ms/step - loss: 0.1865 - accuracy: 0.9541
Epoch 8/10
29479/29479 [==============================] - 156s 5ms/step - loss: 0.1873 - accuracy: 0.9541
Epoch 9/10
29479/29479 [==============================] - 161s 5ms/st[...]<jupyter_text>Now let's train the model using the optimal learning rate. More accuracy is achieved after 4-5 epochs however to save time, we will run the cycle using 2 epochs only. That should give us ~93% accuracy and observed loss (binary crossentropy) of ~0.09<jupyter_code>learner.fit_onecycle(10e-2, 2)<jupyter_output><empty_output><jupyter_text>Autofit function can help optimally train the model without ``overfitting`` it. **Do not run** unless you are willing to spend days (or perhaps weeks) on training<jupyter_code>learner.autofit(10e-2)<jupyter_output>early_stopping automatically enabled at patience=5
reduce_on_plateau automatically enabled at patience=2


begin training using triangular learning rate policy with max lr of 0.1...
Epoch 1/1024
29479/29479 [==============================] - 181s 6ms/step - loss: 0.1889 - accuracy: 0.9541 - val_loss: 0.1869 - val_accuracy: 0.9539
Epoch 2/1024
29479/29479 [==============================] - 175s 6ms/step - loss: 0.1882 - accuracy: 0.9541 - val_loss: 0.1868 - val_accuracy: 0.9539
Epoch 3/1024
29479/29479 [==============================] - 176s 6ms/step - loss: 0.1883 - accuracy: 0.9541 - val_loss: 0.1868 - val_accuracy: 0.9539
Epoch 4/1024
29479/29479 [==============================] - 180s 6ms/step - loss: 0.1883 - accuracy: 0.9541 - val_loss: 0.1868 - val_accuracy: 0.9539

Epoch 00004: Reducing Max LR on Plateau: new max lr will be 0.05 (if not early_stopping).
Epoch 5/1024
29479/29479 [==============================] - 181s 6ms/step - loss: 0.1874 - accuracy: 0.9541 - val_loss: 0.1868 [...]<jupyter_text>Alright, let's save our predictor so we can use it to perform inferences outside of the Jupyter notebook<jupyter_code>predictor = ktrain.get_predictor(learner.model, preproc)
predictor.save('detector_fasttext_plus')
print('MODEL SAVED')<jupyter_output>MODEL SAVED<jupyter_text>It's time for some fun! First, get a predictor instance that uses our pre-trained model<jupyter_code>predictor = ktrain.load_predictor('detector_trained_1m')
new_model = ktrain.get_predictor(predictor.model, predictor.preproc)<jupyter_output><empty_output><jupyter_text>Let's see if it can catch an XSS payload<jupyter_code>text = '<applet onkeydown="alert(1)" contenteditable>test</applet>'
result = new_model.predict(text)
print(result)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1355256160255274, 'fraction_numerical': 0.07613898244992022, 'mean_word_length': 4.913522012578617, 'pattern_counts': {'":': 0, '<': 55, '<?xml version=': 0, '>': 57, 'https://': 7, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '217841', 'license': 'Apache-2.0', 'n_tokens_mistral': 4252, 'n_tokens_neox': 3651, 'n_words': 1097, 'path': 'ml_sqli_detector-nbsvm.ipynb', 'repo_name': 'sstarzh/ml_sqli_detector', 'size': '11282'}