<jupyter_start><jupyter_text>Preparing the training data The idea here is to use the [official AFD training data](https://docs.aws.amazon.com/frauddetector/latest/ug/samples/training_data.zip) and amend it in two dimensions:1. Add a new feature *registration time* that indicates how long the user took to filling and submit the registration form. The basic idea is that bots are much more likely to take much less time to register as they aim to create multiple fake accounts in a short amount of time1. Manipulate the email addresses so that fraudulent sign-ups are much more likely to use a randomised string as user name  Download of data and import of required packages<jupyter_code>!mkdir data
!curl https://docs.aws.amazon.com/frauddetector/latest/ug/samples/training_data.zip --output ./data/training_data.zip
!unzip -o ./data/training_data.zip -d ./data/
!pip install zxcvbn
import pandas as pd
import numpy as np
from scipy.stats import skewnorm
import string
import secrets
from zxcvbn import zxcvbn<jupyter_output><empty_output><jupyter_text>This function creates a random string between 6 and 12 characters long<jupyter_code>def create_random_uname():
    len = np.random.randint(6, 12)
    uname = ''.join(secrets.choice(string.ascii_lowercase) for _ in range(len))
    return uname<jupyter_output><empty_output><jupyter_text>Manipulating the training data<jupyter_code>df = pd.read_csv('data/registration_data_20K_minimum.csv')<jupyter_output><empty_output><jupyter_text>This cell will distribute new labels to the data set -- 90% of the observations will be labelled *legitimate* and 10% will be labelled *fraudulent*.<jupyter_code>df['random1'] = np.random.rand(df.shape[0])
df['random2'] = np.random.rand(df.shape[0])
df['new_label'] = df['random1'].apply(lambda x: 'legit' if x <= .9 else 'fraud')
df['registration_time'] = 0<jupyter_output><empty_output><jupyter_text>This cell will populate the new features (registration_time & user name) for the legitimate observations. The registration time will be distributed in a skew normal distribution with a long tail to account for users that take longer to complete the registration form. The user name for legitimate sign-ups will stay the same as in the original data set 95% of the time. In 5% of the cases the user name will be a random string.<jupyter_code>pd.options.mode.chained_assignment = None 
df_legit = df[df['new_label'] == 'legit']
df_legit['registration_time'] = skewnorm.rvs(10, loc=7000, scale=15000, size=df_legit.shape[0])
df_legit['new_email'] = df_legit['email_address'].str[5:]
df_legit['email_uname'] = df_legit['new_email'].apply(lambda x: x.split('@')[0])
df_legit['email_domain'] = df_legit['new_email'].apply(lambda x: x.split('@')[1])
df_legit['new_uname'] = df_legit.apply(lambda x: x['email_uname'] if x['random2'] <= .95 else create_random_uname(), axis=1)<jupyter_output><empty_output><jupyter_text>For fraudulent observations the registration time will be much shorter (skew noraml distribution with 2000 ms as center) and in 95% of the cases the user name will be a random string.<jupyter_code>df_fraud = df[df['new_label'] == 'fraud']
df_fraud['registration_time'] = skewnorm.rvs(4, loc=2000, scale=2000, size=df_fraud.shape[0])
df_fraud['new_email'] = df_fraud['email_address'].str[5:]
df_fraud['email_uname'] = df_fraud['new_email'].apply(lambda x: x.split('@')[0])
df_fraud['email_domain'] = df_fraud['new_email'].apply(lambda x: x.split('@')[1])
df_fraud['new_uname'] = df_fraud.apply(lambda x: x['email_uname'] if x['random2'] >= .95 else create_random_uname(), axis=1)
df = df_legit.append(df_fraud)
df = df.sort_index()<jupyter_output><empty_output><jupyter_text>In this demo the email domain will be the same for all observations so that the model will be able to pick up on the randomness of the user name as a useful signal. But of course it might be worthwhile to amend the email domain as well to add another dimension to the model.<jupyter_code>df['new_email'] = df['new_uname'] + '@example.com'<jupyter_output><empty_output><jupyter_text>Capping the registration time so that there will be no negative values<jupyter_code>df['registration_time'] = df['registration_time'].apply(lambda x: 100 if x < 100 else x)
df['registration_time'] = df['registration_time'].apply(int)
df['EVENT_LABEL'] = df['new_label']<jupyter_output><empty_output><jupyter_text>Calculates the randomness of the user names, which will be used later to visualise the data along this dimension<jupyter_code>df['randomness_uname'] = df['new_uname'].apply(lambda x: zxcvbn(x)['guesses_log10'] + (np.random.rand() - .5))<jupyter_output><empty_output><jupyter_text>Writing the training data into a csv file<jupyter_code>df_out = df[['ip_address', 'new_email', 'registration_time', 'EVENT_TIMESTAMP', 'new_label']]
df_out.columns = ['ip_address', 'email_address', 'registration_time', 'EVENT_TIMESTAMP', 'EVENT_LABEL']
df_out.to_csv('data/afd_training_data.csv', index=False)
df_out.head()
import matplotlib.pyplot as plt
import seaborn as sns<jupyter_output><empty_output><jupyter_text>Plotting the training data First, the data will be plotted along the registration time feature to visualise the distribution for legitimate vs fraudulent sign-ups<jupyter_code>sns.distplot(df[df['new_label'] == 'legit']['registration_time'])
sns.distplot(df[df['new_label'] == 'fraud']['registration_time'])
plt.legend(labels=['legit', 'bot'])<jupyter_output><empty_output><jupyter_text>Then, the same is done for the randomness of the user names:<jupyter_code>sns.distplot(df[df['new_label'] == 'legit']['randomness_uname'])
sns.distplot(df[df['new_label'] == 'fraud']['randomness_uname'])
plt.legend(labels=['legit', 'bot'])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12296943231441049, 'fraction_numerical': 0.013100436681222707, 'mean_word_length': 7.099009900990099, 'pattern_counts': {'":': 0, '<': 52, '<?xml version=': 0, '>': 50, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '223312', 'license': 'MIT-0', 'n_tokens_mistral': 1976, 'n_tokens_neox': 1871, 'n_words': 535, 'path': 'notebooks/1_prepare_training_data.ipynb', 'repo_name': 'patrick-239/amazon-fraud-detector-bot-detection-demo', 'size': '5725'}