<jupyter_start><jupyter_text>Remember what the doormouse said: Feed your head  Author: <NAME>, Director of AI/ML and Software Development at Fiserv   --- The purpose of this notebook:*   To Inspire*   Exposure to Python*   Explore Machine Learning (ML)--- Subjects:1.   Download Data2.   General CSV Consumption3.   DataFrame Exploration4.   Charting5.   Data Preparation for ML Training6.   ML Training7.   ML Predictions--- How to use this Notebook:  *    Execute each cell of this notebook from top to bottom one at a time. Pro-tip: Shift + Enter executed a cell and moves focus to the next cell. Notes:  This notebook is for demositration. For those of you who are famular with feature enginneering and ethical AI, this is NOT how you would build this model for production use. In the feature engineering phase, we call out and exclude key features that may introduce controvoursey. This is the demoistration.   Lets begin by bringing in some data from a public repository (Download Data)Data courtesy of Kaggle https://www.kaggle.com/kartik2112/fraud-detection?select=fraudTrain.csv and rehosted on Ryan Susman's Google Drive so you dont need to get a Kaggle account.Note: this data may not exist in my Google Drive forever... so you know...<jupyter_code>import gdown
from tqdm import tqdm

data_source = [
  {'source':'https://drive.google.com/uc?id=1Y-mvdNzKKPGhpnB-NMxpVD9iVaZFGsCU',
  'destination':'/content/sample_data/fraudTest.csv',},
  {'source':'https://drive.google.com/uc?id=1Y-mvdNzKKPGhpnB-NMxpVD9iVaZFGsCU',
  'destination':'/content/sample_data/fraudTrain.csv',},
]

print('Downloading...')
for data in tqdm(data_source):
  gdown.download(data['source'], data['destination'], quiet=True)
print('\nDownload Complete')<jupyter_output>Downloading...<jupyter_text>Let's explore the files (General CSV Consumption)<jupyter_code>import pandas as pd
# open the CSVs into memory
train_df = pd.read_csv(r'/content/sample_data/fraudTrain.csv', index_col=0)
test_df = pd.read_csv(r'/content/sample_data/fraudTest.csv', index_col=0)

# add a column ('tick') to each and fill w/ '1', 
# you will understand later when we start grouping and counting rows
train_df['tick'] = 1
test_df['tick'] = 1
# let's look at train_df, this should contain all of our training data for the 
# upcomming Deep Neural Network that we are going to train and predict from

# we can see it contains 1,296,675 records
# our 'tick' at the end to the right there as the last column
train_df<jupyter_output><empty_output><jupyter_text>Show me some things about this data (DataFrame Exploration)<jupyter_code># intristing distrubution of datatypes for each column.
train_df.info()
# .apply(lambda x: format(x, 'f'))    This just removes the scientific notation
train_df['amt'].describe().apply(lambda x: format(x, 'f'))<jupyter_output><empty_output><jupyter_text>Plot some data (Charting)<jupyter_code>import matplotlib.pyplot as plt
# for the sake of exploration, what is the state distribution? 
grouped_states = train_df.groupby(['state'])['tick'].sum().sort_values(ascending=False)
print(f'There are only {len(grouped_states)} in this dataset')
grouped_states.plot.bar(figsize=(10, 4), xlabel='States', ylabel='Count')
# what is the distribution of $$$ (in 1 Standard Deviation) of the entire train set?
train_df[train_df['amt'] < train_df['amt'].std()].boxplot(column=['amt'], return_type='axes')
# what is the distribution of $ (in 1 Standard Deviation(SD)) of the entire train 
# set for cases that are fraud?

# from a feature engineering perspective, it appears that $ is a factor if a 
# transaction is likely to be fraud or not.

figure, axis = plt.subplots(1, 2, figsize=(12, 4))

fraud_df = train_df[train_df['is_fraud'] == 1]
fraud_df[fraud_df['amt'] < fraud_df['amt'].std()].boxplot(
    column=['amt'], return_type='axes', ax=axis[0]).set_title(
        'BoxPlot at 1 SD, Number of $ Per CC Fraud')

not_fraud_df = train_df[train_df['is_fraud'] == 0]
not_fraud_df[not_fraud_df['amt'] < not_fraud_df['amt'].std()].boxplot(
    column=['amt'], return_type='axes', ax=axis[1]).set_title(
        'BoxPlot at 1 SD, Number of $ Per Not CC Fraud')

plt.show()<jupyter_output><empty_output><jupyter_text>Pre-Processing (Data Preperation for ML Training)Below we will be training a deep neural network using the Tensorflow toolkit<jupyter_code># define feature and label columns
# this is where ethics in ML comes into play. I will be excluding Age and Gender
# from the features list. Feel free to add them in if you want to see how that
# influences the model

features = ['merchant', 'category', 'amt', 'city', 'state', 'city_pop', 'job']
label = 'is_fraud'
all_columns = features.copy()
all_columns.append(label)
train_df = train_df[all_columns]
# define a helper method
def df_to_dataset(dataframe, shuffle=True, batch_size=32):
  dataframe = dataframe.copy()
  labels = dataframe.pop(label)
  ds = tf.data.Dataset.from_tensor_slices((dict(dataframe), labels))
  if shuffle:
    ds = ds.shuffle(buffer_size=len(dataframe))
  ds = ds.batch(batch_size)
  return ds
from sklearn.model_selection import train_test_split

# best practice, split the train set into a 20% validation set to measure model 
# performance while training. This way the Test set is strictly for testing

train_split_df, val_df = train_test_split(train_df, test_size=0.2)
import tensorflow as tf

# we need to perform some feature engineering, to do this we will construct
# our datasets in a small batch size for improved performance

batch_size = 5 
train_ds = df_to_dataset(train_split_df, batch_size=batch_size)
val_ds = df_to_dataset(val_df, shuffle=False, batch_size=batch_size)
test_ds = df_to_dataset(test_df, shuffle=False, batch_size=batch_size)
# Grab a sample batch
example_batch = next(iter(train_ds))[0]
from tensorflow import feature_column
from tensorflow.keras import layers

# this method will assist in validating that we have properly built
# the pipeline for the feature

def demo(feature_column):
  feature_layer = layers.DenseFeatures(feature_column)
  print(feature_layer(example_batch).numpy())
feature_columns = []
# because merchant name could be anything, going categorical
merchant = feature_column.categorical_column_with_vocabulary_list(
      'merchant', train_split_df.merchant.unique())
merchant_embedding = feature_column.embedding_column(merchant, dimension=8)
feature_columns.append(merchant_embedding)
demo(merchant_embedding)
# category seems to be categorical
category = feature_column.categorical_column_with_vocabulary_list(
      'category', train_split_df.category.unique())
category_embedding = feature_column.embedding_column(category, dimension=8)
feature_columns.append(category_embedding)
demo(category_embedding)
# amt seems to be all $$$, so we will go with numeric
amt_count = feature_column.numeric_column('amt')
feature_columns.append(amt_count)
demo(amt_count)
# city is categorical
city = feature_column.categorical_column_with_vocabulary_list(
      'city', train_split_df.city.unique())
city_embedding = feature_column.embedding_column(city, dimension=8)
feature_columns.append(city_embedding)
demo(city_embedding)
# easy
state = feature_column.categorical_column_with_vocabulary_list(
      'state', ['AL', 'AK', 'AZ', 'AR', 'CA', 'CO', 'CT', 'DC', 'DE', 'FL', 'GA', 
          'HI', 'ID', 'IL', 'IN', 'IA', 'KS', 'KY', 'LA', 'ME', 'MD', 
          'MA', 'MI', 'MN', 'MS', 'MO', 'MT', 'NE', 'NV', 'NH', 'NJ', 
          'NM', 'NY', 'NC', 'ND', 'OH', 'OK', 'OR', 'PA', 'RI', 'SC', 
          'SD', 'TN', 'TX', 'UT', 'VT', 'VA', 'WA', 'WV', 'WI', 'WY'])

state_one_hot = feature_column.indicator_column(state)
feature_columns.append(state_one_hot)
demo(state_one_hot)
# let's bucketize this one into factors of 10
city_pop = feature_column.numeric_column('city_pop')
city_pop_buckets = feature_column.bucketized_column(city_pop, boundaries=[1, 10, 100,
 1000, 10000, 100000, 1000000, 10000000])
feature_columns.append(city_pop_buckets)
demo(city_pop_buckets)
# job seems to be categorical
job = feature_column.categorical_column_with_vocabulary_list(
      'job', train_split_df.job.unique())
job_embedding = feature_column.embedding_column(job, dimension=8)
feature_columns.append(job_embedding)
demo(job_embedding)<jupyter_output>[[ 0.14925735  0.27286008 -0.31684586 -0.19268544 -0.0485801  -0.42552632
  -0.00516369  0.6947524 ]
 [ 0.16319975  0.2126578   0.3428606   0.04275614  0.0572004  -0.39000604
  -0.08253693  0.00170135]
 [-0.0782868   0.02253897  0.697826    0.05734655 -0.07906716 -0.03483366
  -0.22707267  0.5246777 ]
 [ 0.16633329  0.23916236  0.03819556  0.46795475 -0.00510161 -0.19648775
  -0.39947462  0.47056955]
 [ 0.6882713  -0.26459634  0.06250302 -0.2603796   0.5301043   0.1760384
   0.37745154 -0.27644897]]<jupyter_text>Time to train! (ML Training)Warning, this can be time consuming... 2-3 minutes to run...<jupyter_code># take all of our feature engineering and create our feature layer.
feature_layer = tf.keras.layers.DenseFeatures(feature_columns)
# as we did before, let's use a more standard batch size that won't wreck
# our notebook
batch_size = 32
train_ds = df_to_dataset(train_split_df, batch_size=batch_size)
val_ds = df_to_dataset(val_df, shuffle=False, batch_size=batch_size)
test_ds = df_to_dataset(test_df, shuffle=False, batch_size=batch_size)
# you know all those buzz words about neural networks?  
# Maybe "Deep Neural Networks", here it is!

# a deep neural network

model = tf.keras.Sequential([
  feature_layer, # our features
  layers.Dense(128, activation='relu'), # hidden layer
  layers.Dense(128, activation='relu'), # hidden layer
  layers.Dropout(.1), # throw in some stupidity of 10% to reduce overfitting
  layers.Dense(1) # spit out a single answer (fraud or not fraud)
])

# we will compile the model w/ a methodology to reducing loss, how to optimize 
# and what metrics are avaialbe when training
model.compile(optimizer='adam',
              loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
              metrics=['accuracy'])
# Start Training!  This will take 2-3 minutes.  Only going w/ one Epoch
model.fit(train_ds,
          validation_data=val_ds,
          epochs=1)
# Finally, let's test the model using data the model has never seen before.
loss, accuracy = model.evaluate(test_ds)
print("Accuracy", accuracy)<jupyter_output>WARNING:tensorflow:Layers in a Sequential model should only have a single input tensor, but we receive a <class 'dict'> input: {'trans_date_trans_time': <tf.Tensor 'ExpandDims_18:0' shape=(None, 1) dtype=string>, 'cc_num': <tf.Tensor 'ExpandDims_2:0' shape=(None, 1) dtype=int64>, 'merchant': <tf.Tensor 'ExpandDims_14:0' shape=(None, 1) dtype=string>, 'category': <tf.Tensor 'ExpandDims_1:0' shape=(None, 1) dtype=string>, 'amt': <tf.Tensor 'ExpandDims:0' shape=(None, 1) dtype=float64>, 'first': <tf.Tensor 'ExpandDims_6:0' shape=(None, 1) dtype=string>, 'last': <tf.Tensor 'ExpandDims_9:0' shape=(None, 1) dtype=string>, 'gender': <tf.Tensor 'ExpandDims_7:0' shape=(None, 1) dtype=string>, 'street': <tf.Tensor 'ExpandDims_16:0' shape=(None, 1) dtype=string>, 'city': <tf.Tensor 'ExpandDims_3:0' shape=(None, 1) dtype=string>, 'state': <tf.Tensor 'ExpandDims_15:0' shape=(None, 1) dtype=string>, 'zip': <tf.Tensor 'ExpandDims_21:0' shape=(None, 1) dtype=int64>, 'lat': <tf.Tensor 'ExpandDims_10:0'[...]
{'fraction_non_alphanumeric': 0.12022580929699216, 'fraction_numerical': 0.04463261885860457, 'mean_word_length': 5.416525183927561, 'pattern_counts': {'":': 0, '<': 40, '<?xml version=': 0, '>': 36, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '259066', 'license': 'MIT', 'n_tokens_mistral': 4162, 'n_tokens_neox': 3788, 'n_words': 1107, 'path': 'FeedYourHead.ipynb', 'repo_name': 'darkmatter2222/legendary-computing-machine', 'size': '11342'}