<jupyter_start><jupyter_text>Exploratory Analysis of a small version of the data set<jupyter_code># import useful packages

import pandas as pd
import numpy as np

import matplotlib.pyplot as plt
import matplotlib as mpl
%matplotlib inline

import seaborn as sns
sns.set()

mpl.rcParams['lines.linewidth']=1.5
mpl.rcParams['xtick.labelsize']=13
mpl.rcParams['ytick.labelsize']=13
mpl.rcParams['axes.labelsize']=15
mpl.rcParams['axes.labelweight']='heavy'



import os
import sys<jupyter_output><empty_output><jupyter_text>Add our own packages to the system path, and import packages<jupyter_code>src_dir = os.path.join(os.path.dirname(os.getcwd()),'adfraud')
sys.path.append(src_dir)<jupyter_output><empty_output><jupyter_text>Define where the data is stored, and the data file to load, and read data into a pandas DataFrame<jupyter_code>data_dir = os.path.join(os.path.dirname(os.getcwd()),'data')
data_filename = 'train_sample.csv'
data_location = os.path.join(data_dir,data_filename)
ad_data = pd.read_csv(data_location)<jupyter_output><empty_output><jupyter_text>Let's start getting a very rough overview of the dataFirst looking at the cloumn headings, and then the forst few entries of data to get a feel for the data and data_types in the columns<jupyter_code># get a list of the cloumn names
list(ad_data)
# the first 4 rows of data
ad_data.head(4)
ad_data.shape<jupyter_output><empty_output><jupyter_text>Some initial thoughts: * The ip adresses don't look like normal ip addresses - we probably cannot geotag them (which might have been useful)* attributed_time only exists if there is attribution  - it is therfore also like the label is_attributed - and so would be unfoar to use for training* click_time can probably be converted into 1hour or half-hour periods. i.e replace all events with time between 08:00 and 09:00 with 8:00, regardless of the sepecific day.* All fields except time seem to be categorical. Follwing the above suggestion with the time will lead to the click_time being best treated as nominal, although given a later or earlier time is not neccessarily better/worse and that time occurs on a circle, not linearly, it may be best to treat the field as categorical data.   - treating all the data as categorical could lead to large dimensionality, particularly if use one-hot encoding, it may be that using a hash encoding is better.    - for categorical data there is also the worry that future out-of-sample data won't be predicted well by whatever ML model we build.  check the balanced/imbalanced nature of this dataset<jupyter_code>ad_data['is_attributed'].value_counts()<jupyter_output><empty_output><jupyter_text>So, the data set is clearly imbalanced, there are for more 0 labelled than 1 labelled data.This is already surprising to me - that means most traffic was either fraudulent or potentially so (no downlaod follwing a click)Imprtantly for analysis - this means we have to be careful about how we select data for training from the data setwe may also want to consider whether we care more about sensitivity or specicifity from our model.   Let's start to look into the data itself for some beginning clues<jupyter_code>no_uniq_ips = ad_data['ip'].nunique()
print("number of unique ips = ",no_uniq_ips)
print("fraction of unique ips vs all all ips = ",no_uniq_ips/len(ad_data.index))
print(ad_data['ip'].value_counts().head())
print(ad_data['ip'].value_counts().tail())
ip_counts = ad_data['ip'].value_counts().to_numpy()
print(ip_counts)
plt.semilogy((ip_counts), '*')
plt.xlim([-1000,len(ip_counts)])

plt.xlabel('ip_index')
plt.ylabel('number of visits')

plt.show()
plt.hist(np.log10(ip_counts),bins=15)
plt.xlim([0.0,np.amax(np.log10(ip_counts))])
plt.ylim([-0.0,20000])

plt.xlabel('log10(number of visits by one ip)')
plt.ylabel('counts')

plt.show()<jupyter_output><empty_output><jupyter_text>We can clearly see that most ip addresses make only a handful of clicks, but there are a small number of ips who make a few hundred clicks.factors that may affect this are how frequent that clicks occur. If we were predicting in real time and a new user shows up making many clicks, we may want to keep track of those clicks - it may be that different temporal profiles for clicking patterns can signify true vs false actors.<jupyter_code>ad_data_attr=ad_data[ad_data['is_attributed']==1.0]
ad_data_attr.head()
ad_data_attr['ip'].value_counts().head()
ip_counts = ad_data_attr['ip'].value_counts().to_numpy()
plt.semilogy(ip_counts, '*')
plt.xlabel('ip_index')
plt.ylabel('number of visits')

plt.show()
ad_data_notattr=ad_data[ad_data['is_attributed']==0.0]
ad_data_notattr.head()
ad_data_notattr['ip'].value_counts().head()
ip_counts = ad_data_notattr['ip'].value_counts().to_numpy()
plt.semilogy(ip_counts, '*')
plt.xlabel('ip_index')
plt.ylabel('number of visits')

plt.show()<jupyter_output><empty_output><jupyter_text>So, while we can proabably treat any ip we've seen many times before as fraudulent, the challenge is then, how do we seperate out the fraudulent cases where the ip has only been seen a handful of times. (I would guess this is is bots using vpn/proxy etc to a network of ips obtained nefariously).Since there are many more bad than good actors, cutting out the high usage ips will remove a lot of fraudulent users, but there will still be many more fraudulent than real users.To get a better predictor, we're clearly going to need to use the rest of the data at our disposal.I want to look at how many categories exist in the other data fields, to get a sense of the dimensionality of the problem at hand.<jupyter_code>categoricals=['app','device','os','channel']
for c in categoricals:
    no_uniq = ad_data[c].nunique()
    print("number of unique " + c + " = ",no_uniq)<jupyter_output>number of unique app =  161
number of unique device =  100
number of unique os =  130
number of unique channel =  161<jupyter_text>Number of unique "app" and unique "channel" are the same - some info overlap?I would guess that device and os contain a lot of information overlap. i.e all iphone models running some ios version, samsung running android etc. but then, maybe shouldn't discount this in case say some brands of phone running a certain os is unusual and is actually a sign for bots or something.  DEVICE and OS fields<jupyter_code>dev_os = ad_data.groupby(['device','os']).size().reset_index(name='counts')
ad_data['os'].value_counts().head(10)
ad_data['device'].value_counts().head(10)<jupyter_output><empty_output><jupyter_text>some devices, and some os's, get used a lot more than others. Which makes sense. Are the os's used by the most used devices different from those used by other devices?<jupyter_code>most_used_dev = ad_data[ad_data['device'].isin([0,1,2])][['device','os']]
print("no unique os for most used devices = ", most_used_dev['os'].nunique())
print(most_used_dev['os'].unique())
most_used_dev['os'].value_counts().head(10)
least_used_dev = ad_data[~ad_data['device'].isin([0,1,2])][['device','os']]
print("no unique os for least used devices = ", least_used_dev['os'].nunique())
print(least_used_dev['os'].unique())
least_used_dev['os'].value_counts().head(10)<jupyter_output>no unique os for least used devices =  26
[866 607 748  21  24   0  38  23  29  45  84  13  50  59 113  76  20  12
   5  22  19 199   9 116  67  52]<jupyter_text>So, the most used (top 3) devices use quite a different set of os's to the rest of the devices. All but one row in the table of os 19 were for the top 3 devices.  This could just be due to prevalence of certin brands and associated id's rather than a suggestion that there is a strong predictor for fraud associated with this info. But it could give the option for reducing dimension due to linked info.<jupyter_code>os19 = ad_data[ad_data.os==19]
os19['is_attributed'].value_counts()<jupyter_output><empty_output><jupyter_text>Use of os '19' doesn't seem to be in any way a predictor of ad fraud - rembering that the data set is heavily skewed towards unattributed clicks.Since we are doing an initial quick scan through the data, I'll leave this here and keep it in mind as an area for development during model building. Now, we go on to look at other features  APP and CHANNEL fields let's look to see how the app and channel field are related.<jupyter_code>ad_data['channel'].value_counts().head(10)
ad_data['app'].value_counts().head(10)
modal_channel = ad_data['channel'].mode().iloc[0]

print(ad_data[ad_data['channel']==modal_channel]['app'].value_counts())

print(ad_data[ad_data['channel']==modal_channel]['device'].value_counts())

print(ad_data[ad_data['channel']==modal_channel]['os'].value_counts().head(10))

print(ad_data[ad_data['channel']==modal_channel]['ip'].value_counts().head(10))

print(ad_data[ad_data['channel']==modal_channel]['is_attributed'].value_counts())


modal_app = ad_data['app'].mode().iloc[0]
print(ad_data[ad_data['app']==modal_app]['channel'].value_counts())
print(ad_data[ad_data['app']==modal_app]['device'].value_counts())<jupyter_output>280    7833
137     957
424     722
489     703
480     684
409     681
442     666
115     633
205     613
211     579
173     565
379     557
130     435
135     426
19      422
452     382
371     242
153     235
466     231
417     198
402     149
317     120
30       70
182      67
421      46
113      38
110       9
404       7
456       4
353       3
114       1
22        1
Name: channel, dtype: int64
1     17656
2       613
5         7
59        3
Name: device, dtype: int64<jupyter_text>The most used app uses many channels (although mostly the overall modal channel). The modal channel uses only two apps.<jupyter_code>print(ad_data_attr['channel'].value_counts().head(10))
print(ad_data_notattr['channel'].value_counts().head(10))
print(ad_data_attr['app'].value_counts().head(10))
print(ad_data_notattr['app'].value_counts().head(10))
print(ad_data_attr['device'].value_counts().head(10))
print(ad_data_notattr['device'].value_counts().head(10))
print(ad_data_attr['os'].value_counts().head(10))
print(ad_data_notattr['os'].value_counts().head(10))<jupyter_output>19    41
0     26
24    26
13    25
29    11
21     8
17     6
27     6
22     6
7      5
Name: os, dtype: int64
19    23829
13    21198
17     5226
18     4826
22     4033
10     2813
8      2775
6      2518
15     2392
20     2344
Name: os, dtype: int64<jupyter_text>It looks like the data will be easier to seperate into attributed/not attributed on the basis of app and channel as oppposed to device and os - since there seems to be a lot of overlap of users using the same device/os for not/is attributed cases.  timing of use I'm going to put time into a condensed hourly format (ie all time 08->09 at 08:00, ignoring the exact day, as well as mins and secs).Although, side note:For a real-time analysis system, I would I imagine it would be beneficial to include recent dynamics into the model. e.g. how has a particular ip acted in the last week - could have clicked on ads and not download becuase unsure, but eventually will download or something like that - there may be different patterns for those users compared to fraudulent cases.<jupyter_code>ad_data['click_time'].head(10)
pd.to_datetime(ad_data['click_time'],format='%Y-%m-%d %H:%M:%S').head(10)
# adjust the all rows in click_time into datetime format
ad_data['click_time'] = pd.to_datetime(ad_data['click_time'],format='%Y-%m-%d %H:%M:%S')
# adjust all rows so that click time is just the hour in which click happened
ad_data['click_time'] = ad_data['click_time'].dt.hour<jupyter_output><empty_output><jupyter_text>Now, make two tables, for attributed and unattributed clicks<jupyter_code>ad_data_attr = ad_data[ad_data['is_attributed']==1]
ad_data_notattr = ad_data[ad_data['is_attributed']==0]
attr_click_cts = ad_data_attr['click_time'].value_counts().sort_index()
notattr_click_cts = ad_data_notattr['click_time'].value_counts().sort_index()
attr_click_cts.index
attr_click_cts
def max_norm(a):
    return a/np.amax(a)

plt.fill_between(attr_click_cts.index,0.0,max_norm(attr_click_cts),alpha=0.6,color='g',label='attributed')
plt.fill_between(notattr_click_cts.index,0.0,max_norm(notattr_click_cts),alpha=0.2,color='r',label='not_attributed')
plt.legend(frameon=False)
plt.xlabel('hour')
plt.ylabel('occurences (norm.)')

plt.xlim([0.0,23])
plt.ylim([0.0,1.05])

plt.show()<jupyter_output><empty_output><jupyter_text>The temporal dependence of clicks doesn't look like a great independent predictor for fraudluent clicks.It could be that it performs well when paired with the other features though.It seems a little confusing that all users seems to click throughout the night - but this could be due to the timezone being used not being the one used in the country under investigation. The spiky nature of the attributed data compared with the not attributed data is simply due to fewer data points. Our model:do we actually want to look through the rows a few at a time - to look at temporal profile of users or perhaps apps/channels - and ban particular ip/app/channels as their stats look weird - and ben them for either ever or some period of time?<jupyter_code>modal_ip =  ad_data['ip'].mode().iloc[0]
ad_data_notattr_maxip = ad_data[(ad_data['is_attributed']==0) & (ad_data['ip']==modal_ip)]
notattr_maxip_click_cts = ad_data_notattr_maxip['click_time'].value_counts().sort_index()
plt.fill_between(attr_click_cts.index,0.0,max_norm(attr_click_cts),alpha=0.6,color='g',label='attributed')
plt.fill_between(notattr_click_cts.index,0.0,max_norm(notattr_click_cts),alpha=0.2,color='r',label='not_attributed')
plt.fill_between(notattr_maxip_click_cts.index,0.0,max_norm(notattr_maxip_click_cts),alpha=0.2,color='b',label='not_attributed_maxip')
plt.legend(frameon=False)
plt.xlabel('hour')
plt.ylabel('occurences (norm.)')

plt.xlim([0.0,23])
plt.ylim([0.0,1.05])

plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10366205305651673, 'fraction_numerical': 0.03705305651672434, 'mean_word_length': 5.387200736648251, 'pattern_counts': {'":': 0, '<': 61, '<?xml version=': 0, '>': 62, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '596651', 'license': 'Apache-2.0', 'n_tokens_mistral': 4858, 'n_tokens_neox': 4438, 'n_words': 1512, 'path': 'notebooks/exploratory.ipynb', 'repo_name': 'pdghawk/adfraud', 'size': '13872'}