<jupyter_start><jupyter_text>TalkingData AdTracking Fraud Detection Challengehttps://www.kaggle.com/c/talkingdata-adtracking-fraud-detection1. This competition is a binary classification problem challenge. Kagglers are invited to classify users into "fraudlent" or "not fraudlent" as well as possible2. The evaluation metric is ROC-AUC (the area under a curve on a Receiver Operator Characteristic graph)3. There are three main datasets: train, test_supplement, test (for submission)<jupyter_code>import pandas as pd
import numpy as np

# klearn import
from klearn.eda import PlotlyPlot

train_columns = ['ip', 'app', 'device', 'os', 'channel', 'click_time', 'is_attributed']
test_columns = ['ip', 'app', 'device', 'os', 'channel', 'click_time', 'click_id']
dtype = {
    'ip': 'uint32',
    'app': 'uint16',
    'device': 'uint16',
    'os': 'uint16',
    'channel': 'uint16',
    'is_attributed': 'uint16',
    'click_id': 'uint32'
}
# read data
df_train = pd.read_csv(
    filepath_or_buffer="../data/train.csv",
    usecols=train_columns,
    dtype=dtype,
    low_memory=True,
    parse_dates=['click_time'],
    infer_datetime_format=True,
#     skiprows=range(1, 59709852),
#     nrows=50000
)
df_test = pd.read_csv(
    filepath_or_buffer="../data/test_supplement.csv",
    usecols=test_columns,
    dtype=dtype,
    low_memory=True,
    parse_dates=['click_time'],
    infer_datetime_format=True,
#     skiprows=range(1,633),
#     nrows=10000
)
df_test_submit = pd.read_csv(
    filepath_or_buffer="../data/test.csv",
    usecols=test_columns,
    dtype=dtype,
    low_memory=True,
    parse_dates=['click_time'],
    infer_datetime_format=True,
)<jupyter_output><empty_output><jupyter_text>Let's look at file size and basic info<jupyter_code>df_train.info()
df_test.info()
df_test_submit.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 18790469 entries, 0 to 18790468
Data columns (total 7 columns):
click_id      uint32
ip            uint32
app           uint16
device        uint16
os            uint16
channel       uint16
click_time    datetime64[ns]
dtypes: datetime64[ns](1), uint16(4), uint32(2)
memory usage: 430.1 MB<jupyter_text>Wow! We can see that this is some really big data. For those who don't have a decent amount of memory space in their machine, they need to use some tricks (eg. chunksize loading, hashing tricks, OGD) to get through this challenges.Here are some note about the data:1. We don't need to handle Nans2. Besides 'click_time', all other features are integer type  Looking at the columnsAccording to the data page, our data contains:- ip: ip address of click- app: app id for marketing- device: device type id of user mobile phone (e.g., iphone 6 plus, iphone 7, huawei mate 7, etc.)- os: os version id of user mobile phone- channel: channel id of mobile ad publisher- click_time: timestamp of click (UTC)- attributed_time: if user download the app for after clicking an ad, this is the time of the app download- is_attributed: the target that is to be predicted, indicating the app was downloaded<jupyter_code>df_train.head()<jupyter_output><empty_output><jupyter_text>Some note:1. 'is_attributed' is the target variable (only 1 and 0)2. Data is organized in timestamp sequence (ordered by 'click_time' ASCD)3. Besides 'click_time', all other variables are categorical.<jupyter_code># Let's do unique counts on other variables
cols = ['ip', 'app', 'device', 'os', 'channel']
uniques = [len(df_train[col].unique()) for col in cols]
log_uniques = [np.log(unique) for unique in uniques]
# plot
plotly_plot = PlotlyPlot(
    title='Number of Unique Values Per Feature', 
    yaxis={'title': 'log(unique count)'}, 
    xaxis={'title': 'features'})
plotly_plot.barplot(
    x=cols, 
    y=log_uniques, 
    text=uniques,
    marker_attribute={
        'color': 'lightblue',
        'line': {'width': 2}
    }
)<jupyter_output><empty_output><jupyter_text>If you look at the data samples above, you'll notice that all these variables are encoded - meaning we don't know what the actual value corresponds to - each value has instead been assigned an ID which we're given. This has likely been done because data such as IP addresses are sensitive, although it does unfortunately reduce the amount of feature engineering we can do on these.  Look at the target variable<jupyter_code>true_proportion = (df_train.is_attributed.values == 1).mean()
false_proportion = 1 - true_proportion
data_list = [true_proportion, false_proportion]
text_list = ["{:.2%}".format(v) for v in data_list]
# plot
plotly_plot = PlotlyPlot(
    title='Taget Variable (In Training) Distribution', 
    yaxis={'title': 'proportion'}, 
    xaxis={'title': 'Taget Value'})
plotly_plot.barplot(
    x=['fraudulant', 'not fraudulant'], 
    y=data_list,
    text=text_list,
    marker_attribute={
        'color': 'lightblue',
        'line': {'width': 2}
    },
    **{'width': [0.5, 0.5]}
)<jupyter_output><empty_output><jupyter_text>Wow, that's a really imbalanced dataset. Only 0.2% of the dataset is made up of fraudulent clicks. This means that any models we run on the data will either need to be robust against class imbalance or will require some data resampling.  Look at some time series pattern<jupyter_code># let's round timestamp to hour
df_train['click_hour'] = df_train['click_time'].dt.round('H')
num_clicks = df_train.groupby('click_hour').size()
# plot
plotly_plot = PlotlyPlot(
    title='Number of Clicks (Train) Time Series', 
    yaxis={'title': 'number of clicks'}, 
    xaxis={'title': 'time stamp'})
plotly_plot.lineplot(
    x=num_clicks.index, 
    y=num_clicks, 

)<jupyter_output><empty_output><jupyter_text>There is a definitely pattern in frequency of clicks based on time of day. And this behavior is consistent throughout the time all the way to out-of-sample data. We might want to get 'hour' as one the feature in dataset<jupyter_code># let's round timestamp to hour
df_test['click_hour'] = df_test['click_time'].dt.round('H')
num_clicks = df_test.groupby('click_hour').size()
# plot
plotly_plot = PlotlyPlot(
    title='Number of Clicks (Test) Time Series', 
    yaxis={'title': 'number of clicks'}, 
    xaxis={'title': 'time stamp'})
plotly_plot.lineplot(
    x=num_clicks.index, 
    y=num_clicks, 

)
# let's round timestamp to hour
df_test_submit['click_hour'] = df_test_submit['click_time'].dt.round('H')
num_clicks = df_test_submit.groupby('click_hour').size()
# plot
plotly_plot = PlotlyPlot(
    title='Number of Clicks (Test For Submission) Time Series', 
    yaxis={'title': 'number of clicks'}, 
    xaxis={'title': 'time stamp'})
plotly_plot.lineplot(
    x=num_clicks.index, 
    y=num_clicks, 

)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11500596658711217, 'fraction_numerical': 0.01581145584725537, 'mean_word_length': 4.615577889447236, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '319786', 'license': 'MIT', 'n_tokens_mistral': 2277, 'n_tokens_neox': 2149, 'n_words': 688, 'path': 'eda_nb/basic_EDA.ipynb', 'repo_name': 'KevinLiao159/TalkingData', 'size': '6704'}