<jupyter_start><jupyter_text>Insider Threat Data  HTTP Data<jupyter_code>def get_http_sample():
    names = ['id', 'date', 'user', 'pc', 'url']
    df = pd.read_csv('data/insider_threat/r1/http.csv', names=names)
    df['date'] = pd.to_datetime(df.date, errors='coerce')
    df.set_index('date', inplace=True)
    return df
http_df = get_http_sample()
http_df.head()
# df.set_index('date', inplace=True)
http_df.shape
http_df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 3451665 entries, 2010-01-04 07:08:47 to 2011-05-14 06:10:53
Data columns (total 4 columns):
id      object
user    object
pc      object
url     object
dtypes: object(4)
memory usage: 131.7+ MB<jupyter_text>Aggregate Activity<jupyter_code>daily = http_df.groupby(pd.Grouper(freq='D')).count()
monthly = daily.groupby(pd.Grouper(freq='M')).mean()
daily.plot(kind='line', figsize=(15,5))
monthly.plot(kind='line', figsize=(15,5))<jupyter_output><empty_output><jupyter_text>This sample is from Jan 4, 2010 - May 14, 2011  Let's pull out a unique user and plot their activity<jupyter_code>user_df = http_df[http_df['user'] == 'DTAA/AEG0905']
user_df.head()
user_df.shape
daily = user_df.groupby(pd.Grouper(freq='D')).count()
monthly = daily.groupby(pd.Grouper(freq='W')).mean()
daily.plot(kind='line', figsize=(15,5))
monthly.plot(kind='line', figsize=(15,5))<jupyter_output><empty_output><jupyter_text>Logon<jupyter_code>def get_logon_sample():
    df = pd.read_csv('data/insider_threat/r1/logon.csv')
    df['date'] = pd.to_datetime(df.date, errors='coerce')
    df.set_index('date', inplace=True)
    return df
logon_df = get_logon_sample()
logon_df.head()
logon_df.shape
logon_df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 849579 entries, 2010-01-04 00:10:37 to 2011-05-14 06:18:12
Data columns (total 4 columns):
id          849579 non-null object
user        849579 non-null object
pc          849579 non-null object
activity    849579 non-null object
dtypes: object(4)
memory usage: 32.4+ MB<jupyter_text>Logon data is from Jan 4, 2010 - May 14, 2011<jupyter_code>logon_df.describe()<jupyter_output><empty_output><jupyter_text>Aggregate Activity<jupyter_code>daily = logon_df.groupby(pd.Grouper(freq='D')).count()
monthly = daily.groupby(pd.Grouper(freq='M')).mean()
daily.plot(kind='line', figsize=(15,5))
monthly.plot(kind='line', figsize=(15,5))<jupyter_output><empty_output><jupyter_text>Pull out user w/most logs<jupyter_code>user_df = logon_df[logon_df['user'] == 'DTAA/CGM0994']
user_df.head()
user_df.shape
daily = user_df.groupby(pd.Grouper(freq='D')).count()
monthly = daily.groupby(pd.Grouper(freq='M')).mean()
daily.plot(kind='line', figsize=(15,5))
monthly.plot(kind='line', figsize=(15,5))<jupyter_output><empty_output><jupyter_text>Device<jupyter_code>def get_device_sample():
    df = pd.read_csv('data/insider_threat/r1/device.csv')
    df['date'] = pd.to_datetime(df.date, errors='coerce')
    df.set_index('date', inplace=True)
    return df
device_df = get_device_sample()
device_df.head()
device_df.shape
device_df.describe()<jupyter_output><empty_output><jupyter_text>Aggregate Activity<jupyter_code>daily = device_df.groupby(pd.Grouper(freq='D')).count()
monthly = daily.groupby(pd.Grouper(freq='M')).mean()
monthly.head()
daily.plot(kind='line', figsize=(15,5))
monthly.plot(kind='line', figsize=(15,5))<jupyter_output><empty_output><jupyter_text>Let's pull out a unique user and plot their activity<jupyter_code>user_df = device_df[device_df['user'] == 'DTAA/MQL0135']
user_df.head()
user_df.shape
daily = user_df.groupby(pd.Grouper(freq='D')).count()
monthly = daily.groupby(pd.Grouper(freq='M')).mean()
daily.plot(kind='line', figsize=(15,5))
monthly.plot(kind='line', figsize=(15,5))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.16971141117288854, 'fraction_numerical': 0.04686258935663225, 'mean_word_length': 7.021231422505308, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 41, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '316882', 'license': 'MIT', 'n_tokens_mistral': 1609, 'n_tokens_neox': 1491, 'n_words': 225, 'path': 'tutorials/Insider Threat.ipynb', 'repo_name': 'carriegardner428/cybersecurity_datascience', 'size': '3777'}