<jupyter_start><jupyter_text>Analysis of the Berka datasethttp://lisp.vse.cz/pkdd99/berka.htm<jupyter_code>%pylab inline
import pandas as pd
import pickle
import os
from datetime import date
from scipy import sparse, io
def parse_dataset():
    for filename in os.listdir('original'):
        with open('original/' + filename, 'r') as input_file, open('parsed/' + filename.split('.')[0] + '.csv', 'w') as output_file:
            for line in input_file.readlines():
                print(line.replace('"', '').replace('\n', '').replace(';', ', ').replace(' ', ''), file=output_file)
if not os.path.isdir("parsed"):
    os.mkdir("parsed")
    parse_dataset()<jupyter_output><empty_output><jupyter_text>TransactionsWe are interested in the actual transactions, so we'll analyze the trans.csv file<jupyter_code>df = pd.read_csv('parsed/trans.csv')
print(df.head())
print(df.shape)
print(df.info())<jupyter_output>/usr/local/lib/python3.5/dist-packages/IPython/core/interactiveshell.py:2728: DtypeWarning: Columns (8) have mixed types. Specify dtype option on import or set low_memory=False.
  interactivity=interactivity, compiler=compiler, result=result)<jupyter_text>Number of transactions per accountWe group the accounts and check the number of transactions for each account<jupyter_code>df_tc = df.groupby('account_id', as_index=False).size().rename('transaction_count').reset_index()
df_tc['transaction_count'].describe()<jupyter_output><empty_output><jupyter_text>We have 4500 accounts, with the 25th percentile being at 133 transactions per account. We take only the accounts from the 25th percentile to the 100th<jupyter_code>account_id_list = df_tc[df_tc['transaction_count'] >= 133]['account_id'].tolist()
print('75% of transactions are in', len(account_id_list), 'accounts')<jupyter_output>75% of transactions are in 3381 accounts<jupyter_text>Dates usedWe now check when the starting and ending date of the transactions is<jupyter_code>print('Total days with transactions:', len(df[df['account_id'].isin(account_id_list)]['date'].unique()))
min_date = df['date'].min()
max_date = df['date'].max()
print('Start:', min_date)
print('End:', max_date)<jupyter_output>Total days with transactions: 2191
Start: 930101
End: 981231<jupyter_text>This means that the average 234 transactions are over 2200 days, only the 11% is covered  Analysis of the transactions<jupyter_code>transactions = np.load("usable/transactions.npy")
transactions.shape
print('mean', np.mean(transactions))
print('min', np.min(transactions))
print('max', np.max(transactions))
unique_items, counts = np.unique(np.array(transactions), return_counts=True)
plt.plot(unique_items, counts)<jupyter_output><empty_output><jupyter_text>0 is the value that occurs almost everytime<jupyter_code>sorted_indexes = np.argsort(-counts)
counts = counts[sorted_indexes]
unique_items = unique_items[sorted_indexes]

print('values with highest count:', unique_items[:5], counts[:5])
plt.plot(counts[5:])<jupyter_output>values with highest count: [    0.    -14.6   -30.  -2400.  -1800. ] [9044551   73907    4230    3631    3399]<jupyter_text>The values still follow a standard Zipf's distribution, but the bad thing is that is extremely heavy tailed<jupyter_code>plt.plot(counts[-24000:])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13087248322147652, 'fraction_numerical': 0.0326418547895058, 'mean_word_length': 6.607888631090487, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 30, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '109161', 'license': 'MIT', 'n_tokens_mistral': 1144, 'n_tokens_neox': 1068, 'n_words': 262, 'path': 'datasets/berka_dataset/dataset_analysis.ipynb', 'repo_name': 'HitLuca/Master-thesis', 'size': '3278'}