<jupyter_start><jupyter_text>Load dataset<jupyter_code>path_to_data = 'data/'

training = pd.read_csv(path_to_data + 'training_set.csv', sep=',', header=0)

training_info = pd.read_csv(
    path_to_data + 'training_info.csv', sep=',', parse_dates=True, header=0)

test = pd.read_csv(path_to_data + 'test_set.csv', sep=',', header=0)

test_info = pd.read_csv(path_to_data + 'test_info.csv',
                        sep=',', parse_dates=True, header=0)
path_to_results = 'results/'
train_info, train_email_ids_per_sender, val_info, val_email_ids_per_sender = scoring.get_train_val(training, training_info, train_frac=0.95)
train_body_dict = preprocess.body_dict_from_panda(train_info)
val_body_dict = preprocess.body_dict_from_panda(val_info)<jupyter_output><jupyter_text>Retreive all the know email listThis allows us to filter the emails in the body to potentially only keep the emails that were present as recipients in the training set.<jupyter_code>all_recipients_in_train = preprocess.get_all_recipients_from_df(train_info)
all_recipients_in_train = [rec.lower() for rec in all_recipients_in_train]
print('{nb_rec} were found in the training set'.format(nb_rec=len(all_recipients_in_train)))
val_body_dict = preprocess.body_dict_from_panda(val_info)<jupyter_output><jupyter_text>In body email predictionFinds emails in the body of the mail, and makes prediction that those emails are among the recipients.<jupyter_code>val_mid_sender_dic = preprocess.get_mid_sender_dict(val_email_ids_per_sender)<jupyter_output><empty_output><jupyter_text>Filter emails present in known recipient list<jupyter_code>val_emails_in_content = manualtools.get_filtered_emails_dic(val_body_dict, val_mid_sender_dic, candidate_list=all_recipients_in_train)
val_in_body_predictions = {}
for mid, body in val_body_dict.items():
    if (mid in val_emails_in_content.keys()):
        val_in_body_predictions[mid] = val_emails_in_content[mid]
current_score = scoring.compute_prediction_mad(val_in_body_predictions, val_info)
print(current_score)<jupyter_output>0.0416666666667<jupyter_text>Keep all emails in body (no filter)<jupyter_code>val_emails_in_content = manualtools.get_filtered_emails_dic(val_body_dict, val_mid_sender_dic, candidate_list=None)

val_in_body_predictions = {}
for mid, body in val_body_dict.items():
    if (mid in val_emails_in_content.keys()):
        val_in_body_predictions[mid] = val_emails_in_content[mid]
        
current_score = scoring.compute_prediction_mad(val_in_body_predictions, val_info)
print(current_score)<jupyter_output>0.0180055401662<jupyter_text>We see that filtering out the emails not present in the candidate list significantly improves the results, but that overall the performances are pretty poor.<jupyter_code>train_mid_sender_dic = preprocess.get_mid_sender_dict(train_email_ids_per_sender)
train_emails_in_content = manualtools.get_filtered_emails_dic(train_body_dict, train_mid_sender_dic, candidate_list=None)
train_in_body_predictions = {}
for mid, body in train_body_dict.items():
    if (mid in train_emails_in_content.keys()):
        train_in_body_predictions[mid] = train_emails_in_content[mid]
        
current_score = scoring.compute_prediction_mad(train_in_body_predictions, train_info)
print(current_score)
print('proportion of emails with emails in body : {0}'.format(len(train_in_body_predictions)/len(train_mid_sender_dic)))<jupyter_output>proportion of emails with emails in body : 0.20172813284417843<jupyter_text>For the emails in the training set that contain recipients in the body, we obtain slightly better prediction results.   Keyword prediction- Extracts the string after a given keyword- Finds train emails with matching string sequence- Adds 1 to recipient score for each recipient of the train email- Predicts recipient with highest scores<jupyter_code>keywords = ['From:', 'FW:', 'Subject']
extracted_subject_length = 20

for keyword in keywords:
    keyword_predictions = manualtools.get_keyword_prediction(train_body_dict, val_body_dict,
                                                             train_info, keyword, extracted_length=20)
    current_score = scoring.compute_prediction_mad(keyword_predictions, val_info)
    print('"{key}" score : {score}'.format(key=keyword, score=current_score))<jupyter_output>"From:" score : 0.11337176196376941

"FW:" score : 0.015641293013555786

"Subject" score : 0.04579764121663563<jupyter_text>As expected, the most informative field is 'From:'.We notice that by themselves, when present on the e-mail, none of the keywords allow us (by itself) to make competitive predictions<jupyter_code>test_body_dict = preprocess.body_dict_from_panda(test_info)
training_body_dict = preprocess.body_dict_from_panda(training_info)
for keyword in ['From:']:
    keyword_predictions = manualtools.get_keyword_prediction(training_body_dict, test_body_dict,
                                                             training_info, keyword, extracted_length=30)
with open('ranks_test/frompredictions', 'wb') as outfile:
    pickle.dump(keyword_predictions, outfile)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08174494006681077, 'fraction_numerical': 0.02279426213401454, 'mean_word_length': 6.463343108504398, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 27, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '91698', 'license': 'MIT', 'n_tokens_mistral': 1743, 'n_tokens_neox': 1646, 'n_words': 344, 'path': 'mail_processing.ipynb', 'repo_name': 'GeoffNN/ALTEGRADEnron', 'size': '5089'}