<jupyter_start><jupyter_text>Labeling--- BEFORE YOU START: Rename your Jupyter Notebook so it doesn't cause merge conflictsWe will __assign numbers__ to each categoy as follows:1. `miscl.` -- Any miscelaneous emails that don't belong to one of the categories above: anything that we can't generate a generic response to should go here (eg. attendance issues, cheating, emergencies, medical stuff, ...)2. `conflicts` -- Anything related to midterm/final scheduling conflicts3. `attendance` -- Anything related to attendance of labs/disc/lec4. `assignments` -- Anything related to homework/lab assignments (e.g. submissions, excuses, regrade requests)5. `enrollment` -- Anything related to Calcentral/course enrollment issues6. `internal` -- Anything related to course logistics, hiring (interviews) or other internal adminitrative issues7. `DSP` -- Anything related to dsp---__Keep in mind:__As you read through the emails, treat all emails independently from each other, even if it's in the same thread so we don't accidently classify an email as something if it doesn't actually contain indicators that it belongs in a category!__Example:__Email 1: "okay thanks, i will email you closer to the exam date. as a heads up you will be escorted by a ta from your 294 class to the dsp exam room (it will be in the dsp exam room because of this special circumstance) i.e. you won't have time to go home/take a break if you are accepting this accommodation. please confirm  best"Email 2: "got it. i hope this reply serves as confirmation.  thanks"The first email should be classified as 7 ('DSP') because from just its content alone, that would be our best guess. If we read the earlier thread, we'd know it has to do with Midterm Conflicts, but try not to let that bias your classification to best help our model. Likewise, the second email should be 1 ('miscl') even though we know from the threat the context.<jupyter_code>import pandas as pd
emails = pd.read_csv('..\data\keiko_label.csv')
#for i, em in enumerate(content):
    #print('EMAIL!!! ', i+1, '\n','  ', em, '\n')
# For reference
# 1. `miscl.` 2. `conflicts` 3. `~deprecated~` 4. `hw`  5. `enrollment` 6. `internal` 7. `DSP` 
l = l1+l2+l3+l4+l5+l6+l7+l8+l9+l10+l11+l12+l13+l14+l15+l16+l17+l18+l19+l20+l21+l22+l23+l24+l25+l26+l27+l28+l29+l30+l31
emails['Category'] = l
emails.to_csv('../data/keiko_categorized.csv', index=False, sep = ',', encoding='utf-8')
cat = pd.read_csv('../data/keiko_categorized.csv')
list(cat[cat['Category']==1].index)
content = zip(cat[cat['Category']==1].index, cat[cat['Category']==1]['Body'])
#cat[cat['Category']==1]
#[i for i in content]
l1 = [5,5,5,5,5,3,3,3,3,1]
l2 = [3,3,1,2,2,2,7,1,2,1]
l3 = [4,3,5,3,3,5,5,5,1,1]
l4 = [1,1,1,1,1,1,1,1,7,7]
l5 = [1,1,2,2,1,7,1,1,1,7]
l6 = [2,2,2,2,1,1,2,2,5,4]
l7 = [4,1,4,1,1,1,7,5,5,1]
l8 = [1,1,1,5,5,3,1,7,7,7]
l9 = [1,5,5,5,1,4,1,4,4,4]
l10= [1,1,2,2,2,5,1,2,5,2]
l11= [1,4,4,4,4,5,1,1,5,5]
l12= [5,7,7,1,7,1,5,5,4,4]
l13= [4,1,7,7,7,7,7,4,4,4]
l14= [4,1,5,5,1,5,5,7,1,5]
l15= [5,4,4,4,4,4,4,1,7,7]
l16= [7,7,2,2,5,7,5,1,4,5]
l17= [5,5,2,2,2,2,2,1,2,2]
l18= [2,2,2,2,7,7,1,2,7,1]
l19= [1,1,5,5,3,1,3,3,3,7]
l20= [4,4,4,4,4,4,4,4,4,4]
l21= [5,1,1,5,4,4,4,4,4,4]
l22= [4,4,4,1,3,3,4,4,1,1]
l23= [1,1,5,5,5,5,5,2,5,5]
l24= [2,4,4,4,4,4,4,4,4,1]
l25= [1,4,4,1,4,4,1,1,1,7]
l26= [1,7,4,4,4,4,4,4,4,5]
l27= [5,5,4,4,4,4,4,4,4,4]
l28= [1,5,5,1,1,5,5,4,4,4]
l29= [1,4,1,1,4,4,4,3,3,1]
l30= [1,1,1,1,1,1,1,1,5,5]
l31= [1,1,1,4]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1925287356321839, 'fraction_numerical': 0.12557471264367817, 'mean_word_length': 5.733075435203094, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '317338', 'license': 'MIT', 'n_tokens_mistral': 1659, 'n_tokens_neox': 1573, 'n_words': 401, 'path': 'Data Analysis and Model/exploratory_data_analysis/Labeling_Keiko.ipynb', 'repo_name': 'rexlintc/Autoreply', 'size': '3480'}