<jupyter_start><jupyter_text>Fill the nan value in csv, filtered the concerns without label or text, then extract the text and labels  and put them into a list.<jupyter_code>filename = '~/data/patient_report/all_patient_data.csv'
df = pd.read_csv(filename)
df['Primary / secondary category'].fillna(' ',inplace=True)
df['Full action description'].fillna(' ',inplace=True)
df2 = df[df['Primary / secondary category'] != ' ']
df2 = df2[df2['Full action description'] != ' ']
print(len(df2))
subject = df2['Action subject'].fillna(' ').to_list()
full_text = df2['Full action description'].fillna(' ').to_list()
label = df2['Primary / secondary category'].fillna(' ').to_list()
solution = df2['Resolution response'].fillna(' ').to_list()<jupyter_output>74260<jupyter_text>separate primary category and secondary category, because they were in the same cell of csv<jupyter_code>from sklearn import preprocessing
le = preprocessing.LabelEncoder()
primary =[]
secondary=[]
for i in range(len(label)):
    primary.append(label[i].split('-')[0])
    secondary.append(label[i].split('-')[1])
print(len(primary))
le.fit(primary)
print(le.classes_)<jupyter_output>74260
['Access ' 'Delivery of care ' 'Environment ' 'Finance ']<jupyter_text>Romove all stop words and lematize each words<jupyter_code>from tqdm import tqdm
nlp=spacy.load('en')
lemmatized_text = []
for sent in tqdm(full_text):
    lemmatized_sent = ''
    for token in nlp(sent):
        if not token.is_stop:
            if token.lemma_ != '-PRON-':
                lemmatized_sent+=token.lemma_ + ' '
    lemmatized_text.append(lemmatized_sent)
print(len(lemmatized_text))
print(len(full_text))<jupyter_output><empty_output><jupyter_text>save & load processed data<jupyter_code>count=0
for item in lemmatized_text:
    des.writelines(item+'\n')
    count+=1
print(count)
des.close()
src_lines = open('lemmad_text').readlines()
lemmatized_text2 = []
for item in src_lines:
    lemmatized_text2.append(item)
print(len(lemmatized_text2))<jupyter_output><empty_output><jupyter_text>Use Vectorizer in sklearn to convert text to bag of words vectors, custom stop words were introduced here<jupyter_code>import tools
sw = tools.stop_word()
Count_vectorizer =CountVectorizer(ngram_range=(1,2),min_df = 5,stop_words=sw,max_df = 0.1,token_pattern=r"(?u)\b[a-z]+\b")
Count_vec = Count_vectorizer.fit_transform(lemmatized_text2)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13946711074104912, 'fraction_numerical': 0.013322231473771857, 'mean_word_length': 6.281818181818182, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 19, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '118400', 'license': 'MIT', 'n_tokens_mistral': 849, 'n_tokens_neox': 832, 'n_words': 177, 'path': 'pre-processing.ipynb', 'repo_name': 'centre-for-health-informatics/manual_label_enhanced_LDA', 'size': '2402'}