<jupyter_start><jupyter_text>B2025 - two B2025 valuesB1040 - not a repeat someone else has same DoBB2090 - two B2090 (138, 131 )B2999 - drop as b2091 is first roundB2076 - two (111, 141)B2018 two (86, 143 )1009 - drop as orginal is B1009B1031 - duplicate. Orginal one has a space after B1031.B2033 - two B2033B2043 - two B2043B2039 - two B20392030 -  B2030 originalB2083 - two B2083B2088 - two B2088B2047 - two B20472065 - drop as B2065 is original<jupyter_code>duplicates_b = df[df['7.'].duplicated()]
print(duplicates_b.index)
print(duplicates_b['7.'])
dropindex = [72, 136, 138, 139, 141, 143, 144, 152, 156, 158, 159, 160, 167, 176, 178, 181, 182]
cleaned_df = data.data('questionnaire_data.csv','t2', clean=True, drop_index=dropindex)
cleaned_df.shape
hc = cleaned_df[cleaned_df['7.'].str.contains('B1')]
aan = cleaned_df[cleaned_df['7.'].str.contains('B2')]
print('Number of AAN participants is', aan.shape[0])
print('Number of HC participants is', hc.shape[0])<jupyter_output>Number of AAN participants is 95
Number of HC participants is 39
{'fraction_non_alphanumeric': 0.12702960840496658, 'fraction_numerical': 0.1872015281757402, 'mean_word_length': 5.238095238095238, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 4, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '290411', 'license': 'MIT', 'n_tokens_mistral': 528, 'n_tokens_neox': 399, 'n_words': 115, 'path': 'BB_data/data_processing/data_exploration.ipynb', 'repo_name': 'WMDA/BB_data', 'size': '1047'}