<jupyter_start><jupyter_text>Note that many instances are "Sustaining of General Authorities, Area Seventies, and General Officers"- `t.iloc[1274]['Unnamed: 0']`<jupyter_code>print(t.loc[t.Title == 'The Sustaining of Church Officers'].shape[0])

print(t.loc[t.Title == 'Sustaining of General Authorities, Area Seventies, and General Officers'].shape[0])

t.iloc[1274]['Unnamed: 0']
t[t.columns[-3:]].tail()<jupyter_output><empty_output><jupyter_text>LOOK AT SUMMARY.CSV<jupyter_code>glob.glob('../data/*.csv')
# os.listdir('../data/')
#s for summary
s = pd.read_csv('../data/summary.csv',index_col=0)
print(s.shape)
s.rename(columns={'Month':'Month_letter'},inplace=True)
print(s.columns)
s['Month'] = s['Month_letter'].map({'A':4,'O':10})
s.tail()
s.dtypes<jupyter_output><empty_output><jupyter_text>NOTE THAT THE SUMMARY ONLY GOES UP TO OCTOBER 2018- `print(s.Year.max())`<jupyter_code>temp = t.loc[(t.Year == 2018) & (t.Month == 10)]
print(temp.shape)
print('there is more because it includes sustainings')
temp[['Title','Month']]
# s.loc[(s.Year == 2018) & (s.Month_letter == 'O') & (s.Speaker == '<NAME>')]<jupyter_output><empty_output><jupyter_text>thus 7% of the talks in summary.csv have an incorrect title<jupyter_code>len(comma)
comma = np.array([',' in title for title in t['Title']])
# only 0.07
comma.astype(int).sum() / len(comma)
temp2 = s.loc[(s.Year == 2018) & (s.Month_letter == 'O')]
print(temp2.shape)
temp2[['Speaker','Title']]<jupyter_output>(33, 7)<jupyter_text>one hot encode the topics<jupyter_code>t.head()
# just to verify I'm selecting the appropriate topic column names
topic_column_names = t.iloc[:,range(1,24)].columns
topic_column_names = [int(x) for x in topic_column_names]
topic_column_names == [x for x in range(0,23)]
# just remove the NaN 
# topics_columns = t.fillna(0).iloc[:,range(1,24)].values
topics_columns = t.fillna(0).iloc[:,1:].values
topics_columns = [[x for x in row if x != 0] for row in topics_columns]
t['topic_lists'] = topics_columns
topics_columns = [[x for x in row if x != 0] for row in topics_columns]
# topics_columns<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.18082581869957284, 'fraction_numerical': 0.02800189843379212, 'mean_word_length': 6.58273381294964, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '556702', 'license': 'MIT', 'n_tokens_mistral': 850, 'n_tokens_neox': 793, 'n_words': 173, 'path': 'analysis/merge_data.ipynb', 'repo_name': 'lodeous/gencon-nlp', 'size': '2116'}