<jupyter_start><jupyter_text>NotesHow should we think about this problem? Should topics be extracted one-by-one from each post? (This actually doesn't work using BERT as it wants to find multiple topics when learning) Or, the entire subreddit? (Much more manageable using BERT, but with some caveats). Do we want to include comments in topic modeling? This is also feasible, but comments in different threads can be labeled in the same topic, which feels strange to me. In any case, what's presented here is a view of the entire subreddit.<jupyter_code>topic, probs, topic_model, freq, topic_df = train_bert(corpus_df, 'text')
topic_df
pd.set_option('display.max_rows', None)
freq
topic_nr = freq.iloc[3]["Topic"]
topic_model.get_topic(topic_nr)
topic_model.visualize_topics()
topic_df['pos_sentiment'].hist()
topic_df[topic_df['post_id'] == 'n2n0ax']
final_df = topic_df[['topic_label', 'topic_name', 'neg_sentiment', 'neu_sentiment', 'pos_sentiment', 'compound_sentiment']]
pd.set_option('display.max_rows', None)
final_df.groupby(by=['topic_label', 'topic_name']).mean().reset_index()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11101163831692032, 'fraction_numerical': 0.0026857654431512983, 'mean_word_length': 7.343283582089552, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '547507', 'license': 'MIT', 'n_tokens_mistral': 365, 'n_tokens_neox': 359, 'n_words': 114, 'path': 'notebooks/BERTopic - Andrew 2-1-2022.ipynb', 'repo_name': 'reddit-conflicting-viewpoints/Reddit', 'size': '1117'}