<jupyter_start><jupyter_text>Import Data<jupyter_code>social_anxiety_pre = pd.read_csv('data/socialanxiety_pre_features_tfidf_256.csv')
social_anxiety_post = pd.read_csv('data/socialanxiety_post_features_tfidf_256.csv')<jupyter_output><empty_output><jupyter_text>High-Level Look at Datasets  Preface These datasets contain a very large number of features. For this project we have selected _one_ feature of interest: `substance_use_total`. In order to accomplish our EDA task for Sunday 21st November, we will have to filter our dataset - and associated EDA tasks - to focus exclusively on this feature.<jupyter_code>social_anxiety_pre = social_anxiety_pre.loc[:, ['subreddit', 'author', 'date', 'post', 'substance_use_total']]
social_anxiety_post = social_anxiety_post.loc[:, ['subreddit', 'author', 'date', 'post', 'substance_use_total']]<jupyter_output><empty_output><jupyter_text>'Pre' Dataset<jupyter_code>social_anxiety_pre.head(3)
social_anxiety_pre.tail(3)
print(f'Total number of records in this dataset: {len(social_anxiety_pre)}')
social_anxiety_pre.describe()<jupyter_output>Total number of records in this dataset: 12277<jupyter_text>So it doesn't look like we're missing any values in the `socialanxiety_pre_features_tfidf_256.csv` dataset.Let's take a quick look at the distribution of number of mentions of substance use per post.<jupyter_code>social_anxiety_pre.plot.hist(bins=25)<jupyter_output><empty_output><jupyter_text>'Post' Dataset<jupyter_code>social_anxiety_post.head(3)
social_anxiety_post.tail(3)
print(f'Total number of records in this dataset: {len(social_anxiety_post)}')
social_anxiety_post.describe()<jupyter_output>Total number of records in this dataset: 3738<jupyter_text>So it doesn't look like we're missing any values in the `socialanxiety_post_features_tfidf_256.csv` dataset.Let's take a quick look at the distribution of number of mentions of substance use per post.<jupyter_code>social_anxiety_post.plot.hist(bins=25)<jupyter_output><empty_output><jupyter_text>High-Level Comparison of Pre-Post Datasets<jupyter_code>pd.concat([social_anxiety_pre.describe(), social_anxiety_post.describe()], axis=1)<jupyter_output><empty_output><jupyter_text>As you can see from the table above, there are far fewer records in the 'post' dataset. This is likely due to the fact that the 'post' dataset covers a far shorter period of time. As such, a simple comparison of the means of these two datasets should be taken with a grain of salt. A Z-test of comparison of means may be a good place to start with this dataset to understand if there is any significant difference between the means of these datasets<jupyter_code>print(f"Number of unique authors (posters) in 'pre' dataset: {len(social_anxiety_pre.author.unique())}")
print(f"Number of unique authors (posters) in 'post' dataset: {len(social_anxiety_post.author.unique())}")<jupyter_output>Number of unique authors (posters) in 'pre' dataset: 12277
Number of unique authors (posters) in 'post' dataset: 3738
{'fraction_non_alphanumeric': 0.08954226528566656, 'fraction_numerical': 0.0136986301369863, 'mean_word_length': 7.910714285714286, 'pattern_counts': {'":': 0, '<': 30, '<?xml version=': 0, '>': 30, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '520415', 'license': 'MIT', 'n_tokens_mistral': 1002, 'n_tokens_neox': 940, 'n_words': 286, 'path': 'eda/subreddit/eda-socialanxiety.ipynb', 'repo_name': 'MaeveShi/covid_reddit_behaviour', 'size': '2993'}