<jupyter_start><jupyter_text>free-marketing-watchSearch social media for mentions of brands and collect the comments/tweets/etc.Count mentions of each and perform sentiment analysis on the strings.<jupyter_code>import praw
import pandas as pd
import matplotlib.pyplot as plt
import matplotlib
from secrets import *
from pathlib import Path
from brands import fashion
import re
reddit = praw.Reddit(client_id=client_id,
               client_secret=client_secret,
               user_agent=user_agent)<jupyter_output><empty_output><jupyter_text>Now to get the comments data, put it in a dataframe, and clean the data to get what we want.<jupyter_code>def create_comments_df(subreddit_):
    """Returns a pandas df with the information about comments from this year.

    Inputs
    -----
    str: subreddit to be searched.
    Return
    ------
    Pandas dataframe with all the data from the praw object.m
    """
    subreddit = reddit.subreddit(subreddit_)
    submission_list = subreddit.top(
        time_filter="year", limit=1000
    )  # generator of submissions in the subreddit
    comment_list = []
    for submission in submission_list:
        submission.comments.replace_more(limit=0)
        for comment in submission.comments.list():
            comment_list.append(comment)

    df = pd.DataFrame([vars(comment) for comment in comment_list])
    df2 = df.loc[:,['link_id','id','score','body']]
    df2['Subreddit'] = subreddit_
    return df2

def brand_check(df,brandlist):
    """Checks comment body against a list of brands to see if it mentions any.
       Adds what brand was found if any in the brands column.
       
       Inputs
       ------
       Dataframe you will search over and a list of brands in a separate file. 
       Return
       ------
       Dataframe with column indicating what brand was found in the values.
       """

    df2 = df.body.str.extractall(pat=brandlist, flags=re.VERBOSE)
    df2.dropna(axis=0,how='all',inplace=True)
    return df2<jupyter_output><empty_output><jupyter_text>This takes a long time, probably around 30 minutes per 100,000 comments.<jupyter_code>df = create_comments_df('malefashionadvice')
df<jupyter_output><empty_output><jupyter_text>Run to export the df to csv. Careful about overwriting. Use the mode = 'a' line to add to an existing file.<jupyter_code>p = Path.cwd() / 'data' / 'commentdf.csv'
#df1.to_csv(path_or_buf = p)
df1.to_csv(path_or_buf = p, mode = 'a', header=False)
p = Path.cwd() / 'data' / 'commentdf.csv'
df = pd.read_csv(p)
df<jupyter_output><empty_output><jupyter_text>If you prefer pickling over csv, use these cells for IO<jupyter_code>p = Path.cwd() / 'data' / 'commentdf.pkl'
df.to_pickle(path = p)
df = pd.read_pickle(filepath_or_buffer=p)<jupyter_output><empty_output><jupyter_text>If you read from CSV it created an extra column so this cell drops it.<jupyter_code>df = df.drop(columns =['Unnamed: 0'])
df2 = brand_check(df,fashion)<jupyter_output><empty_output><jupyter_text>Drop any columns that didn't find a match.<jupyter_code>pd.set_option("display.max_rows",50, "display.max_columns", None)
df2.dropna(axis=1,how='all',inplace=True)
df2
overall_counts = df2.count().sort_values(ascending=False)<jupyter_output><empty_output><jupyter_text>Splitting up the male and female subreddits to get demographic data.<jupyter_code>filt = (df['Subreddit'] == 'malefashionadvice')
mfa_df = df[filt]
filt2 = (df['Subreddit'] == 'femalefashionadvice')
ffa_df = df[filt2]
mfa_df = brand_check(mfa_df,fashion)
ffa_df = brand_check(ffa_df,fashion)
mfa_df.dropna(axis=1,how='all',inplace=True)
ffa_df.dropna(axis=1,how='all',inplace=True)
mfa_counts = mfa_df.count().sort_values(ascending=False)
ffa_counts = ffa_df.count().sort_values(ascending=False)
pd.set_option("display.max_rows",117)
counts_df = pd.concat([overall_counts,mfa_counts,ffa_counts],axis='columns',sort=False)<jupyter_output><empty_output><jupyter_text>Final dataframe with the number of mentions for each brand split into overall, r/malefashionadvice, and r/femalefashionadvice. Ready for data visualization.<jupyter_code>counts_df.to_pickle(Path.cwd() / 'data' / 'countsdf.pkl')
#counts_df = pd.read_pickle(Path.cwd() / 'data' / 'countsdf.pkl')
pd.set_option("display.max_rows",75)
counts_df.columns = ['Total','r/malefashionadvice','r/femalefashionadvice']
counts_df.sort_values(by='Total',inplace=True)
counts_df = counts_df.iloc[-30:,1:3]
plt.style.use('ggplot')
ax = counts_df.plot.barh(stacked=True,color={r"r/femalefashionadvice":"tomato",r"r/malefashionadvice":"mediumblue"},title='30 Most Worn Brands of r/MFA and r/FFA in 2020')
plt.xlabel('Mentions from WAYWT threads in 2020')
plt.yticks(fontsize=8)
plt.savefig('waywt.png',dpi=300,bbox_inches="tight")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12010938157341186, 'fraction_numerical': 0.011989903239377367, 'mean_word_length': 5.10397946084724, 'pattern_counts': {'":': 2, '<': 37, '<?xml version=': 0, '>': 37, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '633708', 'license': 'MIT', 'n_tokens_mistral': 1708, 'n_tokens_neox': 1669, 'n_words': 375, 'path': 'free-marketing-watch.ipynb', 'repo_name': 'maybemaby/free-marketing-watch', 'size': '4754'}