<jupyter_start><jupyter_text>IntroductionI noticed that reddit being the expansive website that it is has subreddits for each of the Meyers Briggs personality types. I thought it would be interesting to see some of the differences between population data and the reddit data.The goal of this notebook is to scrape and visualize the data I find. These will be pretty high level and should be taken as a light exercise in Exploritory Data Analysis (EDA) and practice in python.  Ill be using [pandas](pandas.pydata.org/) to handle the data as it is the standard tool in python. [PRAW](http://praw.readthedocs.io/) will help me with parsing the reddit data. Redditmade scraping initially difficult, but I found [this](http://progur.com/2016/09/how-to-create-reddit-bot-using-praw4.html) to be the best way. I believe there are alternatives out there though.Lastly, I hate how matplotlib looks, so I wanted to check out how functional the [best port of ggplot2](https://github.com/has2k1/plotnine) is for python.<jupyter_code>import pandas as pd
import itertools
import praw
import json
import datetime as DT
from plotnine import *
from bokeh.plotting import figure, show
from bokeh.io import output_notebook

# Import Bokeh modules for interactive plotting
import bokeh.charts
import bokeh.io
import bokeh.models
import bokeh.palettes
import bokeh.plotting

output_notebook()<jupyter_output><empty_output><jupyter_text>Although it would probably have been easier to copy and paste a list from the web or to write it out myself, I thought it would be interesting to check out what the almighty python standard library can do. [Itertools](https://docs.python.org/3/library/itertools.html) is pretty handy and `itertools.product` allowed me to generate each personality type from the `dichotomies` list.<jupyter_code># Create list of personality types
dichotomies = [('E', 'I'), ('N', 'S'), ('T', 'F'), ('P', 'J')]
personality_types_tuples = list(itertools.product(*dichotomies))
personality_types = [''.join(pt_tuple) for pt_tuple in personality_types_tuples]<jupyter_output><empty_output><jupyter_text>[Pandas](pandas.pydata.org/) has a really cool ability to [parse an HTML table](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_html.html) just given the url. Even if the URL contains a lot more than just a table! In this case my url has a table showing the distribution of personality types in the general population.<jupyter_code># Get HTML Table
pt_dist_url = 'https://www.careerplanner.com/MB2/TypeInPopulation.cfm'
pt_dist_df = pd.read_html(pt_dist_url)
pt_dist_df = pt_dist_df[-1]

# Properly construct the DataFrame
pt_dist_df[2][0] = pt_dist_df[1][0]
del pt_dist_df[1]
pt_dist_df = pt_dist_df.iloc[1:]
pt_dist_df.columns = ['type', 'pop_freq']
pt_dist_df = pt_dist_df.set_index('type')

# Convert string percentage to numeric
pt_dist_df['pop_freq'] = pd.to_numeric(pt_dist_df['pop_freq'].str.strip('%'))/100<jupyter_output><empty_output><jupyter_text>Here I setup [PRAW](http://praw.readthedocs.io/) which helps me parse reddit. I feed it the subreddits generated from the itertools line above and I put the data into a dataframe.<jupyter_code># Get authentication information
with open('authentication.json') as file:
    auth = json.load(file)

reddit = praw.Reddit(client_id=auth['client_id'], client_secret=auth['client_secret'],
                     password=auth['password'], user_agent=auth['user_agent'],
                     username=auth['username'])

# Create Reddit DataFrame
pt_sr_info_df_columns = ['display_name', 'created_utc', 'description', 'name', 'subscribers', 'gilded']
pt_sr_info_df = pd.DataFrame(columns=pt_sr_info_df_columns, index=personality_types)
pt_sr_info_df.index.name = 'ptypes'
for ptype in personality_types:
    sr_name = reddit.subreddit(ptype).name

    sr_info = next(reddit.info([sr_name]))
    sr_info_vars = vars(sr_info)
    ptype_row = []
    for col in pt_sr_info_df.columns[:-1]:
        ptype_row.append(sr_info_vars[col])

    gilded_count = sum(1 for i in reddit.subreddit(ptype).gilded())
    ptype_row.append(gilded_count)

    pt_sr_info_df.loc[ptype] = ptype_row<jupyter_output><empty_output><jupyter_text>After generating a dataframe from the reddit data, I then merge the dataframes from the general population and generate some interesting features.The main feature of interest is subscribers proportion. The way it works is I find the total number of subscribers and scale that proportionately to the percentage of each of personality types in the population.Note: this isnt a perfect calculation, the issue is that the number of subscribers doesnt necessarily reflect the personality type on reddit. There could be people subscribed to personality types that arent their own, or multiple.<jupyter_code># Concatenating the data
df = pd.concat([pt_sr_info_df, pt_dist_df], axis=1)

# Feature engineering
df['description_length'] = df.description.apply(len)
df['creation_date'] = pd.to_datetime(df.created_utc, unit='s')
df['subreddit_age'] = pd.Timestamp(DT.datetime.now()) - df.creation_date
df['subscribers_percentage'] = df['subscribers']/df['subscribers'].sum()
df['pt_percentage_differences'] = df['pop_freq'] - df['subscribers_percentage']
df['type'] = df.index
df['subscribers_expected'] = df['pop_freq'] * df['subscribers'].sum()
df.reset_index(inplace=True)
df<jupyter_output><empty_output><jupyter_text>Now I have all the data I need, I can start making interesting plots!<jupyter_code># (ggplot(df.sort_values(['subscribers'])) +
#     geom_col(aes('type', 'subscribers', fill='type'), stat="identity") +
#     geom_point(aes('type', y='subscribers_expected')) +
#     guides(fill=False) +
#     ggtitle('Subreddit Subscribers by MBTI Personality Type')+
#     geom_line())

p = figure(title='Subreddit Subscribers by MBTI Personality Type')
p = bokeh.charts.Line(df, x='index', y='subscribers_expected')
p.rect(width=.4, height='subscribers', x='index', y='subscribers')
show(p)
(ggplot(df.reset_index())
 + geom_col(aes('type', 'subscribers'))
 + geom_point(aes('type', y='subscribers_expected'))
#  + geom_path(aes('type', y='subscribers_expected'))
#  + ggtitle('Subreddit Subscribers by MBTI Personality Type')
)
df.sort_values(['subscribers'])
(ggplot(df) +
     aes(x='type', y='subreddit_age', fill='type') +
     geom_bar(stat="identity") +
     guides(fill=False) +
     ggtitle('Subreddit Age by MBTI Personality Type'))
(ggplot(df) +
 aes(x='type', y='pt_percentage_differences', fill='type') +
 geom_bar(stat="identity") +
 guides(fill=False) +
 ggtitle('Subreddit Difference by MBTI Personality Type'))
(ggplot(df) +
 aes(x='type', y='description_length', fill='type') +
 geom_bar(stat="identity") +
 guides(fill=False) +
 ggtitle('Subreddit Difference by MBTI Personality Type'))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11603344579727153, 'fraction_numerical': 0.0038139944257004547, 'mean_word_length': 5.886868686868687, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '34402', 'license': 'MIT', 'n_tokens_mistral': 2236, 'n_tokens_neox': 2205, 'n_words': 632, 'path': 'myers_briggs_reddit_eda.ipynb', 'repo_name': 'datavistics/myers_briggs', 'size': '6817'}