<jupyter_start><jupyter_text>Project 3 - Deep Cleaning Data for Final Use in EDA and ModelingIn this portion of the Project, we perform a deeper cleaning of our data to prep it fully for EDA and Modeling. Here we will determine the key features that are need to best garner inference from our data and then to predict our subreddits. This is singularly the most important part of this project.  Contents- [1.0 Setup - install modules](one)- [2.0 Import Data and Setup Pandas Dataframe](two)- [3.0 Unrool Json to Columns](three)- [4.0 Add Time Series and Subscribers To DataFrame](four)  1.0 Setup - Import Libraries<jupyter_code>import warnings
import pandas as pd
import numpy as np
from datetime import datetime
import json
import ast
# this setting widens how many characters pandas will display in a column:
pd.options.display.max_colwidth = 350<jupyter_output><empty_output><jupyter_text>2.0 Import Data and Setup Pandas Dataframe<jupyter_code>df = pd.read_csv('data/2_Cleaned_IBM_Data/cleaned_ibm.csv')
df.head(5) # Import and Inspect our DataFrame<jupyter_output><empty_output><jupyter_text>3.0 Unspool Response ColumnWe will unspool the dictionary/list in the response column to extract the sentiment and emotional information we want to gather for our analysis.<jupyter_code>test_df = df.copy() # Create a new DataFrame
test_df.tail() # Inspect DataFrame to ensure Copy was done correctly<jupyter_output><empty_output><jupyter_text>> **SENTIMENT:** The label is the overall sentiment of the document (positive, negative, or neutral). The sentiment label is based on the score; a score of 0.0 would indicate that the document is neutral, a positive number would indicate the document is positive, a negative number would indicate the document is negative.> **RELEVANCE** The relevance score ranges from 0.0 to 1.0. The higher the score, the more relevant the keyword.<jupyter_code>warnings.filterwarnings('ignore') # Supress Warnings
test_df["sadness"] = 0 # Assign a Value of '0' to each new column below
test_df["joy"] = 0
test_df["fear"] = 0
test_df["disgust"] = 0
test_df["anger"] = 0
test_df["keyword"] = 0
test_df["relevance_score"] = 0
test_df["sentiment_score"] = 0
test_df.head(1) # Inspect Data Frame
test_df['response'] = test_df['response'].apply(lambda x: ast.literal_eval(x)) # convert to Dictionary
# https://stackoverflow.com/questions/39169718/convert-string-to-dict-then-access-keyvalues-how-to-access-data-in-a-class
for i in range(0,len(test_df)):
    try:
        test_df["sadness"][i] = str(test_df['response'][i]['keywords'][0]['emotion']['sadness'])
        test_df["joy"][i] = str(test_df['response'][i]['keywords'][0]['emotion']['joy'])
        test_df["fear"][i] = str(test_df['response'][i]['keywords'][0]['emotion']['fear'])
        test_df["disgust"][i] = str(test_df['response'][i]['keywords'][0]['emotion']['disgust'])
        test_df["anger"][i] = str(test_df['response'][i]['keywords'][0]['emotion']['anger'])
        test_df["keyword"][i] = str(test_df['response'][i]['keywords'][0]['text'])
        test_df["relevance_score"][i] = str(test_df['response'][i]['keywords'][0]['relevance'])
        test_df["sentiment_score"][i] = str(test_df['response'][i]['keywords'][0]['sentiment']['score'])
    except:
        continue
test_df.tail(2)
test_df.dtypes # Check DataTypes to ensure scores have been converted to Numbers<jupyter_output><empty_output><jupyter_text>> Convert String to Integers<jupyter_code>test_df["sadness"] = test_df["sadness"].astype(float)
test_df["joy"] = test_df["joy"].astype(float)
test_df["fear"] = test_df["fear"].astype(float)
test_df["disgust"] = test_df["disgust"].astype(float)
test_df["anger"] = test_df["anger"].astype(float)
test_df["relevance_score"] = test_df["relevance_score"].astype(float)
test_df["sentiment_score"] = test_df["sentiment_score"].astype(float)
test_df.dtypes<jupyter_output><empty_output><jupyter_text>4.0 Add Date and Subscribers To DataFrame> We add data and Subscriber count to Current DataFrame to prep for Final Analysis<jupyter_code>df_time = pd.read_csv('/data/1_Scraped_Data_IBM/ibm_watson_time.csv') # Import DataFrame
df_time.shape # Check Current Shape of DataFrame
df_time['status_word_count'] = [len(df_time['title'][i].split()) for i in range(0,df_time['title'].shape[0])]
# Split on the spaces and then count the number of words with Lenght in-built function
df_time.shape # Check Current Shape to ensure row was added
df_time = df_time[df_time['status_word_count'] > 5]
# Drop rows with less than 5 words to align with IBM data frame we want to Concat to
df_time.reset_index(inplace = True, drop = True)
# Reset Index and ensure dataframe updates automatically
df_time.shape # Checking shape to ensure this was done correctly
test_df.shape # df_time.shape(0) equals test_df.shape(0) so we are fine
test_df['time'] = df_time['created_utc'] # Add epoch time to new DataFrame
test_df['subscribers'] = df_time['subreddit_subscribers'] # Add Subreddit Subscribers to new time dataframe
test_df[test_df['subreddit'] == 'ProCreate'].head(1) # Inspect the head
test_df['date'] = pd.to_datetime(test_df['time'],unit='s') # Convert Epoch Time to standard date format
test_df.head(1) # Inspect Head to ensure change above executed correctly
test_df.drop(['response','normalized','keyword','time','relevance_score'], axis=1, inplace=True)
# Drop Columns we don't need
test_df.head(1) # Inspect Head to ensure change above executed correctly
# Write the DataFrame you created to a csv called 'final.csv'
test_df.to_csv('data/3_Final_Cleaning/final.csv', index=False)
print('Submission CSV is ready!')<jupyter_output>Submission CSV is ready!
{'fraction_non_alphanumeric': 0.12930117062788224, 'fraction_numerical': 0.012061014544164597, 'mean_word_length': 6.352020860495437, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 29, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '463455', 'license': 'MIT', 'n_tokens_mistral': 1844, 'n_tokens_neox': 1806, 'n_words': 524, 'path': 'code/3_Final_Cleaning_Project_3.ipynb', 'repo_name': 'oaofili/subreddit_predictor_model', 'size': '5638'}