<jupyter_start><jupyter_text>Wrangle & Analyze Data  <NAME><jupyter_code># Import libraries
import tweepy
from tweepy import OAuthHandler
import json
from timeit import default_timer as timer
import pandas as pd
import numpy as np
import seaborn as sns
import urllib.request
import matplotlib.pyplot as plt
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Gather Data<jupyter_code>#read twitter-archieve.csv
df1 = pd.read_csv('twitter-archive-enhanced.csv')
#read image-predictions.tsv
df2 = pd.read_csv('image-predictions.tsv', sep = '\t')
#read jsoon file
df3 = pd.read_json('tweet-json.txt', lines =True)
# Query Twitter API for each tweet in the Twitter archive and save JSON in a text file
# These are hidden to comply with Twitter's API terms and conditions
consumer_key = 'HIDDEN'
consumer_secret = 'HIDDEN'
access_token = 'HIDDEN'
access_secret = 'HIDDEN'

auth = OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_secret)

api = tweepy.API(auth)<jupyter_output><empty_output><jupyter_text>Assess Data<jupyter_code>df1.info() #inspecting
df2.info() #inspectiing data
# inspecting data
df2.head(10)
df3.info() #inspecting df3
df3.head()
df1.describe()
df2.describe()
df3.describe()
all_columns = pd.Series(list(df1) + list (df2)+ list(df3))
all_columns[all_columns.duplicated()]<jupyter_output><empty_output><jupyter_text>Quality- Drop unnecessary columns in df1- Drop unnecessary columns in df2- Extract the columns in df3 which are needed ('id', 'retweet_count', 'favorite_count')- Change 'id' name to 'tweet_id' in df3- Change 'tweet_id' data type to string in df1- Change 'tweet_id' data type to  string in df2- Change 'id' data type to string in df3- Remove "_" and capitalize the first letter for p1, p2, and p3 in df2- Change string to datetime for timestamp in df1- Capitalize first letter in 'names' in df1Tidiness- Merge 3 dogs stage into one single column- Merge 3 columns into one column by 'tweet_id'  Data Cleaning  Quality Issue  Define Drop unnecessary columns in df1_clean<jupyter_code>df1.columns<jupyter_output><empty_output><jupyter_text>Code<jupyter_code>df1_clean = df1.drop(['in_reply_to_status_id', 'in_reply_to_user_id',
       'source', 'retweeted_status_id', 'retweeted_status_user_id',
       'retweeted_status_timestamp', 'expanded_urls'], axis =1)<jupyter_output><empty_output><jupyter_text>Test<jupyter_code>df1_clean.columns<jupyter_output><empty_output><jupyter_text>Define Drop unnecessary columns in df2_clean  Code<jupyter_code>df2_clean = df2.drop(['jpg_url', 'img_num'], axis= 1)<jupyter_output><empty_output><jupyter_text>Test<jupyter_code>df2_clean.columns<jupyter_output><empty_output><jupyter_text>DefineExtract the columns in df3 which are needed  Code<jupyter_code>#extracting columns which are needed
df3_clean = pd.DataFrame(df3, columns = ['id', 'retweet_count', 'favorite_count'])<jupyter_output><empty_output><jupyter_text>Test<jupyter_code>df3_clean.columns<jupyter_output><empty_output><jupyter_text>Define change 'id' name to 'tweet_id' in df3_clean  Code<jupyter_code>df3_clean.rename(columns = {'id': 'tweet_id'}, inplace = True)<jupyter_output><empty_output><jupyter_text>Test<jupyter_code>df3_clean.columns<jupyter_output><empty_output><jupyter_text>Define change tweet_id data type by .astype in df1_clean, df2_clean, and df3_clean  Test<jupyter_code>df1_clean.tweet_id = df1_clean.tweet_id.astype(str)
df2_clean.tweet_id = df2_clean.tweet_id.astype(str)
df3_clean.tweet_id = df3_clean.tweet_id.astype(str)<jupyter_output><empty_output><jupyter_text>Test<jupyter_code>df1_clean.tweet_id.head()
df2_clean.tweet_id.head()
df3_clean.tweet_id.head()<jupyter_output><empty_output><jupyter_text>DefineRemove _ and capitalize the first letter for p1, p2, and p3 in df2  Code<jupyter_code>df2_clean = df2_clean.replace('_', ' ', regex = True)
df2_clean['p1'] = df2_clean['p1'].str.title()
df2_clean['p2'] = df2_clean['p2'].str.title()
df2_clean['p3'] = df2_clean['p3'].str.title()<jupyter_output><empty_output><jupyter_text>Test<jupyter_code>df2_clean.head(100)<jupyter_output><empty_output><jupyter_text>DefineChange string to datetime for timestamp in df1_clean  Code<jupyter_code>#to datetime
df1_clean.timestamp = pd.to_datetime(df1_clean.timestamp)<jupyter_output><empty_output><jupyter_text>Test<jupyter_code>df1_clean.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 2356 entries, 0 to 2355
Data columns (total 10 columns):
tweet_id              2356 non-null object
timestamp             2356 non-null datetime64[ns]
text                  2356 non-null object
rating_numerator      2356 non-null int64
rating_denominator    2356 non-null int64
name                  2356 non-null object
doggo                 2356 non-null object
floofer               2356 non-null object
pupper                2356 non-null object
puppo                 2356 non-null object
dtypes: datetime64[ns](1), int64(2), object(7)
memory usage: 184.1+ KB<jupyter_text>DefineUppercase the first letter in 'names' and dogs' stages by .title and  in df1  Code<jupyter_code>df1_clean['name'] = df1_clean['name'].str.title()
df1_clean['doggo'] = df1_clean['doggo'].str.title()
df1_clean['floofer'] = df1_clean['floofer'].str.title()
df1_clean['pupper'] = df1_clean['pupper'].str.title()
df1_clean['puppo'] = df1_clean['puppo'].str.title()<jupyter_output><empty_output><jupyter_text>Test<jupyter_code>df1_clean['name']<jupyter_output><empty_output><jupyter_text>Tidiness  Define- Merge all three dogs' stages into one single column and then drop the empty rows  Code<jupyter_code>df1_clean.replace('None', np.nan, inplace = True)
df1_clean['stage'] = df1_clean[['doggo', 'floofer', 'pupper', 'puppo']].fillna('').sum(axis=1).astype(str)
#drop the 
df1_clean = df1_clean.drop(['doggo', 'floofer', 'pupper', 'puppo'], axis =1)
df1_clean['name'] = df1_clean['name'].str.title()
df1_clean['stage'].replace('', np.nan, inplace=True)
df1_clean.dropna(subset=['name'])
df1_clean.dropna(subset=['stage'])
df1_clean['stage'].value_counts<jupyter_output><empty_output><jupyter_text>DefineReplace the names 'DoggoPupper" to "Doggo", 'DoggoPuppo', 'Puppo', 'DoggoFloofer', 'Floofer'  Test<jupyter_code>##DoggoPupper should be doggo

df1_clean.stage= df1_clean.stage.replace('DoggoPupper', 'Doggo')
df1_clean.stage= df1_clean.stage.replace('DoggoPuppo', 'Puppo')
df1_clean.stage= df1_clean.stage.replace('DoggoFloofer', 'Floofer')<jupyter_output><empty_output><jupyter_text>Test<jupyter_code>df1_clean['stage'].value_counts()
df1_clean['name'].value_counts()<jupyter_output><empty_output><jupyter_text>DefineMerge all three columns into a single column by 'tweeter_id'  Code<jupyter_code>df_n = pd.merge(df1_clean, df2_clean, on = 'tweet_id', how ='inner')
df_new = pd.merge(df_n, df3_clean, on = 'tweet_id', how ='inner')<jupyter_output><empty_output><jupyter_text>Test<jupyter_code>df_new.columns
df_new.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 2073 entries, 0 to 2072
Data columns (total 18 columns):
tweet_id              2073 non-null object
timestamp             2073 non-null datetime64[ns]
text                  2073 non-null object
rating_numerator      2073 non-null int64
rating_denominator    2073 non-null int64
name                  1496 non-null object
stage                 320 non-null object
p1                    2073 non-null object
p1_conf               2073 non-null float64
p1_dog                2073 non-null bool
p2                    2073 non-null object
p2_conf               2073 non-null float64
p2_dog                2073 non-null bool
p3                    2073 non-null object
p3_conf               2073 non-null float64
p3_dog                2073 non-null bool
retweet_count         2073 non-null int64
favorite_count        2073 non-null int64
dtypes: bool(3), datetime64[ns](1), float64(3), int64(4), object(7)
memory usage: 265.2+ KB<jupyter_text>Data cleaning has been done! And the data type looks fine  Visualize and Analyze<jupyter_code>df_new.describe()<jupyter_output><empty_output><jupyter_text>- At 75 percentile, most dogs get at the scale of 12 on rating numerator.- At 75 percentile, most dogs get at the scale of 11 on rating denominator.- There are more favorite counts than retweet counts.  Most Popular Names<jupyter_code>common_names = df_new['name'].value_counts().nlargest(10)
common_names<jupyter_output><empty_output><jupyter_text>Top 10 dog names<jupyter_code>common_names.plot.bar()<jupyter_output><empty_output><jupyter_text>Unrecorded names with 'A' dogs' have the highest number of names among all other names.  Dog Stages<jupyter_code>dog_stages = df_new['stage'].value_counts()
dog_stages<jupyter_output><empty_output><jupyter_text>Pupper stage has the highest number of dogs<jupyter_code>dog_stages.plot.bar()<jupyter_output><empty_output><jupyter_text>Pupper stage has the highest number of dogs<jupyter_code>df_new.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 2073 entries, 0 to 2072
Data columns (total 18 columns):
tweet_id              2073 non-null object
timestamp             2073 non-null datetime64[ns]
text                  2073 non-null object
rating_numerator      2073 non-null int64
rating_denominator    2073 non-null int64
name                  1496 non-null object
stage                 320 non-null object
p1                    2073 non-null object
p1_conf               2073 non-null float64
p1_dog                2073 non-null bool
p2                    2073 non-null object
p2_conf               2073 non-null float64
p2_dog                2073 non-null bool
p3                    2073 non-null object
p3_conf               2073 non-null float64
p3_dog                2073 non-null bool
retweet_count         2073 non-null int64
favorite_count        2073 non-null int64
dtypes: bool(3), datetime64[ns](1), float64(3), int64(4), object(7)
memory usage: 265.2+ KB<jupyter_text>Favorite Tweets vs Retweets<jupyter_code>df_new['retweet_count'].describe()
df_new['favorite_count'].describe()
df_new.plot(x = 'retweet_count', y= 'favorite_count' , kind = 'scatter', figsize= (20, 20))
plt.show()<jupyter_output><empty_output><jupyter_text>There is a more positive correlation towards the favorite count side.  Favorite tweets vs Timestamp<jupyter_code>df_new.plot(x = 'timestamp', y= 'favorite_count' , kind = 'line', figsize = (20,20))
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10789777094542659, 'fraction_numerical': 0.0441967717140661, 'mean_word_length': 5.051744186046512, 'pattern_counts': {'":': 0, '<': 134, '<?xml version=': 0, '>': 134, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '33', 'dir': 'jupyter-structured-clean-dedup', 'id': '82171', 'license': 'MIT', 'n_tokens_mistral': 3944, 'n_tokens_neox': 3685, 'n_words': 813, 'path': 'wrangle_act.ipynb', 'repo_name': 'ShakhawatHassan/Wrangle-Analyze-Data', 'size': '10418'}