<jupyter_start><jupyter_text>Data analysis of WeRateDogs Twitter account.<jupyter_code>import pandas as pd
import numpy as np
import requests as rq
import matplotlib.pyplot as plt
import seaborn as sns
import matplotlib
%matplotlib inline
import tweepy
import json
import re
from tweepy import OAuthHandler
from timeit import default_timer as timer
from functools import reduce<jupyter_output><empty_output><jupyter_text>1. Gather data  1. The WeRateDogs Twitter archive<jupyter_code>#importing csv file
df_1 = pd.read_csv("twitter-archive-enhanced.csv")
df_1.head()
#getting some information about dataframe
df_1.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 2356 entries, 0 to 2355
Data columns (total 17 columns):
 #   Column                      Non-Null Count  Dtype  
---  ------                      --------------  -----  
 0   tweet_id                    2356 non-null   int64  
 1   in_reply_to_status_id       78 non-null     float64
 2   in_reply_to_user_id         78 non-null     float64
 3   timestamp                   2356 non-null   object 
 4   source                      2356 non-null   object 
 5   text                        2356 non-null   object 
 6   retweeted_status_id         181 non-null    float64
 7   retweeted_status_user_id    181 non-null    float64
 8   retweeted_status_timestamp  181 non-null    object 
 9   expanded_urls               2297 non-null   object 
 10  rating_numerator            2356 non-null   int64  
 11  rating_denominator          2356 non-null   int64  
 12  name                        2356 non-null   object 
 13  doggo                       2356 [...]<jupyter_text>2. The tweet image predictions<jupyter_code>#getting the file of Image predictions programatically
url = "https://d17h27t6h515a5.cloudfront.net/topher/2017/August/599fd2ad_image-predictions/image-predictions.tsv"
response = rq.get(url)

with open('image-predictions.tsv', mode = 'wb') as file:
    file.write(response.content)
#reading the downloaded file
image_p = pd.read_csv('image-predictions.tsv', sep = '\t')
#Retrieving some info
image_p.head()<jupyter_output><empty_output><jupyter_text>3. Twitter API<jupyter_code># I took this code from my udacity class rooom. Because I could not get the Twitter API token.
#https://classroom.udacity.com/nanodegrees/nd002-ent/parts/f55ce890-c08c-46a5-b57f-55a06c1cc6ae/modules/a8fcd18c-b9a5-4852-a7ec-6dbb08ebfe5a/lessons/a8085857-3e28-4fc7-aeb8-da64ccbc2e20/concepts/d7e3de1b-d7a1-4ebc-9d58-beba021a7c29


# Query Twitter API for each tweet in the Twitter archive and save JSON in a text file
# These are hidden to comply with Twitter's API terms and conditions
consumer_key = 'HIDDEN'
consumer_secret = 'HIDDEN'
access_token = 'HIDDEN'
access_secret = 'HIDDEN'

auth = OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_secret)

api = tweepy.API(auth, wait_on_rate_limit=True)

# NOTE TO STUDENT WITH MOBILE VERIFICATION ISSUES:
# df_1 is a DataFrame with the twitter_archive_enhanced.csv file. You may have to
# change line 17 to match the name of your DataFrame with twitter_archive_enhanced.csv
# NOTE TO REVIEWER: this student had mobile verification issues so the following
# Twitter API code was sent to this student from a Udacity instructor
# Tweet IDs for which to gather additional data via Twitter's API
tweet_ids = df_1.tweet_id.values
len(tweet_ids)

# Query Twitter's API for JSON data for each tweet ID in the Twitter archive
count = 0
fails_dict = {}
start = timer()
# Save each tweet's returned JSON as a new line in a .txt file
with open('tweet_json.txt', 'w') as outfile:
    # This loop will likely take 20-30 minutes to run because of Twitter's rate limit
    for tweet_id in tweet_ids:
        count += 1
        print(str(count) + ": " + str(tweet_id))
        try:
            tweet = api.get_status(tweet_id, tweet_mode='extended')
            print("Success")
            json.dump(tweet._json, outfile)
            outfile.write('\n')
        except tweepy.TweepError as e:
            print("Fail")
            fails_dict[tweet_id] = e
            pass
end = timer()
print(end - start)
print(fails_dict)<jupyter_output>1: 892420643555336193
Fail
2: 892177421306343426
Fail
3: 891815181378084864
Fail
4: 891689557279858688
Fail
5: 891327558926688256
Fail
6: 891087950875897856
Fail
7: 890971913173991426
Fail
8: 890729181411237888
Fail
9: 890609185150312448
Fail
10: 890240255349198849
Fail
11: 890006608113172480
Fail
12: 889880896479866881
Fail
13: 889665388333682689
Fail
14: 889638837579907072
Fail
15: 889531135344209921
Fail
16: 889278841981685760
Fail
17: 888917238123831296
Fail
18: 888804989199671297
Fail
19: 888554962724278272
Fail
20: 888202515573088257
Fail
21: 888078434458587136
Fail
22: 887705289381826560
Fail
23: 887517139158093824
Fail
24: 887473957103951883
Fail
25: 887343217045368832
Fail
26: 887101392804085760
Fail
27: 886983233522544640
Fail
28: 886736880519319552
Fail
29: 886680336477933568
Fail
30: 886366144734445568
Fail
31: 886267009285017600
Fail
32: 886258384151887873
Fail
33: 886054160059072513
Fail
34: 885984800019947520
Fail
35: 885528943205470208
Fail
36: 885518971528720385
Fail
3[...]<jupyter_text>Looks like the tokens are expired. Thus I downloaded the file called 'tweet-json.txt' from my classroom via abovementioned link.<jupyter_code>#Extracting the important columns of Json text file. Afterwards I am putting the coulmns into pandas Dataframe so 
#that I can see and assess the data.
my_list = []
with open('tweet-json.txt', encoding = 'utf-8') as file_json:
    for each in file_json:
        json_d = json.loads(each)
        tweet_id = json_d['id']
        favorite_count = json_d['favorite_count']
        retweet_count = json_d['retweet_count']

        my_list.append({'tweet_id': tweet_id,
                        'favorite_count': favorite_count,
                        'retweet_count': retweet_count,
                        })
        
tweet_json = pd.DataFrame(my_list, columns = ['tweet_id', 'favorite_count', 'retweet_count'])

tweet_json.info()
tweet_json.head()<jupyter_output><empty_output><jupyter_text>2. Assess the data  Let us assess first the Twitter archive data<jupyter_code># Gives 10 random samples of dataframe
df_1.sample(10)<jupyter_output><empty_output><jupyter_text>As you can see there are some numbers in retweeted_status_id and retweeted_status_user_id instead of NaNs. This means this data has retweets.<jupyter_code>df_1.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 2356 entries, 0 to 2355
Data columns (total 17 columns):
 #   Column                      Non-Null Count  Dtype  
---  ------                      --------------  -----  
 0   tweet_id                    2356 non-null   int64  
 1   in_reply_to_status_id       78 non-null     float64
 2   in_reply_to_user_id         78 non-null     float64
 3   timestamp                   2356 non-null   object 
 4   source                      2356 non-null   object 
 5   text                        2356 non-null   object 
 6   retweeted_status_id         181 non-null    float64
 7   retweeted_status_user_id    181 non-null    float64
 8   retweeted_status_timestamp  181 non-null    object 
 9   expanded_urls               2297 non-null   object 
 10  rating_numerator            2356 non-null   int64  
 11  rating_denominator          2356 non-null   int64  
 12  name                        2356 non-null   object 
 13  doggo                       2356 [...]<jupyter_text>Timestamp is string.   retweeted_status_timestamp is string.<jupyter_code>#Series containing counts of unique values
df_1.source.value_counts()<jupyter_output><empty_output><jupyter_text>Source column has html tags.<jupyter_code>#I have viewed visually also by excell. So I can see name column fully.
df_1.name
# Selecting name = a OR name = None
df_1[(df_1['name'] == 'a') | (df_1['name'] == 'None')]<jupyter_output><empty_output><jupyter_text>name column has None instead of NaN.<jupyter_code>df_1.name.sample(30)
#When sampling multiple times I have seen name called O. Selecting name = O
df_1[df_1['name'] == 'O']<jupyter_output><empty_output><jupyter_text>There a lot of inaccurute names in this dataframe such as a, an, such, the, by, very, O and so on.<jupyter_code>df_1.doggo.value_counts()
df_1.floofer.value_counts()
df_1.pupper.value_counts()
df_1.puppo.value_counts()<jupyter_output><empty_output><jupyter_text>Ok these four columns have None values instead of NaN.<jupyter_code># Seeking for upper case values of name column.
df_1.loc[df_1['name'].str.isupper()]<jupyter_output><empty_output><jupyter_text>There is a JD name also. What a name :( . Ok So I will replace them with O'Malley and Just Dog.<jupyter_code>#Let see the data tyoe of tweet_id column.
df_1.tweet_id.value_counts()<jupyter_output><empty_output><jupyter_text>Ok I will change tweet_id also. This one should be a string since we are not doing any calculations on it.<jupyter_code>df_1.rating_numerator.sample(30)
df_1.rating_denominator.sample(30)
df_1[df_1['rating_denominator'] == 0]<jupyter_output><empty_output><jupyter_text>This is weird. Denominator cannot be 0. Whereas numerator is 960. Ok this is going to be fun :)  Second I go with Image predictions data.<jupyter_code>image_p.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 2075 entries, 0 to 2074
Data columns (total 12 columns):
 #   Column    Non-Null Count  Dtype  
---  ------    --------------  -----  
 0   tweet_id  2075 non-null   int64  
 1   jpg_url   2075 non-null   object 
 2   img_num   2075 non-null   int64  
 3   p1        2075 non-null   object 
 4   p1_conf   2075 non-null   float64
 5   p1_dog    2075 non-null   bool   
 6   p2        2075 non-null   object 
 7   p2_conf   2075 non-null   float64
 8   p2_dog    2075 non-null   bool   
 9   p3        2075 non-null   object 
 10  p3_conf   2075 non-null   float64
 11  p3_dog    2075 non-null   bool   
dtypes: bool(3), float64(3), int64(2), object(4)
memory usage: 152.1+ KB<jupyter_text>There are 2075 rows. Whereas in df_1 dataframe, there are 2356. This means whether there are some missing data or some tweets do not have images.<jupyter_code>image_p.jpg_url.value_counts()
image_p.img_num.value_counts()
image_p.p1.value_counts()
image_p.p2.value_counts()
image_p.p3.value_counts()<jupyter_output><empty_output><jupyter_text>Here we go. You see the underscores of p1, p2 and p3. In addition, I will create two columns called dog breeds and prediction confidence based on these p columns.<jupyter_code>image_p.head()<jupyter_output><empty_output><jupyter_text>Third is tweet_json dataframe from Twitter API<jupyter_code>#Let us take some samples
tweet_json.sample(10)
tweet_json.info()
tweet_json.tweet_id.value_counts()
tweet_json.favorite_count.value_counts()
tweet_json.retweet_count.value_counts()<jupyter_output><empty_output><jupyter_text>Quality issues:  df_1 DataFrame- Data has retweets since there are some numbers in retweeted_status_id and retweeted_status_user_id instead of NaNs. - timestamp column is string type.- source column has html tags with some text.- name column contains values such as string "None".- name column has awkward names such as a, an, such, the, by, very and so on.- name column has O, JD. It should O'Malley and Just Dog.- tweet_id is integer. It should be string since there are not any calculations.- Wrong data in rating numerator and rating denominator column since tweets include more than one rating or decimal numbers. - Fraction for rating numerator and rating denominator. image_p DataFrame- Remove underscores and make them lowercase of p1, p2 and p3.  Tideness issues:  df_1 DataFrame- 4 different names such as doggo, floofer, pupper, puppo. Change name to dog type.  tweet_json DataFrame- Combine tweet_json with df_1 since data are related to each other.  image_p DataFrame- Combine image_p with df_1 since data of image_p is related to df_1.- Create dog breeds and prediction confidence columns by p1, p1_conf, p1_dog, p2, p2_conf, p2_dog, p3, p3_conf, p3_dog. Drop the p1, p1_conf, p1_dog, p2, p2_conf, p2_dog, p3, p3_conf, p3_dog.  3. Clean the data<jupyter_code>#Copy the data to a new variable. So we can work safely with copied variable.
df_1_copied = df_1.copy()
tweet_json_copied = tweet_json.copy()
image_p_copied = image_p.copy()<jupyter_output><empty_output><jupyter_text>Tidiness:Firstly, I will start cleaning with Tidiness issues since they are more important for beginning. I will solve these two issues since first tidiness issue will be more easy to clean by solving these two issues. Define:- Combine tweet_json with df_1 since data are kinda same.- Combine image_p with df_1 since data of image_p is related to df_1. Solving: Merge dataFrames on tweet_id column.  Code:<jupyter_code>#I will merge these three DataFrames on tweet_id column.
#I will make some list first.
df_3 = [df_1_copied, tweet_json_copied, image_p_copied]

#I will use lambda for merging. I will use reduce funtion.
#https://stackoverflow.com/questions/8689184/nameerror-name-reduce-is-not-defined-in-python
twitter_full = reduce(lambda left, right: pd.merge(left, right, on = 'tweet_id'), df_3)<jupyter_output><empty_output><jupyter_text>Test:<jupyter_code>twitter_full.head()
#There should be 30 columns.
twitter_full.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 2073 entries, 0 to 2072
Data columns (total 30 columns):
 #   Column                      Non-Null Count  Dtype  
---  ------                      --------------  -----  
 0   tweet_id                    2073 non-null   int64  
 1   in_reply_to_status_id       23 non-null     float64
 2   in_reply_to_user_id         23 non-null     float64
 3   timestamp                   2073 non-null   object 
 4   source                      2073 non-null   object 
 5   text                        2073 non-null   object 
 6   retweeted_status_id         79 non-null     float64
 7   retweeted_status_user_id    79 non-null     float64
 8   retweeted_status_timestamp  79 non-null     object 
 9   expanded_urls               2073 non-null   object 
 10  rating_numerator            2073 non-null   int64  
 11  rating_denominator          2073 non-null   int64  
 12  name                        2073 non-null   object 
 13  doggo                       2073 [...]<jupyter_text>Define: - 4 different names such as doggo, floofer, pupper, puppo. Change name to dog type. Solving: Take the dog types from text column and drop the doggo, floofer, pupper, puppo. I will use ReGex(regular expression).   Code:<jupyter_code>#creating new column called dog_type. Extracting doggo, floofer, pupper, puppo from text column. 
#https://stackoverflow.com/questions/54343378/pandas-valueerror-pattern-contains-no-capture-groups
twitter_full['dog_type'] = twitter_full['text'].str.extract('(doggo|floofer|pupper|puppo)')
#Viewing the new column called dog type
twitter_full[['dog_type', 'doggo', 'floofer', 'pupper', 'puppo']].head(10)
#Drop the doggo, floofer, pupper, puppo.
twitter_full = twitter_full.drop(['doggo', 'floofer', 'pupper', 'puppo'], axis = 1)<jupyter_output><empty_output><jupyter_text>Test:<jupyter_code>#There should be 27 columns
twitter_full.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 2073 entries, 0 to 2072
Data columns (total 27 columns):
 #   Column                      Non-Null Count  Dtype  
---  ------                      --------------  -----  
 0   tweet_id                    2073 non-null   int64  
 1   in_reply_to_status_id       23 non-null     float64
 2   in_reply_to_user_id         23 non-null     float64
 3   timestamp                   2073 non-null   object 
 4   source                      2073 non-null   object 
 5   text                        2073 non-null   object 
 6   retweeted_status_id         79 non-null     float64
 7   retweeted_status_user_id    79 non-null     float64
 8   retweeted_status_timestamp  79 non-null     object 
 9   expanded_urls               2073 non-null   object 
 10  rating_numerator            2073 non-null   int64  
 11  rating_denominator          2073 non-null   int64  
 12  name                        2073 non-null   object 
 13  favorite_count              2073 [...]<jupyter_text>Define: - Create dog breeds and prediction confidence columns by p1, p1_conf, p1_dog, p2, p2_conf, p2_dog, p3, p3_conf, p3_dog. Drop the p1, p1_conf, p1_dog, p2, p2_conf, p2_dog, p3, p3_conf, p3_dog.  Solving: If the first prediction was not the dog breed then take the second and so on.    Code:<jupyter_code>#writing a funtion which extracts dog breeds from the columns.
def dog_breed(row):
    if row['p1_dog']:
        return(row['p1'])
    elif row['p2_dog']:
        return(row['p2'])
    elif row['p3_dog']:
        return(row['p3'])
    else:
        return(np.NaN)
#creating new column called dog_breed and applying it into the dataframe.
twitter_full['dog_breed'] = twitter_full.apply(lambda row: dog_breed(row), axis = 1)
#writing a funtion which extracts prediction confidence from the columns.
def pred_conf(row):
    if row['p1_dog']:
        return(row['p1_conf'])
    elif row['p2_dog']:
        return(row['p2_conf'])
    elif row['p3_dog']:
        return(row['p3_conf'])
    else:
        return(np.NaN)
#creating new column called pred_confidence and applying it into the dataframe.
twitter_full['pred_confidence'] = twitter_full.apply(lambda row: pred_conf(row), axis = 1)
#Drop the p1, p1_conf, p1_dog, p2, p2_conf, p2_dog, p3, p3_conf, p3_dog.
twitter_full = twitter_full.drop(['p1', 'p1_conf', 'p1_dog', 'p2', 'p2_conf', 'p2_dog', 'p3', 'p3_conf', 'p3_dog'], axis = 1)<jupyter_output><empty_output><jupyter_text>Test:<jupyter_code>twitter_full.head(5)
#There should be 20 columns
twitter_full.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 2073 entries, 0 to 2072
Data columns (total 20 columns):
 #   Column                      Non-Null Count  Dtype  
---  ------                      --------------  -----  
 0   tweet_id                    2073 non-null   int64  
 1   in_reply_to_status_id       23 non-null     float64
 2   in_reply_to_user_id         23 non-null     float64
 3   timestamp                   2073 non-null   object 
 4   source                      2073 non-null   object 
 5   text                        2073 non-null   object 
 6   retweeted_status_id         79 non-null     float64
 7   retweeted_status_user_id    79 non-null     float64
 8   retweeted_status_timestamp  79 non-null     object 
 9   expanded_urls               2073 non-null   object 
 10  rating_numerator            2073 non-null   int64  
 11  rating_denominator          2073 non-null   int64  
 12  name                        2073 non-null   object 
 13  favorite_count              2073 [...]<jupyter_text>Quality: Define:- Data has retweets since there are some numbers in retweeted_status_id and retweeted_status_user_id instead of NaNs. Solving: Keep the rows where retweeted_status_id has NaN values. Drop the retweeted_status_id, retweeted_status_user_id, retweeted_status_timestamp.  Code:<jupyter_code>#Keping the NaNs.
twitter_full = twitter_full[np.isnan(twitter_full.retweeted_status_id)]
twitter_full.retweeted_status_id
#Dropping retweeted_status_id, retweeted_status_user_id, retweeted_status_timestamp
twitter_full = twitter_full.drop(['retweeted_status_id', 'retweeted_status_user_id', 'retweeted_status_timestamp'], axis = 1)<jupyter_output><empty_output><jupyter_text>Test:<jupyter_code>#There should be 17 columns
twitter_full.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 1994 entries, 0 to 2072
Data columns (total 17 columns):
 #   Column                 Non-Null Count  Dtype  
---  ------                 --------------  -----  
 0   tweet_id               1994 non-null   int64  
 1   in_reply_to_status_id  23 non-null     float64
 2   in_reply_to_user_id    23 non-null     float64
 3   timestamp              1994 non-null   object 
 4   source                 1994 non-null   object 
 5   text                   1994 non-null   object 
 6   expanded_urls          1994 non-null   object 
 7   rating_numerator       1994 non-null   int64  
 8   rating_denominator     1994 non-null   int64  
 9   name                   1994 non-null   object 
 10  favorite_count         1994 non-null   int64  
 11  retweet_count          1994 non-null   int64  
 12  jpg_url                1994 non-null   object 
 13  img_num                1994 non-null   int64  
 14  dog_type               326 non-null    object 
 15  dog_[...]<jupyter_text>Define:- timestamp column is string type. Solving: Convert it into datetime data type.  Code:<jupyter_code>#Get rid of the time zone info from timestamp column
twitter_full['timestamp'] = twitter_full['timestamp'].str.slice(start = 0, stop = -6)
#Convert it into datetime data type
twitter_full['timestamp'] = pd.to_datetime(twitter_full['timestamp'], format = '%Y-%m-%d %H:%M:%S')<jupyter_output><empty_output><jupyter_text>Test:<jupyter_code>#Check the timestamp data type
twitter_full.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 1994 entries, 0 to 2072
Data columns (total 17 columns):
 #   Column                 Non-Null Count  Dtype         
---  ------                 --------------  -----         
 0   tweet_id               1994 non-null   int64         
 1   in_reply_to_status_id  23 non-null     float64       
 2   in_reply_to_user_id    23 non-null     float64       
 3   timestamp              1994 non-null   datetime64[ns]
 4   source                 1994 non-null   object        
 5   text                   1994 non-null   object        
 6   expanded_urls          1994 non-null   object        
 7   rating_numerator       1994 non-null   int64         
 8   rating_denominator     1994 non-null   int64         
 9   name                   1994 non-null   object        
 10  favorite_count         1994 non-null   int64         
 11  retweet_count          1994 non-null   int64         
 12  jpg_url                1994 non-null   object        
 13  img[...]<jupyter_text>Define:- source column has html tags with some text. Solving: Remove the html tags and only leave the name of device.  Code:<jupyter_code>twitter_full.source.value_counts()
#Extracting the only the name. I am using the awesome ReGex.
twitter_full['source'] = twitter_full['source'].str.extract('(Twitter for iPhone|Twitter Web Client|TweetDeck)')<jupyter_output><empty_output><jupyter_text>Test:<jupyter_code>twitter_full.source.value_counts()<jupyter_output><empty_output><jupyter_text>Define:- name column contains values such as string "None".- name column has awkward names such as a, an, such, the, by, very, O, JD and so on.- name column has O, JD. It should O'Malley and Just Dog. Solving: Replace all with NaN where you see some weird names. Replace O with O'Malley and JD with Just Dog.   Code:<jupyter_code>#Let us find the lowercase values. Maybe there are lots of weird names
lower_name =  []
for i in twitter_full.name:
    if i[0].islower() and i not in lower_name:
        lower_name.append(i)
print(lower_name)
#Replacing all lowercase values with NaN
twitter_full.name.replace(lower_name, np.nan, inplace=True)

#Replacing all None's values with NaN
twitter_full.name.replace('None', np.nan, inplace=True)
#Replacing O with O'Malley and JD with Just Dog.
twitter_full.name.replace('O', "O'Malley", inplace=True)
twitter_full.name.replace('JD', "Just Dog", inplace=True)<jupyter_output><empty_output><jupyter_text>Test:<jupyter_code>twitter_full.name.value_counts()
#Checking for the O'Malley and Just Dog.
twitter_full[(twitter_full['name'] == "O'Malley") | (twitter_full['name'] == "<NAME>")]<jupyter_output><empty_output><jupyter_text>Define:- tweet_id is integer. It should be string since there are not any calculations. Solving: Convert it into string data type.  Code:<jupyter_code>#Converting it into string.
twitter_full.tweet_id = twitter_full.tweet_id.astype(str)<jupyter_output><empty_output><jupyter_text>Test:<jupyter_code>twitter_full.tweet_id<jupyter_output><empty_output><jupyter_text>Define:- Wrong data in rating numerator and rating denominator column since tweets include more than one rating or decimal numbers. Solving: Find the ids that have two ratings and correct the rating numerator column and putting it into new column. I will drop the old one.  Code:<jupyter_code>#Finding the more than one ratings for one tweet. Using ReGex.
double_rating_id = twitter_full['tweet_id'][twitter_full.text.str.contains(r'(\d+\.?\d*\/\d+\.?\d*\D+\d+\.?\d*\/\d+\.?\d*)')].tolist()
double_rating_id
#Correcting the rating numerastor column and putting it into new column.
twitter_full['rating_numerator_n'] = twitter_full['rating_numerator']

#Replace the double rating with NaNs. 
twitter_full['rating_numerator_n'] = np.where(twitter_full['tweet_id'].isin(double_rating_id), np.NaN, twitter_full['rating_numerator_n'])
#Check for the NaNs
twitter_full.rating_numerator_n.isnull().sum()
twitter_full.rating_numerator_n.dtype
#Finding the all tweets with decimal numerators
twitter_full[twitter_full.text.str.contains(r'(\d+\.\d+)/(\d+)')]
list_id = twitter_full['tweet_id'][twitter_full.text.str.contains(r'(\d+\.\d+)/(\d+)')].tolist()
list_id
#create new list with decimal ratings
list_num = twitter_full['text'].str.extract(r'(\d+\.\d+)/(\d+)')[0].dropna().tolist()
list_num
#create dictionary with list_id and list_num
dict_n = dict(zip(list_id, list_num))
dict_n
#Use dictionary to correct the wrong numerators.
twitter_full.loc[twitter_full['tweet_id'].isin(dict_n.keys()), 'rating_numerator_n'] = twitter_full['tweet_id'].map(dict_n)
twitter_full.loc[twitter_full['rating_numerator'] != twitter_full['rating_numerator_n']]
#Drop the rating numerator column since we do not need it anymore.
twitter_full = twitter_full.drop(['rating_numerator'], axis=1)<jupyter_output><empty_output><jupyter_text>Test:<jupyter_code>twitter_full.info()
twitter_full[twitter_full.text.str.contains(r'(\d+\.\d+)/(\d+)')]<jupyter_output>/Users/fazliddinibodullaev/opt/anaconda3/lib/python3.7/site-packages/pandas/core/strings.py:1954: UserWarning: This pattern has match groups. To actually get the groups, use str.extract.
  return func(self, *args, **kwargs)<jupyter_text>Define:- Fraction for rating numerator and rating denominator. Solving: create a new column called fraction.  Code:<jupyter_code>#measuring the fraction of two columns.
#https://stackoverflow.com/questions/40353079/pandas-how-to-check-dtype-for-all-columns-in-a-dataframe
twitter_full['fraction'] = twitter_full['rating_numerator_n'].astype(float)/twitter_full['rating_denominator']<jupyter_output><empty_output><jupyter_text>Test:<jupyter_code>twitter_full.fraction.value_counts()<jupyter_output><empty_output><jupyter_text>Define:- Remove underscores and make them lowercase of p1, p2 and p3. Solving: Replace the underscores with space and make them all lowercase  Code:<jupyter_code>twitter_full.dog_breed = twitter_full.dog_breed.str.replace('_', ' ')
twitter_full.dog_breed = twitter_full.dog_breed.str.lower()<jupyter_output><empty_output><jupyter_text>Test:<jupyter_code>twitter_full.dog_breed.value_counts()<jupyter_output><empty_output><jupyter_text>Store the clean data.<jupyter_code>twitter_full.to_csv('twitter_archive_master.csv', encoding='utf-8', index=False)<jupyter_output><empty_output><jupyter_text>Analyzing and visualizing Data<jupyter_code>df = pd.read_csv('twitter_archive_master.csv')
df.info()
df.describe()<jupyter_output><empty_output><jupyter_text>Number of tweets in every month?<jupyter_code>#Changing the data types of variables so we can work with libraries properly
df['source'] = df['source'].astype('category')
df['tweet_id'] = df['tweet_id'].astype('str')
df['timestamp'] = pd.to_datetime(df['timestamp'])
plt.rcParams['figure.figsize'] = (14, 7) #For bigger plot sizes
picked_data = df['tweet_id'].groupby([df['timestamp'].dt.year, df['timestamp'].dt.month]).count()
picked_data.plot(kind = 'line')
plt.title('Number of tweets in every month', size = 15)
plt.xlabel('Time (Year, Month)')
plt.ylabel('Number of tweets')
plt.savefig('number_of_tweets_every_month')<jupyter_output><empty_output><jupyter_text>As you can see, December 2015 has the most of tweets. After that period, number tweets decreased drastically. It carried on decreasing till July 2017.  The most used tool?<jupyter_code>df.source.value_counts()
sns.countplot(data=df, x='source')
plt.title('Sources of Tweets', size=15)
plt.savefig('most_used_source_of_tweets')<jupyter_output><empty_output><jupyter_text>Apparently, the most used source is Iphone. Second and third places are for Twitter Web Client and TweetDeck respectively.  How many dogs have ratings which are over than 10?<jupyter_code>df.rating_numerator_n.value_counts().sort_index().plot(kind = 'bar', color = 'orange')
plt.title('Rating numerator distribution', size = 15)
plt.xlabel('Rating numerator')
plt.ylabel('Rating denominator')
plt.savefig('rating_numerator_distribution')
#Selecting ratings which are over than 10
round(df.rating_numerator_n[df.rating_numerator_n > 10]).count()<jupyter_output><empty_output><jupyter_text>1156 dogs have ratings which are over than 10.  What are the popular dog breeds?<jupyter_code>df.dog_breed.value_counts()[0:10].sort_values(ascending = False).plot(kind= 'bar', color = 'maroon')
plt.ylabel('Number of breeds')
plt.xlabel('Dog breed')
plt.title('Popular 10 dog breeds')
plt.savefig('top_dog_breeds')<jupyter_output><empty_output><jupyter_text>Golden retriever is the most popular breed. Whereas malamute is the least.  The most popular dog names?<jupyter_code>df.name.value_counts()[0:10].sort_values(ascending = False).plot(kind = 'bar', color = 'green')
plt.ylabel('Number of dogs')
plt.xlabel('Names of dogs')
plt.title('Dog names')
plt.savefig('popular_dog_names')<jupyter_output><empty_output><jupyter_text>Charlie is the most popular name. Meanwhile Lucy, Cooper, Oliver and Tucker are also famous.  The most frequent dog stage?<jupyter_code>df.dog_type.value_counts()
sns.countplot(data=df, x='dog_type')
plt.ylabel('Number of dog stages')
plt.xlabel('Names of dog stages')
plt.title('Dog stages')
plt.savefig('frequent_dog_stages')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09454065644070592, 'fraction_numerical': 0.05964044202539997, 'mean_word_length': 3.5471726413679314, 'pattern_counts': {'":': 1, '<': 196, '<?xml version=': 0, '>': 197, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '49', 'dir': 'jupyter-structured-clean-dedup', 'id': '589665', 'license': 'MIT', 'n_tokens_mistral': 11314, 'n_tokens_neox': 9954, 'n_words': 2763, 'path': 'wrangle_act.ipynb', 'repo_name': 'Ibodullaev/Wrangle-and-Analyze-Data', 'size': '30317'}