<jupyter_start><jupyter_text>Gathering Data1. 'twitter_archive_enhanced.csv' (Manually downloading from https://d17h27t6h515a5.cloudfront.net/topher/2017/August/59a4e958_twitter-archive-enhanced/twitter-archive-enhanced.csv)2. 'image-predictions.tsv' obatined programmatically from the given url using requests library3. 'tweet_json.txt' file obtained form the resources section of the project.  1. Gather` image-predictions.tsv` file form the url<jupyter_code># Download 'image-predictions.tsv' file from the link programmatically
url = "https://d17h27t6h515a5.cloudfront.net/topher/2017/August/599fd2ad_image-predictions/image-predictions.tsv"
with requests.Session() as s:
    response = s.get(url)
    if response:
        print('Requests get() method successful')
    else:
        print('Error has occurred')
#check if the encoding is in 'utf-8' or not
    print("Encoding of obtained content: ",response.headers['content-type'].split(';')[-1])
    with open(os.path.join(os.getcwd(), url.split('/')[-1]),'wb') as file:
        file.write(response.content)
        print("Writing of the content successful..!")
image_predictions = pd.read_csv('image-predictions.tsv', delimiter='\t')<jupyter_output><empty_output><jupyter_text>2. Read `twitter_archive_enhanced` csv file<jupyter_code># create dataframe of  'twitter-archive-enhanced' and 'image-predictions' and 'twitter-json' data
twitter_archive_enhanced = pd.read_csv('twitter-archive-enhanced.csv', delimiter=',')<jupyter_output><empty_output><jupyter_text>3. Read `tweet_json` file into a dataframe<jupyter_code>tweet_json = pd.read_json('tweet-json.txt', orient='records', lines=True)<jupyter_output><empty_output><jupyter_text>Assessment  1. `twitter_archive_enhanced`<jupyter_code>twitter_archive_enhanced
twitter_archive_enhanced.columns
twitter_archive_enhanced.sample(5)
twitter_archive_enhanced.info()
print(twitter_archive_enhanced['text'][1042])
print(twitter_archive_enhanced['text'][2339])<jupyter_output>When you're given AUX cord privileges from the back seat and accidentally start blasting an audiobook... both 10/10 https://t.co/gCCrY8P0K9
Oh my. Here you are seeing an Adobe Setter giving birth to twins!!! The world is an amazing place. 11/10 https://t.co/11LvqN4WLq<jupyter_text>Many records of the tweet's columns: doggo, floofer, pupper, puppo , do not contain any value excpet 'None'. and they are string data type. They could be of category data type.<jupyter_code># Check for the duplication of records
twitter_archive_enhanced[twitter_archive_enhanced.duplicated()]
# create a function to obtain value_counts of the columns
def count_values(col_names):
    if type(col_names) is str:
        return twitter_archive_enhanced[col_names].value_counts()
    if type(col_names) is list:
        doggy_data=[]
        for col in col_names:
            doggy_data.append(twitter_archive_enhanced[col].value_counts())
        return doggy_data
source_val_counts = count_values('source')
print(source_val_counts)<jupyter_output><a href="http://twitter.com/download/iphone" rel="nofollow">Twitter for iPhone</a>     2221
<a href="http://vine.co" rel="nofollow">Vine - Make a Scene</a>                          91
<a href="http://twitter.com" rel="nofollow">Twitter Web Client</a>                       33
<a href="https://about.twitter.com/products/tweetdeck" rel="nofollow">TweetDeck</a>      11
Name: source, dtype: int64<jupyter_text>The `source` column contain redundant HTMl informations.<jupyter_code>name_val_counts = count_values('name')
print(name_val_counts)<jupyter_output>None       745
a           55
Charlie     12
Cooper      11
Oliver      11
Lucy        11
Penny       10
Lola        10
Tucker      10
Winston      9
Bo           9
Sadie        8
the          8
Toby         7
Buddy        7
Daisy        7
an           7
Bailey       7
Oscar        6
Dave         6
Milo         6
Stanley      6
Jax          6
Leo          6
Koda         6
Scout        6
Rusty        6
Jack         6
Bella        6
George       5
          ... 
Baron        1
Mitch        1
Maya         1
Mary         1
Jareld       1
Batdog       1
Edgar        1
Timmy        1
Mingus       1
Rueben       1
Theo         1
Clyde        1
Benny        1
Buddah       1
Tupawc       1
Tuco         1
Flash        1
Mike         1
Tiger        1
Taco         1
Chef         1
Bilbo        1
Murphy       1
Blipson      1
Ember        1
Shiloh       1
Pepper       1
Glacier      1
Norman       1
Fiji         1
Name: name, Length: 957, dtype: int64<jupyter_text>The `name` column contain wrong names or literary words/characters like 'a', 'Bo','the',and 'an' .<jupyter_code>numer_val_counts = count_values('rating_numerator')
print(numer_val_counts)
# check the text and url in it of the index 1869
print(twitter_archive_enhanced.query('rating_numerator == 1')['text'][1869])<jupyter_output>What kind of person sends in a picture without a dog in it? 1/10 just because that's a nice table https://t.co/RDXCfk8hK0<jupyter_text>`rating_numerator` does not seem to contain any quality or tidy issues<jupyter_code>denom_val_counts = count_values('rating_denominator')
print(denom_val_counts)
twitter_archive_enhanced.query('rating_denominator == 11')
print(twitter_archive_enhanced.query('rating_denominator == 11')['text'][1068])<jupyter_output>After so many requests, this is Bretagne. She was the last surviving 9/11 search dog, and our second ever 14/10. RIP https://t.co/XAVDNDaVgQ<jupyter_text>The numerator and denominator contains the wrong values comparing with the `text` column. 9/11 refers to the event of plane crashes in New York but the true value is actually 14/10 (numerator/denominator) for the index 1068.<jupyter_code># Dog stages frequency counts
dog_stages = ["doggo", "floofer", "pupper", "puppo"]
dog_stages_val_counts = count_values(dog_stages)
print("Doggo's value counts: ", dog_stages_val_counts[0])
print("==================================================")
print("\nFloofer's value counts: ", dog_stages_val_counts[0])
print("==================================================")
print("\nPupper's value counts: ", dog_stages_val_counts[0])
print("==================================================")
print("\nPupper's value counts: ", dog_stages_val_counts[0])
print("==================================================")<jupyter_output>Doggo's value counts:  None     2259
doggo      97
Name: doggo, dtype: int64
==================================================

Floofer's value counts:  None     2259
doggo      97
Name: doggo, dtype: int64
==================================================

Pupper's value counts:  None     2259
doggo      97
Name: doggo, dtype: int64
==================================================

Pupper's value counts:  None     2259
doggo      97
Name: doggo, dtype: int64
==================================================<jupyter_text>2. `image_predictions`<jupyter_code>image_predictions
image_predictions.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 2075 entries, 0 to 2074
Data columns (total 12 columns):
tweet_id    2075 non-null int64
jpg_url     2075 non-null object
img_num     2075 non-null int64
p1          2075 non-null object
p1_conf     2075 non-null float64
p1_dog      2075 non-null bool
p2          2075 non-null object
p2_conf     2075 non-null float64
p2_dog      2075 non-null bool
p3          2075 non-null object
p3_conf     2075 non-null float64
p3_dog      2075 non-null bool
dtypes: bool(3), float64(3), int64(2), object(4)
memory usage: 152.1+ KB<jupyter_text>- There are no null values in the dataframe<jupyter_code># Check the validity of the url of the image
print(image_predictions['jpg_url'][2074])
# check for duplicity in tweet_id's
image_predictions[image_predictions.tweet_id.duplicated()]
# check for duplicity in jpg_urls
image_predictions.duplicated(subset=['jpg_url']).sum()<jupyter_output><empty_output><jupyter_text>The images of other animals except dogs are also availableThere is inconsisitency in the predicted string's letter case.Some are lower case while others are not.The predicted dog breed is of `object` (i.e string) datatype, category type may be suitable.Also the jpg_url contain duplicated data.  3. `tweet-json` data<jupyter_code>tweet_json
tweet_json.columns
tweet_json.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 2354 entries, 0 to 2353
Data columns (total 31 columns):
contributors                     0 non-null float64
coordinates                      0 non-null float64
created_at                       2354 non-null datetime64[ns]
display_text_range               2354 non-null object
entities                         2354 non-null object
extended_entities                2073 non-null object
favorite_count                   2354 non-null int64
favorited                        2354 non-null bool
full_text                        2354 non-null object
geo                              0 non-null float64
id                               2354 non-null int64
id_str                           2354 non-null int64
in_reply_to_screen_name          78 non-null object
in_reply_to_status_id            78 non-null float64
in_reply_to_status_id_str        78 non-null float64
in_reply_to_user_id              78 non-null float64
in_reply_to_user_id_str          78 n[...]<jupyter_text>There are 0 non-null data in `contributors` , `coordinates` and `geo` columns whereas only 1 non-null data in `place` columns.<jupyter_code>tweet_json['full_text'][0]
tweet_json['extended_entities'][0]
tweet_json['lang'].value_counts()<jupyter_output><empty_output><jupyter_text>11. `lang` column is of `object` datatype.It may be category type.12.  Remove unwanted column names **Quality Issues**`twitter_archive_enhanced`- `timestamp` is a string (i.e object) which should be `datetime` type.- Unnecessary retweet records.- Many records of the tweet's columns: doggo, floofer, pupper, puppo , do not contain any value excpet 'None'. and they are string data type. They could be of category data type.- The `source` column contain redundant HTMl informations.- The `name` column contain wrong names or literary words/characters like 'a', 'Bo','the',and 'an' .- The numerator and denominator contains the wrong values when compared with the `text` column.`image_predictions`- There are duplicated jpg_urls.- There is inconsisitency in the predicted string's letter case. Some are lower case while others are not.- The predicted dog breed is of `object` (i.e string) datatype, category type may be suitable.- The underscore symbol in place of space in the dog bredd's name`tweet-json` - Unwanted columns- There are 0 non-null data in `contributors` , `coordinates` and `geo` columns whereas only 1 non-null data in    `place` columns. **Tidiness Issues**`twitter_archive_enhanced`- The dog stages: `doggo`, `floofer`, `pupper`, `puppo` are in different columns.`twitter_json`- the `display_text_range` contains a list of numerical data`image_predictions`- There are three different predictions breeds and their confidence values in three different columns`All three`- Merge all three dataframe into a master dataframe  Data Cleaning  Quality 1: `twitter_archive_enhanced` Datatype conversion **Define:** Convert `timestamp` from string to datetime type and dogs stages: [`doggo, floofer, pupper, puppo`] to category type **Code**<jupyter_code>from datetime import datetime
tae_clean = twitter_archive_enhanced.copy()
# convert timestamp's datatype to 'datetime' type
tae_clean['timestamp'] =  pd.to_datetime(tae_clean['timestamp'])
tae_clean['retweeted_status_timestamp'] = pd.to_datetime(tae_clean['retweeted_status_timestamp'])
# convert dog stages name columns into category type
tae_clean['doggo']= tae_clean['doggo'].astype("category")
tae_clean['floofer']= tae_clean['floofer'].astype("category")
tae_clean['pupper']= tae_clean['pupper'].astype("category")
tae_clean['puppo']= tae_clean['puppo'].astype("category")<jupyter_output><empty_output><jupyter_text>**Test**<jupyter_code># oberve the changes made on the datatype
tae_clean.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 2356 entries, 0 to 2355
Data columns (total 17 columns):
tweet_id                      2356 non-null int64
in_reply_to_status_id         78 non-null float64
in_reply_to_user_id           78 non-null float64
timestamp                     2356 non-null datetime64[ns]
source                        2356 non-null object
text                          2356 non-null object
retweeted_status_id           181 non-null float64
retweeted_status_user_id      181 non-null float64
retweeted_status_timestamp    181 non-null datetime64[ns]
expanded_urls                 2297 non-null object
rating_numerator              2356 non-null int64
rating_denominator            2356 non-null int64
name                          2356 non-null object
doggo                         2356 non-null category
floofer                       2356 non-null category
pupper                        2356 non-null category
puppo                         2356 non-null category
dtypes: [...]<jupyter_text>Quality 2 :`twitter_archive_enhanced` Unnecessary Retweet Data **Define**  Dropping the records of 181 non-null retweets along with the columns**Code**<jupyter_code># update the dataframe with only the null values of retweets
tae_clean = tae_clean.query("retweeted_status_user_id == 'NAN'")
# Before removing
tae_clean.columns
# Drop all the unnecessary columns of dataframe
tae_clean.drop(['retweeted_status_user_id','retweeted_status_id','retweeted_status_timestamp'],axis=1,inplace=True)<jupyter_output><empty_output><jupyter_text>**Test**<jupyter_code># After removing retweet columns
tae_clean.columns<jupyter_output><empty_output><jupyter_text>Quality 3: `image_archive_enhanced` 's Unwanted data removal**Define:**  The html tags and elements are supefluous to what we require so they may be removed apart from the source data. **Code**<jupyter_code># Original unique elements in 'source' column
tae_clean['source'].value_counts()
# check the html data and possbile approach to extract the source info
print(tae_clean.source[0])
print(tae_clean.source[972])
# Approach to extract all 'source' name
print("\n",tae_clean.source[972].split('>')[-2][:-3])
# Extract 'source' name from the html tag
source =[]
for html in tae_clean.source:
    source.append(html.split('>')[-2][:-3])
# make sure the source list is of same length as that of tae-clean['source'] series
len(source) == len(tae_clean['source'])
# Update the 'source' column with the extracted source info from the original column
tae_clean['source'] = source<jupyter_output><empty_output><jupyter_text>**Test**<jupyter_code>tae_clean['source'].sample(5)
tae_clean['source'].value_counts()<jupyter_output><empty_output><jupyter_text>Quality 4: Inappropraite and lowercase name in `name` column**Define:** The names in the names column contain lowercase and unnatural names like 'a', 'an', 'the, 'None' , which must be replaced with real dog names.**Code**<jupyter_code>tae_clean['name'].value_counts()
# Find the lowercase names, which may be inappropraite names
lowercase_dog_names=[]
for n in tae_clean['name']:
    if n.islower() and n not in lowercase_dog_names:
        lowercase_dog_names.append(n)
print(lowercase_dog_names)<jupyter_output>['such', 'a', 'quite', 'not', 'one', 'incredibly', 'very', 'my', 'his', 'an', 'actually', 'just', 'getting', 'mad', 'this', 'unacceptable', 'all', 'old', 'infuriating', 'the', 'by', 'officially', 'life', 'light', 'space']<jupyter_text>- Here, all the lowercase names are inappropriate names of dogs.<jupyter_code># Replace these inappropriate names with 'NaN' and 'None' with 'Nan' as well.
tae_clean['name'].replace(lowercase_dog_names, np.nan, inplace= True)
tae_clean['name'].replace('None', np.nan, inplace=True)<jupyter_output><empty_output><jupyter_text>**Test**<jupyter_code># After cleaning inappropriate names
tae_clean.name.value_counts()<jupyter_output><empty_output><jupyter_text>Quality 5: `twitter_archive_enhanced` numerator values correction**Define**The numerator/denominator values in the `text` column do not correspond to the `rating_numerator`**Code**The numerator decimals in the text are extracted to change the numerator's datatype from int to float<jupyter_code>tae_clean.columns
# make a copy of the data for this task
num_denom = tae_clean.copy()
# create a regex pattern that corresponds to values like '13.5/10' and extract them to a list
dec_val_pattern = "(\d+\.\d*\/\d+)"
dec_val = num_denom[num_denom.text.str.contains(dec_val_pattern)]
dec_numer = []
for record in dec_val['text']:
    first_split = record.split('/')
    dec_numer.append(first_split[0].split()[-1])
print(dec_numer)
# creating a list of tweet_id corresponding to the decimal numerator obtained above
tweet_id_val = dec_val['tweet_id'].tolist()
tweet_id_val
# Update the rating_numerator values in tae_clean data for the above 'tweet_id'
for i in range(len(tweet_id_val)):
    tae_clean.loc[(tae_clean['tweet_id']== tweet_id_val[i]), ['rating_numerator']] = float(dec_numer[i])
tae_clean.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 2175 entries, 0 to 2355
Data columns (total 14 columns):
tweet_id                 2175 non-null int64
in_reply_to_status_id    78 non-null float64
in_reply_to_user_id      78 non-null float64
timestamp                2175 non-null datetime64[ns]
source                   2175 non-null object
text                     2175 non-null object
expanded_urls            2117 non-null object
rating_numerator         2175 non-null float64
rating_denominator       2175 non-null int64
name                     1391 non-null object
doggo                    2175 non-null category
floofer                  2175 non-null category
pupper                   2175 non-null category
puppo                    2175 non-null category
dtypes: category(4), datetime64[ns](1), float64(3), int64(2), object(4)
memory usage: 275.8+ KB<jupyter_text>Quality 6 :`twitter-json`  drop columns**Define**- Drop superfluous column and extract necessary columns from the twitter json data  set**Code**<jupyter_code>tweet_json.columns
# Tak eonly the required columns from the twitter_json dataset
twitter_api = tweet_json[['id', 'display_text_range', 'favorite_count', 'retweet_count', 'retweeted']]<jupyter_output><empty_output><jupyter_text>**Test**<jupyter_code>twitter_api.head()<jupyter_output><empty_output><jupyter_text>Quality 7:`image_predictions` : Drop duplicate jpg_urls **Define:** Drop the duplicated urls in the jpg_urls columns**Code**<jupyter_code>image_predictions.sample(3)
image_predictions[image_predictions.duplicated(subset=['jpg_url'])]
# drop duplicate data of jpg_url
image_predictions.drop_duplicates(subset=['jpg_url'], keep='last', inplace=True)<jupyter_output><empty_output><jupyter_text>**Test**<jupyter_code># check if there are duplicated data or not
image_predictions[image_predictions.duplicated(subset=['jpg_url'])]<jupyter_output><empty_output><jupyter_text>Quality 8: `image_prediction`'s prediction datatype **Define**Change the prediction p1,p2 and p3 into `category` datatype**Code**<jupyter_code>image_predictions.info()
image_predictions['p1'] = image_predictions['p1'] .astype('category')
image_predictions['p2'] = image_predictions['p2'] .astype('category')
image_predictions['p3'] = image_predictions['p3'] .astype('category')<jupyter_output><empty_output><jupyter_text>**Test**<jupyter_code>image_predictions.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 2009 entries, 0 to 2074
Data columns (total 12 columns):
tweet_id    2009 non-null int64
jpg_url     2009 non-null object
img_num     2009 non-null int64
p1          2009 non-null category
p1_conf     2009 non-null float64
p1_dog      2009 non-null bool
p2          2009 non-null category
p2_conf     2009 non-null float64
p2_dog      2009 non-null bool
p3          2009 non-null category
p3_conf     2009 non-null float64
p3_dog      2009 non-null bool
dtypes: bool(3), category(3), float64(3), int64(2), object(1)
memory usage: 186.8+ KB<jupyter_text>Quality 9: `image_prediction` Inconsistent Dog breeds name **Define**Some of the breeds name predicted are written in lowercase whereas others in uppercase. Remove the underscore sign from between the words.**Code**<jupyter_code>image_predictions['p1'].replace(to_replace="_", value=r' ',regex=True,inplace=True)
image_predictions['p2'].replace(to_replace="_", value=r' ',regex=True,inplace=True)
image_predictions['p3'].replace(to_replace="_", value=r' ',regex=True,inplace=True)
image_predictions['p1'] = image_predictions.p1.str.capitalize()
image_predictions['p2'] = image_predictions.p1.str.capitalize()
image_predictions['p3'] = image_predictions.p1.str.capitalize()<jupyter_output><empty_output><jupyter_text>**Test**<jupyter_code>image_predictions.sample(3)<jupyter_output><empty_output><jupyter_text>Tidiness 1 : `image_predictions` Single prediction dog breed with high confidence **Define** Take only the dog name with highest confidence value by comparing the bool values and confidence values**Code**<jupyter_code># function to extract dog's breed
def get_dog_breed(record):
    if record['p1_dog'] == True:
        return record['p1']#, record['p1_conf']
    elif record['p2_dog'] == True:
        return record['p2']#, record['p2_conf']
    elif record['p3_dog'] == True:
        return record['p3']#, record['p3_conf']
    else:
        return np.nan
# function to extract the same dog's confidence level
def get_dog_confidence(record):
    if record['p1_dog'] == True:
        return record['p1_conf']
    elif record['p2_dog'] == True:
        return record['p2_conf']
    elif record['p3_dog'] == True:
        return record['p3_conf']
    else:
        return np.nan#, np.nan
# Extract dog's breed name and confidence value
breed= image_predictions.apply(lambda row: get_dog_breed(row), axis = 1)
confidence_level = image_predictions.apply(lambda row: get_dog_confidence(row), axis = 1)
# Assign those above list to a new columns in image_predictions data
image_predictions['breed_predicted'] = breed
image_predictions['confidence'] = confidence_level
# Drop unwanted columns from image_predictions
image_predictions.drop(['p1', 'p1_conf', 'p1_dog', 'p2', 'p2_conf', 'p2_dog', 'p3', 'p3_conf', 'p3_dog'], axis=1,inplace=True)<jupyter_output><empty_output><jupyter_text>**Test**<jupyter_code>image_predictions.columns<jupyter_output><empty_output><jupyter_text>Tidiness 2: `twitter_json` data: Create two more columns**Define:** Divide display_text_range column into two columns: min_text_range, and max_text_range**Code**<jupyter_code>twitter_api.info()
# apply function to index first value for min_text_range and second value for max_text_range
twitter_api['min_text_range'] = twitter_api['display_text_range'].apply(lambda x: x[0])
twitter_api['max_text_range'] = twitter_api['display_text_range'].apply(lambda x: x[1])
twitter_api.drop('display_text_range', axis=1, inplace=True)
twitter_api.head()<jupyter_output><empty_output><jupyter_text>Tidiness 3:`twitter_archive_enhanced` dogs stages name category**Define:** The slang words for dogs stages like `doggo`,`floofer`, `pupper`, `puppo` have different columns that can be made into one.**Code**<jupyter_code>tae_clean.columns
melting_cols = tae_clean.loc[:,'doggo':'puppo']
remaining_cols = tae_clean.loc[:,'tweet_id':'name']

tae_clean = pd.melt(tae_clean, id_vars = remaining_cols, value_vars= melting_cols, var_name= 'loving_name', value_name='stage_name' )
# drop var_name : 'loving_names'
tae_clean.drop('loving_name',axis=1, inplace=True)
tae_clean['stage_name'].value_counts()
# convert 'slang_names' to category type
tae_clean['stage_name'] = tae_clean['stage_name'].astype('category')
tae_clean[tae_clean.duplicated(subset='tweet_id')]
# As the first duplicated values are all 'None' for 'stage_name', we will keep the last one
tae_clean = tae_clean.sort_values('stage_name').drop_duplicates('tweet_id', keep = 'last')
# Check if duplicated rows are deleted or not
tae_clean[tae_clean.duplicated(subset='tweet_id')]
# Replace 'None' value to 'Nan'
tae_clean['stage_name'].replace('None', np.nan, inplace=True)
# Check if 'None' value is deleted or not
tae_clean['stage_name'].value_counts()<jupyter_output><empty_output><jupyter_text>**Test**<jupyter_code>tae_clean.sample(3)<jupyter_output><empty_output><jupyter_text>Tidiness 3: Merge all 3 datasets**Define**Merge all three data sets into a master data set.**Code**<jupyter_code># First merge two cleaned data sets: 'tae_clean'i.e twitter_archive_enhanced and 'twitter_api'.
twitter_master_temp = pd.merge(left=tae_clean,right=twitter_api, left_on='tweet_id', right_on='id', how = 'left')
# Then merge above temporary master (twitter_master_temp) with 'image_predicitons' data set
twitter_master = pd.merge(left=twitter_master_temp,right=image_predictions, left_on='tweet_id', right_on='tweet_id', how = 'left')<jupyter_output><empty_output><jupyter_text>Quality 10: Final cleaning, drop `id` column and make a combined `rating`<jupyter_code># Drop the extra 'id' column
twitter_master.drop('id', axis=1, inplace=True)
# make a combine rating of numerator and denominator
twitter_master['rating'] = twitter_master['rating_numerator']/twitter_master['rating_denominator']
twitter_master.rating.head<jupyter_output><empty_output><jupyter_text>Storing the master data to a csv file<jupyter_code># Save to csv file 'twitter_master_archive.csv'
twitter_master.to_csv('twitter_master_archive.csv',index=False)<jupyter_output><empty_output><jupyter_text>Analyzing and Visualization  Insight 1 : The most frequent dog breed that was tweeted about<jupyter_code># twitter_df = pd.read_csv('twitter_master_archive.csv', delimiter=',')
twitter_df = twitter_master.copy()
twitter_df.head()
twitter_df.columns
twitter_df.breed_predicted.value_counts()[:15]
fig, ax = plt.subplots(figsize = (8,6))
ax = sns.countplot(x = "breed_predicted", data = twitter_df, order=twitter_df.breed_predicted.value_counts().iloc[:15].index, palette = "Reds_r")
plt.xticks(rotation=90)
plt.title("Absolute frequency of top 15 dog breeds")<jupyter_output><empty_output><jupyter_text>Now we want to have better insights based on proportions rather than just frequency count<jupyter_code>twitter_df.shape[0]
# Ref: https://classroom.udacity.com/nanodegrees/nd002-ent/parts/9c2eb5bf-2bda-4e0c-85bf-d997082a8571/modules/9b9bd2f7-e891-4545-adb0-e57eb8efba81/lessons/24200b64-07aa-4526-b558-df1ae31faa7c/concepts/bd9948e9-eda5-48b7-8c6f-e97192a9ac48

# get proportion taken by most common group for derivation
fig, ax = plt.subplots(figsize = (10,6))
n_points = twitter_df.shape[0]
max_count = twitter_df.breed_predicted.value_counts().max()
max_prop = max_count / n_points

# generate tick mark locations and names
tick_props = np.arange(0, max_prop, 0.004)
tick_names = ['{:0.2f}'.format(v) for v in tick_props]

# create the plot
sns.countplot(data = twitter_df, x = 'breed_predicted',order=twitter_df.breed_predicted.value_counts().iloc[:15].index, palette = "Reds_r")
plt.yticks(tick_props * n_points, tick_names)
plt.ylabel('proportion')
plt.xticks(rotation=90)
plt.title("Relative frequency of top 15 dog breed")<jupyter_output><empty_output><jupyter_text>**Insight 1 conclusion:** Golden retriever is the most frequently tweeted dog breed.  Insight 2: Rating over the years<jupyter_code>twitter_df.columns
# extract year and rating from twitter_df
year = twitter_df['timestamp'].dt.year
rating = twitter_df['rating']

# inintialize a pandas dataframe
year_rating = pd.DataFrame()
year_rating['year'] = year
year_rating['rating'] = rating
# As there was an infinite value, we replace it with nan value and then that row is dropped
year_rating['rating'].replace(np.inf, np.nan, inplace=True)
year_rating.dropna(axis=0, inplace=True)
year_rating['rating'].describe()
sns.pointplot(x='year',y='rating',data=year_rating)
plt.title('Rating over the years with their average value')<jupyter_output><empty_output><jupyter_text>**Insight 2 conclusion:** The rating increased from 2015 to 2017 with almost 0.6 difference on their average value.  Insight 3: realtion between dogs stages , favourite_count, and retweet_count **Question 1: Which breed and stage name of dogs has the highest ever favorite and retweet count in a single tweet?**<jupyter_code>coveted_cols = [['stage_name', 'favorite_count','retweet_count', 'retweeted']]
# Create a new dataframe and fill it with data from twitter_df
dog_stage_df = pd.DataFrame()
for col in coveted_cols:
    dog_stage_df[col] = twitter_df[col]
# value counts of stage names
dog_stage_df.stage_name.value_counts()
# Check which stage_name and index hsa the highest favorite count
idx_max_fav = dog_stage_df.favorite_count.idxmax()
dog_stage_df[dog_stage_df.index == idx_max_fav ]
# Check which stage_name and index hsa the highest retweet count
dog_stage_df[dog_stage_df.index == dog_stage_df.retweet_count.idxmax()]
# Breed with highest favorite count in a single tweet
twitter_df[twitter_df.index == 2158]['breed_predicted']
# # Breed with highest retweet count in a single tweet
twitter_df[twitter_df.index == 1876]['breed_predicted']<jupyter_output><empty_output><jupyter_text>**Relation between retweet and favorite counts****Question 2: How are favorite count and retweet count related?**<jupyter_code>fig, ax = plt.subplots(figsize=(10,6))
sns.regplot(x="favorite_count", y="retweet_count", data=dog_stage_df)
plt.title("Regression plot")
corr = dog_stage_df.corr().loc['retweet_count','favorite_count']
print("Correlation between'favorite_count' and  'retweet_count' is: ", corr)<jupyter_output>Correlation between'favorite_count' and  'retweet_count' is:  0.911731653356
{'fraction_non_alphanumeric': 0.11842636180228648, 'fraction_numerical': 0.03133826496301278, 'mean_word_length': 5.022883758606723, 'pattern_counts': {'":': 0, '<': 177, '<?xml version=': 0, '>': 179, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '44', 'dir': 'jupyter-structured-clean-dedup', 'id': '282262', 'license': 'MIT', 'n_tokens_mistral': 10214, 'n_tokens_neox': 9496, 'n_words': 2526, 'path': 'Part 4 Data Wrangling/project-4 wrangling/wrangle_act.ipynb', 'repo_name': 'bimarsha7/Data-analysis', 'size': '29740'}