<jupyter_start><jupyter_text>@WeRateDogs  Gather<jupyter_code>import pandas as pd
import requests
import io
import tweepy
import timeit
import time
import json
import numpy as np
import matplotlib.pyplot as plt
# Read haded CSV into DataFrame
twitter_archive_enhanced = pd.read_csv('twitter-archive-enhanced.csv')
# Request image_predictions.tsv into DataFrame
url = 'https://d17h27t6h515a5.cloudfront.net/topher/2017/August/599fd2ad_image-predictions/image-predictions.tsv'
with requests.Session() as s:
    data = s.get(url)    
    decoded_content = data.content.decode('utf-8')
    image_predictions = pd.read_csv(io.StringIO(data.text), '\t')
# Using the tweet IDs in the WeRateDogs Twitter archive,
tweet_id = twitter_archive_enhanced.tweet_id.tolist()
# query the Twitter API...
consumer_key = 'xxx'
consumer_secret = 'xxx'
access_token = 'xxx'
access_secret = 'xxx'
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_secret)
twitter = tweepy.API(auth, wait_on_rate_limit=True, wait_on_rate_limit_notify=True)
# for each tweet's JSON data using Python's Tweepy library...
json_list = []
status_errors = []
for x in tweet_id:
    try:
        start = timeit.timeit()
        tweet = api.get_status(x, tweet_mode='extended')
        json_list.append(tweet._json)
        print(tweet.full_text)
        time.sleep(0.5)
        end = timeit.timeit()
        print(end - start)
    except Exception as e:
        print('\033[93m' + str(x) + ": " + str(e) + '\033[0m')
        status_errors.append(x)
# and store each tweet's entire set of JSON data in a file called tweet_json.txt file. 
# Each tweet's JSON data should be written to its own line.
with open('tweet_json.txt', 'w') as file:
    for item in json_list:
        json.dump(item, file)
        file.write('\n')
# Then read this .txt file line by line into a pandas DataFrame with (at minimum): 
# tweet ID, retweet count, and favorite count.
df_list = []
with open('tweet_json.txt') as file:
    for line in file:
        json_line = json.loads(line)
        id = json_line['id']
        retweet_count = json_line['retweet_count']
        favorite_count = json_line['favorite_count']
        retweeted = json_line['retweeted']
        created_at = json_line['created_at']
        # Append to list of dictionaries
        df_list.append({'tweet_id': id,
                        'retweet_count': retweet_count,
                        'favorite_count': favorite_count,
                        'retweeted': retweeted,
                        'created_at': created_at})
    social_data = pd.DataFrame(df_list, columns = ['tweet_id', 'retweet_count', 'favorite_count', 'retweeted', 'created_at'])<jupyter_output><empty_output><jupyter_text>Assess<jupyter_code>twitter_archive_enhanced.head()
twitter_archive_enhanced.info()
twitter_archive_enhanced['retweeted_status_id'].notnull().value_counts()
twitter_archive_enhanced.loc[twitter_archive_enhanced['retweeted_status_id'].notnull()]
twitter_archive_enhanced['retweeted_status_user_id'].notnull().value_counts()
twitter_archive_enhanced['retweeted_status_user_id'].notnull().value_counts()
twitter_archive_enhanced.loc[(twitter_archive_enhanced.rating_denominator != 10) & \
                             (twitter_archive_enhanced.expanded_urls.notnull()) & \
                             (twitter_archive_enhanced.retweeted_status_id.isnull())]                
twitter_archive_enhanced.loc[twitter_archive_enhanced.text.str.contains('\d\.\d\/'),['tweet_id','text','rating_numerator']]
twitter_archive_enhanced.sample(5)
twitter_archive_enhanced.tweet_id.duplicated().value_counts()
image_predictions.info()
image_predictions.sample(10)
image_predictions.tweet_id.duplicated().value_counts()
social_data.info()
social_data.head()
social_data.sample(7)
social_data.retweeted.value_counts()
social_data.tweet_id.duplicated().value_counts()
social_data.describe()<jupyter_output><empty_output><jupyter_text>Quality *twitter_archive_enhanced* table- Some tweets have more than one value for the dog stages.- In columns `retweeted_status_id`, `retweeted_status_user_id`, and `retweeted_status_timestamp` there are 181 not-null values (numbers like 8.860537e+17). In other words, not all are dog ratings and some are retweets.- In the column `expanded_urls` there are 59 null values.- Issues with fractional rating numerators: tweet_is's 883482846933004288 and 681340665377193984.- Wrong rating for `tweet_id` 716439118184652801: 50/50 instead of 11/10.- Wrong rating for `tweet_id` 682962037429899265: 7/11 instead of 10/10.- Wrong rating for `tweet_id` 666287406224695296: 1/2 instead of 9/10.- Wrong variable types for columns `timestamp, tweet_id, in_reply_to_status_id, in_reply_to_user_id, timestamp, retweeted_status_id, and retweeted_status_user_id`.- In columns `doggo`, `floofer`, `pupper` and `puppo`, values 'None' instead of nulls. *image_predictions* table- Missing data: only 2075 entries out of 2356.- Three predictions per tweet.- Some predictions are not dogs.- Non descriptive column headers `px`, `px_conf`, `px_dog`. *social_data* table- Missing data: only 1729 entries out of 2356. Tidyness *twitter_archive_enhanced* table- Four variables (columns `doggo`, `floofer`, `pupper` and `puppo`) instead of one: `dog_stage` (categorical). *image_predictions* table- Nine variables (columns `p1`, `p1_conf`, `p1_dog`, `p2`, `p2_conf`, `p2_dog`, `p3`, `p3_conf`, `p3_dog`) instead of four: `prediction_number` (categorical), `predicted_breed` (onject), `prediction_confidence` (float) and `is_dog` (boolean).- This data should go into `twitter_archive_enhanced` table. *social_data* table- This data should go into `twitter_archive_enhanced` table.  Clean<jupyter_code># Create copy
twitter_archive_clean = twitter_archive_enhanced.copy()
image_predictions_clean = image_predictions.copy()
social_data_clean = social_data.copy()<jupyter_output><empty_output><jupyter_text>Missing data: Both cases will be addressed by using a `inner merge` later on, given that we cannot do anything to gather that missing data.  *social_data* table- This data should go into `twitter_archive_enhanced` table.  Define1. Create a list with the columns to transfer from `social_data_clean` to `twitter_archive_clean`.2. Join (left) those columns of `social_data_clean` to `twitter_archive_clean` on `tweet_id`.3. Delete `social_data_clean`.  Code<jupyter_code># Code goes here
cols = ['tweet_id', 'retweet_count', 'favorite_count']
twitter_archive_clean = pd.merge(twitter_archive_clean, social_data_clean[cols], on=('tweet_id'), how='inner')
del social_data_clean<jupyter_output><empty_output><jupyter_text>Test<jupyter_code># Test code goes here
twitter_archive_clean.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 1729 entries, 0 to 1728
Data columns (total 19 columns):
tweet_id                      1729 non-null int64
in_reply_to_status_id         65 non-null float64
in_reply_to_user_id           65 non-null float64
timestamp                     1729 non-null object
source                        1729 non-null object
text                          1729 non-null object
retweeted_status_id           151 non-null float64
retweeted_status_user_id      151 non-null float64
retweeted_status_timestamp    151 non-null object
expanded_urls                 1679 non-null object
rating_numerator              1729 non-null int64
rating_denominator            1729 non-null int64
name                          1729 non-null object
doggo                         1729 non-null object
floofer                       1729 non-null object
pupper                        1729 non-null object
puppo                         1729 non-null object
retweet_count                 17[...]<jupyter_text>*image_predictions* table- Nine variables (columns `p1`, `p1_conf`, `p1_dog`, `p2`, `p2_conf`, `p2_dog`, `p3`, `p3_conf`, `p3_dog`) instead of four: `prediction_number` (categorical), `predicted_breed` (onject), `prediction_confidence` (float) and `is_dog` (boolean).  Define1. Change the name of the columns to be modified: `p1, p1_conf, p1_dog, p2, p2_conf, p2_dog, p3, p3_conf and p3_dog`.2. Use `wide_to_long` to modifie the dataframe structure.  Code<jupyter_code>image_predictions_clean = image_predictions_clean.rename(index=str, columns={"p1": "predicted_breed1", 
                                                   "p1_conf": "prediction_confidence1",
                                                   "p1_dog": "is_dog1",
                                                   "p2": "predicted_breed2", 
                                                   "p2_conf": "prediction_confidence2",
                                                   "p2_dog": "is_dog2",
                                                   "p3": "predicted_breed3", 
                                                   "p3_conf": "prediction_confidence3",
                                                   "p3_dog": "is_dog3",
                                                  })
image_predictions_clean = pd.wide_to_long(image_predictions_clean, 
                                          stubnames=['predicted_breed', 'prediction_confidence', 'is_dog'], 
                                          i='tweet_id', j='prediction_number')\
.reset_index()\
.sort_values(by=['tweet_id', 'prediction_number'])<jupyter_output><empty_output><jupyter_text>Test<jupyter_code>image_predictions_clean.head()
image_predictions_clean.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 6225 entries, 0 to 6224
Data columns (total 7 columns):
tweet_id                 6225 non-null int64
prediction_number        6225 non-null object
img_num                  6225 non-null int64
jpg_url                  6225 non-null object
predicted_breed          6225 non-null object
prediction_confidence    6225 non-null float64
is_dog                   6225 non-null bool
dtypes: bool(1), float64(1), int64(2), object(3)
memory usage: 346.5+ KB<jupyter_text>*image_predictions* table- This data should go into `twitter_archive_enhanced` table.  Define 1. Drop every prediction that isn't a dog.2. For each `tweet_id` select the prediction with the lowest `prediction_number`.3. Join (left) relevant columns (`predicted_breed` and `prediction_confidence`) to `twitter_archive_clean` on `tweet_id`.  Code<jupyter_code># Code here
image_predictions_clean = image_predictions_clean.loc[image_predictions_clean['is_dog']]
image_predictions_clean = image_predictions_clean.sort_values("prediction_number").groupby("tweet_id", as_index=False).first()\
.sort_values(by='tweet_id')
cols = ['tweet_id', 'predicted_breed', 'prediction_confidence']
twitter_archive_clean = pd.merge(twitter_archive_clean, image_predictions_clean[cols], on=('tweet_id'), how='inner')<jupyter_output><empty_output><jupyter_text>Test<jupyter_code># Test code here
twitter_archive_clean.info()
twitter_archive_clean.sample(5)
image_predictions_clean.prediction_confidence.min().astype(float)
image_predictions_clean.loc[image_predictions_clean['prediction_confidence'] == 1.00288e-05]<jupyter_output><empty_output><jupyter_text>*twitter_archive_enhanced* table- Four variables (columns `doggo`, `floofer`, `pupper` and `puppo`) instead of one: `dog_stage` (categorical).  Define 1. Create a new pivot df called `twitter_archive_clean_dog_stages`.2. Melt this df into these columns: `tweet_id`, `dog_stage`.3. Drop all the rows with value `None`.4. Join (left) this df into `twtitter_archive_clean` on `tweet_id`.5. Drop the columns `doggo`, `floofer`, `pupper` and `puppo`.6. Delete df `twitter_archive_clean_dog_stages`.  Code<jupyter_code># Code goes here
cols = ['tweet_id', 'doggo', 'floofer', 'pupper', 'puppo']
twitter_archive_clean_dog_stages = twitter_archive_clean[cols]
twitter_archive_clean_dog_stages = pd.melt(twitter_archive_clean_dog_stages, id_vars=['tweet_id'], value_name='dog_stage')
twitter_archive_clean_dog_stages = twitter_archive_clean_dog_stages[twitter_archive_clean_dog_stages.dog_stage != 'None']
twitter_archive_clean = pd.merge(twitter_archive_clean, twitter_archive_clean_dog_stages[['tweet_id', 'dog_stage']], on='tweet_id', how='left')
twitter_archive_clean = twitter_archive_clean.drop(['doggo', 'floofer', 'pupper', 'puppo'], axis=1)
del twitter_archive_clean_dog_stages<jupyter_output><empty_output><jupyter_text>Test<jupyter_code># Test code goes here
twitter_archive_clean.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 1299 entries, 0 to 1298
Data columns (total 18 columns):
tweet_id                      1299 non-null int64
in_reply_to_status_id         18 non-null float64
in_reply_to_user_id           18 non-null float64
timestamp                     1299 non-null object
source                        1299 non-null object
text                          1299 non-null object
retweeted_status_id           54 non-null float64
retweeted_status_user_id      54 non-null float64
retweeted_status_timestamp    54 non-null object
expanded_urls                 1299 non-null object
rating_numerator              1299 non-null int64
rating_denominator            1299 non-null int64
name                          1299 non-null object
retweet_count                 1299 non-null int64
favorite_count                1299 non-null int64
predicted_breed               1299 non-null object
prediction_confidence         1299 non-null float64
dog_stage                     236 no[...]<jupyter_text>*twitter_archive_enhanced* table- Some tweets have more than one value for the dog stages.  Define 1. Find all the duplicated `tweet_id`'s.2. Address the values directly, one by one.3. Delete duplicates `tweet_id`.  Code<jupyter_code># Code goes here
twitter_archive_clean.loc[twitter_archive_clean.tweet_id.duplicated()]
twitter_archive_clean.loc[twitter_archive_clean.tweet_id == 855851453814013952]
twitter_archive_clean.at[137,'dog_stage'] = 'puppo'
twitter_archive_clean = twitter_archive_clean.drop([138])
twitter_archive_clean.loc[twitter_archive_clean.tweet_id == 854010172552949760]
twitter_archive_clean = twitter_archive_clean.drop([145])
twitter_archive_clean.loc[twitter_archive_clean.tweet_id == 817777686764523521]
twitter_archive_clean.at[322,'dog_stage'] = 'pupper'
twitter_archive_clean = twitter_archive_clean.drop([323])
twitter_archive_clean.loc[twitter_archive_clean.tweet_id == 808106460588765185]
twitter_archive_clean.at[383,'dog_stage'] = 'doggo&pupper'
twitter_archive_clean = twitter_archive_clean.drop([384])
twitter_archive_clean.loc[twitter_archive_clean.tweet_id == 802265048156610565]
twitter_archive_clean.at[408,'dog_stage'] = 'doggo&pupper'
twitter_archive_clean = twitter_archive_clean.drop([409])
twitter_archive_clean.loc[twitter_archive_clean.tweet_id == 801115127852503040]
twitter_archive_clean.at[414,'dog_stage'] = 'pupper'
twitter_archive_clean = twitter_archive_clean.drop([415])
twitter_archive_clean.loc[twitter_archive_clean.tweet_id == 775898661951791106]
twitter_archive_clean.at[562,'dog_stage'] = 'doggo&pupper'
twitter_archive_clean = twitter_archive_clean.drop([563])
twitter_archive_clean.loc[twitter_archive_clean.tweet_id == 770093767776997377]
twitter_archive_clean.at[597,'dog_stage'] = 'doggo&pupper'
twitter_archive_clean = twitter_archive_clean.drop([598])
twitter_archive_clean.loc[twitter_archive_clean.tweet_id == 733109485275860992]
twitter_archive_clean.at[614,'dog_stage'] = 'doggo&pupper'
twitter_archive_clean = twitter_archive_clean.drop([615])<jupyter_output><empty_output><jupyter_text>Test<jupyter_code># Test code here
twitter_archive_clean[twitter_archive_clean.tweet_id.duplicated()]
twitter_archive_clean.dog_stage.value_counts()
twitter_archive_clean.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 1290 entries, 0 to 1298
Data columns (total 18 columns):
tweet_id                      1290 non-null int64
in_reply_to_status_id         17 non-null float64
in_reply_to_user_id           17 non-null float64
timestamp                     1290 non-null object
source                        1290 non-null object
text                          1290 non-null object
retweeted_status_id           52 non-null float64
retweeted_status_user_id      52 non-null float64
retweeted_status_timestamp    52 non-null object
expanded_urls                 1290 non-null object
rating_numerator              1290 non-null int64
rating_denominator            1290 non-null int64
name                          1290 non-null object
retweet_count                 1290 non-null int64
favorite_count                1290 non-null int64
predicted_breed               1290 non-null object
prediction_confidence         1290 non-null float64
dog_stage                     227 no[...]<jupyter_text>*twitter_archive_enhanced* table- In columns `retweeted_status_id`, `retweeted_status_user_id`, and `retweeted_status_timestamp` there are 181 not-null values (numbers like 8.860537e+17). i.e., Not all are dog ratings and some are retweets.  Define 1. Drop rows where `retweeted_status_id` is not null.2. Drop columns `retweeted_status_id`, `retweeted_status_user_id` and `retweeted_status_timestamp`.  Code<jupyter_code># Code goes here
twitter_archive_clean = twitter_archive_clean.loc[twitter_archive_clean.retweeted_status_id.isnull()]
twitter_archive_clean = twitter_archive_clean.drop(['retweeted_status_id', 'retweeted_status_user_id', 'retweeted_status_timestamp'], axis=1)<jupyter_output><empty_output><jupyter_text>Test<jupyter_code># Test code here
twitter_archive_clean.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 1238 entries, 0 to 1298
Data columns (total 15 columns):
tweet_id                 1238 non-null int64
in_reply_to_status_id    17 non-null float64
in_reply_to_user_id      17 non-null float64
timestamp                1238 non-null object
source                   1238 non-null object
text                     1238 non-null object
expanded_urls            1238 non-null object
rating_numerator         1238 non-null int64
rating_denominator       1238 non-null int64
name                     1238 non-null object
retweet_count            1238 non-null int64
favorite_count           1238 non-null int64
predicted_breed          1238 non-null object
prediction_confidence    1238 non-null float64
dog_stage                217 non-null object
dtypes: float64(3), int64(5), object(7)
memory usage: 154.8+ KB<jupyter_text>*twitter_archive_enhanced* table- Issues with fractional rating numerators: tweet_is's 883482846933004288 and 681340665377193984.  Define 1. Using regex, extract correct numerator from `text` into new column `rating_numerator_clean`. (float)2. Compare `rating_numerator` and `rating_numerator_clean` to see if it solves issues for tweet_id's 883482846933004288 and 681340665377193984.3. If there are other difference, adjust the code until it fully represents the real rating.4. Drop colum `rating_numerator`.5. Rename column `rating_numerator_clean` to `rating_numerator`. (float)  Code<jupyter_code># Code goes here
twitter_archive_clean['rating_numerator_clean'] = twitter_archive_clean.text.str.extract('(\d*\.\d+|\d+)\/\d+', expand=True).astype(float)
twitter_archive_clean.loc[(twitter_archive_clean.tweet_id == 883482846933004288)]
twitter_archive_clean = twitter_archive_clean.drop('rating_numerator', axis=1)
twitter_archive_clean = twitter_archive_clean.rename(index=str, columns={'rating_numerator_clean': 'rating_numerator'})<jupyter_output><empty_output><jupyter_text>Test<jupyter_code># Test Code here
twitter_archive_clean.loc[(twitter_archive_clean.tweet_id == 883482846933004288)]
list(twitter_archive_clean)<jupyter_output><empty_output><jupyter_text>*twitter_archive_enhanced* table- Wrong rating for `tweet_id` 716439118184652801: 50/50 instead of 11/10.- Wrong rating for `tweet_id` 682962037429899265: 7/11 instead of 10/10.- Wrong rating for `tweet_id` 666287406224695296: 1/2 instead of 9/10.  Define 1. Locate each tweet_id's index.2. Manually change `rating_numerator` and `ratin_denominator` using `at`.  Code<jupyter_code># Code goes here
twitter_archive_clean.loc[twitter_archive_clean.tweet_id == 716439118184652801, \
                          ['tweet_id', 'rating_numerator', 'rating_denominator']]
twitter_archive_clean.at['689','rating_numerator'] = 11
twitter_archive_clean.at['689','rating_denominator'] = 10
twitter_archive_clean.loc[twitter_archive_clean.tweet_id == 682962037429899265, \
                          ['tweet_id', 'rating_numerator', 'rating_denominator']]
twitter_archive_clean.at['1038','rating_numerator'] = 10
twitter_archive_clean.at['1038','rating_denominator'] = 10
twitter_archive_clean.loc[twitter_archive_clean.tweet_id == 666287406224695296, \
                          ['tweet_id', 'rating_numerator', 'rating_denominator']]<jupyter_output><empty_output><jupyter_text>Test<jupyter_code># Test code goes here
twitter_archive_clean.loc[(twitter_archive_clean.tweet_id == 716439118184652801), \
                          ['tweet_id', 'rating_numerator', 'rating_denominator']]
twitter_archive_clean.loc[(twitter_archive_clean.tweet_id == 682962037429899265), \
                          ['tweet_id', 'rating_numerator', 'rating_denominator']]<jupyter_output><empty_output><jupyter_text>*twitter_archive_enhanced* table- Wrong variable types for columns `timestamp, tweet_id, in_reply_to_status_id, in_reply_to_user_id, timestamp, retweeted_status_id, and retweeted_status_user_id`.  Define 1. `timestamp`: Change type to datetime.2. `tweet_id, in_reply_to_status_id, in_reply_to_user_id, retweeted_status_id, and retweeted_status_user_id`: Change type to object.  Code<jupyter_code># Code goes here
twitter_archive_clean.timestamp = pd.to_datetime(twitter_archive_clean.timestamp)
twitter_archive_clean.tweet_id = twitter_archive_clean.tweet_id.astype(object)
twitter_archive_clean.in_reply_to_status_id = twitter_archive_clean.in_reply_to_status_id.astype(object)
twitter_archive_clean.in_reply_to_user_id = twitter_archive_clean.in_reply_to_user_id.astype(object)<jupyter_output><empty_output><jupyter_text>Test<jupyter_code># Test code here
twitter_archive_clean.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Index: 1239 entries, 0 to 689
Data columns (total 15 columns):
tweet_id                 1238 non-null object
in_reply_to_status_id    17 non-null object
in_reply_to_user_id      17 non-null object
timestamp                1238 non-null datetime64[ns]
source                   1238 non-null object
text                     1238 non-null object
expanded_urls            1238 non-null object
rating_denominator       1238 non-null float64
name                     1238 non-null object
retweet_count            1238 non-null float64
favorite_count           1238 non-null float64
predicted_breed          1238 non-null object
prediction_confidence    1238 non-null float64
dog_stage                217 non-null object
rating_numerator         1239 non-null float64
dtypes: datetime64[ns](1), float64(5), object(9)
memory usage: 194.9+ KB<jupyter_text>Storing<jupyter_code>twitter_archive_clean.to_csv('twitter_archive_master.csv', encoding='utf-8')<jupyter_output><empty_output><jupyter_text>Analysis and visualization<jupyter_code>twitter_archive_master = pd.read_csv('twitter_archive_master.csv')
twitter_archive_master.describe()
rating = twitter_archive_master['rating_numerator']/twitter_archive_master['rating_denominator']*100
rating.describe()<jupyter_output><empty_output><jupyter_text>**\Insight 1:** The average rating is 111%.<jupyter_code>twitter_archive_master.dog_stage.describe()<jupyter_output><empty_output><jupyter_text>**Insight 2:** Most of the tweets have _pupper_ as dog stage (69%).<jupyter_code>twitter_archive_master.predicted_breed.describe()
twitter_archive_master.predicted_breed.value_counts().head(4)
twitter_archive_clean.loc[twitter_archive_clean.retweet_count >= 3621.750000]['predicted_breed'].value_counts().head(4)
twitter_archive_clean.loc[twitter_archive_clean.favorite_count >= 14410.250000]['predicted_breed'].value_counts().head(4)
twitter_archive_clean.loc[(twitter_archive_clean.rating_numerator/twitter_archive_clean.rating_denominator) >= 1.2]['predicted_breed'].value_counts().head(4)<jupyter_output><empty_output><jupyter_text>**Insight 3:** There are 121 different predicted breeds, beign the most frequent `golden_retriever`, with more than 9% of ocurrencies, followed by `Labrador_retriever`, `Pembroke` and `Chihuahua`. This same distribution is present in the 1st quartile of `retweet_count`, `favorite_count` and `rating`.<jupyter_code>%matplotlib inline

plt.scatter(rating, twitter_archive_master['favorite_count'], alpha=0.1)
plt.xlim(xmin=0,xmax=150)
plt.ylim(ymax=60000)
%matplotlib inline

plt.scatter(rating, twitter_archive_master['retweet_count'], alpha=0.1)
plt.xlim(xmin=0,xmax=150)
plt.ylim(ymax=20000)<jupyter_output><empty_output><jupyter_text>**Insight 4:** There seems to be a positive correlation between the rating and the favorite_count, and between rating and the retweet_count.<jupyter_code>%matplotlib inline

plt.scatter(twitter_archive_master.text.str.len(), twitter_archive_master['favorite_count'], alpha=0.1)
plt.ylim(ymax=45000)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09484160025375679, 'fraction_numerical': 0.05439911185123508, 'mean_word_length': 4.382415706359368, 'pattern_counts': {'":': 10, '<': 109, '<?xml version=': 0, '>': 112, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 22, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '27', 'dir': 'jupyter-structured-clean-dedup', 'id': '599818', 'license': 'MIT', 'n_tokens_mistral': 8964, 'n_tokens_neox': 7985, 'n_words': 1702, 'path': 'wrangle_act.ipynb', 'repo_name': 'josematiasarevalo/wrangling_dogs-rates-breeds', 'size': '25221'}