<jupyter_start><jupyter_text>Wrangle and Analyze Data: We Rate Dogs ([@dog_rates]("https://twitter.com/dog_rates"))A data analysis project focused on data wrangling efforts. Table of Contents- [Introduction](intro)- [Gather](gather)- [Assess](assess)  - Detect and document at least **eight (8) quality issues** and **two (2) tidiness issues**- [Clean](clean)- [Storing, Analyzing and Visualizing Data](storing_analyzing_and_visualizing)  - At least **three (3) insights** and **one (1) visualization** must be produced- [Wrangling Efforts Report](wranglingeffortsreport)- [Communicate Findings Report](communicatefindingsreport)- [References](references)   IntroductionReal-world data rarely comes clean. Using Python and its libraries, I will gather data from a variety of sources and in a variety of formats, assess its quality and tidiness, then clean it. This is called data wrangling. I will document my wrangling efforts in a Jupyter Notebook, plus showcase them through analyses and visualizations using Python (and its libraries) and/or SQL.The dataset that I will be wrangling (and analyzing and visualizing) is the tweet archive of Twitter user [@dog_rates]("https://twitter.com/dog_rates"), also known as [WeRateDogs]("https://twitter.com/dog_rates"). WeRateDogs is a Twitter account that rates people's dogs with a humorous comment about the dog. These ratings almost always have a denominator of 10. The numerators, though? Almost always greater than 10. 11/10, 12/10, 13/10, etc. Why? Because "they're good dogs Brent." WeRateDogs has over 4 million followers and has received international media coverage.WeRateDogs [downloaded their Twitter archive]("https://help.twitter.com/en/managing-your-account/how-to-download-your-twitter-archive") and sent it to Udacity via email exclusively for you to use in this project. This archive contains basic tweet data (tweet ID, timestamp, text, etc.) for all 5000+ of their tweets as they stood on August 1, 2017  GatherI will be gathering data from these three resources:   1. The [WeRateDogs]("https://twitter.com/dog_rates") Twitter archive. The *twitter_archive_enhanced.csv* file was given.   2.  The tweet image predictions, i.e., what breed of dog (or other object, animal, etc.) is present in each tweet according to a neural network. This file was provided.   3. Twitter API and Python's Tweepy library to gather each tweet's retweet count and favorite or like count at minimum, and any additional data I find interesting. I will be generating this using my Twitter API key, secrets, and tokens.<jupyter_code># import necessary libaries
import pandas as pd
import numpy as np
# loading the WeRateDogs twitter archive data
archive = pd.read_csv('twitter-archive-enhanced.csv')

archive.head()
# downloading the image prediction data programmatically
import requests
predicted_breeds_url = 'https://d17h27t6h515a5.cloudfront.net/topher/2017/August/599fd2ad_image-predictions/image-predictions.tsv'
response = requests.get(predicted_breeds_url)
with open('image_predictions.tsv', 'wb') as f:
    f.write(response.content)
    
# load image prediction data
image_predictions = pd.read_csv("image_predictions.tsv", sep='\t')
image_predictions.head()
import tweepy 
consumer_key = 'HIDDEN' 
consumer_secret = 'HIDDEN' 
access_token = 'HIDDEN' 
access_secret = 'HIDDEN' 
# this secures my authentification codes above
auth = tweepy.OAuthHandler(consumer_key, consumer_secret) 
auth.set_access_token(access_token, access_secret) 
#this connects me to Twitter API
api = tweepy.API(auth)
api.me()
api = tweepy.API(auth, parser=tweepy.parsers.JSONParser())
import json
import time

start = time.time()

tweet_ids = archive.tweet_id.values

tweets_data = []
tweet_success = []
tweet_failure = []

for tweet_id in tweet_ids:
        try:
            data = api.get_status(tweet_id, tweet_mode='extended', 
                                  wait_on_rate_limit = True, 
                                  wait_on_rate_limit_notify = True)
            tweets_data.append(data)
            tweet_success.append(tweet_id)
        except:
            tweet_failure.append(tweet_id)
            print(tweet_id)
         

        
end = time.time()

print(end - start)

# storing data to tweet_json.txt
with open('tweet_json.txt', mode = 'w') as file:
    json.dump(tweets_data, file)
# reading the API data stored in json file
df3 = pd.read_json('tweet_json.txt')
df3['tweet_id'] = tweet_success
df3 = df3[['tweet_id', 'retweet_count', 'favorite_count']]
df3.head()<jupyter_output><empty_output><jupyter_text>Assess<jupyter_code>archive
image_predictions
df3
archive.info()
image_predictions.info()
df3.info()
archive.describe()
image_predictions.describe()
df3.describe()
all_columns = pd.Series(list(archive) + list(image_predictions) + list(df3))
all_columns[all_columns.duplicated()]
list(archive)
list(image_predictions)
archive[archive['retweeted_status_user_id'].isnull()]
archive.name.value_counts()
archive.rating_numerator.value_counts()
archive.rating_denominator.value_counts()
archive.sample(50)
image_predictions.p1.value_counts()
image_predictions.p2.value_counts()
image_predictions.p3.value_counts()
archive.sample(5)
image_predictions.sample(5)
from IPython.display import Image 
Image(url = 'https://pbs.twimg.com/media/CU8v-rdXIAId12Z.jpg')
sum(archive.rating_numerator.isnull())
sum(archive.rating_denominator.isnull())
sum(archive.timestamp.isnull())<jupyter_output><empty_output><jupyter_text>Quality `archive` table- Missing data in (in_reply_to_status_id, in_reply_to_user_id, retweeted_status_id, retweeted_status_user_id, retweeted_status_timestamp, expanded_url) columns- Timestamp has +0000 at the end. Not in the right format- Erroneous datatype (timestamp and retweeted status timestamp should be integer instead of string)- Tweet id is integer instead of string- Lowercase dog names such as *a, an, the, quite, my, such etc.* are unusual. all dog names in lowercase - Text shows there are retweets or replies in the data- Extraction of numerator and denominator are incorrect `image_predictions` table- Tweet id is integer instead of string- Names in Algorithm's predictions p1, p2, and p3 are sentence case sometimes, lowercase other times - Compound names in p1, p2 and p3 columns have underscore sometimes, hyphen other times `df3` table- Tweet id is integer instead of string Tidiness- One variable in four columns in `archive` table (doggo, floofer, pupper and puppo)- df3 should be part of the `archive` table  Clean Now it's time to fix the quality issues and tidiness issues spotted above<jupyter_code>archive_clean = archive.copy()
image_predictions_clean = image_predictions.copy()
df3_clean = df3.copy()<jupyter_output><empty_output><jupyter_text>Quality  Erroneous Data Format**`archive`**: Timestamp has +0000 at the end, making it recorded in pandas as object. Not in the right format DefineChange timestamp to datetime format Code<jupyter_code># remove +0000 from timestamp
archive_clean['timestamp'] = archive_clean['timestamp'].str.slice(start=0, stop=-6)
# convert timestamp to datetime
archive_clean['timestamp'] = pd.to_datetime(archive_clean['timestamp'], 
                                            format = "%Y-%m-%d %H:%M:%S")<jupyter_output><empty_output><jupyter_text>Test<jupyter_code>archive_clean.info()
archive_clean.head(2)<jupyter_output><empty_output><jupyter_text>Erroneous Datatype**`archive`: Tweet id is integer instead of string****`image_predictions`: Tweet id is integer instead of string****`df3`: Tweet id is integer instead of string** DefineChange tweet id datatype to string  Code<jupyter_code>archive_clean['tweet_id'] = archive_clean['tweet_id'].astype(str)
image_predictions_clean['tweet_id'] = image_predictions_clean['tweet_id'].astype(str)
df3_clean['tweet_id'] = df3_clean['tweet_id'].astype(str)<jupyter_output><empty_output><jupyter_text>Test<jupyter_code>archive_clean.info()
image_predictions_clean.info()
df3_clean.info()
archive_clean.head(1)<jupyter_output><empty_output><jupyter_text>Inconsistency In Name**`archive`**:Names such as a, an, the, quite, my, such etc. are unusual. all dog names in lowercase DefineConvert all non-dog names to np.nan. All non-dog names start with lowercase Code<jupyter_code>mask = archive_clean.name.str.islower()
column_name = 'name'
archive_clean.loc[mask, column_name] = np.nan
archive_clean.name.value_counts()
archive_clean[archive_clean.name.isnull()]<jupyter_output><empty_output><jupyter_text>`archives` table: **There are retweets and replies in the dataset** DefineFilter the null values for the three columns related to retweets and check to verify them. Code<jupyter_code>archive_clean = archive_clean[archive_clean.retweeted_status_id.isnull()]
archive_clean = archive_clean[archive_clean.retweeted_status_user_id.isnull()]
archive_clean = archive_clean[archive_clean.retweeted_status_timestamp.isnull()]
archive_clean = archive_clean[archive_clean.in_reply_to_status_id.isnull()]
archive_clean = archive_clean[archive_clean.in_reply_to_user_id.isnull()]<jupyter_output><empty_output><jupyter_text>Test<jupyter_code>archive_clean.retweeted_status_id.notnull().sum()
archive_clean.retweeted_status_user_id.notnull().sum()
archive_clean.retweeted_status_timestamp.notnull().sum()
archive_clean.in_reply_to_status_id.notnull().sum()
archive_clean.in_reply_to_user_id.notnull().sum()<jupyter_output><empty_output><jupyter_text>`archive` table: **The extraction of numerator and denominator are incorrect** DefineThe extraction of the numerator and the denominator didn't seem to work fine, as floats were not extracted correctly. The extraction will be executed again and the results shall be stored in the DataFrame. Code<jupyter_code>rating = archive_clean.text.str.extract('((?:\d+\.)?\d+)\/(\d+)', expand = True)
rating.columns = ['rating_numerator', 'rating_denominator']
rating['rating_numerator'] = rating['rating_numerator'].astype(float)<jupyter_output><empty_output><jupyter_text>Test<jupyter_code>archive_clean.rating_numerator.value_counts()
archive_clean.info()
archive_clean.head(3)<jupyter_output><empty_output><jupyter_text>`image_predictions`: **Names in Algorithm's predictions p1, p2, and p3 are sentence case sometimes, lowercase other times** DefineChange the breed names to the same format for consistency Code<jupyter_code># change the image predictions' breed names to the same format for consistency
image_predictions_clean[["p1", "p2", "p3"]] = image_predictions_clean[["p1", "p2", "p3"]].apply(lambda x: x.str.lower(), axis=1)
image_predictions_clean[["p1", "p2", "p3"]]<jupyter_output><empty_output><jupyter_text>Test<jupyter_code>image_predictions_clean.p1.describe
image_predictions_clean.p2.describe
image_predictions_clean.p3.describe<jupyter_output><empty_output><jupyter_text>`image_predictions`: **Compound names in p1, p2 and p3 columns have underscore sometimes, hyphen other times** DefineUse pandas apply function to make changes to compound names with hypen and underscore for consistency Code<jupyter_code>image_predictions_clean[["p1", "p2", "p3"]].apply(lambda x: x.str.replace("_", " ").str.title, axis=1)
image_predictions_clean[["p1", "p2", "p3"]]<jupyter_output><empty_output><jupyter_text>Test<jupyter_code>image_predictions_clean.p1.describe
image_predictions_clean[["p1", "p2", "p3"]].apply(lambda x: x.str.replace("_", "-").str.title, axis=1)
image_predictions_clean[["p1", "p2", "p3"]]<jupyter_output><empty_output><jupyter_text>Test<jupyter_code>image_predictions_clean[["p1", "p2", "p3"]].describe<jupyter_output><empty_output><jupyter_text>Missing Data**`archive`: Missing data in (in_reply_to_status_id, in_reply_to_user_id, retweeted_status_id, retweeted_status_user_id, retweeted_status_timestamp, expanded_urls) columns** DefineDrop the columns in which missing data are present in the `archive` table. Since they are not going to be used in my analysis. Code<jupyter_code># remove the columns in archive with missing data
archive_clean = archive_clean.drop(['in_reply_to_status_id', 'in_reply_to_user_id', 
                              'retweeted_status_id', 
                              'retweeted_status_user_id', 
                              'retweeted_status_timestamp', 
                              'expanded_urls'], axis = 1)<jupyter_output><empty_output><jupyter_text>Test<jupyter_code>archive_clean.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 2097 entries, 0 to 2355
Data columns (total 11 columns):
tweet_id              2097 non-null object
timestamp             2097 non-null datetime64[ns]
source                2097 non-null object
text                  2097 non-null object
rating_numerator      2097 non-null int64
rating_denominator    2097 non-null int64
name                  1993 non-null object
doggo                 2097 non-null object
floofer               2097 non-null object
pupper                2097 non-null object
puppo                 2097 non-null object
dtypes: datetime64[ns](1), int64(2), object(8)
memory usage: 196.6+ KB<jupyter_text>Tidiness **Several columns in archive table contain similar variables** DefineReplace *None* variables in the dog stage columns, then combine all stages to one column, separate the joint multiple stages and then convert the missing values to nan. In the end, drop individual stages from the archive table.  Code<jupyter_code># handle None variables in the dog stage columns
archive_clean.doggo.replace('None', '', inplace=True)
archive_clean.floofer.replace('None', '', inplace=True)
archive_clean.pupper.replace('None', '', inplace=True)
archive_clean.puppo.replace('None', '', inplace=True)
# merge all stages into one column
archive_clean['dog_stage'] = archive_clean.doggo + archive_clean.floofer + archive_clean.pupper + archive_clean.puppo
# drop individual stage columns
archive_clean = archive_clean.drop(['doggo', 'floofer', 'pupper', 'puppo'], axis = 1)
# handle tweets with multiple stages
archive_clean.loc[archive_clean.dog_stage == 'doggopupper', 'dog_stage'] = 'doggo, pupper'
archive_clean.loc[archive_clean.dog_stage == 'doggopuppo', 'dog_stage'] = 'doggo, puppo'
archive_clean.loc[archive_clean.dog_stage == 'doggofloofer', 'dog_stage'] = 'doggo, floofer'
# handle missing values
archive_clean.loc[archive_clean.dog_stage == '', 'dog_stage'] = np.nan<jupyter_output><empty_output><jupyter_text>Test<jupyter_code>archive_clean.info()
archive_clean.dog_stage.value_counts()<jupyter_output><empty_output><jupyter_text>**Dog breed predictions are not in should be a variable** DefineWrite a loop to select dog breed in the predictions variable Code<jupyter_code># writing a loop to select dog breed
dog_prediction = []
for i in range (len(image_predictions_clean)):
    if image_predictions_clean['p1_dog'][i] == True:
        dog_prediction.append(image_predictions_clean.p1[i])
    elif image_predictions_clean['p2_dog'][i] == True:
        dog_prediction.append(image_predictions_clean.p2[i])
    elif image_predictions_clean['p3_dog'][i] == True:
        dog_prediction.append(image_predictions_clean.p3[i])
    else:
        dog_prediction.append("no prediction")
# create a new column from dog prediction list
image_predictions_clean['predictions'] = dog_prediction<jupyter_output><empty_output><jupyter_text>Test<jupyter_code># check prediction
image_predictions_clean[['predictions', 'p1_dog', 'p1']]
image_predictions_clean.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 2075 entries, 0 to 2074
Data columns (total 13 columns):
tweet_id       2075 non-null object
jpg_url        2075 non-null object
img_num        2075 non-null int64
p1             2075 non-null object
p1_conf        2075 non-null float64
p1_dog         2075 non-null bool
p2             2075 non-null object
p2_conf        2075 non-null float64
p2_dog         2075 non-null bool
p3             2075 non-null object
p3_conf        2075 non-null float64
p3_dog         2075 non-null bool
predictions    2075 non-null object
dtypes: bool(3), float64(3), int64(1), object(6)
memory usage: 168.3+ KB<jupyter_text>**All the table should be merged as one**  DefineMerge df3_clean with archive_clean Code<jupyter_code># check if the DataFrames have duplicates first
archive_clean['tweet_id'].duplicated().sum()
image_predictions_clean['tweet_id'].duplicated().sum()
df3_clean['tweet_id'].duplicated().sum()
# good! no duplicate. now we can combine the DataFrames
we_rate_dogs = pd.merge(archive_clean, 
                        df3_clean, on = 'tweet_id', how = 'left')<jupyter_output><empty_output><jupyter_text>Test<jupyter_code>we_rate_dogs.head()
we_rate_dogs.info()
# checking for duplicates again
we_rate_dogs['tweet_id'].duplicated().sum()<jupyter_output><empty_output><jupyter_text>**All the table should be merged as one**  DefineMerge image_predictions_clean with we_rate_dogs Code<jupyter_code>we_rate_dogs2 = pd.merge(we_rate_dogs, 
                        image_predictions_clean, on = 'tweet_id', how = 'left')<jupyter_output><empty_output><jupyter_text>Test<jupyter_code>we_rate_dogs2.head(2)
we_rate_dogs2.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 2097 entries, 0 to 2096
Data columns (total 22 columns):
tweet_id              2097 non-null object
timestamp             2097 non-null datetime64[ns]
source                2097 non-null object
text                  2097 non-null object
rating_numerator      2097 non-null int64
rating_denominator    2097 non-null int64
name                  1993 non-null object
dog_stage             336 non-null object
retweet_count         2088 non-null float64
favorite_count        2088 non-null float64
jpg_url               1971 non-null object
img_num               1971 non-null float64
p1                    1971 non-null object
p1_conf               1971 non-null float64
p1_dog                1971 non-null object
p2                    1971 non-null object
p2_conf               1971 non-null float64
p2_dog                1971 non-null object
p3                    1971 non-null object
p3_conf               1971 non-null float64
p3_dog                [...]<jupyter_text>Storing, Analyzing and Visualizing DataStore the clean DataFrame in a csv file named `twitter_archive_master.csv`Analyze and visualize my wrangled data. At least, **three (3) insights and one(1) visualization** must be producedDrawing conclusions and creating visuals to communicate results. The following questions are addressed**Q1:** What are the features that influence retweet count and favorite count?**Q2:** Is rating influenced by dog stage? What are the dog stages with the highest rating?**Q3:** What is the most popular dog name?<jupyter_code># store the clean DataFrame in a csv file named twitter_archive_master.csv
we_rate_dogs2.to_csv('twitter_archive_master.csv', encoding='utf-8', index=False)
# import necessary libaries
import datetime 
import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns

# load the clean data
df = pd.read_csv('twitter_archive_master.csv')<jupyter_output><empty_output><jupyter_text>Exploring with Visuals<jupyter_code># create rating column
df['rating'] = df.rating_numerator/df.rating_denominator
#create new column for dog breeds substituting single breeds for "other"
# value counts for all the dog breeds
vc = df.predictions.value_counts()
# get the breeds that have a value count of less than 10
singles = vc[vc<10].index.tolist()
# new column for dog breeds
df['breed_group'] = df['predictions']
# replace strings in single list with string 'other'
df['breed_group'].replace(singles, 'other', inplace = True)
# get the average rating only for each breed
rating2 = df.groupby('breed_group').rating.mean()
rating2
# the index for this series is the breed
# the value is the average rating
# plotting these directly in pandas:
# plot average rating of breed group and dog stage
fig, axes = plt.subplots(1,2)
rating = df.groupby('breed_group').rating.mean()
rating.plot.barh(ax = axes[1], align = 'center', color = ['lavender', 'lightsteelblue', 'cornflowerblue', 'royalblue', 'midnightblue'], figsize = [5,11])
rating2 = df.groupby('dog_stage').rating.mean()
rating2.plot.barh(ax = axes[0], align = 'center', color = ['lavender', 'lightsteelblue', 'cornflowerblue', 'royalblue', 'midnightblue'])
plt.subplots_adjust(wspace = 0.4)
fig.set_figwidth(25)
plt.tight_layout()
plt.show()<jupyter_output><empty_output><jupyter_text>Dog breed with **no prediction** which is where no dog breed exist, has the highest average rating. Average rating for dog breeds is below 1.4<jupyter_code># plot average favorite count for dog stage and breed group
fig, axes = plt.subplots(1,2, figsize = [5,11])

rating = df.groupby('dog_stage').favorite_count.mean()
rating.plot.barh(ax = axes[0],align = 'center', color = ['lavender', 'lightsteelblue', 'cornflowerblue', 'royalblue', 'midnightblue'])

rating2 = df.groupby('breed_group').favorite_count.mean()
rating2.plot.barh(ax = axes[1],align = 'center', color = ['lavender', 'lightsteelblue', 'cornflowerblue', 'royalblue', 'midnightblue'])

plt.subplots_adjust(wspace =0.4)
fig.set_figwidth(25)

plt.tight_layout()

plt.show()<jupyter_output><empty_output><jupyter_text>There is a big difference in average favorite count for the combined dog stage **doggo, puppo** compare to other stages. This factor is subject to further findings<jupyter_code># plot average retweet count for dog stage and breed group
fig, axes = plt.subplots(1,2, figsize = [5,11])

rating = df.groupby('dog_stage').retweet_count.mean()
rating.plot.barh(ax = axes[0],align = 'center', color = ['lavender', 'lightsteelblue', 'cornflowerblue', 'royalblue', 'midnightblue'])

rating2 = df.groupby('breed_group').retweet_count.mean()
rating2.plot.barh(ax = axes[1],align = 'center', color = ['lavender', 'lightsteelblue', 'cornflowerblue', 'royalblue', 'midnightblue'])

plt.subplots_adjust(wspace =0.4)
fig.set_figwidth(25)

plt.tight_layout()

plt.show()<jupyter_output><empty_output><jupyter_text>There is a big difference in average retweet count for the combined dog stage **doggo, puppo** compare to other stages. This factor is also subject to further findings in our analysis **Q1:** What are the features that influence retweet count and favorite count?<jupyter_code>p_breed_retweet = df[~df['breed_group'].isin(['no prediction'])].retweet_count.mean()
p_breed_favorite = df[~df['breed_group'].isin(['no prediction'])].favorite_count.mean()
p_stage_retweet = df[~df['breed_group'].isin(['no prediction'])].retweet_count.mean()
print('average retweet count for all dog breeds except no prediction = %f' %df[~df['breed_group'].isin(['no prediction'])].retweet_count.mean())
print('average favorite count for all dog breeds except no prediction = %f' %df[~df['breed_group'].isin(['no prediction'])].favorite_count.mean())
print('average rating for all dog breeds under no prediction = %f' %df[df['predictions'].isin(['no prediction'])].rating.mean())
print('average rating for all dog breeds under other = %f' %df[df['predictions'].isin(singles)].rating.mean())
# correlation between retweet count and favorite count
plt.scatter(df.retweet_count, df.favorite_count)
plt.title('Correlation Between Retweet and Favourite')
plt.xlabel('Retweet Count')
plt.ylabel('Favorite Count')

plt.show();<jupyter_output><empty_output><jupyter_text>There is a positive correlation between retweet count and favorite count<jupyter_code>df = df.astype({'dog_stage': 'category'})
# install a pip package in the current Jupyter kernel to update the seaborn libary
import sys
!{sys.executable} -m pip install seaborn -U
sns.scatterplot(data = df, x='retweet_count', y='favorite_count', hue = df.dog_stage.tolist());<jupyter_output><empty_output><jupyter_text>Here, we can see that there is a positive correlation between retweet count and favorite count across the dog stages, with **doggo** having the strongestTherefore, dog stage influences the reetweet count and favorite count  **Q2:** Is rating influenced by dog stage? What are the dog stages with the highest rating?<jupyter_code>sns.scatterplot(data = df, x = 'rating', y = 'dog_stage');<jupyter_output><empty_output><jupyter_text>Now let's check the relationship between rating and each dog stage excluding the outliers<jupyter_code># filter out the outliers in rating
rating_within = df[df['rating']<2.5]
sns.scatterplot( data = rating_within, x = 'rating' , y = 'dog_stage');<jupyter_output><empty_output><jupyter_text>This visual shows that **doggo**, **pupper** and **puppo** have the highest rating **Q3:** What are the most popular dog names and dog breed?<jupyter_code># df.name.value_counts()
df.name.value_counts()[1:10].plot('barh', figsize=(15,8), color = 'cornflowerblue', title='Most Popular Dog Name').set_xlabel("Dog Count");<jupyter_output><empty_output><jupyter_text>Lucy and Charlie are the most popular dog names followed by Cooper and Oliver.<jupyter_code>df.breed_group.value_counts()[2:12].plot('barh', figsize=(15,8), color = 'cornflowerblue', title='Most Popular Breed').set_xlabel("Dog Count");<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10722973242413367, 'fraction_numerical': 0.020337360928340712, 'mean_word_length': 5.453422542460113, 'pattern_counts': {'":': 0, '<': 151, '<?xml version=': 0, '>': 149, 'https://': 7, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '37', 'dir': 'jupyter-structured-clean-dedup', 'id': '504146', 'license': 'MIT', 'n_tokens_mistral': 8246, 'n_tokens_neox': 7805, 'n_words': 2179, 'path': 'wrangle_act.ipynb', 'repo_name': 'Bron-analytics/Wrangle-A-Dataset---We-Rate-Dogs-Twitter-Dataset', 'size': '25077'}