<jupyter_start><jupyter_text>Gather<jupyter_code>twitter_archive = pd.read_csv('twitter-archive-enhanced.csv')
r = requests.get('https://d17h27t6h515a5.cloudfront.net/topher/2017/August/599fd2ad_image-predictions/image-predictions.tsv')
with open("image_predictions.tsv", mode = 'wb') as outfile:
    outfile.write(r.content)
image_preds = pd.read_csv('image_predictions.tsv', sep='\t')
# Done, so comment out so that we don't do it again!

import tweepy
import json

# consumer_key =
# consumer_secret =
# access_token =
# access_secret =

# auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
# auth.set_access_token(access_token, access_secret)

# api = tweepy.API(auth, wait_on_rate_limit=True)

# i=0
# with open('tweet_json.txt', 'w') as outfile:
#     for tweet_id in twitter_archive.tweet_id.values:
#         i+=1
#         print("{} tweets processed".format(i))
#         try:
#             tweet = api.get_status(tweet_id, tweet_mode='extended')
#             json.dump(tweet._json, outfile) #json.dump(tweet._json, outfile)
#             outfile.write("\n")
#         except: # tweepy.TweepError as e:
#             print("Tweet {} skipped!".format(tweet_id))
#             pass
data = []
with open('tweet_json.txt', 'r') as f:
    for line in f:
        data.append(json.loads(line))
# data is list of dictionaries
tweet_json = pd.DataFrame(data)<jupyter_output><empty_output><jupyter_text>Assess<jupyter_code>twitter_archive.head()
image_preds.head()
tweet_json.head()
twitter_archive.sample(5)
# twitter_archive.sample(5)
twitter_archive.info()
twitter_archive.rating_numerator.describe()
twitter_archive.rating_denominator.describe()
twitter_archive.name.value_counts()
twitter_archive[twitter_archive.tweet_id.duplicated()]
image_preds.head()
image_preds[~image_preds.p1_dog]<jupyter_output><empty_output><jupyter_text>Quality `twitter_archive` table- Some of the tweets are retweets, which should be removed- Not all denominator_rating values are 10 (some are a multiple of 10 that is intentional for multiple dogs; others are simply incorrect)- Not all numerators are correct (none of the decimal numerators are captured correctly; other times the whole rating is simply wrong)- Numerator_rating column should be of data type float, to accommodate the few decimal ratings- There is a tweet about plagarism that is not a rating and doesn't belong in the dataset (835152434251116546)- Not all dog names are correct- There are tweets that are replies (not original we-rate-dogs tweets) that should be removed- timestamp column should be in datetime format- Don't need columns source, expanded_urls `tweet_json` table- We only need the id, retweet_count, and favorite_count columns `image_preds` table- Don't need columns jpg_url and img_num- Inconsistent capitalization of dog breed predictions (columns p1, p2, p3)  Tidiness- All data can be merged into 1 dataframe- Dog types should be listed in one column in the `twitter_archive` table  Clean<jupyter_code>twitter_archive_clean = twitter_archive.copy()
tweet_json_clean = tweet_json.copy()
image_preds_clean = image_preds.copy()<jupyter_output><empty_output><jupyter_text>Quality  `twitter_archive`: Table included retweets  DefineRemove all of the tweets that are retweets.  Code<jupyter_code>twitter_archive_clean = twitter_archive_clean[twitter_archive_clean.retweeted_status_id.isnull()]<jupyter_output><empty_output><jupyter_text>Test<jupyter_code>twitter_archive_clean.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 2175 entries, 0 to 2355
Data columns (total 17 columns):
tweet_id                      2175 non-null int64
in_reply_to_status_id         78 non-null float64
in_reply_to_user_id           78 non-null float64
timestamp                     2175 non-null object
source                        2175 non-null object
text                          2175 non-null object
retweeted_status_id           0 non-null float64
retweeted_status_user_id      0 non-null float64
retweeted_status_timestamp    0 non-null object
expanded_urls                 2117 non-null object
rating_numerator              2175 non-null int64
rating_denominator            2175 non-null int64
name                          2175 non-null object
doggo                         2175 non-null object
floofer                       2175 non-null object
pupper                        2175 non-null object
puppo                         2175 non-null object
dtypes: float64(4), int64(3), object(1[...]<jupyter_text>Now that all the retweets are gone, we don't need the columns pertaining to retweet info, so drop those.<jupyter_code>twitter_archive_clean.drop(['retweeted_status_id','retweeted_status_user_id','retweeted_status_timestamp'],axis=1,inplace=True)
twitter_archive_clean.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 2175 entries, 0 to 2355
Data columns (total 14 columns):
tweet_id                 2175 non-null int64
in_reply_to_status_id    78 non-null float64
in_reply_to_user_id      78 non-null float64
timestamp                2175 non-null object
source                   2175 non-null object
text                     2175 non-null object
expanded_urls            2117 non-null object
rating_numerator         2175 non-null int64
rating_denominator       2175 non-null int64
name                     2175 non-null object
doggo                    2175 non-null object
floofer                  2175 non-null object
pupper                   2175 non-null object
puppo                    2175 non-null object
dtypes: float64(2), int64(3), object(9)
memory usage: 254.9+ KB<jupyter_text>`twitter_archive`: Table includes non-original tweets (replies)  DefineRemove all of the tweets that are only replies to other tweets.  Code<jupyter_code>twitter_archive_clean = twitter_archive_clean[twitter_archive_clean.in_reply_to_status_id.isnull()]<jupyter_output><empty_output><jupyter_text>Test<jupyter_code>twitter_archive_clean.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 2097 entries, 0 to 2355
Data columns (total 14 columns):
tweet_id                 2097 non-null int64
in_reply_to_status_id    0 non-null float64
in_reply_to_user_id      0 non-null float64
timestamp                2097 non-null object
source                   2097 non-null object
text                     2097 non-null object
expanded_urls            2094 non-null object
rating_numerator         2097 non-null int64
rating_denominator       2097 non-null int64
name                     2097 non-null object
doggo                    2097 non-null object
floofer                  2097 non-null object
pupper                   2097 non-null object
puppo                    2097 non-null object
dtypes: float64(2), int64(3), object(9)
memory usage: 245.7+ KB<jupyter_text>Now that all the reply tweets are gone, we don't need the columns pertaining to reply anymore. Remove those.<jupyter_code>twitter_archive_clean.drop(['in_reply_to_status_id','in_reply_to_user_id'],axis=1,inplace=True)
twitter_archive_clean.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 2097 entries, 0 to 2355
Data columns (total 12 columns):
tweet_id              2097 non-null int64
timestamp             2097 non-null object
source                2097 non-null object
text                  2097 non-null object
expanded_urls         2094 non-null object
rating_numerator      2097 non-null int64
rating_denominator    2097 non-null int64
name                  2097 non-null object
doggo                 2097 non-null object
floofer               2097 non-null object
pupper                2097 non-null object
puppo                 2097 non-null object
dtypes: int64(3), object(9)
memory usage: 213.0+ KB<jupyter_text>`twitter_archive`: Table includes incorrect ratings in both the numerator and denominator for various reasons  DefineRe-extract both numerator and denominator from the text column. These 2 issues will be fixed together.In the process, we are making numerator_rating type float instead of int.  Code<jupyter_code>twitter_archive_clean[['extra1','extra2']] = twitter_archive_clean.text.str.extract('((?:\d+\.)?\d+)\/(\d+)',expand=True)
fixed_num = twitter_archive_clean[twitter_archive_clean.rating_numerator.astype(float)!=twitter_archive_clean.extra1.astype(float)]
fixed_num
twitter_archive_clean.rating_numerator = twitter_archive_clean.extra1
twitter_archive_clean.drop('extra1',axis=1,inplace=True)
twitter_archive_clean.drop('extra2',axis=1,inplace=True)<jupyter_output><empty_output><jupyter_text>Now see which denominators remain that are not equal to the expected value of 10.<jupyter_code>wrong_denominator = twitter_archive_clean[twitter_archive_clean.rating_denominator!=10]
wrong_denominator<jupyter_output><empty_output><jupyter_text>Since there's only 17 with a denominator other than 10, I visually inspected each one by looking at the text. Most of them have a rating for multiple dogs that can be reduced to something out of 10. So, I'll fix that programmatically so the denominator is 10. There's a handful of tweets where I grabbed the wrong fraction (or numbers with a slash between them). I can change these manually. This one is just wrong. It actually has no rating, so I'll remove it.<jupyter_code>twitter_archive_clean.loc[516]
twitter_archive_clean.drop(516,inplace=True)<jupyter_output><empty_output><jupyter_text>The following I'm fixing manually:<jupyter_code>twitter_archive_clean['rating_numerator'][1068] = 14
twitter_archive_clean['rating_denominator'][1068] = 10

twitter_archive_clean['rating_numerator'][1165] = 13
twitter_archive_clean['rating_denominator'][1165] = 10

twitter_archive_clean['rating_numerator'][1202] = 11
twitter_archive_clean['rating_denominator'][1202] = 10

twitter_archive_clean['rating_numerator'][1662] = 10
twitter_archive_clean['rating_denominator'][1662] = 10

twitter_archive_clean['rating_numerator'][2335] = 9
twitter_archive_clean['rating_denominator'][2335] = 10
wrong_denominator = twitter_archive_clean[twitter_archive_clean.rating_denominator!=10]
wrong_denominator<jupyter_output><empty_output><jupyter_text>Make sure numerator and denominator are numbers and not strings.<jupyter_code>twitter_archive_clean.info()
twitter_archive_clean.rating_numerator = twitter_archive_clean.rating_numerator.astype(float)<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 2096 entries, 0 to 2355
Data columns (total 12 columns):
tweet_id              2096 non-null int64
timestamp             2096 non-null object
source                2096 non-null object
text                  2096 non-null object
expanded_urls         2093 non-null object
rating_numerator      2096 non-null object
rating_denominator    2096 non-null int64
name                  2096 non-null object
doggo                 2096 non-null object
floofer               2096 non-null object
pupper                2096 non-null object
puppo                 2096 non-null object
dtypes: int64(2), object(10)
memory usage: 292.9+ KB<jupyter_text>For tweets that have multiple dogs, for which the rating is a multiple of something out of 10, reduce it to have a denominator of 10, so there's a consistent format and we can compare ratings more easily.<jupyter_code>for i in wrong_denominator.index:
    multiple = twitter_archive_clean['rating_denominator'][i]/10
    twitter_archive_clean['rating_denominator'][i] = 10
    twitter_archive_clean['rating_numerator'][i] = twitter_archive_clean['rating_numerator'][i]/multiple<jupyter_output>/opt/conda/lib/python3.6/site-packages/ipykernel_launcher.py:3: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  This is separate from the ipykernel package so we can avoid doing imports until
/opt/conda/lib/python3.6/site-packages/ipykernel_launcher.py:4: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  after removing the cwd from sys.path.<jupyter_text>Test<jupyter_code>wrong_denominator = twitter_archive_clean[twitter_archive_clean.rating_denominator!=10]
wrong_denominator<jupyter_output><empty_output><jupyter_text>`twitter_archive`: Table includes a post about plagarism that doesn't belong  DefineRemove this tweet (835152434251116546)  Code<jupyter_code>plagarism_tweet = twitter_archive_clean[twitter_archive_clean.tweet_id==835152434251116546]
twitter_archive_clean.drop(plagarism_tweet.index,inplace=True)<jupyter_output><empty_output><jupyter_text>`twitter_archive`: timestamp is not in datetime format  DefineChange timestamp to datetime format.  Code<jupyter_code>twitter_archive_clean.timestamp = pd.to_datetime(twitter_archive_clean.timestamp)<jupyter_output><empty_output><jupyter_text>Test<jupyter_code>twitter_archive_clean.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 2095 entries, 0 to 2355
Data columns (total 12 columns):
tweet_id              2095 non-null int64
timestamp             2095 non-null datetime64[ns]
source                2095 non-null object
text                  2095 non-null object
expanded_urls         2092 non-null object
rating_numerator      2095 non-null float64
rating_denominator    2095 non-null int64
name                  2095 non-null object
doggo                 2095 non-null object
floofer               2095 non-null object
pupper                2095 non-null object
puppo                 2095 non-null object
dtypes: datetime64[ns](1), float64(1), int64(2), object(8)
memory usage: 212.8+ KB<jupyter_text>`twitter_archive`: Some names are not correct  DefineRe-extract names from text column to get rid of cases of "a", "the", and "an" being counted as names.  Code<jupyter_code>twitter_archive_clean['name_fixed'] = twitter_archive_clean.text.str.extract('(?:named |This is |name is |Meet |Say hello to )([A-Z][a-z]+)',expand=True)
twitter_archive_clean[['name','name_fixed']].sample(10)<jupyter_output><empty_output><jupyter_text>I found cases that my code misses. I'll just fix the 2 I found manually.<jupyter_code>twitter_archive_clean.name_fixed.loc[775] = 'O\'Malley'
# Devon and Fronq with accent on o
twitter_archive_clean['name_fixed'][915] = twitter_archive_clean['name'][915]
twitter_archive_clean['name_fixed'][1559] = twitter_archive_clean['name'][1559]<jupyter_output>/opt/conda/lib/python3.6/site-packages/ipykernel_launcher.py:2: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  
/opt/conda/lib/python3.6/site-packages/ipykernel_launcher.py:3: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  This is separate from the ipykernel package so we can avoid doing imports until<jupyter_text>Now replace name column with my temporary name_fixed and drop name_fixed.Also replace NaNs in name with None, representing no name given rather than a missing value.<jupyter_code>twitter_archive_clean.name=twitter_archive_clean.name_fixed
twitter_archive_clean.drop('name_fixed',axis=1,inplace=True)
twitter_archive_clean.name.fillna('None',inplace=True)
twitter_archive_clean.info()
twitter_archive_clean.name.value_counts()<jupyter_output><empty_output><jupyter_text>`image_preds` Table: Inconsistent capitalization of dog breed predictions in columns p1, p2, p3  DefineMake all breeds/predictions lowercase.  Code<jupyter_code>image_preds_clean.p1 = image_preds_clean.p1.str.lower()
image_preds_clean.p2 = image_preds_clean.p2.str.lower()
image_preds_clean.p3 = image_preds_clean.p3.str.lower()<jupyter_output><empty_output><jupyter_text>Test<jupyter_code>image_preds_clean.sample(5)<jupyter_output><empty_output><jupyter_text>There are unneeded columns in all three tables  DefineRemove these columns.  Code<jupyter_code>twitter_archive_clean.drop(['source','expanded_urls'], axis=1, inplace=True)
image_preds_clean.drop(['jpg_url','img_num'], axis=1, inplace=True)
cols_to_drop = tweet_json_clean.columns.difference(['id','favorite_count','retweet_count'])
tweet_json_clean.drop(cols_to_drop, axis=1,inplace=True)<jupyter_output><empty_output><jupyter_text>Test<jupyter_code>twitter_archive_clean.info()
image_preds_clean.info()
tweet_json_clean.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 2331 entries, 0 to 2330
Data columns (total 3 columns):
favorite_count    2331 non-null int64
id                2331 non-null int64
retweet_count     2331 non-null int64
dtypes: int64(3)
memory usage: 54.7 KB<jupyter_text>Tidiness  `twitter_archive`: The dog stages are spread across different variable columns  DefineMake one column 'dog_stages' in favor of the 4 stages listed as columns.  Code<jupyter_code>twitter_archive_clean[['doggo','floofer','pupper','puppo']] = twitter_archive_clean[['doggo','floofer','pupper','puppo']].replace("None",np.nan)
twitter_archive_clean["dog_stages"] = twitter_archive_clean['doggo'].fillna('') + twitter_archive_clean['floofer'].fillna('') + twitter_archive_clean['pupper'].fillna('') + twitter_archive_clean['puppo'].fillna('')
valid_stages = ['pupper','doggo','puppo','floofer','none']

twitter_archive_clean['dog_stages'].replace('','none',inplace=True)
twitter_archive_clean['dog_stages'] = twitter_archive_clean['dog_stages'].replace(["doggopupper","doggofloofer","doggopuppo"],"multiple")
twitter_archive_clean.drop(['pupper','doggo','floofer','puppo'],axis=1,inplace=True)<jupyter_output><empty_output><jupyter_text>Test<jupyter_code>twitter_archive_clean.head(10)<jupyter_output><empty_output><jupyter_text>All three tables should be merged into one  DefineMerge them, only keeping rows that are present in each table (inner join).  Code<jupyter_code>twitter_archive_master = twitter_archive_clean.merge(tweet_json_clean,left_on='tweet_id', right_on='id',how='inner')
twitter_archive_master.drop('id',axis=1,inplace=True)
image_preds_clean.rename(columns={'tweet_id': 'id'}, inplace=True)
twitter_archive_master = twitter_archive_master.merge(image_preds_clean,left_on='tweet_id',right_on='id',how='inner')<jupyter_output><empty_output><jupyter_text>Test<jupyter_code>twitter_archive_master.info()
twitter_archive_master.head()
# save
twitter_archive_master.to_csv('twitter_archive_master.csv',index=False)<jupyter_output><empty_output><jupyter_text>Exploratory Data Analysis<jupyter_code>%matplotlib inline
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>How does the average dog rating change over time?<jupyter_code>twitter_archive_master.timestamp.describe()
twitter_archive_master.sort_values('timestamp')
plt.scatter(data=twitter_archive_master,x='tweet_id',y='rating_numerator',alpha=1/5);<jupyter_output><empty_output><jupyter_text>Looks like there's 2 outliers we should get rid of.<jupyter_code>outliers = twitter_archive_master[twitter_archive_master.rating_numerator>250]
twitter_archive_master.drop(outliers.index, inplace=True)
plt.scatter(data=twitter_archive_master,x='tweet_id',y='rating_numerator',alpha=1/5);
plt.xlabel('Tweet ID')
plt.ylabel('Rating out of 10')
plt.title('Rating over Time')<jupyter_output><empty_output><jupyter_text>This is interesting, and as I suspected. At least up to the time for which we have tweet data, the ratings tend to get higher. In particular, somewhere around the tweet that starts with 78, which corresponds to around September of 2016, We Rate Dogs had stopped giving any dogs lower than 10/10 (I suspect the 6/10 is another outlier/mistake).  Is there a correlation between tweet rating, retweet count, and favorite count?<jupyter_code>twitter_archive_master.rating_numerator.value_counts()<jupyter_output><empty_output><jupyter_text>For this analysis, I will remove the tweets that have a non-integer numerator rating, since there's only 4 of them, so I can treat the rating variable as discrete.<jupyter_code># twitter_archive_master_decimals = twitter_archive_master[(twitter_archive_master.rating_numerator==9.75) | (twitter_archive_master.rating_numerator==11.26) | (twitter_archive_master.rating_numerator==13.5) | (twitter_archive_master.rating_numerator==11.27)]
# twitter_archive_master_ints = twitter_archive_master.drop(twitter_archive_master_decimals.index)
# twitter_archive_master_ints.rating_numerator = twitter_archive_master_ints.rating_numerator.astype('int')
# twitter_archive_master_ints.info()
plt.scatter(data=twitter_archive_master,x='favorite_count',y='retweet_count',c='rating_numerator',cmap='viridis');
cbar = plt.colorbar()
plt.xlabel('Favorite Count');
plt.ylabel('Retweet Count');
cbar.set_label('Rating out of 10')
plt.title('Tweet Retweet Count vs. Favorite Count vs. Rating');
np.corrcoef([twitter_archive_master['favorite_count'],twitter_archive_master['retweet_count'],twitter_archive_master['rating_numerator']])<jupyter_output><empty_output><jupyter_text>There appears to be a strong positive correlation (r=0.93) between favorite_count and retweet count. The correlation between the rating and favorite count, and between rating and retweet count, on the other hand, are moderate (r=0.4 and r=0.3, respectively). This is reflected in the multivariate scatterplot above.  Which dog breeds are most popular?<jupyter_code>twitter_archive_master.head(10)<jupyter_output><empty_output><jupyter_text>First, make a new column for predicted dog breed, based on the most likely predictions out of the three given that are actually dog breeds. If no valid breeds are given, remove those for this analysis.<jupyter_code>twitter_archive_master['pred_breed'] = twitter_archive_master.p1
twitter_archive_master.head()
p1_not_dog = twitter_archive_master[~twitter_archive_master.p1_dog]
p1_not_dog
twitter_archive_master.pred_breed[p1_not_dog.index] = twitter_archive_master.p2[p1_not_dog.index]
p12_not_dog = twitter_archive_master[~twitter_archive_master.p1_dog & ~twitter_archive_master.p2_dog]
p12_not_dog
twitter_archive_master.pred_breed[p12_not_dog.index] = twitter_archive_master.p3[p12_not_dog.index]
p123_not_dog = twitter_archive_master[~twitter_archive_master.p1_dog & ~twitter_archive_master.p2_dog & ~twitter_archive_master.p3_dog]
p123_not_dog
twitter_archive_master_breeds = twitter_archive_master.drop(p123_not_dog.index)
twitter_archive_master_breeds.sample(10)<jupyter_output><empty_output><jupyter_text>To make comparison of breeds more statistically fair, only consider those that are in at least 20 different tweets.<jupyter_code>twitter_archive_master_breeds.pred_breed.value_counts()
top_tweeted_breeds = twitter_archive_master_breeds.pred_breed.value_counts()[twitter_archive_master_breeds.pred_breed.value_counts().values>=20]
top_tweeted_breeds
top_tweeted_breeds.index
top_breeds = top_tweeted_breeds.index.tolist()
top_breeds
twitter_archive_master_top_breeds = twitter_archive_master_breeds[twitter_archive_master_breeds.pred_breed.isin(top_breeds)]
twitter_archive_master_top_breeds.sample(5)<jupyter_output><empty_output><jupyter_text>Calculate popularity based on average rating, retweet count, and favorite count.<jupyter_code>mean_ratings_breed = twitter_archive_master_top_breeds.groupby('pred_breed').rating_numerator.mean().nlargest(3)
mean_ratings_breed
mean_retweet_breed = twitter_archive_master_top_breeds.groupby('pred_breed').retweet_count.mean().nlargest(3)
mean_retweet_breed
mean_retweet_breed = twitter_archive_master_top_breeds.groupby('pred_breed').favorite_count.mean().nlargest(3)
mean_retweet_breed<jupyter_output><empty_output><jupyter_text>The top 3 most popularly represented breeds in We Rate Dogs' tweets are Golden Retriever, Laborator Retriever, and Pembroke (aka Corgi). Filtering down to just the top 20 most-tweeted breeds, the top 3 highest-rated breeds were Samoyed, Golden Retriever, and Pembroke. The top 3 most retweeted breeds were French Bulldog, Cocker Spaniel, and Eskimo Dog. Finally, the top 3 most-favorite'd breeds were French Bulldog, Cocker Spaniel, and Samoyed.  What are the most popular dog names?<jupyter_code>twitter_archive_master.name.value_counts()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0883581844994265, 'fraction_numerical': 0.03502375880714403, 'mean_word_length': 5.473879607531159, 'pattern_counts': {'":': 0, '<': 159, '<?xml version=': 0, '>': 161, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '40', 'dir': 'jupyter-structured-clean-dedup', 'id': '104079', 'license': 'CNRI-Python', 'n_tokens_mistral': 8357, 'n_tokens_neox': 7680, 'n_words': 1809, 'path': 'wrangle_act.ipynb', 'repo_name': 'dagmaraw/WeRateDogs', 'size': '24412'}