<jupyter_start><jupyter_text>Graphing the number of favorites to @infinite_scream over timeBy <NAME> (@staeiou), Released CC-BY 4.0 & MIT License  Setup Installing dependencies<jupyter_code>!pip install tweepy pandas seaborn<jupyter_output>Requirement already satisfied: tweepy in /home/staeiou/conda/lib/python3.5/site-packages
Requirement already satisfied: pandas in /home/staeiou/conda/lib/python3.5/site-packages
Requirement already satisfied: seaborn in /home/staeiou/conda/lib/python3.5/site-packages
Requirement already satisfied: requests>=2.4.3 in /home/staeiou/conda/lib/python3.5/site-packages (from tweepy)
Requirement already satisfied: requests-oauthlib>=0.4.1 in /home/staeiou/conda/lib/python3.5/site-packages (from tweepy)
Requirement already satisfied: six>=1.7.3 in /home/staeiou/conda/lib/python3.5/site-packages (from tweepy)
Requirement already satisfied: python-dateutil>=2 in /home/staeiou/conda/lib/python3.5/site-packages (from pandas)
Requirement already satisfied: pytz>=2011k in /home/staeiou/conda/lib/python3.5/site-packages (from pandas)
Requirement already satisfied: numpy>=1.7.0 in /home/staeiou/conda/lib/python3.5/site-packages (from pandas)
Requirement already satisfied: oauthlib>=0.6.2 in /home/stae[...]<jupyter_text>Importing libraries<jupyter_code>import random
import twitter_login  # a file containing my API keys
import tweepy

import pandas as pd
import matplotlib
%matplotlib inline
import matplotlib.pyplot as plt
import seaborn as sns
import datetime
import pytz<jupyter_output><empty_output><jupyter_text>Authenticate with Twitter<jupyter_code>CONSUMER_KEY = twitter_login.CONSUMER_KEY
CONSUMER_SECRET = twitter_login.CONSUMER_SECRET
ACCESS_TOKEN = twitter_login.ACCESS_TOKEN
ACCESS_TOKEN_SECRET = twitter_login.ACCESS_TOKEN_SECRET

# Authenticate

auth = tweepy.OAuthHandler(CONSUMER_KEY, CONSUMER_SECRET)
auth.set_access_token(ACCESS_TOKEN, ACCESS_TOKEN_SECRET)
api = tweepy.API(auth)<jupyter_output><empty_output><jupyter_text>Getting the tweetsThis is limited to the last ~3200 tweets from the API. :(<jupyter_code># by yanofsky, https://gist.github.com/yanofsky/5436496

screen_name="infinite_scream"

#initialize a list to hold all the tweepy Tweets
alltweets = []

#make initial request for most recent tweets (200 is the maximum allowed count)
new_tweets = api.user_timeline(screen_name = screen_name,count=200)

#save most recent tweets
alltweets.extend(new_tweets)

#save the id of the oldest tweet less one
oldest = alltweets[-1].id - 1

#keep grabbing tweets until there are no tweets left to grab
while len(new_tweets) > 0:
    print("getting tweets before %s" % (oldest))

    #all subsiquent requests use the max_id param to prevent duplicates
    new_tweets = api.user_timeline(screen_name = screen_name,count=200,max_id=oldest)

    #save most recent tweets
    alltweets.extend(new_tweets)

    #update the id of the oldest tweet less one
    oldest = alltweets[-1].id - 1

    print("...%s tweets downloaded so far" % (len(alltweets)))<jupyter_output>getting tweets before 890520937354993663
...400 tweets downloaded so far
getting tweets before 890165966184882175
...600 tweets downloaded so far
getting tweets before 889821186279997441
...800 tweets downloaded so far
getting tweets before 889589662125363199
...1000 tweets downloaded so far
getting tweets before 889144233315250176
...1200 tweets downloaded so far
getting tweets before 888713905316515839
...1400 tweets downloaded so far
getting tweets before 888281045912526847
...1600 tweets downloaded so far
getting tweets before 887906074371731455
...1800 tweets downloaded so far
getting tweets before 887478259469910015
...2000 tweets downloaded so far
getting tweets before 887100750219890688
...2200 tweets downloaded so far
getting tweets before 886717121975201791
...2400 tweets downloaded so far
getting tweets before 886506830297485311
...2600 tweets downloaded so far
getting tweets before 886081529079836672
...2800 tweets downloaded so far
getting tweets before 885670229136879615
[...]<jupyter_text>Data processingFirst, go through all the tweets and pull out a few key variables, putting it into a two-dimensional array.<jupyter_code>tweets = [[tweet.id_str, tweet.created_at, tweet.favorite_count, tweet.retweet_count,\
           tweet.favorite_count + tweet.retweet_count, tweet.in_reply_to_screen_name is not None,\
           tweet.text, len(tweet.text)] for tweet in alltweets]<jupyter_output><empty_output><jupyter_text>Convert to a pandas DataFrame for easy processing<jupyter_code>tweets_df = pd.DataFrame.from_dict(tweets)
tweets_df.columns = ["id_str", "created_at", "favorite_count", "retweet_count", "fav_rt_count", \
                     "is_reply", "tweet_text", "tweet_length"]<jupyter_output><empty_output><jupyter_text>Peek into this dataframe, pulling out rows 1000 to 1005<jupyter_code>tweets_df[1000:1005]<jupyter_output><empty_output><jupyter_text>Filter out tweets that were replies to specific users.<jupyter_code>tweets_noreplies_df = tweets_df[tweets_df['is_reply'] == False]<jupyter_output><empty_output><jupyter_text>Time is in UTC / GMT-0 from the API, convert to US/Pacific<jupyter_code>fav_count = tweets_noreplies_df.set_index("created_at")["fav_rt_count"]
fav_count = fav_count.tz_localize(pytz.utc).tz_convert(pytz.timezone('US/Pacific'))
fav_count[0:10]<jupyter_output><empty_output><jupyter_text>Visualization<jupyter_code>ax = fav_count.plot(figsize=[14,8], fontsize=16)
ax.set_title("Engagement to @infinite_scream's tweets over time", {"fontsize":20})
ax.set_ylabel("Number of favorites + retweets", {"fontsize":20})
ax.set_xlabel("Time (Pacific Standard Time / GMT-8)", {"fontsize":20})
xlim_start, xlim_end = plt.xlim()
xlim_end = xlim_end + .25
plt.xlim(xlim_start, xlim_end)<jupyter_output><empty_output><jupyter_text>Output to file<jupyter_code>tweets_df.index.name = "tweet_num"
tweets_df.to_csv("infinite_scream.csv")
tweets_df.to_json("infinite_scream.json")
!head infinite_scream.csv<jupyter_output>tweet_num,id_str,created_at,favorite_count,retweet_count,fav_rt_count,is_reply,tweet_text,tweet_length
0,890792730653339648,2017-07-28 04:35:11,0,0,0,True,@citizen_born AAAAAAAAAAAAAAAAHHHHHH,36
1,890792730653274112,2017-07-28 04:35:11,0,0,0,True,@LeslieBattenND AAAAAAAAAAAAAAAAHH,34
2,890792730644840449,2017-07-28 04:35:11,0,0,0,True,@AngryNIdaho AAAAAAAAAAAAAAAAAHHHHHHH,37
3,890792593658871808,2017-07-28 04:34:39,11,16,27,False,AAAAAAHHHHHHH,13
4,890791471674253312,2017-07-28 04:30:11,2,1,3,True,@bazelonbethany AAAAAAAAAAAAAAAHHHHHHHH,39
5,890790211311071233,2017-07-28 04:25:11,0,0,0,True,@parsaguin AAAAAAHHHHHHHHHHHHHHH,32
6,890790079735762945,2017-07-28 04:24:39,64,67,131,False,AAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHH,35
7,890788957214167040,2017-07-28 04:20:12,0,0,0,True,@YerbaDog AHHHHH,16
8,890788957126103040,2017-07-28 04:20:12,0,0,0,True,@TheDannyBarry AAAAAAAAAAAAAHHHHHHHHH,37
{'fraction_non_alphanumeric': 0.10024662701291165, 'fraction_numerical': 0.10866096039460323, 'mean_word_length': 6.798642533936651, 'pattern_counts': {'":': 3, '<': 43, '<?xml version=': 0, '>': 51, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 12, 'word_list_counts': {'cursed_substrings.json': 14, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '644465', 'license': 'MIT', 'n_tokens_mistral': 3038, 'n_tokens_neox': 2508, 'n_words': 524, 'path': 'infinite_scream/2017-07-27/infinite_scream.ipynb', 'repo_name': 'staeiou/assorted-notebooks', 'size': '6903'}