<jupyter_start><jupyter_text>Intro This notebook demonstrates the use of the tweeteater functions. The goal of the tweeteater library is to provide functions that allow simple loading of large Twitter datasets collected from the streaming API. It particularly allows to extract nested observations of engagements.<jupyter_code>from main import *<jupyter_output><empty_output><jupyter_text>Load some tweets + extract engagements<jupyter_code># the directory with the tweets 
tweet_directory = 'sample_data'
# the extension of the tweet files in the directory
extension = '.jsonl'
# where we want to save output
output_directory = 'sample_output'<jupyter_output><empty_output><jupyter_text>At first, we now want to load all tweets in the directory, filter them and extract some attributes. We can filter on two conditions, 1. being that we only keep certain types of tweet, 2. we can provide a custom subsetting function. Regarding 1.: a tweet can be a a) retweet of a tweet, b) a replie to a tweet c) a quote of a tweet or d) just an original tweet, that is a tweet that is neither a) - c). a) - c) are not necessarily mutually exclusive. We decide that we want to keep all tweettypes<jupyter_code>keep_tweettypes = ['original', 'retweet', 'reply', 'quote']<jupyter_output><empty_output><jupyter_text>We now only want to keep tweets made by certain users, which are in this case political accounts, and hence define a custom subset function that acts directly on the tweet object<jupyter_code>pt = 'sample_data/screenname2party.json'
politicians_screennames = set([scn for scn, party in load_json(pt).items()])
subset_func = lambda tweet: safe_get(tweet, *('user', 'screen_name')) in politicians_screennames<jupyter_output><empty_output><jupyter_text>And we want to extract certain attributes:<jupyter_code>attributes = ['created_at', 'user.screen_name', 'user.name', 
              'user.favourites_count', 'text', 'lang']<jupyter_output><empty_output><jupyter_text>Now we are ready and can go:<jupyter_code>tweets, engagements =  do_main(tweet_directory=tweet_directory,
                               extension=extension,
                               output_directory=output_directory,
                               keep_tweettypes=keep_tweettypes,
                               subset_func=subset_func, 
                               attributes=attributes)<jupyter_output>0%|          | 0/2 [00:00<?, ?it/s]<jupyter_text>Investigate  ...the tweets<jupyter_code>tweets.head()<jupyter_output><empty_output><jupyter_text>...the engagements<jupyter_code>engagements.head()<jupyter_output><empty_output><jupyter_text>...the distribution of tweettypes<jupyter_code>import seaborn as sns 
import matplotlib.pyplot as plt
%matplotlib inline

tweettypes_df = pd.DataFrame({'tweet type': [str(x) for x in tweets['tweettypes']]})
fig, ax = plt.subplots(1, 1, figsize=(10, 4))
sns.countplot(data=tweettypes_df, y='tweet type', ax=ax)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08989523487664752, 'fraction_numerical': 0.005407232173031429, 'mean_word_length': 4.681381957773512, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 36, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '627037', 'license': 'MIT', 'n_tokens_mistral': 923, 'n_tokens_neox': 889, 'n_words': 283, 'path': 'illustrative-notebook.ipynb', 'repo_name': 'richardknudsen/tweeteater', 'size': '2959'}