<jupyter_start><jupyter_text>Biorefinery Tweets Dataset Descriptive Statistics  Questions1. First tweet about biorefineries    1. When did the fist tweet about biorefineries appear?    2. What did it say?    3. Who did it say it?     4. To what field does the author belong?2. Time & fequency: how many tweetas about biorefinery are per year? How does it compare to all tweets?2. Oldest tweet in relation to the network age3. Proportions of discussions / retweets with sole tweets / unfavorited4. Growth of the subject level of discussion, in relation to the growth of the network usage5. Are there clusters of frequent interacting actors?    1. Is it possible to link it with _real-world_ links? (associations, contracts, commercial relations)6. What is noise in a tweeter feed? Has it been defined, measured, operationalized?7. Proportion of personal and organization accounts  Compiled answers (2021-05-30 to ...)1. First tweet about biorefineries    1. The first tweet was on 2015-09-19 02:38:56 UTC    2. It said "Good morning, I've been busy to get this day going. Writing, reading, figuring out things around biotech, clean tech an biorefineries.    3. It was tweeted by <NAME> (@digitalvillages)    4. He is at "Business developmnet at KK-net" (maybe his own company), same as when he tweet it. Maybe he is a sort of trader or venture capitalist.2.   Imports<jupyter_code>import pandas as pd
from matplotlib import pyplot as plt
import numpy as np<jupyter_output>Error in callback <function _enable_matplotlib_integration.<locals>.configure_once at 0x7faacd7da680> (for post_run_cell):<jupyter_text>Importing and combining the data Tweets containing the keywords "biorefinery" or "biorefineries" up until 2021-....Tweets where downloaded and combined localy, the following script loads and merges two pikle files: one with individual tweets, aand another with each user information. From both datasets, columns irrelevant for the study where removed.<jupyter_code>def load_pickle_and_combine(tweets_pkl, users_pkl, on_column, rename_cols):
    tweets = pd.read_pickle(tweets_pkl)
    tweets_df = pd.DataFrame(tweets)  # sequence suggested by Albert

    users = pd.read_pickle(users_pkl)
    users_df = pd.DataFrame(users)

    del tweets
    del users

    users_df['user_id'] = users_df.index
    tweets_df['id'] = tweets_df.index
    biorefinery_df = pd.merge(tweets_df, users_df, on=on_column)
    biorefinery_df.rename(columns=rename_cols, inplace=True)
    biorefinery_df.set_index('id', inplace=True)

    del tweets_df
    del users_df

    return biorefinery_df


tweets_pkl_file = '../data/biorefinery_tweets_df_cleaned.pkl4'
users_pkl_file = '../data/biorefinery_users_df_cleaned.pkl4' # seralized to protocol 4 to work with Python v3.7
join_on_column = 'user_id'
columns_renaming = {
    'created_at_x': 'tweet_created_at',
    'created_at_y': 'user_created_at',
    'favourites_count': 'user_favourites_count'
}

df = load_pickle_and_combine(tweets_pkl_file, users_pkl_file, join_on_column, columns_renaming)

del tweets_pkl_file
del users_pkl_file
del join_on_column
del columns_renaming
#testing

df.info()
print("\n-------------\n")
df.describe()<jupyter_output><empty_output><jupyter_text>Frequency and growth New DF counting the number of tweets about BR per year, and adding the Twitter network size, as million daily active users<jupyter_code>percentiles = list(range(10, 100, 10))
percentiles = [x / 100 for x in percentiles]


description = df.describe(percentiles=percentiles, include='all', datetime_is_numeric=True)

df_grupped_year = df.groupby(df['tweet_created_at'].dt.year)['full_text'].count()

df_grupped_year = pd.DataFrame(df_grupped_year)

twitter_MAU = {
    "2007": np.NaN,
    "2008": np.NaN,
    "2009": np.NaN,
    "2010": 54,
    "2011": 117,
    "2012": 185,
    "2013": 241,
    "2014": 288,
    "2015": 305,
    "2016": 318,
    "2017": 330,
    "2018": 321,
    "2019": 330,
    "2020": 353,
    "2021": np.NaN
}  # source: Twitter financial statements and https://backlinko.com/twitter-users accessed on 2021-05-30

df_grupped_year.reset_index(inplace=True)

df_grupped_year.rename(
    columns={'tweet_created_at': 'Year', 'full_text': 'Biorefinery tweets'},
    inplace=True
    )

df_grupped_year['Twitter million daily active users'] = twitter_MAU.values()<jupyter_output><empty_output><jupyter_text>Plotting and comparing the evolution.<jupyter_code># plotting figures by creating aexs object
# using subplots() function
fig, ax = plt.subplots(figsize = (10, 5))
plt.title('BR tweets & network size per year')
  
# using the twinx() for creating another
# axes object for secondry y-Axis
ax2 = ax.twinx()
ax.plot(df_grupped_year['Year'], df_grupped_year['Biorefinery tweets'], color = 'b')
ax2.plot(df_grupped_year['Year'], df_grupped_year['Twitter million daily active users'], color = 'g')
  
# giving labels to the axises
ax.set_xlabel('Year', color = 'k')
ax.set_ylabel('Number of tweets', color = 'k')
  
# secondary y-axis label
ax2.set_ylabel('million daily active users', color = 'k')


# ax.set_ylabel('Number of tweets containing "biorefinery"')
ax.set_ylim(0, 4000)
ax2.set_ylim(0, 400)

# defining display layout 
plt.tight_layout()
  
# show plot
plt.show()
df_grupped_year<jupyter_output><empty_output><jupyter_text>Questions about time growth2. Time & fequency: how many tweetas about biorefinery are per year? How does it compare to all tweets?3. Oldest tweet in relation to the network age Answers from previos plot and table2. Tweets count grwo from the start of the network, in a similar slope as the network, but stabilize and start a slow descend in 2014, while the network keeps slowly growing.3. The oldest tweet is from 2007, just a year after the network was created.  Moving on..Q: Proportions of discussions / retweets with sole tweets / unfavoritedGroupng and counting:- is_quote_status- retweet_count- favorite_count- reply_count- quote_countI should define what each variable means.And some sub-questions:- % of<jupyter_code>reactions_df = df[["tweet_created_at", "is_quote_status", "retweet_count", "favorite_count", "reply_count", "quote_count"]]
reactions_df.info()
result = pd.DataFrame(reactions_df.astype(bool).sum(axis=0))

print(result)

tweet_location = df.groupby('location', sort=False)['user_id'].count().sort_values(ascending=False).head(30)

print(tweet_location)<jupyter_output>location
                              6859
Madrid                         906
London                         490
Brussels, Belgium              457
USA                            379
Global                         346
Canada                         293
Hürth, Nordrhein-Westfalen     271
Washington, D.C.               269
Europe                         254
Helsinki                       252
Ireland                        239
Washington, DC                 235
United States                  228
Grand Forks, ND                204
Fredericton, New Brunswick     203
UK                             199
India                          189
Worldwide                      181
United States of America       162
Stockholm, Sweden              162
Germany                        159
Finland                        134
Netherlands                    133
New York                       133
New York City                  130
México                         127
Los Angeles, CA                127
South Afric[...]
{'fraction_non_alphanumeric': 0.08637963715136746, 'fraction_numerical': 0.038180341186027617, 'mean_word_length': 3.3787788974510966, 'pattern_counts': {'":': 15, '<': 22, '<?xml version=': 0, '>': 22, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '443026', 'license': 'MIT', 'n_tokens_mistral': 2540, 'n_tokens_neox': 2253, 'n_words': 730, 'path': 'notebooks/descriptive_statistics.ipynb', 'repo_name': 'RM35M4419/Biorefinery_field_analysis', 'size': '7392'}