<jupyter_start><jupyter_text>*****************************************************************  The Social Web - Instructors <NAME>, <NAME>- Exercises for Hands-on session 1- 6 February 2020 11:00 - 12:45                 - NU-5B-21, NU-6A-04, NU-6B-20, NU-6C-39, NU-6C-40                             ***************************************************************** Prerequisites:- Python 3.7- Python packages: twitter, prettytable, matplotlibFirst you need to know how to retrieve some social web data. Exercises 1 and 2 will show you how to retrieve trends and search results from Twitter. But let's check first if we're running a sufficiently new version of Python:<jupyter_code>import platform
import sys
print("This jupyter notebook is running on Python " + platform.python_version())
# It's good practice to assert packages requirements at the beginning of a script:
assert sys.version_info >= (3, 6) # Tested with Python==3.7.5<jupyter_output><empty_output><jupyter_text>Let's install now the required packages for this hands on session:<jupyter_code># If you're using a virtualenv, make sure its activated before running 
# this cell!
!pip install twitter PrettyTable matplotlib<jupyter_output><empty_output><jupyter_text>Part 1: Authorizing an application to access Twitter account data (from Example 1-1/9-1 in Mining the Social Web):1. Make sure to add your mobile phone number to your private twitter profile.2. Go to https://apps.twitter.com/ and click on "create an app". Twitter will prompt you to create a *developer account*.3. You'll receive an *account confirmation* email with a link. Follow it and create an app. 4. Once the app is created, you'll see a "Keys and Token" item on the top left-hand side corner of the webpage. This values will be needed to fill in the next cell.<jupyter_code>import twitter # Tell Python to use the twitter package
CONSUMER_KEY = ''
CONSUMER_SECRET = ''
# to get the oauth credential you need to click on the 'Generate access token' button:
OAUTH_TOKEN = '' 
OAUTH_TOKEN_SECRET = ''
auth = twitter.oauth.OAuth(OAUTH_TOKEN, OAUTH_TOKEN_SECRET,CONSUMER_KEY, CONSUMER_SECRET)
twitter_api = twitter.Twitter(auth=auth)
print(twitter_api) 
# Nothing to see by displaying twitter_api except that it's now a defined variable<jupyter_output><empty_output><jupyter_text>1.1: Retrieving twitter search trends (from Example 1-2/9-2 in Mining the Social Web)<jupyter_code>WORLD_WOE_ID = 1 # The "Yahoo! Where On Earth ID" for the entire world
world_trends = twitter_api.trends.place(_id=WORLD_WOE_ID) # get back a callable
print(world_trends)<jupyter_output><empty_output><jupyter_text>Task 1Find out how WORLD_WOE_IDs are defined by Yahoo and try to use others in a query. What kind of differences do you find between the worldwide trends and the local trends?<jupyter_code># Your code here
#
#
#
#
#<jupyter_output><empty_output><jupyter_text>1.2: Retrieving search results (from Example 1-5/9-3 in Mining the Social Web):<jupyter_code>q = '#ThrowbackThursday' # XXX: Set this variable to a trending topic, or anything else you like. 
count = 100 # number of results to retrieve

# See https://dev.twitter.com/docs/api/1.1/get/search/tweets for more info

search_results = twitter_api.search.tweets(q=q, count=count) # search for your query 'q' 100 times
statuses = search_results['statuses'] # extract the tweets found

# The following code allows you to print in a nice format the contents of search_results
import json
print(type(statuses))
print(json.dumps(statuses[0], indent=1))<jupyter_output><empty_output><jupyter_text>Task 2In the cell below, create a second variable (e.g. `statuses2`) that holds the results of a query other than the one presented above. Think about a query that would yield very different results than the first one, for example one that may yield a shorter output or about a different topic.<jupyter_code># your code here
#
#
#
#
#
#
print()<jupyter_output><empty_output><jupyter_text>Part 2: Extracting text, screen names, and hashtags from tweets (from Example 1-6 in Mining the Social Web)Simply printing all the search results to screen is nice, but to really start analysing them, it is handy to select the interesting parts and store them in a different structure such as a list. In this example you are using a thing called "List Comprehension". 2.1 List ComprehensionsList comprehension is a powerful construct that allows to succinctly build a list.With it you can process items from any iterable (e.g. dictionaries, lists, tuples, iterators...) and output a list while optionally performing an operation on each value.Here's a few examples:<jupyter_code># double all values from 0 to 9
double_list = [i*2 for i in range(10)]
# raise to the power of 2, but only if the number is uneven
power_even_list = [i**2 for i in range(10) if i%2!=0]
# clean strings in a tuple
stripped_lines = [x.strip() for x in ('The\n', 'Social\n', 'Web\n')]
# return length of each string in stripped_lines
len_str_lines = [len(s) for s in stripped_lines]
# finally, we can nest list comprehensions to flatten a list of lists:
list_of_lists = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
range_9 = [x for y in list_of_lists for x in y]

print(double_list)
print(power_even_list)
print(stripped_lines)
print(len_str_lines)
print(range_9)<jupyter_output><empty_output><jupyter_text>2.2 Parsing text, screen names and hashtags from tweets*(from Example 1-6 in Mining the Social Web)*Hereafter, we'll be creating a variable `status_texts` of type list. \The list will be filled with the `text` elements from each `status`, whereas `status` comes from looping through all `statuses` in the `search_results` list (1.2). \Look up the list comprehensions in your Python reference materials to make sure you understand what's happening here.<jupyter_code>status_texts = [ status['text'] for status in statuses ]
# the escape character "\" allows for the list comprehension to continue
# on a new line. While not strictly necessary, it makes code more readable
# for your fellow programmers.
screen_names = [ user_mention['screen_name'] for status in statuses \
                for user_mention in status['entities']['user_mentions'] ]

hashtags = [ hashtag['text'] for status in statuses \
        for hashtag in status['entities']['hashtags'] ]

# Compute a collection of all words from all tweets
words = [ w for t in status_texts for w in t.split() ] #split the string on the empty spaces

# Explore the first 5 items for each...
print(json.dumps(status_texts[0:5], indent=1))
print(json.dumps(screen_names[0:5], indent=1))
print(json.dumps(hashtags[0:5], indent=1))
print(json.dumps(words[0:5], indent=1))<jupyter_output><empty_output><jupyter_text>Task 3You are now ready to parse usernames, hashtags and text from the results you previously obtained in Task 2 (e.g. `statuses_2`). While doing it, make sure to leave the variables created in 2.2 untouched. Instead, create your own variable names, which you'll be using soon.<jupyter_code># your code here
#
#
#
#<jupyter_output><empty_output><jupyter_text>Part 3: Creating a basic frequency distribution from words in tweets*(from Examples 1-7 in Mining the Social Web)* In the cell below we display the 10 most common hashtag instances:<jupyter_code>from collections import Counter

for item in [words, screen_names, hashtags]:
    c = Counter(item)
    
print(c.most_common()[:10]) # top 10<jupyter_output><empty_output><jupyter_text>Your output should look something like this: \`[('ThrowbackThursday', 34), ('throwbackthursday', 11), ('TBT', 6), ('ThrowBackThursday', 6), ('Trivia', 3), ('madoka_magica', 2), ('New', 2), ('EURO2020', 2), ('artists', 2)]` Task 4Show hashtags frequency for results that you obtained in Task 3. Think about possible explanations for the different results you get from the analyses for the different queries.<jupyter_code># You code here
#
#
#
#
#<jupyter_output><empty_output><jupyter_text>3.1 Storing your resultsSo far, we have been storing the data in working memory. Often it's handy to store your data to disk so you can retrieve it in a next session. The pickle module lets you do exactly that, by serializing data in a binary format.<jupyter_code>import pickle

filepath = "my_data.pickle"
# this indented python syntax is broadly defined as "context manager".
# This means that everything happening under its indentation will use f
# as file handle to filepath. The Shortand `wb` stands for "write binary",
# which is how we serialize data to disk.
with open(filepath, "wb") as f:
    pickle.dump(words, f) # write the contents of list 'words' to file 'f'
    
# Note that, after the end of the indented block, the file is automatically closed.
# Hence, no memory resource on your system is wasted idly.<jupyter_output><empty_output><jupyter_text>If you browse to your working directory, you should find a file there named "myData.pickle". You can open this in a text editor, or load its contents back into a variable to do some more analyses on.<jupyter_code># open the myData.pickle file and store its contents into variable 'words'

with open(filepath, "rb") as f:
    words = pickle.load(f)
print(words)<jupyter_output><empty_output><jupyter_text>3.2 Using prettytable to display tuples in a nice way<jupyter_code>from prettytable import PrettyTable
for label, data in (('Word', words),
                    ('Screen Name', screen_names),
                    ('Hashtag', hashtags)):
    pt = PrettyTable(field_names=[label, 'Count'])
    c = Counter(data)
    [ pt.add_row(kv) for kv in c.most_common()[:10] ]
    pt.align[label], pt.align['Count'] = 'l', 'r' # Set column alignment
    print(pt)<jupyter_output><empty_output><jupyter_text>3.3 Calculating lexical diversity for tweets *(from Example 1-9 in Mining the Social Web)*:<jupyter_code># Define a function for computing lexical diversity
def lexical_diversity(tokens):
    return 1.0*len(set(tokens))/len(tokens)

# Define a function for computing the average number of words per tweet
def average_words(statuses):
    total_words = sum([ len(s.split()) for s in statuses ])
    return 1.0*total_words/len(statuses) 

# Let's use these functions:

print(lexical_diversity(words))
print(lexical_diversity(screen_names))
print(lexical_diversity(hashtags))
print(average_words(status_texts))<jupyter_output><empty_output><jupyter_text>Task 5: What do the printed numbers indicate? Try to explain them.(*Double click this cell to write your answer*)  3.4 Finding the most popular retweets *(from Example 1-10 in Mining the Social Web):*<jupyter_code>retweet_cnt_un_txt = [(status['retweet_count'],
    status['retweeted_status']['user']['screen_name'],
    status['text'])# Store out a tuple of these three values
    for status in statuses # for each status
    if 'retweeted_status' in status # ... so long as the status meets this condition.
    ]

# Slice off the first 5 from the sorted results and display each item in the tuple
pt = PrettyTable(field_names=['Count', 'Screen Name', 'Text'])
[ pt.add_row(row) for row in sorted(retweet_cnt_un_txt, reverse=True)[:5] ]
pt.max_width['Text'] = 50
pt.align= 'l'
print(pt)<jupyter_output><empty_output><jupyter_text>3.5 Looking up users who have retweeted a status *(from Example 1-11 in Mining the Social Web):*<jupyter_code>_retweets = twitter_api.statuses.retweets(id=1224140327688249349) # Get the original tweet id for a tweet from its retweeted_status node and insert it here
print("Users who've retweeted the tweet:\n")
print([r['user']['screen_name'] for r in _retweets])<jupyter_output><empty_output><jupyter_text>Task 6 (advanced)If you have a Twitter account with a nontrivial number of tweets, request your historical tweet archive from your account settings and analyze it. \The export of your account data includes files organized by time period in a convenient JSON format. See the README.txt file included in the downloaded archive for more details. What are the most common terms that appear in your tweets? \Who do you retweet the most often? \How many of your tweets are retweeted (and why do you think this is the case)?  3.6 Plotting frequencies of words *(from Example 1-12 in Mining the Social Web)*In the previous exercises we have been looking at the text from the tweets, but when you retrieved the results, you retrieved much more information about the tweets, such as the username of the person who shared this tweet with the world. You can use this information to find out who retweets whom in our examples.<jupyter_code>word_counts = sorted(Counter(words).values(), reverse=True)
import matplotlib.pyplot as plt
plt.loglog(word_counts)
plt.ylabel("Freq")
plt.xlabel("Word Rank")
plt.show()<jupyter_output><empty_output><jupyter_text>Generating histograms of words, screen names, and hashtags *(from Example 1-13 in Mining the Social Web):*<jupyter_code>counts = [count for count, _, _ in retweets]
plt.hist(counts)
plt.title("Retweets")
plt.xlabel('Bins (number of times retweeted)')
plt.ylabel('Number of tweets in bin')
print(counts)
plt.show()
# extra: seaborn plots with a one-liner.
!pip install seaborn
import seaborn as sns

sns.distplot(counts, kde=False, rug=True);<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09899863450159307, 'fraction_numerical': 0.01570323167956304, 'mean_word_length': 5.183395872420262, 'pattern_counts': {'":': 0, '<': 83, '<?xml version=': 0, '>': 84, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '574007', 'license': 'MIT', 'n_tokens_mistral': 4156, 'n_tokens_neox': 3897, 'n_words': 1611, 'path': 'Hands-on_1_TwitterAPI.ipynb', 'repo_name': 'mknw/thesocialweb2020', 'size': '13199'}