<jupyter_start><jupyter_text>Programming and Data Analytics 1 2021/2022Sant'Anna School of Advanced Studies, Pisa, ItalyCourse responsible<NAME> <EMAIL>Co-lecturer <NAME> <EMAIL>--- Lecture 5:  Creation of word clouds for COVID-related online news ---  Computing and visulatizing the most important words in online news This example shows that functions can hide a lot of complexity.* E.g., we can download remote data by just invoking a function* In particular, in this example we will:    1.	Download all the articles the online service [NewsAPI](https://newsapi.org/)    2.	Combine the articles into one document (`str`)    3.	Clean data    4.	Compute word-frequency pairs    5.	Visualize the analysis in a Word Cloud      *Some definitions:*A **web service** is an application run by a web server that can be called from a series of URLs that will return their data in a format intended for parsing by a "generic" computer program rather than by a browser. As a result, web services can use something like XML (especially SOAP or so) or JSON is used.An **Application Programming Interface (API)** allows two systems to communicate with one another. An API exactly defines the methods for one software program to interact with the other. ![](images/web_server_web_service.png) This document has been __created by <NAME>__ and elaborated by <NAME>.  The modules we need to use<jupyter_code>%pip install newsapi-python
%pip install wordcloud
# integrating News API into your Python application 
from newsapi import NewsApiClient 

# creating wordclouds into your Python application 
from wordcloud import WordCloud, STOPWORDS

#  In Python, string.punctuation will give the all sets of punctuation. 
import string

import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>__NewsApiClient__ is a local client for the online service [__NewsAPI__](https://newsapi.org/) that allows you to get news from [eveywhere in the world](https://newsapi.org/sources)* Intuitively, an online service is like an online function running in a remote server (computer)* A local client is a piece of code runnning in your machine that communicates with the online service  * It allows you to easily interact with the online service  * The interaction is typically done through a set of message exchanges as prescribed by the APIs of the service* We need a `api_key` that is a univoque identifier   * necessary when making requests to be identified  * can be obtained registering [here](https://newsapi.org/register) There exist many online services accessible with Python (e.g. [Twitter API](https://www.tweepy.org/))* Most of them follow this api-key approach* E.g. to enforce subscriptions  Implementing two functions for data cleaning __Stop words__ are usually the most common words in any text ( like “the”, “of”, “to” and “and”), * They don’t tell us much about the actual content in a text* These are the words we want to ignore - this is our data cleaningWe want to find the words that will help us differentiate a text from texts that are about different subjects. * __We will filter out the common words__.* Of course, we are going to use a function for doing this Furthermore, we want to ignore punctuation * In Python, string.punctuation will give the all sets of punctuation.* We are going to use it to delete all punctuation from a string<jupyter_code># to remove STOPWORDS from a list of strings
def remove_stopwords(wordlist):
    return [w for w in wordlist if w not in STOPWORDS]

# to remove Punctuations from a string
def remove_punctuations(my_str): 
    no_punct = ""
    for char in my_str:
        if char  not in string.punctuation:
            no_punct = no_punct + char
    return no_punct

# for testing 
remove_stopwords(remove_punctuations('Andrea likes pizza and Python!! and you?').split())<jupyter_output><empty_output><jupyter_text>Data loading We first get the most relevant articles (top 100) about coronavirus, published in usa-today* This looks like a [normal function invocation](https://newsapi.org/docs/client-libraries/python)* But under the hood a lot happens to contact the remote service (https://newsapi.org/v2/everything?q=coronavirus&sources=usa-today&language=en&pageSize=100&sortBy=relevancy&apiKey=<KEY>)Luckily, the function `get_everything` hides this complexity to us<jupyter_code>newsapi = NewsApiClient(api_key='<KEY>')
json_data = newsapi.get_everything(q='coronavirus',        # All the articles that contain this word
                                    language='en',
                                    sources = 'usa-today', # The newspaper of interest
                                    #sources = 'cnn', # The newspaper of interest
                                    #sources = 'bbc-news', # The newspaper of interest
                                    #sources = 'bbc-news,the-verge'
                                    #
                                    page_size=100,           # Get 100 articles
                                    sort_by='relevancy',
                                    
                                   
                                    )
#There exist many more parameters allowing us the ask for more articles, different sources, etc<jupyter_output><empty_output><jupyter_text>The return value `json_data` is just a dictionary collecting information about the remote request<jupyter_code>print(type(json_data))
print()
print(json_data.keys())
print()
print('The status of the request is',json_data['status'],'There are ',json_data['totalResults'],'articles matching the request.')
print('We got',len(json_data['articles']),'articles')
print()<jupyter_output><empty_output><jupyter_text>We can ignore all the metadata (i.e. the data about the remote request) and focus on the actual articles only<jupyter_code>articles = json_data['articles'] 

#Get the description of the first article
content = articles[0]['description']
print('The content:')
print(content)
print('The content is long',len(content),"characters")<jupyter_output><empty_output><jupyter_text>Now let's combine the content of all the news into variable `contents`<jupyter_code>def combine_articles(list_of_articles):
    contents = ''
    for article in list_of_articles:
         #  check if key 'description' has Non-None value in dictionary 
        if article['description']:
            contents += article['description']+" "
            
    return contents

contents = combine_articles(articles)
print('Overall we have',len(contents),'characters')<jupyter_output><empty_output><jupyter_text>Data cleaning, manipulation and visualization  Now begins the funny part1. Let's clean the data removing not interesting words2. Let's compute the frequency of use of words3. Let's visualize graphically the computed analysis - Word Cloud<jupyter_code># 1 cleaning data
def clean_data(content_to_clean):
    print('Before cleaning')
    print(content_to_clean[:600])
    print()
    # simple text normalization: string in lower case
    content_to_clean = content_to_clean.lower()
    # remove punctuations
    cleaned_content = remove_punctuations(content_to_clean).split()
    # remove stopwords
    cleaned_content = remove_stopwords(cleaned_content)
    print('After cleaning')
    print(cleaned_content[:200])
    print()
    return cleaned_content

cleaned_content = clean_data(contents)
# 2 computing word-frequency pairs
def analyze_data(cleaned_content):
    wordcount = {}
    for w in cleaned_content:
        if w in wordcount:
            wordcount[w] += 1
        else:
            wordcount[w] = 1
    return wordcount

wordcount = analyze_data(cleaned_content)
# 3 visualizing the analysisis in a Word Cloud
def visualize_data(wordcount):
    wc = WordCloud(width=1200,height=600, background_color='white').generate_from_frequencies(wordcount)
    plt.figure(figsize=(20,10))
    plt.imshow(wc)
    plt.axis("off")
    plt.show()

visualize_data(wordcount)<jupyter_output><empty_output><jupyter_text>Let's put everything in a function, so that we can run it easily for different sources<jupyter_code>def make_world_cloud(query,source,n_articles=50):
    """
    Download relevant 'n_article' articles that contain 'query' from 'source' newspaper using NewsAPI
    and plot a wordcloud with the most common words in the corpus.
    :param query: 
        Keywords or phrases to search for in the article title and body.
    :param source: 
        the news source or blog you want headlines from
    :param n_articles:
        The number of articles to return (maximum 100)
    """
    # 1.  Download all the articles the online service NewsAPI
    newsapi = NewsApiClient(api_key='<KEY>')
    json_data = newsapi.get_everything(q=query,           # All the articles that contain this word
                                       language='en',
                                       sources=source,      # The newspaper of interest
                                       page_size=n_articles,  # Get 50 articles (by default)
                                       sort_by='relevancy'
                                    )
    
    # 2. Combine the articles into one document (string)
    contents = combine_articles(json_data['articles'])
    
    # 3. cleaning data
    cleaned_content = clean_data(contents)

    # 4. computing word-frequency pairs
    wordcount = analyze_data(cleaned_content)
 
    # 5. Visualizing the analysis in a Word Cloud
    visualize_data(wordcount)
make_world_cloud('coronavirus','the-verge',100)
make_world_cloud('coronavirus','bbc-news',100)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0754697286012526, 'fraction_numerical': 0.007515657620041753, 'mean_word_length': 3.427449168207024, 'pattern_counts': {'":': 0, '<': 42, '<?xml version=': 0, '>': 42, 'https://': 7, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 19, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '346376', 'license': 'MIT', 'n_tokens_mistral': 2633, 'n_tokens_neox': 2493, 'n_words': 984, 'path': 'PDA/jupyter/jupyterNotebooks/.ipynb_checkpoints/05WordCloudCovid-checkpoint.ipynb', 'repo_name': 'EMbeDS-education/StatsAndComputing20212022', 'size': '9725'}