<jupyter_start><jupyter_text>Machine Learning With Text : Sentiment Analysis This is the first cut at a series of exercises to conduct sentiment analysis for the Microsoft Reactor Machine Learning initiative. Machine learning does not just have to be about numbers. We can turn text into quantifiable data to be able to look at things like document similarity and clusters. We can also engage Natural Language Processing (NLP) tools to perform sentiment analysis, named entity recognition and more.Consider the explosion of data we face and then realize that we have similar issues with documents in the form of news stories or magazine articles. You either need to read very fast (or constantly) or rely on some reporting source to tell you what is going on. It is clear that not every news story is unbiased and sometimes there may be positive or negative spin on a story, so how do you look behind the scenes to see what is going on?We can increasingly rely on machine learning systems to help us process documents and extract information and other value from them. If you work in a particular industry, you could start to see what is being said about your industry. If you have an investment portfolio, you could imagine letting software read the news and draw your attention to positive or negative swings in the reporting on indeustries or products you're invested in. If you work for an organization, you could get a sense of what people were saying about it on social media.In our exercise, we will use an RSS feed of the news headlines from Microsoft Money as a source of data. As time passes, you could store and assess what is the general sentiment of the news and begin to ask questions like: Who is being mentioned most often? Are positive and negative things being said about these individuals, organizations or places?PLEASE NOTE: THE RESULTS OF THIS EXERCISE WILL CHANGE OVER TIME AND DO NOT CONSTITUTE LEGAL OR FINANCIAL ADVICE. PLEASE CONSULT WITH YOUR OWN EXPERTS BEFORE MAKING ANY DECISIONS ON WHAT YOU READ IN THE NEWS OR SEE HERE.You will need to install BeautifulSoup, nltk and feedparser.  Notes from Sarah: Can you include links to the documentation for these packages?  Let's also include the pip install commands in `code` style int he markdown. In particular I had to do a `pip install beautifulsoup4`<jupyter_code>import feedparser
import requests
import hashlib
import tempfile
import os
from bs4 import BeautifulSoup
from nltk.sentiment.vader import SentimentIntensityAnalyzer
import nltk
from nltk import tokenize
import numpy as np

# We need to load the lexicons that the VADER algorithm uses to assess sentiment.

nltk.download('punkt')
nltk.download('vader_lexicon')<jupyter_output>[nltk_data] Downloading package punkt to
[nltk_data]     C:\Users\sguthals\AppData\Roaming\nltk_data...
[nltk_data]   Unzipping tokenizers\punkt.zip.
[nltk_data] Downloading package vader_lexicon to
[nltk_data]     C:\Users\sguthals\AppData\Roaming\nltk_data...<jupyter_text>We do not want to hammer any websites or RSS feeds, so the first couple of functions will cache the feeds locally in a temporary directory. This first one makes sure that the temporary directory we are expecting exists.<jupyter_code>def createTempDirIfNecessary() :
    tempDir = tempfile.gettempdir()
    path = os.path.join(tempDir, "msft-reactor-ml")

    if(not os.path.exists(path)) :
        os.mkdir(path)

    return path<jupyter_output><empty_output><jupyter_text>We hash a URL and use that as the basis of the file identity in our temporary directory. If we already have a file with this name, we won't try to fetch it again. Removing the temporary file will force it to be re-fetched.TODO:* Fetch hashed files if they are over some threshold of days old to keep the feed relatively up-to-date.<jupyter_code>def fetchUrlIfNecessary(url) :
    path = createTempDirIfNecessary()

    hashed = hashlib.sha256(url.encode()).hexdigest()

    file = os.path.join(path, f"{hashed}.in.txt")

    if(not os.path.exists(file)) :
        response = requests.get(url)
        with open(file, mode='wb') as localfile:
            localfile.write(response.content)

    return file<jupyter_output><empty_output><jupyter_text>This function strips non-text-related structures from the fetched HTML files (e.g. scripts, style tags, links, etc.) to result in relatively clean text.<jupyter_code># https://stackoverflow.com/questions/30565404/remove-all-style-scripts-and-html-tags-from-an-html-page/30565420

def cleanMe(html):
    soup = BeautifulSoup(html, "html.parser") # create a new bs4 object from the html data loaded
    for script in soup(["script", "style", "a", "li"]): # remove all javascript and stylesheet code
        script.extract()
    # get text
    text = soup.get_text()
    # break into lines and remove leading and trailing space on each
    lines = (line.strip() for line in text.splitlines())
    # break multi-headlines into a line each
    chunks = (phrase.strip() for line in lines for phrase in line.split("  "))
    # drop blank lines
    text = '\n'.join(chunk for chunk in chunks if chunk)
    return text<jupyter_output><empty_output><jupyter_text>This function iterates over all of the feed elements, fetches them if need be, cleans up the text and dumps out the text files for future processing. Notes from Sarah:  I got a unicode error on this one. should we add a try-catch or something?  Also missing comments in the following cells.<jupyter_code>feedFile = fetchUrlIfNecessary("http://rss.msn.com/en-us/money?feedoutput=rss")
feed = feedparser.parse(f"file:///{feedFile}")

results = []

for post in feed.entries:
    print(f"Fetching: {post.link}")
    
    entryFile = fetchUrlIfNecessary(post.link)

    with open (entryFile, "r" ) as myfile:
        data = myfile.read()
        data = cleanMe(data)

        outfile = open(f"{entryFile}.out.txt", "w")
        outfile.write(data)
        outfile.close()<jupyter_output>Fetching: https://www.msn.com/en-us/money/companies/us-companies-got-emergency-government-loans-despite-having-months-of-cash/ar-BB13Kwsf?srcref=rss<jupyter_text>This is a sample cell for showing the steps for analysis of one text file. We iterate over the sentences and print out the Vader sentiment polarity scores per sentence.<jupyter_code>from nltk import tokenize

feedFile = fetchUrlIfNecessary("http://rss.msn.com/en-us/money?feedoutput=rss")
feed = feedparser.parse(f"file:///{feedFile}")
postFile = fetchUrlIfNecessary(feed.entries[0].link)

with open(postFile, "r") as f:
    data = f.read()
    data = cleanMe(data)
    sentences = tokenize.sent_tokenize(data)

    sid = SentimentIntensityAnalyzer()
    for sentence in sentences:
        print(sentence)
        ss = sid.polarity_scores(sentence)
        for k in sorted(ss):
            print('{0}: {1}, '.format(k, ss[k]), end='')
        print()<jupyter_output><empty_output><jupyter_text>We are going to start by iterating over all of the cached feeds and do sentiment analysis on the feed entry titles with an analyzer from the NLTK. This tool calculates the positive, negative, neutral and compound scores per headline. We accumulate the results for further analysis later to be put into a Pandas DataFrame.<jupyter_code>feedFile = fetchUrlIfNecessary("http://rss.msn.com/en-us/money?feedoutput=rss")
feed = feedparser.parse(f"file:///{feedFile}")

results = []
sid = SentimentIntensityAnalyzer()

for post in feed.entries:
    print(f"Analyzing: {post.title}")

    scores = sid.polarity_scores(post.title)
    scores['title'] = post.title

    print(f"compound: {scores['compound']}")
    print(f"neg: {scores['neg']}")
    print(f"neu: {scores['neu']}")
    print(f"pos: {scores['pos']}")
    print()

    results.append(scores)<jupyter_output>Analyzing: A mortgage break isn't the boon many people expected
compound: 0.0
neg: 0.0
neu: 1.0
pos: 0.0

Analyzing: The latest housing numbers are worse than they look
compound: -0.4767
neg: 0.279
neu: 0.721
pos: 0.0

Analyzing: These states don't tax unemployment benefits
compound: 0.0572
neg: 0.254
neu: 0.466
pos: 0.28

Analyzing: Surprising things the world is buying in lockdown
compound: 0.2732
neg: 0.0
neu: 0.769
pos: 0.231

Analyzing: Harvard prof reaps 17,000% return on early vaccine bet
compound: 0.0
neg: 0.0
neu: 1.0
pos: 0.0

Analyzing: The pandemic is pushing Americans into early retirement
compound: 0.0
neg: 0.0
neu: 1.0
pos: 0.0

Analyzing: Credit cards cutting limits for people facing tough times
compound: 0.1531
neg: 0.259
neu: 0.517
pos: 0.224

Analyzing: Stocks waver on news of possible virus treatment
compound: 0.0
neg: 0.0
neu: 1.0
pos: 0.0

Analyzing: These things might not survive the pandemic
compound: 0.0
neg: 0.0
neu: 1.0
pos: 0.0

Analyzing: 'I did not sign up[...]<jupyter_text>As you may have seen in the Microsoft Reactor Data Science workshops, Pandas DataFrames are useful for manipulating data. We will build an instance from our collected results.<jupyter_code>import pandas as pd 
df = pd.DataFrame.from_records(results)
df.head()<jupyter_output><empty_output><jupyter_text>The sentiment analysis tool defines a range of values between -1.0 and 1.0 representating very negative to very positive values. This function establishes a threshold below which, we will consider to be a negative score. Anything above the threshold will be considered a positive score. Values in-between the positive and negative scores, we will be consider them to be neutral. We will return a score of 1 for positive, 0 for neutral and -1 for negative results. Feel free to change these thresholds to what makes sense to you.<jupyter_code>def label_lambda(x):
    retVal = 0

    if (x < 0.3) & (x > -0.3):
        retVal = 0
    elif x >= 0.3:
        retVal = 1
    else:
        retVal = -1

    return retVal<jupyter_output><empty_output><jupyter_text>We will add a label feature to our DataFrame by applying our lambda function to the compund sentiment rating.<jupyter_code>df['label'] = df['compound'].apply(label_lambda)
df.head()<jupyter_output><empty_output><jupyter_text>Grab the numeric columns for cluster analysis.  Notes from Sarah:  At this point explain why you're doing this and why you aren't including the -1, 0, 1 numerical column<jupyter_code>df2 = df[['compound', 'neg', 'neu', 'pos']]
df2.head()<jupyter_output><empty_output><jupyter_text>Cluster the sentences based on their numbers using SciKit-Learn and its K-Means implementation. Consider the implications of different numbers of clusters.  Notes from Sarah:  Please include a sentence or two on the goal of this. It may seem obvious, but just to make sure the point is clear.<jupyter_code>from sklearn.cluster import KMeans
kmeans = KMeans(n_clusters=3)
kmeans.fit(df2.values)

kmeans.labels_<jupyter_output><empty_output><jupyter_text>It might be interesting to visualize the clusters and compare the texts. For now, let's just add the label feature to our DataFrame as well.  Notes from Sarah  Again, include a sentence or two on why. What is the goal here? What made you want to add the lable feature back tot he dataframe? And didn't we already have the label feature? Did you mean add the cluster?<jupyter_code>df['cluster'] = kmeans.labels_
df.head()<jupyter_output><empty_output><jupyter_text>We can analyze the average compound sentiment ratings for each of the clusters. Unsurprisingly, we have found clusters of mostly positive, neutral or negative texts.  Notes from Sarah:  Can you add more explanation here about how we clustered without our -1, 0 ,1 labeling and we can see a similar clusting happening as a result of the algorithm? Maybe looking at the actual comparison between our thresholds and the models?<jupyter_code>cluster_zero = df[df.cluster==0]['compound'].mean()
cluster_one = df[df.cluster==1]['compound'].mean()
cluster_two = df[df.cluster==2]['compound'].mean()

print(f"Average compound score for cluster 0: {cluster_zero}")
print(f"Average compound score for cluster 1: {cluster_one}")
print(f"Average compound score for cluster 2: {cluster_two}")<jupyter_output>Average compound score for cluster 0: -0.4147136363636364
Average compound score for cluster 1: 0.004927419354838709
Average compound score for cluster 2: 0.34895600000000004<jupyter_text>Pandas allows us to easily look at average values for all of our features, not just the compound values.<jupyter_code># Average values for all metrics in cluster 0
df[df.cluster==0].mean()
# Average values for all metrics in cluster 1
df[df.cluster==1].mean()<jupyter_output><empty_output><jupyter_text>Let's visualize the clusters using the popular Matplotlib library. We need to import the functionality, but we also need to tell Jupyter Notebooks to render the results inline.<jupyter_code>import matplotlib.pyplot as plt
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Notes from Sarah:  Can you describe the code below?<jupyter_code>plt.scatter(df['pos'], df['neg'], s=(100 * df['compound'] + 50), c=df['label'])
from mpl_toolkits.mplot3d import Axes3D

estimators = [('k_means_3', KMeans(n_clusters=3)),
              ('k_means_5', KMeans(n_clusters=5)),
              ('k_means_8', KMeans(n_clusters=8))]

X = df2 = df[['neg', 'neu', 'pos']].values

fignum = 1
titles = ['3 clusters', '5 clusters', '8 clusters']
for name, est in estimators:
    fig = plt.figure(fignum, figsize=(4, 3))
    ax = Axes3D(fig, rect=[0, 0, .95, 1], elev=48, azim=134)
    est.fit(X)
    labels = est.labels_

    ax.scatter(X[:, 0], X[:, 2], X[:, 1],
               c=labels.astype(np.float), edgecolor='k')

    ax.w_xaxis.set_ticklabels([])
    ax.w_yaxis.set_ticklabels([])
    ax.w_zaxis.set_ticklabels([])
    ax.set_xlabel('neg')
    ax.set_ylabel('pos')
    ax.set_zlabel('neu')
    ax.set_title(titles[fignum - 1])
    ax.dist = 12
    fignum = fignum + 1<jupyter_output><empty_output><jupyter_text>Notes from Sarah:  Missing some lines here about what insight the learner might gain from those visualizations. We are now going to use Microsoft's Cognitive Services Text Analytics Services for even more functionality that can scale up to larger datasets. You will need to install the Python Text Analytics client:`pip install --upgrade azure-cognitiveservices-language-textanalytics`The following block will require you supply a key and endpoint that you would have been given when you registered for a Free Account. Please supply these values for you below. These will then be used to authenticate the client to the service. You will be able to use the client in cells below.NOTE: PLEASE DO NOT CHECK IN JUPYTER NOTEBOOKS WITH PRIVATE KEYS IN THEM TO CHANGE MANAGEMENT SYSTEMS SUCH AS GIT!!! Notes from Sarah:  Since we used the sentiment analysis from nltk earlier, can you add a few sentences as to why someone would use that then but use cognitive services now?<jupyter_code># key = ADD_YOUR_KEY_HERE
# endpoint = ADD_YOUR_ENDPOINT_URL_HERE

import os
from azure.cognitiveservices.language.textanalytics import TextAnalyticsClient
from msrest.authentication import CognitiveServicesCredentials

def authenticateClient():
    credentials = CognitiveServicesCredentials(key)
    text_analytics_client = TextAnalyticsClient(
        endpoint=endpoint, credentials=credentials)
    return text_analytics_client

client = authenticateClient()<jupyter_output><empty_output><jupyter_text>We are now going to process our headlines through the Text Analytics APIs.The general structure the APIs expect is something like:{    "id" : 0,    "text" : "The body of text you wish to analyze."}So, we will build this structure. Some of the services want to know the source language (e.g. 'en', 'jp', 'fr'). We do not have this information yet, so we will rely on the language detection service to decide this for us. After this we conduct a sentiment analysis on the body of the text. This will give us a numeric summary between 0.0 (negative) and 1.0 (positive).Finally, we will find the key phrases in the titles as well.<jupyter_code>feedFile = fetchUrlIfNecessary("http://rss.msn.com/en-us/money?feedoutput=rss")
feed = feedparser.parse(f"file:///{feedFile}")

documents = []
i = 0

for post in feed.entries:
    print(f"Analyzing: {post.title}")
    documents.append({'text' : post.title, "id" : i})
    i += 1

response = client.detect_language(documents=documents)

for document in response.documents:
   documents[int(document.id)]['language'] = document.detected_languages[0].iso6391_name

response = client.sentiment(documents=documents)

for document in response.documents:
    documents[int(document.id)]['sentiment'] = document.score

response = client.key_phrases(documents=documents)

for document in response.documents:
    documents[int(document.id)]['keyphrases'] = document.key_phrases

print(documents)<jupyter_output>yzing: 20 industries that will never be the same after the coronavirus
Analyzing: How grocery stores have fared during the pandemic
Analyzing: These retailers and restaurants may not survive coronavirus
Analyzing: Iconic US brands that aren't made in America
Analyzing: Cities where home prices plummeted as early as February
Analyzing: Predictions on when we will bounce back from COVID-19
Analyzing: How America's biggest companies are impacted by COVID-19
Analyzing: Netflix's incredible rise from rentals to ruling the world
Analyzing: The number of COVID-19 cases per capita in each state
Analyzing: These products are coming back to store shelves soon
Analyzing: 10 things to know about filing for unemployment
Analyzing: 50 remote jobs that can pay well
Analyzing: Where the coronavirus recession will likely hit hardest
Analyzing: Restaurant chains that are closing locations
Analyzing: These states are the most financially prepared for a recession
Analyzing: Countries with the most debt
An[...]<jupyter_text>One of the interesting things we can do is find references to specific individuals, organizations or places. These discovered entities will be tagged as being of type 'Person' or 'Organization'. This is useful metadata that we could use to do further analysis. We will filter out less meaningful types such as 'Quantity' or 'DateTime'. We add these entity types to our document records.<jupyter_code>response = client.entities(documents=documents)

for document in response.documents:
    entities = []

    for entity in document.entities:
        if entity.type != 'Quantity' and entity.type != 'DateTime':
            entities.append({'name' : entity.name, 'type' : entity.type, 'subtype' : entity.sub_type })

    documents[int(document.id)]['entities'] = entities

print(documents)<jupyter_output>'id': 20, 'language': 'en', 'sentiment': 0.5, 'keyphrases': ['news of possible virus treatment', 'Stocks waver'], 'entities': []}, {'text': 'The latest housing numbers are worse than they look', 'id': 21, 'language': 'en', 'sentiment': 0.00920066237449646, 'keyphrases': ['latest housing numbers'], 'entities': []}, {'text': "Whopping jobless numbers still don't tell the whole story", 'id': 22, 'language': 'en', 'sentiment': 0.21789979934692383, 'keyphrases': ['Whopping jobless numbers', 'story'], 'entities': []}, {'text': 'Millions join unemployed ranks as virus ravages economy', 'id': 23, 'language': 'en', 'sentiment': 0.1355624794960022, 'keyphrases': ['unemployed ranks', 'virus ravages economy', 'Millions'], 'entities': []}, {'text': 'State and city workers could be the next wave of layoffs', 'id': 24, 'language': 'en', 'sentiment': 0.5, 'keyphrases': ['city workers', 'wave of layoffs', 'State'], 'entities': [{'name': 'State', 'type': 'Organization', 'subtype': None}]}, {'text': 'Oil[...]<jupyter_text>We will allow people to ask questions about positive or negative references to different types of entities.<jupyter_code>def addReference(refs, key, sentiment, text):
   currentRefs = refs.get(key, {'sentiment' : [], 'headline' : []})
   currentRefs['sentiment'].append(document['sentiment'])
   currentRefs['headline'].append(document['text'])
   refs[key] = currentRefs

def averageSentimentValue(ratings) :
    return np.array(ratings).mean()

def entitySummary(entityType, refs, polarity):
    print(f"{entityType} with the most {polarity} coverage:")

    for entity in refs.keys() :
        print('---------------')
        print(f"Name: {entity}")
        avg = averageSentimentValue(refs[entity]['sentiment'])
        print(f" .  Average sentiment analysis: {avg}")
        headlines = refs[entity]['headline']
        print(" .  Headlines:")
        for h in headlines:
            print(f"      {h}")

    print()<jupyter_output><empty_output><jupyter_text>We will now keep track of which entities are referenced the most and who has the most positive and negative references.<jupyter_code>entityCount = {}
entityTypeCount = {}
posPersonRefs = {}
negPersonRefs = {}
posOrgRefs = {}
negOrgRefs = {}

for document in documents:
    for entity in document['entities']:
        name = entity['name']
        t = entity['type']
        count = entityCount.get(name, 0)
        entityCount[name] = count + 1
        count = entityTypeCount.get(t, 0)
        entityTypeCount[t] = count + 1

        if document['sentiment'] >= 0.6:
            # Add the positive references
            if t == 'Person' :
                print(f"Adding + person reference for {name}")
                addReference(posPersonRefs, name, document['sentiment'], document['text'])
            elif t == 'Organization' :
                print(f"Adding + org reference for {name}")
                addReference(posOrgRefs, name, document['sentiment'], document['text'])

        elif document['sentiment'] <= 0.4:
            # Add the negative references
            if t == 'Person' :
                print(f"Adding - person reference for {name}")
                addReference(negPersonRefs, name, document['sentiment'], document['text'])
            elif t == 'Organization' :
                print(f"Adding - org reference for {name}")
                addReference(negOrgRefs, name, document['sentiment'], document['text'])<jupyter_output>Adding - org reference for Costco
Adding - org reference for Netflix
Adding + org reference for Chipotle Mexican Grill
Adding - org reference for UK
Adding - org reference for Delta Airlines
Adding + org reference for Nasdaq
Adding + org reference for Saving & Spending
Adding - org reference for Costco
Adding + org reference for LinkedIn
Adding + org reference for Chipotle
Adding + org reference for Netflix
Adding + person reference for <NAME>
Adding + org reference for Iconic US
Adding + org reference for Netflix
Adding + org reference for Company (military unit)<jupyter_text>We can now summarize the most positive and negative mentions of People and Organizations.<jupyter_code>entitySummary('People', posPersonRefs, 'positive')
entitySummary('People', negPersonRefs, 'negative')
entitySummary('Organizations', posOrgRefs, 'positive')
entitySummary('Organizations', negOrgRefs, 'negative')<jupyter_output>People with the most positive coverage:
---------------
Name: <NAME>
 .  Average sentiment analysis: 0.9144583344459534
 .  Headlines:
      Stocks of 'good corporate citizens' such as Chipotle, Netflix are thriving, <NAME> says

People with the most negative coverage:

Organizations with the most positive coverage:
---------------
Name: <NAME>
 .  Average sentiment analysis: 0.868740439414978
 .  Headlines:
      Chipotle handling pandemic much better than other chains
---------------
Name: Nasdaq
 .  Average sentiment analysis: 0.8922896385192871
 .  Headlines:
      Nasdaq 'hopeful' that larger IPOs are around the corner
---------------
Name: <NAME>
 .  Average sentiment analysis: 0.7526739239692688
 .  Headlines:
      Saving & Spending
---------------
Name: LinkedIn
 .  Average sentiment analysis: 0.7435328960418701
 .  Headlines:
      LinkedIn: These companies are hiring now
---------------
Name: Chipotle
 .  Average sentiment analysis: 0.91445[...]<jupyter_text>We are still operating in Jupyter notebooks, so we can do some Python code to visualize which entities are mentioned the most in the news.<jupyter_code>topEntities = {}
n = 10

count = 0

for w in sorted(entityCount, key=entityCount.get, reverse=True):
   topEntities[w] = entityCount[w]
   count = count + 1
   if count > n:
       break

fig, axs = plt.subplots(1)
fig.suptitle('Entities in the News')
labels = list(topEntities.keys())
axs.bar(labels, topEntities.values())
axs.set_xticklabels(labels, rotation=45)
plt.show()<jupyter_output><empty_output><jupyter_text>We could also keep track of which types of entities are being mentioned in the news.<jupyter_code>fig, axs = plt.subplots(1)
fig.suptitle('Entity Types in the News')
axs.bar(list(entityTypeCount.keys()), entityTypeCount.values())
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09657119806373538, 'fraction_numerical': 0.01871722468737394, 'mean_word_length': 4.39756150664054, 'pattern_counts': {'":': 0, '<': 100, '<?xml version=': 0, '>': 102, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '25', 'dir': 'jupyter-structured-clean-dedup', 'id': '566552', 'license': 'MIT', 'n_tokens_mistral': 7647, 'n_tokens_neox': 7136, 'n_words': 2791, 'path': 'workshop-resources/data-science-and-machine-learning/Machine_Learning_1/financial-project/Sentiment-Analysis.ipynb', 'repo_name': 'runewake2/Reactors', 'size': '24829'}