<jupyter_start><jupyter_text>[View in Colaboratory](https://colab.research.google.com/github/schwaaweb/aimlds1_11-NLP/blob/master/Th11_CC_DJ_NLP_Coding_Challenge__4.ipynb)  Coding Challenge 4: Natural Language Processing In this Coding Challenge, you will decipher the sentiment within a large set of tweets regarding a specific topic on Twitter in an autonomous fashion. **Challenge ****1)** Construct a Twitter Sentiment Analyzer. The Sentiment Analyzer will search twitter for a list of tweets about any topic of choice, then it will analyze each tweet to ascertain whether a positive or negative emotion is expressed in the tweet**2)** Create a graph of polarity to determine whether the trend over time is a positive one or a negative one. *Hint*: you will have to define a StreamListener instance to continuously listen in on a specific topic and then graph the polarity to showcase the trend (i.e. positive or negative)***Setup***:Before you can work on the challenge, the following items need to be taken care of:1) Install dependencies: a) pip install tweepy - library for accessing the twitter API, b) textblob (pip install textblob) which will facilitate the actual task of performing sentiment analysis.2) You need data - tweepy lets you get data live from the Twitter API, which you have to register for. a) Go to https://apps.twitter.com, b) click on "Create New Application", c) Under Application details, enter name, description, website details (can be anything), d) click on "Create Application" and e) make a note of the tokens on the "Keys and Access Tokens" tab.If you are blocked on the Twitter API/tweepy step, it is okay to manually get or code some test tweets to still use for sentiment analysis, and revisit the API step later. Good luck!<jupyter_code>%%time
!conda install -c conda-forge textblob tweepy --yes<jupyter_output>Solving environment: done

## Package Plan ##

  environment location: /Users/darwinm/anaconda3

  added / updated specs: 
    - textblob
    - tweepy


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    blinker-1.4                |             py_0          13 KB  conda-forge
    conda-4.5.4                |           py36_0         622 KB  conda-forge
    oauthlib-2.1.0             |             py_0          69 KB  conda-forge
    tweepy-3.5.0               |           py36_0          50 KB  conda-forge
    requests-oauthlib-0.8.0    |           py36_1          26 KB  conda-forge
    textblob-0.15.1            |             py_0         597 KB  conda-forge
    pyjwt-1.6.4                |             py_0          16 KB  conda-forge
    certifi-2018.4.16          |           py36_0         142 KB  conda-forge
    ------------------------------------------------------------
              [...]<jupyter_text>App Info:Name: schwaaweb_twt_sent.20180614Description: simple sentiment analysisWebsite: https://amazing-euclid-b565e6.netlify.com/          _____________          Consumer Key           Consumer Key (API Key)	hse80rQnATyXi7lPgMXXoJRmPConsumer Secret (API Secret)	-------------------------------Access Level	Read and write (modify app permissions)Owner	schwaawebOwner ID	3110959311         -----------------         Access Token         Access Token	<PASSWORD>-4u4cJDHgP1xUODq9ZeDul4j8YL4SIgjFkVC57VUAccess Token Secret	--------------------------Access Level	Read and writeOwner	schwaawebOwner ID	3110959311 **1)** Construct a Twitter Sentiment Analyzer. The Sentiment Analyzer will search twitter for a list of tweets about any topic of choice, then it will analyze each tweet to ascertain whether a positive or negative emotion is expressed in the tweet [Tutorial](https://dev.to/rodolfoferro/sentiment-analysis-on-trumpss-tweets-using-python-)<jupyter_code>%%time
# General:
import tweepy           # To consume Twitter's API
import pandas as pd     # To handle data
import numpy as np      # For number computing

# For plotting and visualization:
from IPython.display import display
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
# We import our access keys:
from credentials import *    # This will allow us to use the keys as variables

# API's setup:
def twitter_setup():
    """
    Utility function to setup the Twitter's API
    with our access keys provided.
    """
    # Authentication and access using keys:
    auth = tweepy.OAuthHandler(CONSUMER_KEY, CONSUMER_SECRET)
    auth.set_access_token(ACCESS_TOKEN, ACCESS_SECRET)

    # Return API with authentication:
    api = tweepy.API(auth)
    return api
# We create an extractor object:
extractor = twitter_setup()

# We create a tweet list as follows:
tweets = extractor.user_timeline(screen_name="kanyewest", count=200)
print("Number of tweets extracted: {}.\n".format(len(tweets)))

# We print the most recent 5 tweets:
print("5 recent tweets:\n")
for tweet in tweets[:5]:
    print(tweet.text)
    print()
df = pd.DataFrame(data=[tweet.text for tweet in tweets], columns=['Tweets'])
# Tweepy
print(dir(tweets[0]))
print(tweets[0].id)
print(tweets[0].created_at)
print(tweets[0].source)
print(tweets[0].favorite_count)
print(tweets[0].retweet_count)
print(tweets[0].geo)
print(tweets[0].coordinates)
print(tweets[0].entities)
df['len']  = np.array([len(tweet.text) for tweet in tweets])
df['ID']   = np.array([tweet.id for tweet in tweets])
df['Date'] = np.array([tweet.created_at for tweet in tweets])
df['Source'] = np.array([tweet.source for tweet in tweets])
df['Likes']  = np.array([tweet.favorite_count for tweet in tweets])
df['RTs']    = np.array([tweet.retweet_count for tweet in tweets])
df.head()
mean = np.mean(df['len'])
print("The lenght's average in tweets: {}".format(mean))
# We extract the tweet with more FAVs and more RTs:

fav_max = np.max(df['Likes'])
rt_max  = np.max(df['RTs'])

fav = df[df.Likes == fav_max].index[0]
rt  = df[df.RTs == rt_max].index[0]

# Max FAVs:
print("The tweet with more likes is: \n{}".format(df['Tweets'][fav]))
print("Number of likes: {}".format(fav_max))
print("{} characters.\n".format(df['len'][fav]))

# Max RTs:
print("The tweet with more retweets is: \n{}".format(df['Tweets'][rt]))
print("Number of retweets: {}".format(rt_max))
print("{} characters.\n".format(df['len'][rt]))
# We create time series for data:

tlen = pd.Series(data=df['len'].values, index=df['Date'])
tfav = pd.Series(data=df['Likes'].values, index=df['Date'])
tret = pd.Series(data=df['RTs'].values, index=df['Date'])
# Lenghts along time:
tlen.plot(figsize=(16,4), color='r');
# Likes vs retweets visualization:
tfav.plot(figsize=(16,4), label="Likes", legend=True)
tret.plot(figsize=(16,4), label="Retweets", legend=True);
# We obtain all possible sources:
sources = []
for source in df['Source']:
    if source not in sources:
        sources.append(source)

# We print sources list:
print("Creation of content sources:")
for source in sources:
    print("* {}".format(source))
# We create a numpy vector mapped to labels:
percent = np.zeros(len(sources))

for source in df['Source']:
    for index in range(len(sources)):
        if source == sources[index]:
            percent[index] += 1
            pass

percent /= 100

# Pie chart:
pie_chart = pd.Series(percent, index=sources, name='Sources')
pie_chart.plot.pie(fontsize=11, autopct='%.2f', figsize=(6, 6));
from textblob import TextBlob
import re

def clean_tweet(tweet):
    '''
    Utility function to clean the text in a tweet by removing 
    links and special characters using regex.
    '''
    return ' '.join(re.sub("(@[A-Za-z0-9]+)|([^0-9A-Za-z \t])|(\w+:\/\/\S+)", " ", tweet).split())

def analize_sentiment(tweet):
    '''
    Utility function to classify the polarity of a tweet
    using textblob.
    '''
    analysis = TextBlob(clean_tweet(tweet))
    if analysis.sentiment.polarity > 0:
        return 1
    elif analysis.sentiment.polarity == 0:
        return 0
    else:
        return -1
# We create a column with the result of the analysis:
df['SA'] = np.array([ analize_sentiment(tweet) for tweet in df['Tweets'] ])

# We display the updated dataframe with the new column:
display(df.head(10))
# We construct lists with classified tweets:

pos_tweets = [ tweet for index, tweet in enumerate(df['Tweets']) if df['SA'][index] > 0]
neu_tweets = [ tweet for index, tweet in enumerate(df['Tweets']) if df['SA'][index] == 0]
neg_tweets = [ tweet for index, tweet in enumerate(df['Tweets']) if df['SA'][index] < 0]

print("Percentage of positive tweets: {}%".format(len(pos_tweets)*100/len(df['Tweets'])))
print("Percentage of neutral tweets: {}%".format(len(neu_tweets)*100/len(df['Tweets'])))
print("Percentage de negative tweets: {}%".format(len(neg_tweets)*100/len(df['Tweets'])))<jupyter_output>Percentage of positive tweets: 29.0%
Percentage of neutral tweets: 63.5%
Percentage de negative tweets: 7.5%
{'fraction_non_alphanumeric': 0.13455691926584845, 'fraction_numerical': 0.02094358743384754, 'mean_word_length': 3.3861728395061728, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 10, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '327821', 'license': 'Unlicense', 'n_tokens_mistral': 2950, 'n_tokens_neox': 2744, 'n_words': 900, 'path': 'Th11_CC_DJ_NLP_Coding_Challenge__4.ipynb', 'repo_name': 'schwaaweb/aimlds1_11-NLP', 'size': '8881'}