<jupyter_start><jupyter_text>Unit 12 - Tales from the Crypto---  1. Sentiment AnalysisUse the [newsapi](https://newsapi.org/) to pull the latest news articles for Bitcoin and Ethereum and create a DataFrame of sentiment scores for each coin.Use descriptive statistics to answer the following questions:1. Which coin had the highest mean positive score?2. Which coin had the highest negative score?3. Which coin had the highest positive score?<jupyter_code># Initial imports
import os
import pandas as pd
from dotenv import load_dotenv
load_dotenv()
import nltk as nltk
nltk.download('vader_lexicon')
from nltk.sentiment.vader import SentimentIntensityAnalyzer
analyzer = SentimentIntensityAnalyzer()

%matplotlib inline
# Read your api key environment variable
api_key = os.getenv("NEWS_API_KEY")
# Create a newsapi client
from newsapi import NewsApiClient
newsapi = NewsApiClient(api_key=api_key)
# Fetch the Bitcoin news articles
btc_news1 = pd.DataFrame(newsapi.get_everything(q="bitcoin", language ="en",sort_by='relevancy',page = 1)['articles'])
btc_news2 = pd.DataFrame(newsapi.get_everything(q="bitcoin", language ="en",sort_by='relevancy',page = 2)['articles'])
btc_news3 = pd.DataFrame(newsapi.get_everything(q="bitcoin", language ="en",sort_by='relevancy',page = 3)['articles'])
btc_news4 = pd.DataFrame(newsapi.get_everything(q="bitcoin", language ="en",sort_by='relevancy',page = 4)['articles'])
btc_news5 = pd.DataFrame(newsapi.get_everything(q="bitcoin", language ="en",sort_by='relevancy',page = 5)['articles'])

btc_news = pd.concat([btc_news1,btc_news2,btc_news3,btc_news4,btc_news5])
btc_news
# Fetch the Ethereum news articles
eth_news = newsapi.get_everything(q="ethereum", language ="en",sort_by='relevancy')

eth_news1 = pd.DataFrame(newsapi.get_everything(q="ethereum", language ="en",sort_by='relevancy',page = 1)['articles'])
eth_news2 = pd.DataFrame(newsapi.get_everything(q="ethereum", language ="en",sort_by='relevancy',page = 2)['articles'])
eth_news3 = pd.DataFrame(newsapi.get_everything(q="ethereum", language ="en",sort_by='relevancy',page = 3)['articles'])
eth_news4 = pd.DataFrame(newsapi.get_everything(q="ethereum", language ="en",sort_by='relevancy',page = 4)['articles'])
eth_news5 = pd.DataFrame(newsapi.get_everything(q="ethereum", language ="en",sort_by='relevancy',page = 5)['articles'])

eth_news = pd.concat([eth_news1,eth_news2,eth_news3,eth_news4,eth_news5])
eth_news.head()
# Create the Bitcoin sentiment scores DataFrame
btc_sentiment = []

for content in btc_news['content']:
    sentiment = analyzer.polarity_scores(content)
    compound = sentiment["compound"]
    pos = sentiment["pos"]
    neu = sentiment["neu"]
    neg = sentiment["neg"]
        
    btc_sentiment.append({
            "text": content,
            "compound": compound,
            "positive": pos,
            "negative": neg,
            "neutral": neu   
        })

# Create DataFrame
btc_df = pd.DataFrame(btc_sentiment)

# Reorder DataFrame columns
cols = ["text", "compound", "positive", "negative", "neutral"]
btc_df = btc_df[cols]

btc_df.head()
# Create the Bitcoin sentiment scores DataFrame
eth_sentiment = []

for content in eth_news['content']:
    sentiment = analyzer.polarity_scores(content)
    compound = sentiment["compound"]
    pos = sentiment["pos"]
    neu = sentiment["neu"]
    neg = sentiment["neg"]
        
    eth_sentiment.append({
            "text": content,
            "compound": compound,
            "positive": pos,
            "negative": neg,
            "neutral": neu   
        })

# Create DataFrame
eth_df = pd.DataFrame(eth_sentiment)

# Reorder DataFrame columns
cols = ["text", "compound", "positive", "negative", "neutral"]
eth_df = eth_df[cols]

eth_df.head()
# Describe the Bitcoin Sentiment
btc_df.describe()
# Describe the Ethereum Sentiment
eth_df.describe()<jupyter_output><empty_output><jupyter_text>Questions:Q: Which coin had the highest mean positive score?*__A: Etheruem had the highest mean positive score, 5.49% vs 4.68% for Bitcoin.__*Q: Which coin had the highest compound score?*__A: Ethereum had the highest mean compound score (12.84% vs 5.92% respectively), as well as the highest compound score for an individual article (84.81% vs 84.02% respectively).__*Q. Which coin had the highest positive score?*__A: Ethereum had the highest positive score (29% vs 28.2% for ETH)__*  2. Natural Language Processing---   TokenizerIn this section, you will use NLTK and Python to tokenize the text for each coin. Be sure to:1. Lowercase each word.2. Remove Punctuation.3. Remove Stopwords.<jupyter_code>from nltk.tokenize import word_tokenize, sent_tokenize
from nltk.corpus import reuters, stopwords
from nltk.stem import WordNetLemmatizer, PorterStemmer
from string import punctuation
import re
btc_string=" ".join(btc_news['content'].astype(str))
# Create a list of stopwords
sentence = sent_tokenize(btc_string)
words = word_tokenize(btc_string)
sw = set(stopwords.words('english'))
first_result = [word.lower() for word in words if word.lower() not in sw]

# unsure if the list has to be expanded 
# Expand list with char reuters
# Instantiate the lemmatizer
lemmatizer = WordNetLemmatizer()
# Complete the tokenizer function
def tokenizer(text):
    """Tokenizes text."""
    
    # Remove the punctuation from text
    regex = re.compile("[^a-zA-Z ]")
    re_clean = regex.sub('', text)

    # Create a tokenized list of the words
    words = word_tokenize(re_clean)    
    
    # Define the stop words
    sw = stopwords.words('english')
    # Additional words as they appear in the top 10 most common words and add little to no value
    sw_addon = {'char', 'reuters'}
    
    # Lemmatize words into root words
    lem = [lemmatizer.lemmatize(word) for word in words]
   
    # Convert the words to lowercase
    output = [word.lower() for word in lem if word.lower() not in sw]
    
    # unsure how to add the custom list - .union(sw_addon) - to be pasted behind sw. in the previous line, error: union is not recognized
    
    return output
# Create a new tokens column for Bitcoin
btc_token = btc_news['content'].apply(tokenizer)
btc_df['token'] = btc_token.values
btc_df.head()
# Create a new tokens column for Ethereum
eth_token = eth_news['content'].apply(tokenizer)
eth_df['token'] = eth_token.values
eth_df.head()<jupyter_output><empty_output><jupyter_text>---  NGrams and Frequency AnalysisIn this section you will look at the ngrams and word frequency for each coin. 1. Use NLTK to produce the n-grams for N = 2. 2. List the top 10 words for each coin.<jupyter_code>from collections import Counter
from nltk import ngrams
# Create btc_string of all tokenized words per article 
btc_string = " ".join(btc_df['token'].astype(str))
# Tokenize btc_string to ensure you have one big string 
btc_string_processed = tokenizer(btc_string)
# Generate the Bitcoin N-grams where N=2
btc_counts = Counter(ngrams(btc_string_processed, n=2))
print(dict(btc_counts))
# Generate the Ethereum N-grams where N=2
# Create eth_string of all tokenized words per article 
eth_string = " ".join(eth_df['token'].astype(str))
# Tokenize eth_string to ensure you have one big string 
eth_string_processed = tokenizer(eth_string)
# Generate the Ethereum N-grams where N=2
eth_counts = Counter(ngrams(eth_string_processed, n=2))
print(dict(eth_counts))
# Function token_count generates the top 10 words for a given coin
def token_count(tokens, N=10):
    """Returns the top N tokens from the frequency count"""
    return Counter(tokens).most_common(N)
# Use token_count to get the top 10 words for Bitcoin

# Create string with individual words
btc_counts_indiv = Counter(ngrams(btc_string_processed, n=1))
# Apply token_count function
token_count(btc_counts_indiv)

#Alternative solution
# print(dict(btc_counts_indiv.most_common(10)))
# Use token_count to get the top 10 words for Ethereum

# Create string with individual words
eth_counts_indiv = Counter(ngrams(eth_string_processed, n=1))
# Apply token_count function
token_count(eth_counts_indiv)

#Alternative solution
# print(dict(eth_counts_indiv.most_common(10)))<jupyter_output><empty_output><jupyter_text>---  Word CloudsIn this section, you will generate word clouds for each coin to summarize the news for each coin<jupyter_code>from wordcloud import WordCloud
import matplotlib.pyplot as plt
plt.style.use('seaborn-whitegrid')
import matplotlib as mpl
mpl.rcParams['figure.figsize'] = [20.0, 10.0]
# Generate the Bitcoin word cloud

# confused - btc_string_processed is already a string (error given - expects a string like object)

wc = WordCloud().generate(btc_string_processed)
plt.imshow(wc)
# Generate the Ethereum word cloud

wc = WordCloud().generate(eth_string_processed)
plt.imshow(wc)<jupyter_output><empty_output><jupyter_text>--- 3. Named Entity RecognitionIn this section, you will build a named entity recognition model for both Bitcoin and Ethereum, then visualize the tags using SpaCy.<jupyter_code>import spacy
from spacy import displacy
# Download the language model for SpaCy
# !python -m spacy download en_core_web_sm
# Load the spaCy model
nlp = spacy.load('en_core_web_sm')<jupyter_output><empty_output><jupyter_text>--- Bitcoin NER<jupyter_code># Concatenate all of the Bitcoin text together
#already done in btc_string
# Run the NER processor on all of the text
btc_NER = nlp(btc_string)

# Add a title to the document
# YOUR CODE HERE!
# Render the visualization
displacy.render(btc_NER, style='ent')
# List all Entities
# all geopolitical entities like in the example. Not sure if that's the question
print([ent.text for ent in btc_NER.ents if ent.label_ == 'GPE'])<jupyter_output>['florida', 'china', 'germany', 'china', "'york'", 'china', 'china', 'paris', "'salvador'", "'york'", "'york'", 'london', "'york'", "'york'", 'china', 'america', "'york'", 'london', 'china', "'beach'", 'washington', "'york'"]<jupyter_text>---  Ethereum NER<jupyter_code># Concatenate all of the Ethereum text together
# already done in eth_string
# Run the NER processor on all of the text
eth_NER = nlp(eth_string)

# Add a title to the document
# YOUR CODE HERE!
# Render the visualization
displacy.render(eth_NER, style='ent')
# List all Entities
# all geopolitical entities like in the example. Not sure if that's the question
print([ent.text for ent in eth_NER.ents if ent.label_ == 'GPE'])<jupyter_output>["'york'", 'china', 'london', 'britain', 'london', 'london', 'staffmumbai', "'york'", 'london', 'london', 'shanghai', 'china', 'beijing', 'india']
{'fraction_non_alphanumeric': 0.11251784864350309, 'fraction_numerical': 0.009424083769633508, 'mean_word_length': 5.242424242424242, 'pattern_counts': {'":': 10, '<': 27, '<?xml version=': 0, '>': 27, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '476339', 'license': 'ADSL', 'n_tokens_mistral': 3550, 'n_tokens_neox': 3379, 'n_words': 1027, 'path': 'Starter_Code/crypto_sentiment.ipynb', 'repo_name': 'jcm-co/homework-CryptoTales', 'size': '10505'}