<jupyter_start><jupyter_text>News Headlines SentimentUse the news api to pull the latest news articles for bitcoin and ethereum and create a DataFrame of sentiment scores for each coin. Use descriptive statistics to answer the following questions:1. Which coin had the highest mean positive score?2. Which coin had the highest negative score?3. Which coin had the highest positive score?<jupyter_code># Read your api key environment variable
api_key = os.getenv("news_api")
# Create a newsapi client
newsapi = NewsApiClient(api_key=api_key)
# Fetch the Bitcoin news articles
bitcoin_headlines = newsapi.get_everything(
    q="bitcoin",
    language="en",
    page_size=100,
    sort_by="relevancy"
)

# Print total articles
print(f"Total articles about Bitcoin: {bitcoin_headlines['totalResults']}")

# Show sample article
bitcoin_headlines["articles"][20]
# Fetch the Ethereum news articles
ethereum_headlines = newsapi.get_everything(
    q="ethereum",
    language="en",
    page_size=100,
    sort_by="relevancy"
)

# Print total articles
print(f"Total articles about Ethereum: {ethereum_headlines['totalResults']}")

# Show sample article
ethereum_headlines["articles"][20]
# Create the Bitcoin sentiment scores DataFrame
bitcoin_sentiments = []

for article in bitcoin_headlines["articles"]:
    try:
        text = article["content"]
        sentiment = analyzer.polarity_scores(text)
        compound = sentiment["compound"]
        pos = sentiment["pos"]
        neu = sentiment["neu"]
        neg = sentiment["neg"]
        
        bitcoin_sentiments.append({
            "text": text,
            "compound": compound,
            "positive": pos,
            "negative": neg,
            "neutral": neu
            
        })
        
    except AttributeError:
        pass
    
# Create DataFrame
bitcoin_df = pd.DataFrame(bitcoin_sentiments)

# Reorder DataFrame columns
cols = ["compound", "positive", "negative", "neutral", "text"]
bitcoin_df = bitcoin_df[cols]

bitcoin_df.head()
# Create the ethereum sentiment scores DataFrame
ethereum_sentiments = []

for article in ethereum_headlines["articles"]:
    try:
        text = article["content"]
        sentiment = analyzer.polarity_scores(text)
        compound = sentiment["compound"]
        pos = sentiment["pos"]
        neu = sentiment["neu"]
        neg = sentiment["neg"]
        
        ethereum_sentiments.append({
            "text": text,
            "compound": compound,
            "positive": pos,
            "negative": neg,
            "neutral": neu
            
        })
        
    except AttributeError:
        pass
    
# Create DataFrame
ethereum_df = pd.DataFrame(ethereum_sentiments)

# Reorder DataFrame columns
cols = ["compound", "positive", "negative", "neutral", "text"]
ethereum_df = ethereum_df[cols]

ethereum_df.head()
# Describe the Bitcoin Sentiment
bitcoin_df.describe()
# Describe the Ethereum Sentiment
ethereum_df.describe()<jupyter_output><empty_output><jupyter_text>Questions:Q: Which coin had the highest mean positive score?A: Bitcoin but by a mere .00003 so it is not significant and both are by all standards the same. Q: Which coin had the highest compound score?A: Ethereum has a higher mean compound score. Q. Which coin had the highest positive score?A: Bitcoin has the highest postive score. Q: Which coin had the highest negative score?A: Bitcoin has the highest negative score.  ---  TokenizerIn this section, you will use NLTK and Python to tokenize the text for each coin. Be sure to:1. Lowercase each word2. Remove Punctuation3. Remove Stopwords<jupyter_code>from nltk.tokenize import word_tokenize, sent_tokenize
from nltk.corpus import stopwords
from nltk.stem import WordNetLemmatizer, PorterStemmer
from string import punctuation
import re
# Expand the default stopwords list if necessary
import nltk
nltk.download('wordnet')
  
# Complete the tokenizer function
lemmatizer = WordNetLemmatizer()
def tokenizer(text):
    """Tokenizes text."""
    sw = set(stopwords.words('english'))
    regex = re.compile("[^a-zA-Z ]")
    re_clean = regex.sub('', str(text))
    words = word_tokenize(re_clean)
    lem = [lemmatizer.lemmatize(word) for word in words]
    tokens = [word.lower() for word in lem if word.lower() not in sw]
    return tokens
# Create a new tokens column for bitcoin
bitcoin_text = bitcoin_df['text'].tolist()
sentence_tokenized = [sent_tokenize(i) for i in bitcoin_text]
word_tokenized = []
for story in sentence_tokenized:
    words = []
    for sent in story:
        words = tokenizer(story)
    # append all words for each article to the word_tokenized list
    word_tokenized.append(words)
bitcoin_df['bitcoin_tokens'] = word_tokenized    
bitcoin_df
# Create a new tokens column for ethereum
ethereum_text = ethereum_df['text'].tolist()
sentence_tokenized = [sent_tokenize(i) for i in ethereum_text]
word_tokenized = []
for story in sentence_tokenized:
    words = []
    for sent in story:
        words = tokenizer(story)
    # append all words for each article to the word_tokenized list
    word_tokenized.append(words)
ethereum_df['ethereum_tokens'] = word_tokenized  
ethereum_df<jupyter_output><empty_output><jupyter_text>---  NGrams and Frequency AnalysisIn this section you will look at the ngrams and word frequency for each coin. 1. Use NLTK to produce the n-grams for N = 2. 2. List the top 10 words for each coin.<jupyter_code>from collections import Counter
from nltk import ngrams
# Generate the Bitcoin N-grams where N=2
def bigram_counter(corpus): 
    # Combine all articles in corpus into one large string
    big_string = ' '.join(corpus)
    processed = tokenizer(big_string)
    bigrams = ngrams(processed, n=2)
    count = dict(Counter(bigrams))
    return pd.DataFrame(list(count.items()), columns=['bigram', 'count']).sort_values(by=['count'], ascending=False)
bitcoin_bigrams = bigram_counter(bitcoin_df['text'])
bitcoin_bigrams
# Generate the Ethereum N-grams where N=2
ethereum_bigrams = bigram_counter(ethereum_df['text'])
ethereum_bigrams
# Use the token_count function to generate the top 10 words from each coin
def token_count(tokens, N=10):
    """Returns the top N tokens from the frequency count"""
    big_string = ' '.join(tokens)
    tokens = tokenizer(big_string)
    return Counter(tokens).most_common(N)
# Get the top 10 words for Bitcoin
bitcoin_10 = token_count(bitcoin_df['text'])
bitcoin_10
# Get the top 10 words for Ethereum
ethereum_10 = token_count(ethereum_df['text'])
ethereum_10<jupyter_output><empty_output><jupyter_text>Word CloudsIn this section, you will generate word clouds for each coin to summarize the news for each coin<jupyter_code>from wordcloud import WordCloud
import matplotlib.pyplot as plt
plt.style.use('seaborn-whitegrid')
import matplotlib as mpl
mpl.rcParams['figure.figsize'] = [20.0, 10.0]
bitcoin_df['bitcoin_tokens']
# Generate the Bitcoin word cloud
bt_entities = bitcoin_df['bitcoin_tokens'].astype(str)
wc = WordCloud().generate(' '.join(bt_entities))
plt.imshow(wc)
# Generate the Ethereum word cloud
et_entities = ethereum_df['ethereum_tokens'].astype(str)
wc = WordCloud().generate(' '.join(et_entities))
plt.imshow(wc)<jupyter_output><empty_output><jupyter_text>Named Entity RecognitionIn this section, you will build a named entity recognition model for both coins and visualize the tags using SpaCy.<jupyter_code>import spacy
from spacy import displacy
# Optional - download a language model for SpaCy
# !python -m spacy download en_core_web_sm
# Load the spaCy model
nlp = spacy.load('en_core_web_sm')<jupyter_output><empty_output><jupyter_text>Bitcoin NER<jupyter_code># Concatenate all of the bitcoin text together
bitcoin_all_text = bitcoin_df['text'].str.cat(sep=', ')
bitcoin_all_text[:3000]
# Run the NER processor on all of the text
bt_doc = nlp(bitcoin_all_text)
# Add a title to the document
bt_doc.user_data["title"] = "Bitcoin NER"
# Render the visualization
displacy.render(bt_doc, style='ent')
# List all Entities
bt_entities = [(ent.text, ent.label_) for ent in bt_doc.ents]

for x in bt_entities:
    print(x[0], x[1])<jupyter_output>Less than two miles QUANTITY
Reykjavik airport FAC
tens of thousands CARDINAL
Bitcoin PERSON
HTC ORG
Bitcoin PERSON
Three hundred and thirty-seven CARDINAL
last year DATE
U.S. GPE
South Korean NORP
Welcome FAC
Video NORP
Bitcoin PERSON
France GPE
first ORDINAL
The Next Web WORK_OF_ART
French NORP
Le Ministère de lÉducation Nationale ORG
Complete Blockchain and Ethereum Programmer ORG
just $24 MONEY
Bitcoin PERSON
<NAME> PERSON
Bitcoin PERSON
Bitcoin PERSON
Today DATE
<NAME> PERSON
Bitcoin PERSON
the past 24 hours TIME
Baudriallard PERSON
<NAME> PERSON
Bitcoin BTC PERSON
Bitcoin PERSON
Today DATE
<NAME> PERSON
Bitcoin PERSON
the past 24 hours TIME
Nietzsche PERSON
Time ORG
<NAME> PERSON
Bitcoin BTC PERSON
Bitcoin PERSON
Today DATE
<NAME> PERSON
Bitcoin PERSON
the past 24 hours TIME
Machiavelli PERSON
<NAME> PERSON
Bitcoin BTC PERSON
Bitcoin PERSON
Today DATE
<NAME> PERSON
Bitcoin PERSON
the past 24 hours TIM[...]<jupyter_text>---  Ethereum NER<jupyter_code># Concatenate all of the bitcoin text together
ethereum_all_text =  ethereum_df['text'].str.cat(sep=', ')
ethereum_all_text[:3000]
# Run the NER processor on all of the text
et_doc = nlp(ethereum_all_text)
# Add a title to the document
et_doc.user_data["title"] = "Ethereum NER"
# Render the visualization
displacy.render(et_doc, style='ent')
# List all Entities
et_entities = [(ent.text, ent.label_) for ent in et_doc.ents]

for x in et_entities:
    print(x[0], x[1])<jupyter_output>Complete Blockchain and Ethereum Programmer ORG
just $24 MONEY
Bitcoin PERSON
Disrupt Berlin PERSON
December 11 and December 12 DATE
Bitcoin PERSON
HTC ORG
second ORDINAL
The Complete Blockchain and Ethereum Programming Bundle EVENT
30 MONEY
Everybodys ORG
Ethereum Foundation's ORG
Devcon 5 FAC
Osaka GPE
Japan GPE
OpenLibra ORG
Facebook ORG
OpenLibra ORG
Libra ORG
Blockchain PERSON
the year 2009 DATE
<NAME> PERSON
Nakamoto PERSON
Bitcoin PERSON
Blockchai GPE
Reuters ORG
Bitcoin PERSON
as much as 9% PERCENT
Wednesday DATE
June DATE
Facebook ORG
<NAME>'s PERSON
Libra GPE
Zuckerberg GPE
Swiss NORP
countrys NORP
Swiss NORP
Block ORG
Opera ORG
one CARDINAL
Opera Mobile ORG
XRP ORG
Bitcoin PERSON
XRP ORG
third ORDINAL
the day DATE
six CARDINAL
Bitcoin PERSON
the Complete Blockchain and Ethereum Programmer Bundle ORG
just $29 MONEY
first ORDINAL
last week DATE
Chinese NORP
2021 DATE
first ORDINAL
last year DATE
Chinese NORP
Tencent GPE
countrys NORP
British Columbia GPE
Ca[...]
{'fraction_non_alphanumeric': 0.08247129163898644, 'fraction_numerical': 0.008826041567808675, 'mean_word_length': 4.5668251452720545, 'pattern_counts': {'":': 10, '<': 37, '<?xml version=': 0, '>': 37, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '492056', 'license': 'ADSL', 'n_tokens_mistral': 3533, 'n_tokens_neox': 3318, 'n_words': 1119, 'path': 'code/crypto_sentiment.ipynb', 'repo_name': 'irayje/nlp_homework', 'size': '10637'}