<jupyter_start><jupyter_text>Twitter Sentiment  In the first [notebook](01_tweets_mining.ipynb), we have downloaded tweets for different cryptocurrencies. We will next perform [sentiment analysis](https://en.wikipedia.org/wiki/Sentiment_analysis:~:text=Sentiment%20analysis%20(also%20known%20as,affective%20states%20and%20subjective%20information.) on the tweets so as to understand the mood of the public towards each cryptocurrency, and eventually see if the sentiments could be used to predict the returns of the cryptocurrency.As a quickstart to [Natural Language Programming](https://en.wikipedia.org/wiki/Natural-language_programming), we will be using [TextBlob](https://textblob.readthedocs.io/en/dev/quickstart.html) to extract the polarity of the tweets. We will revisit polarity again later in the notebook.  Do note that we are unable to share the downloaded tweets due to Twitter's regulation on its data. Hence this notebook will only serve as a reference on how we can obtain the sentiments.<jupyter_code>import glob
import os
import re

import nltk
import pandas as pd

nltk.download("stopwords")
nltk.download("wordnet")
from nltk.corpus import stopwords
from nltk.stem import SnowballStemmer, WordNetLemmatizer
from nltk.tokenize import RegexpTokenizer
from textblob import TextBlob<jupyter_output><empty_output><jupyter_text>As the mined tweets are split into files by the currency, the below list is used to iterate through the currency files for processing.<jupyter_code>currency = [
    "BTC",
    "ETH",
    "USDT",
    "XRP",
    "BCH",
    "ADA",
    "BSV",
    "LTC",
    "LINK",
    "BNB",
    "EOS",
    "TRON",
]
def process_ccy(curr):
    pd_df = pd.concat(
        [pd.read_csv(f) for f in glob.glob(f"clean_data/{curr}*.csv", recursive=True)],
        ignore_index=True,
    )

    print(f"Number of records loaded for {curr}", len(pd_df))
    pd_df[["coin_symbol", "tweet_id", "created_at", "date", "hour"]].to_csv(
        f"clean_data/ccy_tweets/{curr}_tweets.csv", index=False
    )

    return pd_df<jupyter_output><empty_output><jupyter_text>We combine all the downloaded tweets into one dataframe.<jupyter_code>combined_df = pd.concat(
    [process_ccy(c) for c in currency],
    ignore_index=True,
)<jupyter_output><empty_output><jupyter_text>Since we repeated the data mining multiple times, we expect duplication of tweets. We will keep the latest mined as the number of followers and retweets can change.<jupyter_code>combined_df.sort_values(by=["mined_at"], inplace=True, ignore_index=True)
combined_df.drop_duplicates(
    subset=["tweet_id"], inplace=True, keep="last", ignore_index=True
)
combined_df.head(5)<jupyter_output><empty_output><jupyter_text>As `tweet_id` is somewhat similar to timestamp, we sort the tweets by the id to get it in chronological order (though it doesn't really matter much here).<jupyter_code>combined_df.sort_values(by=["tweet_id"], inplace=True)
len(combined_df)<jupyter_output><empty_output><jupyter_text>PreprocessingWe used the minimal text processing in this notebook to turn the text into a canonical, machine-readable format.  The text normalization process will include:- converting all letters into lower case- Lemmatization and stemming the words - remove stop words<jupyter_code>stop_words = stopwords.words("english")
stemmer = SnowballStemmer("english")
lemmatizer = WordNetLemmatizer()

tokenizer = RegexpTokenizer(r"\w+")
def preprocess(text):
    words = [lemmatizer.lemmatize(w) for w in text if w not in stop_words]
    stem_text = " ".join([stemmer.stem(i) for i in words])
    return stem_text
combined_df["processed text"] = combined_df["text"].apply(
    lambda x: preprocess(tokenizer.tokenize(x.lower()))
)<jupyter_output><empty_output><jupyter_text>Getting sentimentWe create a TextBlob and obtain the polarity from its [sentiment](https://textblob.readthedocs.io/en/dev/api_reference.htmltextblob.blob.TextBlob.sentiment) property.<jupyter_code>def getSentiment(tweet):
    analysis = TextBlob(tweet)
    return analysis.sentiment.polarity  # work out sentiment
combined_df["polarity"] = combined_df["processed text"].apply(lambda x: getSentiment(x))
combined_df.head(5)<jupyter_output><empty_output><jupyter_text>The polarity is a value within -1.0 and 1.0 where 0 shows a neutral post, and likely to be a bot posting.  Negative polarity indicates a negative posting and likewise, postive polarity indicates a positive mood.  We create a `sentiment` column to store these classifications.<jupyter_code>combined_df["sentiment"] = combined_df["polarity"].apply(
    lambda s: "Postive" if s > 0 else ("Neutral" if s == 0 else "Negative")
)
combined_df.to_csv("full_tweets.csv", index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10021186440677966, 'fraction_numerical': 0.0057203389830508475, 'mean_word_length': 5.7250712250712255, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 34, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '198617', 'license': 'Apache-2.0', 'n_tokens_mistral': 1551, 'n_tokens_neox': 1465, 'n_words': 440, 'path': 'notebooks/twitter/02_sentiment.ipynb', 'repo_name': 'tiphanieqch/notebooks', 'size': '4720'}