<jupyter_start><jupyter_text>Sentiment Analysis IntroductionWhen it comes to text data, there are a few popular techniques that enable us to extract useful information from data. Analyzing text and generating results is it different from dealing with numerical data. One such basic technique is sentiment analysis. A few key points to remember with sentiment analysis are:1. **TextBlob Module:** Linguistic researchers have labeled the sentiment of words based on their domain expertise. Sentiment of words can vary based on where it is in a sentence. The TextBlob module allows us to take advantage of these labels.2. **Sentiment Labels:** Each word in a corpus is labeled in terms of polarity and subjectivity (there are more labels as well, but we're going to ignore them for now). A corpus' sentiment is the average of these. - **Polarity:** How positive or negative a word is. -1 is very negative. +1 is very positive. -  **Subjectivity:** How subjective, or opinionated a word is. 0 is fact. +1 is very much an opinion.Let's take a look at the sentiment of the various transcripts, both overall and throughout the comedy routine. Sentiment of Routine<jupyter_code># reading corpus
import pandas as pd
data = pd.read_pickle('pickle files/corpus.pkl')
data<jupyter_output><empty_output><jupyter_text>**NOTE:** We are using corpus for sentiment analysis and not document term matrix because the later works as a bag of words i.e. text words are in unordered fashion in that. Where as a corpus proves useful as it preserves the word order.<jupyter_code># creating lambda function to find polarity and subjectivity
from textblob import TextBlob

pol = lambda x: TextBlob(x).sentiment.polarity
sub = lambda x: TextBlob(x).sentiment.subjectivity

data['polarity'] = data['transcript'].apply(pol)
data['subjectivity'] = data['transcript'].apply(sub)
data
# plotting the results
import matplotlib.pyplot as plt

plt.rcParams['figure.figsize'] = [10, 8]

for index, comedian in enumerate(data.index):
    x = data.polarity.loc[comedian]
    y = data.subjectivity.loc[comedian]
    plt.scatter(x, y, color = 'red')
    plt.text(x+.001, y+ .001, data['full_name'][index], fontsize = 10)
    plt.xlim(-.01, .12)
    
plt.title('Sentiment Analysis', fontsize = 20)
plt.xlabel('<--------Negative-----------------------------Positive-------->', fontsize = 15)
plt.ylabel('<-------Fact---------------------Opinionated--------->', fontsize = 15)

plt.show()<jupyter_output><empty_output><jupyter_text>Sentiment of Routine Over TimeInstead of sentiment analysis for complete routine, let us check for the sentiments of different comedians over time during the routine.<jupyter_code># necessary imports
import numpy as np
import math

# splitting text into n equal parts for analysis
def split_text(text, n = 10):
    # calculate the length pf whole text, size of each splitted text and
    # starting point of each split
    length = len(text)
    size = math.floor(length/n)
    start = np.arange(0, length, size)
    
    # pulling out equally sized pieces of text and putting into list
    split_list = []
    for split in range(n):
        split_list.append(text[start[split]:start[split] + size])
    return split_list
# creating list to store the splitted text
list_of_splits = []
for text in data['transcript']:
    split = split_text(text)
    list_of_splits.append(split)
    
list_of_splits[0]
# calculating the polarity for each chunk of text
pol_transcript = []
for split in list_of_splits:
    pol_piece = []
    for p in split:
        pol_piece.append(TextBlob(p).sentiment.polarity)
    pol_transcript.append(pol_piece)
    
pol_transcript
# plotting the graph of polarity for one comedian
plt.plot(pol_transcript[0])
plt.title(data['full_name'][0])
plt.show()
# plotting the curve for all comedians
plt.rcParams['figure.figsize'] = [16, 12]

for index, comedian in enumerate(data.index):    
    plt.subplot(3, 4, index+1)
    plt.plot(pol_transcript[index])
    plt.plot(np.arange(0,10), np.zeros(10))
    plt.title(data['full_name'][index])
    plt.ylim(ymin=-.2, ymax=.3)
    
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11129970902036858, 'fraction_numerical': 0.01139670223084384, 'mean_word_length': 4.080049261083744, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '277204', 'license': 'MIT', 'n_tokens_mistral': 1256, 'n_tokens_neox': 1181, 'n_words': 462, 'path': 'Natural Language Processing/3-Sentiment-Analysis.ipynb', 'repo_name': 'riddhi-jain/Way-to-Deep-Learning', 'size': '4124'}