<jupyter_start><jupyter_text>Sentiment Classification & How To "Frame Problems" for a Neural Networkby <NAME>modified by <NAME>- **Trask Twitter**: @iamtrask- **Trask Blog**: http://iamtrask.github.io- **Maynes Twitter**: @nathanmaynes  Concepts We Will Use- neural networks, forward and back-propagation- stochastic gradient descent- mean squared error- and train/test splits Where to Get Help if You Need it- Extra Reading Material - [Grokking Deep Learning](https://www.manning.com/books/grokking-deep-learning)- Shoot a tweet @iamtrask or @nathanmaynes  Curate a Dataset<jupyter_code>def pretty_print_review_and_label(i):
    print(labels[i] + "\t:\t" + reviews[i][:80] + "...")

g = open('1000_reviews.txt','r') # What we know!
reviews = list(map(lambda x:x[:-1],g.readlines()))
g.close()

g = open('1000_labels.txt','r') # What we WANT to know!
labels = list(map(lambda x:x[:-1].upper(),g.readlines()))
g.close()
len(reviews)
reviews[0]
labels[0]<jupyter_output><empty_output><jupyter_text>Develop a Predictive Theory<jupyter_code>print("labels.txt \t : \t reviews.txt\n")
pretty_print_review_and_label(2137)
pretty_print_review_and_label(12816)
pretty_print_review_and_label(6267)
pretty_print_review_and_label(21934)
pretty_print_review_and_label(5297)
pretty_print_review_and_label(4998)<jupyter_output>labels.txt 	 : 	 reviews.txt

NEGATIVE	:	this movie is terrible but it has some good effects .  ...
POSITIVE	:	adrian pasdar is excellent is this film . he makes a fascinating woman .  ...
NEGATIVE	:	comment this movie is impossible . is terrible  very improbable  bad interpretat...
POSITIVE	:	excellent episode movie ala pulp fiction .  days   suicides . it doesnt get more...
NEGATIVE	:	if you haven  t seen this  it  s terrible . it is pure trash . i saw this about ...
POSITIVE	:	this schiffer guy is a real genius  the movie is of excellent quality and both e...<jupyter_text>Quick Theory Validation<jupyter_code>from collections import Counter
import numpy as np
positive_counts = Counter()
negative_counts = Counter()
total_counts = Counter()
for i in range(len(reviews)):
    if(labels[i] == 'POSITIVE'):
        for word in reviews[i].split(" "):
            positive_counts[word] += 1
            total_counts[word] += 1
    else:
        for word in reviews[i].split(" "):
            negative_counts[word] += 1
            total_counts[word] += 1
positive_counts.most_common(10)
pos_neg_ratios = Counter()

for term,cnt in list(total_counts.most_common()):
    if(cnt > 100):
        pos_neg_ratio = positive_counts[term] / float(negative_counts[term]+1)
        pos_neg_ratios[term] = pos_neg_ratio

for word,ratio in pos_neg_ratios.most_common():
    if(ratio > 1):
        pos_neg_ratios[word] = np.log(ratio)
    else:
        pos_neg_ratios[word] = -np.log((1 / (ratio+0.01)))
# words most frequently seen in a review with a "POSITIVE" label
pos_neg_ratios.most_common(10)
# words most frequently seen in a review with a "NEGATIVE" label
list(reversed(pos_neg_ratios.most_common(10)))<jupyter_output><empty_output><jupyter_text>Transforming Text into Numbers<jupyter_code>from IPython.display import Image

review = "This was a horrible, terrible movie."

Image(filename='sentiment_network.png')
review = "The movie was excellent"

Image(filename='sentiment_network_pos.png')<jupyter_output><empty_output><jupyter_text>Creating the Input/Output Data<jupyter_code>vocab = set(total_counts.keys())
vocab_size = len(vocab)
print(vocab_size)
list(vocab)[0:10]
import numpy as np

layer_0 = np.zeros((1,vocab_size))
layer_0
from IPython.display import Image
Image(filename='sentiment_network.png')
word2index = {}

for i,word in enumerate(vocab):
    word2index[word] = i
word2index

# Dictionary slicing function to explore the word2index dictionary
def slice_dict(d):
    return {k:v for k,v in d.items() if (v < 11) } # return key, values for first ten entries
slice_dict(word2index)
def update_input_layer(review):
    
    global layer_0
    
    # clear out previous state, reset the layer to be all 0s
    layer_0 *= 0
    for word in review.split(" "):
        layer_0[0][word2index[word]] += 1

update_input_layer(reviews[0])
layer_0
def get_target_for_label(label):
    if(label == 'POSITIVE'):
        return 1
    else:
        return 0
labels[0]
get_target_for_label(labels[0])
labels[1]
get_target_for_label(labels[1])<jupyter_output><empty_output><jupyter_text>Project 3: Building a Neural Network<jupyter_code># Initialize the Neural Network for Sentiment Analysis
# Psuedo Code
# define our neural network class, its an object:
#     def the initialization function, parameters are self, reviews, labels, hidden_nodes, learning_rate:
#         set the random seed generator
#         preprocess the data, normalize it, vectorize it
#         initialize the network with the following parameters, size of our vocab list, number of hidden nodes, number of output nodes, learning rate
#     def our preprocess function, parameters are self, reviews, label
#         create a set of the words in reviews
#         create a set of the words in labels (In this problem there are only 2)
#         create a dictionary of the words in reviews
#         create a dictionary of the words in labels
#     def our initialize function with the parameters self, input_nodes, hidden_nodes, output_nodes, learning_rate
#         set the number of input nodes, hidden_nodes, and output nodes
#         set the learning_rate
#         set the workspace layer. best practice to allocate the memory and initialize the array with zeros
#     def our input update layer function with parameters self and a review - this function will be processing one review at a time
#         clear out previous workspace layer by setting values to zero
#         loop through words in each review, if found increment the count in workspace layer
#     def our label classification function, parameter is layer
#         if label is positive return 1 else 0
#     def sigmoid function
#     def sigmoid function derivative
#     def train function with parameters self, train_reviews, train_labels
#         for each review in the training set
#             set the input layer
#             set the hidden layer
#             set the output layer
#             implement a forward pass
#             implement a backwards pass
#             update the training weights
#     def test function with parameters self, test_reviews, test_labels
#         for each review in the testing set
#             call the run function and compare the predicted label with the actual label
#     def run function with parameters self, review
#         set the input layer
#         set the output layer
#         if the output layer is greater than .5 assign the review a positive label else a negative label
import time
import sys
import numpy as np

class SentimentNetwork:
    def __init__(self, reviews,labels,hidden_nodes = 10, learning_rate = 0.1):
       
        # set our random number generator 
        np.random.seed(1)
    
        self.pre_process_data(reviews, labels)
        
        self.init_network(len(self.review_vocab),hidden_nodes, 1, learning_rate)
        
        
    def pre_process_data(self, reviews, labels):
        
        review_vocab = set()
        for review in reviews:
            for word in review.split(" "):
                review_vocab.add(word)
        self.review_vocab = list(review_vocab)
        
        label_vocab = set()
        for label in labels:
            label_vocab.add(label)
        
        self.label_vocab = list(label_vocab)
        
        self.review_vocab_size = len(self.review_vocab)
        self.label_vocab_size = len(self.label_vocab)
                
        self.word2index = {}
        for i, word in enumerate(self.review_vocab):
            self.word2index[word] = i
        
        self.label2index = {}
        for i, label in enumerate(self.label_vocab):
            self.label2index[label] = i
         
        
    def init_network(self, input_nodes, hidden_nodes, output_nodes, learning_rate):
        # Set number of nodes in input, hidden and output layers.
        self.input_nodes = input_nodes
        self.hidden_nodes = hidden_nodes
        self.output_nodes = output_nodes

        # Initialize weights
        self.weights_0_1 = np.zeros((self.input_nodes,self.hidden_nodes))
    
        self.weights_1_2 = np.random.normal(0.0, self.output_nodes**-0.5, 
                                                (self.hidden_nodes, self.output_nodes))
        
        self.learning_rate = learning_rate
        
        self.layer_0 = np.zeros((1,input_nodes))
    
        
    def update_input_layer(self,review):

        # clear out previous state, reset the layer to be all 0s
        self.layer_0 *= 0
        for word in review.split(" "):
            if(word in self.word2index.keys()):
                self.layer_0[0][self.word2index[word]] += 1   
            
    def get_target_for_label(self,label):
        if(label == 'POSITIVE'):
            return 1
        else:
            return 0
        
    def sigmoid(self,x):
        return 1 / (1 + np.exp(-x))
    
    
    def sigmoid_output_2_derivative(self,output):
        return output * (1 - output)
    
    def train(self, training_reviews, training_labels):
        # Test to make sure the reviews and labels are the same size
        assert(len(training_reviews) == len(training_labels))
        
        # Counter value to check how many preditions we get right before and after training
        correct_so_far = 0
        
        # Timer function 
        start = time.time()
        
        for i in range(len(training_reviews)):
            
            review = training_reviews[i]
            label = training_labels[i]
            
            #### Implement the forward pass here ####
            ### Forward pass ###

            # Input Layer
            self.update_input_layer(review)

            # Hidden layer
            layer_1 = self.layer_0.dot(self.weights_0_1)

            # Output layer
            layer_2 = self.sigmoid(layer_1.dot(self.weights_1_2))

            #### Implement the backward pass here ####
            ### Backward pass ###

            # Output error
            layer_2_error = layer_2 - self.get_target_for_label(label) # Output layer error is the difference between desired target and actual output.
            layer_2_delta = layer_2_error * self.sigmoid_output_2_derivative(layer_2)

            # Backpropagated error
            layer_1_error = layer_2_delta.dot(self.weights_1_2.T) # errors propagated to the hidden layer
            layer_1_delta = layer_1_error # hidden layer gradients - no nonlinearity so it's the same as the error

            # Update the weights
            self.weights_1_2 -= layer_1.T.dot(layer_2_delta) * self.learning_rate # update hidden-to-output weights with gradient descent step
            self.weights_0_1 -= self.layer_0.T.dot(layer_1_delta) * self.learning_rate # update input-to-hidden weights with gradient descent step

            if(np.abs(layer_2_error) < 0.5):
                correct_so_far += 1
            
            # Timing statistics
            reviews_per_second = i / float(time.time() - start)
            
            sys.stdout.write("\rProgress:" + str(100 * i/float(len(training_reviews)))[:4] + "% Speed(reviews/sec):" + str(reviews_per_second)[0:5] + " #Correct:" + str(correct_so_far) + " #Trained:" + str(i+1) + " Training Accuracy:" + str(correct_so_far * 100 / float(i+1))[:4] + "%")
            if(i % 2500 == 0):
                print("")
    
    def test(self, testing_reviews, testing_labels):
        
        correct = 0
        
        start = time.time()
        
        for i in range(len(testing_reviews)):
            pred = self.run(testing_reviews[i])
            if(pred == testing_labels[i]):
                correct += 1
            
            reviews_per_second = i / float(time.time() - start)
            
            sys.stdout.write("\rProgress:" + str(100 * i/float(len(testing_reviews)))[:4] \
                             + "% Speed(reviews/sec):" + str(reviews_per_second)[0:5] \
                            + "% #Correct:" + str(correct) + " #Tested:" + str(i+1) + " Testing Accuracy:" + str(correct * 100 / float(i+1))[:4] + "%")
    
    def run(self, review):
        
        # Input Layer
        self.update_input_layer(review.lower())

        # Hidden layer
        layer_1 = self.layer_0.dot(self.weights_0_1)

        # Output layer
        layer_2 = self.sigmoid(layer_1.dot(self.weights_1_2))
        
        if(layer_2[0] > 0.5):
            return "POSITIVE"
        else:
            return "NEGATIVE"<jupyter_output><empty_output><jupyter_text>Call Our Sentiment Network<jupyter_code>mlp = SentimentNetwork(reviews[:-333],labels[:-333], learning_rate=0.1)
# evaluate our model before training (just to show how horrible it is)
mlp.test(reviews[-333:],labels[-333:])
# train the network
mlp.train(reviews[:-333],labels[:-333])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09431366554570468, 'fraction_numerical': 0.017043717517578723, 'mean_word_length': 2.709951800396938, 'pattern_counts': {'":': 0, '<': 32, '<?xml version=': 0, '>': 33, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '557361', 'license': 'MIT', 'n_tokens_mistral': 4071, 'n_tokens_neox': 3899, 'n_words': 1111, 'path': 'Sentiment_Classification_W_Movie_Reviews.ipynb', 'repo_name': 'nmaynes/sentiment-network', 'size': '13097'}