<jupyter_start><jupyter_text>Gap Framework - Data Preparation for Sentiment Analysis PDSG: Boardgame rating and commentThe Boardgame rating/comment dataset is in a CSV format. To start you will need to read in the raw data from the CSV file using a csv reader. Fortunately, Python has a CSV reader. Let's start by importing the reader.<jupyter_code># Import Python's CSV parser module
import csv<jupyter_output><empty_output><jupyter_text>The Python csv reader creates a generator. For convenience, since you may want to re-read the file more than once as you practice preparing the data, let's make a function to create the generator.Note: The boardgame rating/comment dataset has unicode characters in it. So we need to open the file with the encoding 'utf-8'. If we don't the reader will throw an exception when it encounters a unicode character sequence.<jupyter_code># Create a generator for reading each row as a list in the CSV file.
def load():
    # make sure to open with Unicode encoding (there are unicode chars in the dataset)
    f = open('boardgames.csv', encoding='utf-8')
    r = csv.reader(f, delimiter=',')
    return r<jupyter_output><empty_output><jupyter_text>Keep only the data I am interested in.For my purposes, I am only interested in the rating and the comment. The entry and game ID will not contribute to my model, so I want to toss them out. I will simply create a new dataset where I will copy only the rating and comment over.<jupyter_code># Make a new dataset of just the rating and comment
dataset = []

# Let's read each row to assemble a new dataset
header = True
for row in load():
    # First row is a header, so let's skip it.
    if header:
        header = False
        continue
        
    # some of the ratings are floating point values, like 8.5 or 6.2. We will make them all ints for convinence. 
    rating = int(float(row[2]))

    dataset.append( { 'rating': rating, 'comment': row[3] } )<jupyter_output><empty_output><jupyter_text>Let's check that we really create the new dataset as we expect.<jupyter_code>print("Number of rows", len(dataset))
print("Row 0", dataset[0])
print("Row 2", dataset[2])<jupyter_output><empty_output><jupyter_text>This first row shows some of the issues you will need to consider:    - Punctuation sequences like +++ and *** may indicate the user is emphasizing the word.    - All caps (e.g., BAD) may indicate the user is emphasizing the word.    - ! may mean the user is elevating their statement.    So why is this an issue? The general rule of thumb in NLP preprocessing is to remove punctuation and lowercase words. Hum, that might cause us to lose valuable information. We will explore this, along with other things.  Preprocessing with GapOkay, there are a zillion ways one could code preprocessing text for sentiment analysis. There are no shortage of blogs.I will show you how to use **Gap** as an alternative to writing each line of preprocessing code by hand, and do it in a few simple steps. We start by first importing the SYNTAX module from **Gap**. This module does syntactically analysis for Text.<jupyter_code># Import Words from Gap's Syntax module
from gapnlp.syntax import Words<jupyter_output><empty_output><jupyter_text>I will use the Words class to demonstrate methods to preprocess (i.e., prepare) the text for machine learning. Like many applications of NLP, sentiment analysis has its own unique challenges:    - Reviews contain slang    - Repetitive punctuation and all CAPS are used as emphasis.    - Short-hand, incomplete sentences, higher frequency of spelling errors (not a concern to someone entering a review).    I will start with a simple example of a single 'positive' phrase; but in the phrase, the reviewer has used punctuation and all caps to emphasis words.First, we preprocess the traditional way: remove punctuation, lowercase, stem and stopword removal.<jupyter_code># This is our helper function to make it easier to display preprocessed text w/o tagging
def towords(words):
    for word in words:
        print(word['word'], ' ')<jupyter_output><empty_output><jupyter_text>Standard MethodLet's process and print the NLP tokenized string according to the 'standard rule of thumb'<jupyter_code>w = Words('+++ My favourite game of all TIME!!')
towords(w.words)<jupyter_output><empty_output><jupyter_text>OMG. What's wrong with this. Well, we lost stuff that might be important, like:    - TIME was in all CAPS    - +++ punctuation was used as emphasis    - favorite was spelled according to UK spelling.  Bare ModeLet's do the opposite and just keep everything. In **Gap**, that's the parameter *bare=True*.<jupyter_code>w = Words('+++ My favourite game of all TIME!!', bare=True)
towords(w.words)

# Note that tag 14 is an acronymn<jupyter_output><empty_output><jupyter_text>Okay, that might be better. But favourite won't match other occurrences of the American version favorite, and TIME won't match time and Time, etc. Perhaps your neural network will learn the relationship. But why burden the neural network? It has plenty of other important things to learn, like predicting the rating!  Keeping PunctuationOkay, let's go back to step 0. We start by doing the standard rule of thumb, but this time we keep punctuation by setting the keyword parameter *punct* to True, since it might indicate an emphasis.<jupyter_code>w = Words('+++ My favourite game of all TIME!!', punct=True)
towords(w.words)<jupyter_output><empty_output><jupyter_text>Okay, that's better. But we are still missing important things. Note how time is lowercased. But if we look at it's tag, it will tell us that it was uppercased (value 14). Also the punctuation all have tags as punctuation (value 23) or symbol (value 24). Interesting, we don't need to write code to tell if something was uppercased or if we have sequence of punctuation. We can just look at the tags.<jupyter_code>print(w.words)<jupyter_output><empty_output><jupyter_text>Sentiment WordsWe are still missing something. It's the word 'favourite' that indicates the sentiment! And it's in UK spelling. **Gap** has a word dictionary of US and UK spellings, slang and misspellings of words that (or may) indicate a sentiment. Let's tell **Gap** to keep these words by setting the keyword parameter *sentiment* to True.<jupyter_code>w = Words('+++ My favourite game of all TIME!!', punct=True, sentiment=True)
print(w.words)<jupyter_output><empty_output><jupyter_text>Let's take a close look at the NLP sequence. Aah, eventhough time is lowercase, the tag (14) indicates it was all CAPS. We kept the punctuation. And look, we retain the word favorite, and it's in US spelling.  NegationOkay, sometimes people use a negation. That is, they use a positive word like 'great' or 'good', but proceed it was a negation, like 'not' or 'never'. **Gap** recognizes this as well. Let's try one.<jupyter_code>w = Words("Did not like the game.", sentiment=True, punct=True)
print(w.words)

# Note the positive word Like is negated by not, so it was removed.<jupyter_output><empty_output><jupyter_text>Ahh, **Gap** dropped the otherwise positive word 'like' and simply kept the negative word 'not'.  Contractions**Gap** handles contractions as well. For example, the not might be part of a contraction like: don't, can't, won't, isn't, etc.<jupyter_code>w = Words("Didn't like the game.", sentiment=True, punct=True)
print(w.words)

# Handles contractions too.<jupyter_output><empty_output><jupyter_text>So many ways to write a word**Gap** has builtin and 3rd party stemmers and lemmatizers to match the same word with different word endings. For example, the words 'best' and 'better' are recognized as the same.<jupyter_code>w = Words('it is a great game', sentiment=True, punct=True)
print(w.words)
w = Words('it is the greatest game', sentiment=True, punct=True)
print(w.words)<jupyter_output><empty_output><jupyter_text>Spelling ErrorsReviews are notorious for spelling errors! **Gap** an implementation of the (Peter) Norvig speller builtin for English (and French, Spanish, German and Italian) language spell checker. Using the spell parameter, **Gap** will lookup each word in the pyaspeller dictionary. If the word is not found and the *Norvig* speller finds a replacement, the misspelled word is replaced.Note in the example, the misspelled word 'grat' is replaced with 'great'.<jupyter_code>w = Words('The game was graat!', sentiment=True, punct=True)
print(w.words)
w = Words('The game was graat!', sentiment=True, punct=True, spell='en')
print(w.words)<jupyter_output><empty_output><jupyter_text>Prepare for Neural NetworkFirst, **Gap** wasn't focused for sentiment analysis, but for extracting and processing text from PDF documents. So we need to do some last steps by hand that in the future will be incorporated into **Gap**, these include:1. Add special markers.2. Reduce token sequence to fixed length.3. Convert words to integers.  Special MarkersLet's start defining special markers, some of which I will explain their purpose further into the code-along.<jupyter_code>pad      = '<PAD>'
emphasis = '<EMP>'
positive = '<POS>'
negative = '<NEG>'<jupyter_output><empty_output><jupyter_text>Let's know create a function that will insert some of our special markers, as follows:    1. If a word has a negative sentiment tag, insert     2. If a word has a positive sentiment tag, insert     3. If a word is all caps, insert a     4. If + or * occurs two or more times in sequence, replace with a     5. If a ! mark appears, replace with a     6. Otherwise, remove all remaining punctuation and symbols.<jupyter_code>from syntax import Vocabulary

def prepare(comment):
    """ Convert text to NLP sequence """
    ret = []
    
    # Create words object from text: keep punctuation and sentiment words
    try:
        words = Words(comment, punct=True, sentiment=True)
    except:
        return
    
    # Reconstruct word list, adding special markers
    last = None
    for word in words.words:
        # Add special <EMP> marker for all caps words
        if word['tag'] == Vocabulary.ACRONYM:
            ret.append(emphasis)
        # Add special <POS> marker for positive words
        if word['tag'] == Vocabulary.POSITIVE:
            ret.append(positive)
        # Add special <NEG> marker for negative words
        elif word['tag'] == Vocabulary.NEGATIVE:
            ret.append(negative)
        # Drop punctuation, unless its an exclamation mark
        elif word['tag'] == Vocabulary.PUNCT:
            # Add <EMP> if exclamation mark
            if word['word'] == '!':
                ret.append(emphasis)
            continue
        elif word['tag'] == Vocabulary.SYMBOL:
            # Drop symbols, unless + or * as multiple sequence
            if word['word'] in ['+', '*']:
                # Add <EMP> if multiple sequence
                if last == word['word']:
                    ret.append(emphasis)
            continue
        ret.append(word['word'])
        # remember the last word
        last = word['word']
        
    return ret<jupyter_output><empty_output><jupyter_text>Let's now update each comment in the dataset with our special markers using the prepare function.Note, since the contents of lists are mutable in Python, we do not need to create a new list. We can replace the previous element in the list with a new element in place (no copy).<jupyter_code># Annotate each row with special markers
for row in dataset:
    comment = prepare(row['comment'])
    row['comment'] = comment
dataset[1]<jupyter_output><empty_output><jupyter_text>Truncating SizeSo we plan to send our sequences into a Recurrent Neural Network (RNN), like a LSTM. Best practice is that each sequence of tokens we input is the same length. So we need to pick a length, and anything above that length we truncate and below that length we add the special marker  - alas, the purpose of the  special marker.    Let's set the sequence length to 15.<jupyter_code>seq_len = 15

# Resize each comment to be exactly 15 tokens
for row in dataset:
    try:
        rlen = len(row['comment'])
    except:
        continue
    # If comment is > 15 tokens, then  truncate
    if rlen > 15:
        row['comment'] = row['comment'][0:15]
    # If less than 15 tokens, then add paddimg
    elif rlen < 15:
        for _ in range(rlen, 15+1):
            row['comment'].append('<PAD>')<jupyter_output><empty_output><jupyter_text>Let's look at a couple of rows and see what changed.    - Row 1: Was 16 elements, we dropped the last element .    - Row 2: Was 13 elements, we added two  elements.<jupyter_code>dataset[1]
dataset[2]<jupyter_output><empty_output><jupyter_text>Convert Words to IntegersWell, inputs to neural networks are numbers not words! Yeaks, what next.Well, we need to make a dictionary that maps each unique word to a unique integer and the replace the words in our inputs with the corresponding integer value.In Python, we do that with a dictionary object. Let's start by initializing our dictionary with the special markers.<jupyter_code>word2int = {}
word2int['<PAD>'] = 0
word2int['<EMP>'] = 1
word2int['<POS>'] = 2
word2int['<NEG>'] = 3<jupyter_output><empty_output><jupyter_text>Let's now build a dictionary of all the words in our dataset, mapping each word to a unique integer<jupyter_code>index = 4 # next integer value of word to add to dictionary

# Walk thru each row in the dataset
for row in dataset:
    # Get the list of preprocessed words for this row's comment
    words = row['comment']
    # For each word, we will see if we need to add it to the dictionary
    for word in words:
        # looks like this word is not in the dictionary!
        if word not in word2int:
            # Add the word and set its value to the next index sequence
            word2int[word] = index
            index += 1<jupyter_output><empty_output><jupyter_text>Let's do some verification that this step work.<jupyter_code>print("Number of Unique Words:", len(word2int))
print("Mapping of the word fun:", word2int['fun'])
# Update each row in the dataset, replacing words with their unique integer value
for row in dataset:
    comment = []
    # for each word in the comment, lookup its integer value
    for word in row['comment']:
        # replace the word with its unique integer value
        comment.append(word2int[word])
    # replace the comment with the list of integer values
    row['comment'] = comment<jupyter_output><empty_output><jupyter_text>Let's look at a couple of rows and see what changed.<jupyter_code>dataset[1]
dataset[2]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0802608994164092, 'fraction_numerical': 0.0054926192928252664, 'mean_word_length': 4.078800557880056, 'pattern_counts': {'":': 0, '<': 112, '<?xml version=': 0, '>': 113, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '24', 'dir': 'jupyter-structured-clean-dedup', 'id': '327244', 'license': 'Apache-2.0', 'n_tokens_mistral': 4165, 'n_tokens_neox': 3972, 'n_words': 1904, 'path': 'train/session 4.ipynb', 'repo_name': 'gapml/NLP', 'size': '14565'}