<jupyter_start><jupyter_text>In this chapter you will:* Clean and prepare text data* Build feature vectors from text documents* Train a machine learning model to classify positive and negative movie reviews* Work with large text datasets using out-of-core learning<jupyter_code>## Will be working with movie reviews from IMDB database
## Dataset is 50,000 reviews labeled as positive or negative
## Positive was rated with more than 6 stars on IMDb

## Read movie reviews into a Dataframe- may take 10 minutes

import pyprind
import pandas as pd
import os
pbar = pyprind.ProgBar(50000)
labels = {'pos':1, 'neg':0}
df = pd.DataFrame()
for s in ('test', 'train'):
    for l in ('pos', 'neg'):
        path = './aclImdb/%s/%s' % (s,l)
        for file in os.listdir(path):
            with open(os.path.join(path, file), 'r') as infile: 
                txt = infile.read()
                df = df.append([[txt, labels[l]]], ignore_index = True)
                pbar.update()
df.columns = ['review', 'sentiment']
import numpy as np 
np.random.seed(0)
df = df.reindex(np.random.permutation(df.index))
df.to_csv('./movie_data.csv', index = False)
df = pd.read_csv('./movie_data.csv')
df.head(3)<jupyter_output><empty_output><jupyter_text>Bag-of-words model1. Create a vocabulary of unique tokens i.e., words from the entire set of documents2. Construct a feature vector from each document that contains the counts of how often each word occurs in the particular documentThis will result in sparse vectors<jupyter_code>### Transforming words into feautre vectors

import numpy as np
from sklearn.feature_extraction.text import CountVectorizer
count = CountVectorizer()
docs = np.array([
    'The sun is shining',
    'The weather is sweet',
    'The sun is shining and the weather is sweet'])
bag = count.fit_transform(docs)
print (count.vocabulary_)
print(bag.toarray())
## referred to as "raw term frequencies": tf(t,d) the number of times
## term t appeared in document d
### Assessing word relevancy via term frequency-inverse document frequeny
## TF-idf - downweights frequently occurring words
## Defined as the product of term freqency and inverse document frequency
## Inverse doc frequency is

## log [(n_d)/(1+df(d,t))]
## where n_d is the total number of docuements 
## and df(d,t) is the number of docs d that contain term t

## Sci-kit learn has a transofrmer for this

from sklearn.feature_extraction.text import TfidfTransformer
tfidf = TfidfTransformer()
np.set_printoptions(precision=2)
print(tfidf.fit_transform(count.fit_transform(docs)).toarray())<jupyter_output>[[ 0.    0.43  0.56  0.56  0.    0.43  0.  ]
 [ 0.    0.43  0.    0.    0.56  0.43  0.56]
 [ 0.4   0.48  0.31  0.31  0.31  0.48  0.31]]<jupyter_text>Cleaning Text Data<jupyter_code>## strip out all unwanted characters

df.loc[0, 'review'][-50:]
### Had to edit from book- mistake in book near .join
import re
def preprocessor(text):
    text = re.sub('<[^>]*>', '', text)
    emoticons = re.findall('(?::|;|=)(?:-)?(?:\)|\(|D|P)', text)
    text = re.sub('[\W]+', ' ', text.lower()) + ' '.join(emoticons).replace('-', '')
    return text
preprocessor(df.loc[0, 'review'][-50:])
preprocessor("</a>This :) is :( a test :-) !")
df['review']= df['review'].apply(preprocessor)<jupyter_output><empty_output><jupyter_text>Processing documents into tokens<jupyter_code>## Need to figure out how to split the text into individual elements
## Can tokenize words by splitting at the whitespace characters

def tokenizer(text):
    return text.split()

tokenizer('runners like running and thus they run')
## Word stemming is taking the word root and mapping words that are similar
## nltk uses the porter stemming alorithm

from nltk.stem.porter import PorterStemmer
porter = PorterStemmer()
def tokenizer_porter(text):
    return [porter.stem(word) for word in text.split()]

tokenizer_porter('runners like running and thus they run')
## lemmatizaiton aims to obtain the canonical forms of individual words 
## stemming and lemmatization have little effect on performance

### Stop word removal 
## Because they are so common, stop words only have a minimal effect
## on the classification 

import nltk
nltk.download('stopwords')


from nltk.corpus import stopwords
stop = stopwords.words('english')
[w for w in tokenizer_porter('a runner likes running and runs a lot')[-10:] if w not in stop]<jupyter_output><empty_output><jupyter_text>Training a logistic regression model for document classification<jupyter_code>## Divide dataframe into 25,000 training and 25,000 test

X_train = df.loc[:25000, 'review'].values
y_train = df.loc[:25000, 'sentiment'].values
X_test = df.loc[25000:, 'review'].values
y_test = df.loc[25000:, 'sentiment'].values
from sklearn.grid_search import GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.feature_extraction.text import TfidfVectorizer

tfidf = TfidfVectorizer(strip_accents = None,
                       lowercase = False,
                       preprocessor = None)
param_grid = [{'vect__ngram_range': [(1,1)],
              'vect__stop_words': [stop, None],
              'vect__tokenizer': [tokenizer, tokenizer_porter],
              'clf__penalty': ['l1', 'l2'],
              'clf__C': [1.0, 10.0, 100.0]},
             {'vect__ngram_range': [(1,1)],
              'vect__stop_words': [stop, None],
              'vect__tokenizer': [tokenizer, tokenizer_porter],
              'vect__use_idf': [False],
              'vect__norm':[None],
              'clf__penalty': ['l1', 'l2'],
              'clf__C': [1.0, 10.0, 100.0]}]

lr_tfidf = Pipeline([('vect', tfidf), ('clf', LogisticRegression(random_state = 0))])

gs_lr_tfidf = GridSearchCV(lr_tfidf, param_grid, scoring='accuracy', cv = 5, verbose = 1, n_jobs = -1)
gs_lr_tfidf.fit(X_train, y_train)
print('Best Parameter set: %s ' % gs_lr_tfidf.best_params_)
print('CV Accuracy: %.3f' % gs_lr_tfidf.best_score_)
clf = gs_lr_tfidf.best_estimator_
print('Test Accuracy: %.3f' % clf.score(X_test, y_test))<jupyter_output><empty_output><jupyter_text>Naieve bayes classifiers are also popular for this kind of work. Can read about them in:S.Raschka Naive Bayes and Text Classification I - introduction and Theory. Computing Research Repository (CoRR), abs/1410.5329, 2014. Http://arxiv.org/pdf/1410.5329v3.pdf  Out of core learningCan stream little bits of data at a time to train the model and update the estimates<jupyter_code>import numpy as np
import re
from nltk.corpus import stopwords
stop = stopwords.words('english')
def tokenizer(text):
    text = re.sub('<[*>]*.', '', text)
    emoticons = re.findall('(?::|;|=)(?:-)?(?:\)|\(|D|P)', text.lower())
    text = re.sub('[\W]+', ' ', text.lower()) + ' '.join(emoticons).replace('-', '')
    tokenized = [w for w in text.split() if w not in stop]
    return tokenized
def stream_docs(path):
    with open(path, 'r') as csv:
        next(csv)
        for line in csv:
            text, label = line[:-3], int (line[-2])
            yield text, label
next(stream_docs(path='./movie_data.csv'))
def get_minibatch(doc_stream, size):
    docs ,y = [], []
    try:
        for _ in range(size):
            text, label = next(doc_stream)
            docs.append(text)
            y.append(label)
    except StopIteration:
        return None, None
    return docs, y
### Use a hashing trick to be able to calculate counts out of memory

from sklearn.feature_extraction.text import HashingVectorizer
from sklearn.linear_model import SGDClassifier
vect = HashingVectorizer(decode_error='ignore',
                        n_features = 2**21,
                        preprocessor=None,
                        tokenizer = tokenizer)

clf = SGDClassifier(loss = 'log', random_state = 1, n_iter = 1)
doc_stream = stream_docs(path='./movie_data.csv')
## Initialized progress bar with 45 minibatches of 1000 docs each
## use the las 5000 for performance 

import pyprind
pbar = pyprind.ProgBar(45)
classes = np.array([0,1])
for _ in range(45):
    X_train, y_train = get_minibatch(doc_stream, size = 1000)
    if not X_train:
        break
    X_train = vect.transform(X_train)
    clf.partial_fit(X_train, y_train, classes = classes)
    pbar.update()
X_test, y_test = get_minibatch(doc_stream, size = 5000)
X_test = vect.transform(X_test)
print('Accuracy: %.3f' % clf.score(X_test, y_test))
## Add last 5000 docs to update the model
clf = clf.partial_fit(X_test, y_test)<jupyter_output>/Users/andrew.moskowitz/anaconda2/lib/python2.7/site-packages/sklearn/linear_model/stochastic_gradient.py:117: DeprecationWarning: n_iter parameter is deprecated in 0.19 and will be removed in 0.21. Use max_iter and tol instead.
  DeprecationWarning)<jupyter_text>A Popular extension of this model that accounts for structre and grammar is the LDA or Latent Dirichlet allocationWord2vec is a more modern application of the bag-of-words model     uses neural networks to automatically learn relationships betweenw ords  Chapter 9 Embedding a Machine Learning Model into a Web Application Session was kept open at the suggestion of the authors as we use the same model that was geneated in the previous chapterOne way for "model persistence" (being able to reuse a trained model) is serializing and deserializing our python objects. this allows us to save and reload the current state of our model.<jupyter_code>import pickle
import os
dest = os.path.join('movieclassifier', 'pkl_objects')
if not os.path.exists(dest):
    os.makedirs(dest)
pickle.dump(stop,
           open(os.path.join(dest, 'stopwords.pkl'), 'wb'), protocol = 2)
pickle.dump(clf, open(os.path.join(dest, 'classifier.pkl'), 'wb'), protocol = 2)<jupyter_output><empty_output><jupyter_text>The next bit to test the serializer and vetorizer was done in an ipython session  Setting up a SQLite database for data storage<jupyter_code>### Create a new sql lite database inside movieclassifier to
### collect optional feedback about predictions from users

import sqlite3
import os
conn = sqlite3.connect('reviews.sqlite')
c = conn.cursor()
c.execute('CREATE TABLE review_db (review TEXT, sentiment INTEGER, date TEXT)')
example1 = 'I love this movie'
c.execute("INSERT INTO review_db (review, sentiment, date) VALUES (?, ?, DATETIME('now'))", (example1, 1))
example2 = 'I disliked this movie'
c.execute("INSERT INTO review_db (review, sentiment, date) VALUES (?, ?, DATETIME('now'))", (example2, 0))
conn.commit()
conn.close()
conn = sqlite3.connect('reviews.sqlite')
c = conn.cursor()
c.execute("SELECT * FROM review_db where date BETWEEN '2018-01-01 00:00:00' AND DATETIME('now')")
results = c.fetchall()
conn.close()
print(results)
import flask
import wtforms<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11933887384442991, 'fraction_numerical': 0.022130917919506957, 'mean_word_length': 4.102429728442115, 'pattern_counts': {'":': 1, '<': 34, '<?xml version=': 0, '>': 35, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '9643', 'license': 'MIT', 'n_tokens_mistral': 3621, 'n_tokens_neox': 3394, 'n_words': 1098, 'path': 'Chapter 8 Sentiment Analysis + Chapter 9 Embedding in Web App.ipynb', 'repo_name': 'almoskowitz/Raschka_MLBook', 'size': '10709'}