<jupyter_start><jupyter_text>And the test data.<jupyter_code>test_data_df.head()<jupyter_output><empty_output><jupyter_text>Let's count how many labels do we have for each sentiment class.<jupyter_code>train_data_df.Sentiment.value_counts()<jupyter_output><empty_output><jupyter_text>Finally, let's calculate the average number of words per sentence. We could do the following using a list comprehension with the number of words per sentence.<jupyter_code>import numpy as np 

np.mean([len(s.split(" ")) for s in train_data_df.Text])<jupyter_output><empty_output><jupyter_text>First we need to init the vectorizer. We need to remove puntuations, lowercase, remove stop words, and stem words. All these steps can be directly performed by `CountVectorizer` if we pass the right parameter values. We can do as follows.<jupyter_code>import re, nltk
from sklearn.feature_extraction.text import CountVectorizer        
from nltk.stem.porter import PorterStemmer

stemmer = PorterStemmer()
def stem_tokens(tokens, stemmer):
    stemmed = []
    for item in tokens:
        stemmed.append(stemmer.stem(item))
    return stemmed

def tokenize(text):
    # remove non letters
    text = re.sub("[^a-zA-Z]", " ", text)
    # tokenize
    tokens = nltk.word_tokenize(text)
    # stem
    stems = stem_tokens(tokens, stemmer)
    return stems
######## 

vectorizer = CountVectorizer(
    analyzer = 'word',
    tokenizer = tokenize,
    lowercase = True,
    stop_words = 'english',
    max_features = 100
)
corpus_data_features = vectorizer.fit_transform(train_data_df.Text.tolist() + test_data_df.Text.tolist())<jupyter_output><empty_output><jupyter_text>Numpy arrays are easy to work with, so convert the result to an array.<jupyter_code>corpus_data_features_nd = corpus_data_features.toarray()
corpus_data_features_nd.shape
vocab = vectorizer.get_feature_names()
print(vocab)<jupyter_output>['act', 'actor', 'actual', 'ani', 'anoth', 'bad', 'becaus', 'befor', 'believ', 'best', 'better', 'br', 'cast', 'charact', 'come', 'comedi', 'day', 'did', 'didn', 'direct', 'director', 'doe', 'doesn', 'don', 'effect', 'end', 'enjoy', 'everi', 'fact', 'feel', 'film', 'funni', 'girl', 'good', 'got', 'great', 'guy', 'ha', 'hi', 'horror', 'just', 'kill', 'know', 'life', 'like', 'littl', 'live', 'look', 'lot', 'love', 'm', 'make', 'man', 'mani', 'minut', 'movi', 'music', 'new', 'noth', 'old', 'onli', 'origin', 'peopl', 'perform', 'play', 'plot', 'point', 'pretti', 'quit', 'real', 'realli', 'role', 's', 'say', 'scene', 'seen', 'set', 'someth', 'star', 'start', 'stori', 't', 'thi', 'thing', 'think', 'thought', 'time', 'tri', 'turn', 'use', 've', 'veri', 'wa', 'want', 'watch', 'way', 'whi', 'work', 'world', 'year']<jupyter_text>We can also print the counts of each word in the vocabulary as follows.<jupyter_code>dist = np.sum(corpus_data_features_nd, axis=0)

for tag, count in zip(vocab, dist):
    print(count, tag)<jupyter_output>17494 act
13599 actor
10011 actual
15048 ani
8594 anoth
18544 bad
17715 becaus
8525 befor
7841 believ
12630 best
11459 better
201954 br
8762 cast
28364 charact
13260 come
7430 comedi
7850 day
12624 did
8768 didn
7450 direct
10050 director
11673 doe
8876 doesn
17661 don
7248 effect
19100 end
8490 enjoy
7968 everi
7358 fact
10353 feel
95890 film
8768 funni
7881 girl
30195 good
7234 got
18404 great
9257 guy
33400 ha
57725 hi
7507 horror
35186 just
7419 kill
15170 know
12938 life
45210 like
12435 littl
8718 live
19951 look
9723 lot
18221 love
10176 m
30035 make
11943 man
13486 mani
7441 minut
103284 movi
8587 music
8102 new
8396 noth
8811 old
23241 onli
7630 origin
18384 peopl
10733 perform
17374 play
13809 plot
8068 point
7258 pretti
7498 quit
9442 real
23095 realli
8386 role
129794 s
14992 say
21452 scene
13378 seen
8033 set
10219 someth
8450 star
8145 start
25285 stori
68322 t
151030 thi
16512 thing
17546 think
7604 thought
31968 time
12535 tri
7623 turn
10228 use
10244 ve
27729 veri
95[...]<jupyter_text>A bag-of-words linear classifier<jupyter_code>from sklearn.cross_validation import train_test_split

# remember that corpus_data_features_nd contains all of our original train and test data, so we need to exclude
# the unlabeled test entries
X_train, X_test, y_train, y_test  = train_test_split(
    corpus_data_features_nd[0:len(train_data_df)], 
    train_data_df.Sentiment,
    train_size=0.85, 
    random_state=1234)
print(X_test[0])<jupyter_output>[1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 1 1 0 0 0
 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 1 0 0 0 1 0 0 1 0 1 0 1 0
 0 0 0 0 0 0 1 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0]<jupyter_text>Now we are ready to train our classifier.<jupyter_code>from sklearn.linear_model import LogisticRegression

log_model = LogisticRegression()
log_model = log_model.fit(X=X_train, y=y_train)<jupyter_output><empty_output><jupyter_text>Now we use the classifier to label our evaluation set. We can use either `predict` for classes or `predict_proba` for probabilities.<jupyter_code>y_pred = log_model.predict(X_test)
from sklearn.metrics import classification_report

print(classification_report(y_test, y_pred))<jupyter_output>precision    recall  f1-score   support

   NEGATIVE       0.87      0.86      0.87      1920
   POSITIVE       0.85      0.87      0.86      1830

avg / total       0.86      0.86      0.86      3750<jupyter_text>Finally, we can re-train our model with all the training data and use it for sentiment classification with the original (unlabeled) test set.<jupyter_code>log_model = LogisticRegression()
log_model = log_model.fit(X=corpus_data_features_nd[0:len(train_data_df)], y=train_data_df.Sentiment)

test_pred = log_model.predict(corpus_data_features_nd[len(train_data_df):])
actual_pred = test_data_df["Sentiment"].tolist()

print(classification_report(test_pred, actual_pred))<jupyter_output>precision    recall  f1-score   support

   NEGATIVE       0.86      0.85      0.86     12584
   POSITIVE       0.85      0.86      0.86     12416

avg / total       0.86      0.86      0.86     25000
{'fraction_non_alphanumeric': 0.10825596816976127, 'fraction_numerical': 0.10377984084880637, 'mean_word_length': 4.519670631290028, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 36, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 15, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '407367', 'license': 'Apache-2.0', 'n_tokens_mistral': 2591, 'n_tokens_neox': 2157, 'n_words': 714, 'path': 'romil/logistic_regression-with-imdb.ipynb', 'repo_name': 'romil93/SentimentAnalysis-CSCI544-Fall2016-', 'size': '6032'}