<jupyter_start><jupyter_text>SENTIMENT ANALYSIS -- (WITH HOMEMADE DATA!) (via [these docs](http://www.nltk.org/howto/sentiment.html))  |  10-06-19  STEP 1: Import ALL the things<jupyter_code>from nltk.classify import NaiveBayesClassifier
from nltk.corpus import subjectivity
from nltk.sentiment import SentimentAnalyzer
from nltk.sentiment.util import *<jupyter_output><empty_output><jupyter_text>STEP 2: Import the fake data you created just for this project<jupyter_code>import os
import pandas as pd
negative = os.listdir('AI_NEG/')
positive = os.listdir('AI_POS/')
positive_alltext = []
for file in positive:
    f=open('AI_POS/'+file)
    content=f.read()
    positive_alltext.append(content)
    f.close()

negative_alltext = []
for file in negative:
    f=open('AI_NEG/'+file)
    content=f.read()
    negative_alltext.append(content)
    f.close()<jupyter_output><empty_output><jupyter_text>STEP 2b: Tokenize and clean the data<jupyter_code>from nltk.tokenize import word_tokenize

def get_tokens(sentence):
    tokens = word_tokenize(sentence)
    clean_tokens = [word.lower() for word in tokens if word.isalpha()]
    return clean_tokens

negative_alltext_tokens = [get_tokens(sentence) for sentence in negative_alltext]
positive_alltext_tokens = [get_tokens(sentence) for sentence in positive_alltext]
neg_docs = [(sent, 'neg') for sent in negative_alltext_tokens]
pos_docs = [(sent, 'pos') for sent in positive_alltext_tokens]<jupyter_output><empty_output><jupyter_text>STEP 3: Create `test` and `train` for both `subj` and `obj`<jupyter_code>train_neg_docs = neg_docs[:4]
test_neg_docs = neg_docs[4:5]
train_pos_docs = pos_docs[:4]
test_pos_docs = pos_docs[4:5]<jupyter_output><empty_output><jupyter_text>STEP 4: Combine the two `test` and `train` sets<jupyter_code>training_docs = train_neg_docs + train_pos_docs
testing_docs = test_neg_docs + test_pos_docs<jupyter_output><empty_output><jupyter_text>STEP 5: Use `SentimentAnalyzer` to mark negation in training docs<jupyter_code>sentim_analyzer = SentimentAnalyzer()
all_words_neg = sentim_analyzer.all_words([mark_negation(doc) for doc in training_docs])
all_words_neg<jupyter_output><empty_output><jupyter_text>Note how this sentiment analyzer is SUPPOSED to mark everything after a negation word with '_NEG'However, we do not have enough data in our 10 text file dataset to actually run this successfully  STEP 6: Use `unigram_word_feats` to get unigrams features<jupyter_code>unigram_feats = sentim_analyzer.unigram_word_feats(all_words_neg)
len(unigram_feats)<jupyter_output><empty_output><jupyter_text>STEP 7: Use `add_feat_extractor` to get a feature-value representation of our data  Apply to both `training_set` and `testing_set`<jupyter_code>sentim_analyzer.add_feat_extractor(extract_unigram_feats, unigrams=unigram_feats)
training_set = sentim_analyzer.apply_features(training_docs)
training_set[:1]
test_set = sentim_analyzer.apply_features(testing_docs)
test_set[:1]<jupyter_output><empty_output><jupyter_text>STEP 8: FINAL STEP!! We use Naive Bayes to create a trainer and FINALLY classify our data!<jupyter_code>trainer = NaiveBayesClassifier.train
classifier = sentim_analyzer.train(trainer, training_set)
for key,value in sorted(sentim_analyzer.evaluate(test_set).items()):
    print('{0}: {1}'.format(key,value))<jupyter_output>Evaluating NaiveBayesClassifier results...
Accuracy: 1.0
F-measure [neg]: 1.0
F-measure [pos]: 1.0
Precision [neg]: 1.0
Precision [pos]: 1.0
Recall [neg]: 1.0
Recall [pos]: 1.0
{'fraction_non_alphanumeric': 0.10120412844036697, 'fraction_numerical': 0.011754587155963303, 'mean_word_length': 7.366906474820144, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 36, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '575802', 'license': 'MIT', 'n_tokens_mistral': 1297, 'n_tokens_neox': 1243, 'n_words': 278, 'path': 'assets/all_html/2019_10_06_HW1_viathedocs_kdata.ipynb', 'repo_name': 'dskw1/dskw1.github.io', 'size': '3488'}