<jupyter_start><jupyter_text>Jupyter Notebook Playground<jupyter_code>print("hello world!")
my_variable = 101
print(my_variable)<jupyter_output><empty_output><jupyter_text>Text Classification Tutorial Credit to: https://stackabuse.com/text-classification-with-python-and-scikit-learn/<jupyter_code>import numpy as np
import re
import nltk
from sklearn.datasets import load_files
import pickle
from nltk.corpus import stopwords
movie_data = load_files(r"review_polarity/txt_sentoken")
X, y = movie_data.data, movie_data.target<jupyter_output><empty_output><jupyter_text>Preprocessing The Data<jupyter_code>documents = []

from nltk.stem import WordNetLemmatizer


stemmer = WordNetLemmatizer()

for sen in range(0, len(X)):
    # Remove all the special characters
    document = re.sub(r'\W', ' ', str(X[sen]))
    
    # remove all single characters
    document = re.sub(r'\s+[a-zA-Z]\s+', ' ', document)
    
    # Remove single characters from the start
    document = re.sub(r'\^[a-zA-Z]\s+', ' ', document) 
    
    # Substituting multiple spaces with single space
    document = re.sub(r'\s+', ' ', document, flags=re.I)
    
    # Removing prefixed 'b'
    document = re.sub(r'^b\s+', '', document)
    
    # Converting to Lowercase
    document = document.lower()
    
    # Lemmatization
    document = document.split()

    document = [stemmer.lemmatize(word) for word in document]
    document = ' '.join(document)
    
    documents.append(document)<jupyter_output><empty_output><jupyter_text>Feature Engineering<jupyter_code>from sklearn.feature_extraction.text import CountVectorizer
vectorizer = CountVectorizer(max_features=1500, min_df=5, max_df=0.7, stop_words=stopwords.words('english'))
X1 = vectorizer.fit_transform(documents).toarray()<jupyter_output><empty_output><jupyter_text>CountVectorizer makes very unique word a feature for the ml model. Parameters:- We take a max of 1500 features (the most common, highest occuring 1500 unique words)- min_df is number of documents they appear in at minimum- max_df is the maximum percentage of documents containing this word- Finally, stop_words removes any very commons words in the english language<jupyter_code>from sklearn.feature_extraction.text import TfidfTransformer
tfidfconverter = TfidfTransformer()
X2 = tfidfconverter.fit_transform(X1).toarray()<jupyter_output><empty_output><jupyter_text>TfidfTransformer turns the count array earlier into an array of weights for each unique word based on the document:- TF-IDF weight is a weight often used in information retrieval and text mining- This weight is a statistical measure used to evaluate how important a word is to a document in a collection or corpus- The importance increases proportionally to the number of times a word appears in the document but is offset by the frequency of the word in the corpus- Variations of the tf-idf weighting scheme are often used by search engines as a central tool in scoring and ranking a document's relevance given a user query  Splitting The Data (Train-Test Split)<jupyter_code>from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X2, y, test_size=0.2, random_state=0)<jupyter_output><empty_output><jupyter_text>Training The Model<jupyter_code>from sklearn.ensemble import RandomForestClassifier
classifier = RandomForestClassifier(n_estimators=1000, random_state=0)
classifier.fit(X_train, y_train) 
y_pred = classifier.predict(X_test)<jupyter_output><empty_output><jupyter_text>Random Forest example:  Pros of random forest:- Great predictive performance for binary classification- They provide a reliable feature importance estimate- They offer efficient estimates of the test error without incurring the cost of repeated model training associated with cross-validation- Handles thousands of input variables without variable deletionCons of random forest:- An ensemble model is inherently less interpretable than an individual decision tree- Training a large number of deep trees can have high computational costs (but can be parallelized) and use a lot of memory- Predictions are slower, which may create challenges for applicationsMore considerations: - https://github.com/TayariAmine/ML_cheat_sheet/wiki/Random-forest-Pros-and-Cons- https://www.oreilly.com/library/view/hands-on-machine-learning/9781789346411/e17de38e-421e-4577-afc3-efdd4e02a468.xhtml  Evaluating The Model<jupyter_code>from sklearn.metrics import classification_report, confusion_matrix, accuracy_score

print("CONFUSION MATRIX: \n{}".format(confusion_matrix(y_test,y_pred)))
print("\nACCURACY: {}".format(accuracy_score(y_test, y_pred)))<jupyter_output><empty_output><jupyter_text><jupyter_code>print(classification_report(y_test,y_pred))<jupyter_output><empty_output><jupyter_text>Saving The Model<jupyter_code>with open('text_classifier', 'wb') as picklefile:
    pickle.dump(classifier,picklefile)
with open('text_classifier', 'rb') as training_model:
    model = pickle.load(training_model)
y_pred2 = model.predict(X_test)

print(confusion_matrix(y_test, y_pred2))
print(classification_report(y_test, y_pred2))
print(accuracy_score(y_test, y_pred2))<jupyter_output><empty_output><jupyter_text>Try It Yourself!<jupyter_code>import numpy as np
import re
import nltk
from sklearn.datasets import load_files
import pickle
from nltk.corpus import stopwords

from sklearn.feature_extraction.text import CountVectorizer
from nltk.stem import WordNetLemmatizer
from sklearn.feature_extraction.text import TfidfTransformer

movie_data = load_files(r"review_polarity/txt_sentoken")
X, y = movie_data.data, movie_data.target

# Preprocessing functions
def preprocessing(input_X):

    stemmer = WordNetLemmatizer()
    documents = []

    for sen in range(0, len(input_X)):
        # Remove all the special characters
        document = re.sub(r'\W', ' ', str(input_X[sen]))

        # remove all single characters
        document = re.sub(r'\s+[a-zA-Z]\s+', ' ', document)

        # Remove single characters from the start
        document = re.sub(r'\^[a-zA-Z]\s+', ' ', document) 

        # Substituting multiple spaces with single space
        document = re.sub(r'\s+', ' ', document, flags=re.I)

        # Removing prefixed 'b'
        document = re.sub(r'^b\s+', '', document)

        # Converting to Lowercase
        document = document.lower()

        # Lemmatization
        document = document.split()

        document = [stemmer.lemmatize(word) for word in document]
        document = ' '.join(document)

        documents.append(document)

    vectorizer = CountVectorizer(max_features=1500, min_df=5, max_df=0.7, stop_words=stopwords.words('english'))
    X1 = vectorizer.fit_transform(documents).toarray()

    tfidfconverter = TfidfTransformer()
    X2 = tfidfconverter.fit_transform(X1).toarray()
    
    return X2, documents

with open('text_classifier', 'rb') as training_model:
    model = pickle.load(training_model)<jupyter_output><empty_output><jupyter_text>Change the sentences in the 'new_X_test' array below!<jupyter_code>new_X_test = ['I loved this movie so much!', 'This movie was bad and had terrible actors']
count = len(new_X_test)
new_X_test.extend(X)

new_X2, new_documents = preprocessing(new_X_test)
for i in range(count):
    review = new_documents[i]
    prediction = model.predict([new_X2[i]])
    print("review: {}, pred: {}".format(review, prediction))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0957962813257882, 'fraction_numerical': 0.010778765831312315, 'mean_word_length': 5.285351397121084, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 49, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '509233', 'license': 'MIT', 'n_tokens_mistral': 2403, 'n_tokens_neox': 2285, 'n_words': 612, 'path': 'NLP_Workshop_May2020/HandsOnWorkshop_May2020.ipynb', 'repo_name': 'chriskok/MLAI-CoP', 'size': '7422'}