<jupyter_start><jupyter_text>More NLP Truncated Singular Value Decomposition and Dimensionality ReductionWhen processing text we end up with feature sets that are large! There is up to one feature per different word in our text sample, far larger than a typical feature set that we're used to. One thing we can do when vectorizing is just to cap the number of features we end up with, but that doesn't seem to be the most sophisticated approach. TSVD is one thing that we can do to chop down the feature set - or reduce the dimensions - with a little more thought.  Dimensionality ReductionDimensionality reduction is a common technique in machine learning, it does its name - reduces the dimensions in our feature data.   Load Dataset from Last Time<jupyter_code>#Load Data
df = pd.read_csv("data/spam.csv", encoding="ISO-8859-1")
df.drop(columns={"Unnamed: 2", "Unnamed: 3", "Unnamed: 4"}, inplace=True)
df.rename(columns={"v1":"target", "v2":"text"}, inplace=True)
#TF-IDF
vec_tf = TfidfVectorizer(sublinear_tf=True, ngram_range=(1,4), stop_words="english", strip_accents="unicode")
tmp = vec_tf.fit_transform(df["text"])
tok_cols = vec_tf.get_feature_names()
tok_df = pd.DataFrame(tmp.toarray(), columns=tok_cols)
print("original:", df["text"].shape)
print("vectorized:", tmp.shape)
y = df["target"]
X = df["text"]
tok_df.sample(5)<jupyter_output>original: (5572,)
vectorized: (5572, 89635)<jupyter_text>LSA - Latent Semantic AnalysisThe TSVD performs somehting called latent semantic analysis. The process of LSA and the math behind it are not something we need to explore in detail. (LSA is often called LSI - Latent Semantic Indexing)The idea of LSA is that it can generate "concepts" in the text. These concepts are found by looking at which terms occur in which documents - documents that have the same terms repeated are likely related to the same concept; other documents that share other words with those documents are likely on the same concept as well.  An important part is the word "Latent" - i.e. the patterns detected are hidden, not explicit in the data.   Implement SVD to Trim DatasetWe are starting with LOTS of feature inputs. Below we can loop through several models of different number of remaining components to see the accuracy depending on the number of features we keep in the feature set. The truncated part of truncated SVD trims the featureset down to the most significant features. We started with a lot of features - we can make predictions that are close to as accurate with far fewer, hopefully!<jupyter_code># get a list of models to evaluate
def get_models():
	models = dict()
	for i in range(1,10):
		n = i*50
		steps = [('svd', TruncatedSVD(n_components=n)), ('m', LogisticRegression())]
		models[str(n)] = Pipeline(steps=steps)
	return models
 
# evaluate a give model using cross-validation
def evaluate_model(model, X, y):
	#Splits cut for speed
	cv = RepeatedStratifiedKFold(n_splits=2, n_repeats=1)
	scores = cross_val_score(model, X, y, scoring='accuracy', cv=cv, n_jobs=-1, error_score='raise')
	return scores
 
models = get_models()
# evaluate the models and store results
results, names = list(), list()
for name, model in models.items():
	scores = evaluate_model(model, tok_df, y)
	results.append(scores)
	names.append(name)
# plot model performance for comparison

plt.boxplot(results, labels=names, showmeans=True)
plt.xticks(rotation=45)
plt.show()
tf_idf = TfidfVectorizer(sublinear_tf=True, ngram_range=(1,3), stop_words='english', strip_accents='unicode')
mp_vec = tf_idf.fit_transform(df['text'])

tok_cols2 = tf_idf.get_feature_names()
tmp_df = pd.DataFrame(mp_vec.toarray(), columns=tok_cols2)
tmp_df.head()
X_train, X_test, y_train, y_test = train_test_split(tmp_df, y)
svd_temp = TruncatedSVD(n_components=25)
pipe_steps = [('svd', svd_temp), ('model', RandomForestClassifier)]
pipe_test = Pipeline(steps=pipe_steps)

pipe_test.fit(X_train, y_train)
pipe_test.score(X_test, y_test)<jupyter_output><empty_output><jupyter_text>Exercise - Truncated SVDTry to use the same text for predictions from the newsgroups last time. Try to use the TSVD with a limited number of components and see if the accuracy can stay similar to what we got last time.<jupyter_code>from sklearn.datasets import fetch_20newsgroups
remove = ("headers", "footers", "quotes")
categories = ["alt.atheism", "talk.religion.misc"]

data_train = fetch_20newsgroups(
    subset="train", categories=categories, shuffle=True, random_state=42, remove=remove)

data_test = fetch_20newsgroups(
    subset="test", categories=categories, shuffle=True, random_state=42, remove=remove)
# Vectorize and prep datasets

# Create Models<jupyter_output><empty_output><jupyter_text>Word2Vec and ClassificationIn addition to calculating things solely directly from our data, we can also use some external tools that can help create embeddings that are a little better (hopefully). This is also a neural network running behind the scenes to help us out. Word2Vec is an algorithm made by Google that can help process text and produce embeddings. Word2Vec looks for associations of words that occur with each other.  Word2Vec in ProcessWord2Vec generates its embeddings by looking at words in a sentence, and the surrounding words in that same sentence.  GensimGensim is a package that we can install that has an implementation of Word2Vec that we can use pretty easily.<jupyter_code>from gensim.models import Word2Vec
import nltk
for package in ['stopwords','punkt','wordnet']:
    nltk.download(package) 
from nltk.corpus import stopwords 
from nltk.tokenize import word_tokenize
stop_words    = set(stopwords.words('english'))<jupyter_output><empty_output><jupyter_text>TokenizerSince we are not using the vecorizer from sklearn, we need to provide our own tokenization. We can use the nltk based one from last time. We can also do any other types of processing here that we may want - stemming, customized stop words, etc... For this one I chopped out any 1 character tokens and added a regex filter to get rid of punctuation.<jupyter_code>class lemmaTokenizer(object):
    def __init__(self, stop_words):
        self.stop_words = stop_words
        from nltk.stem import WordNetLemmatizer
        self.lemmatizer = WordNetLemmatizer()
    def __call__(self, doc):
        tokens = word_tokenize(doc)
        filtered_tok = []
        for tok in tokens:
            if tok not in stop_words:
                tok = re.sub('\W+','', tok) #Punctuation strip
                tmp = self.lemmatizer.lemmatize(tok)
                if len(tmp) >= 2:
                    filtered_tok.append(tmp)
        return filtered_tok<jupyter_output><empty_output><jupyter_text>Create Clan Text - Tokenize and LemmatizePrep some data. The "second half" of the dataframe is what we can use with the Word2Vec prediction models - we have cleaned up lists of tokens as well as translating the targets to 1 and 0.<jupyter_code>tok = lemmaTokenizer(stop_words)
df["clean_text"] = df["text"].apply(lambda x: tok(x))
df["target2"] = pd.get_dummies(df["target"], drop_first=True)
df.head()<jupyter_output><empty_output><jupyter_text>Create Word2VecWe can train our Word2Vec model with our cleaned up data. This will have Word2Vec do its magic behind the scenes and perform the training. W2V works in one of two ways, which are roughly opposites of each other, when doing this training: Continuous Bag of Words: looks at a window around each target word to try to predict surrounding words. Skip-Gram: looks at words surrounding target to try to predict it. We'll revisit the details of this stuff later on when we look at neural networks, since W2V is a neural network algorithm, it will make more sense in context. Note: this training is not making a model that we are using to make predictions. This is training inside the W2V algorithm to generate representations of our tokens.<jupyter_code># create Word2vec model
model = Word2Vec(df['clean_text'],min_count=1, vector_size=200, sg=1)
#min_count=1 means word should be present at least across all documents,
#if min_count=2 means if the word is present less than 2 times across all the documents then we shouldn't consider it

#combination of word and its vector
w2v = dict(zip(model.wv.index_to_key, model.wv.vectors))  

#for converting sentence to vectors/numbers from word vectors result by Word2Vec
class MeanEmbeddingVectorizer(object):
    def __init__(self, word2vec):
        self.word2vec = word2vec
        # if a text is empty we should return a vector of zeros
        # with the same dimensionality as all the other vectors
        self.dim = len(next(iter(word2vec.values())))

    def fit(self, X, y):
        return self

    def transform(self, X):
        return np.array([
            np.mean([self.word2vec[w] for w in words if w in self.word2vec]
                    or [np.zeros(self.dim)], axis=0)
            for words in X
        ])<jupyter_output><empty_output><jupyter_text>Word2Vec ModelEach word in the vocabulary now has a vector representing it - of size 200. We can make a dataframe and see each token in our text and its vector representation. This vector is the internal representation of each token that is generated by Word2Vec. This is how the algorithm calculates things like similarity...<jupyter_code>tmp = pd.DataFrame(w2v)
vectors = model.wv
tmp.head()<jupyter_output><empty_output><jupyter_text>SimilarityOne of the things that Word2Vec allows us to do is to look at the similarity of words. This similarity is calculated via the cosine distance of the vectors. Cosine similarity is a technique to calculate the distance between two vectors - smaller distance, more similar. ![Cosine Similarity](images/cosine_sim.png "Cosine Similarity" )Once the vectors are derived by in the training process, these similarity calculations are pretty easy and quick. Note: the similarites here are calculated by the values derived from our trained model. So they are based on the relationships in our text. Word2Vec and other NLP packages also commonly have pretrained models that can be downloaded that are based on large amounts of text. Words may be represented very differently in those vs whatever we train here - the more data we have, the more consistent they'll be; the more "unique" our text is, the more different it will be.<jupyter_code>#Find the most similar word to anything in our vocabulary 
vectors.most_similar("think")[0:3]
# We can also see how similar different words are. 
vectors.similarity("think", "determine")<jupyter_output><empty_output><jupyter_text>Make PredictionsWe can take our actual data now and transform it through the Word2Vec model that we've made. This will generate our smaller feature set that we can build our models from.One of the things that the MeanEmbeddingVectorizer does is to collapse the data down to those 200 dimensions in the vector.<jupyter_code>#SPLITTING THE TRAINING DATASET INTO TRAINING AND VALIDATION
 
# Split data - using the new dataframe parts that we cleaned up. 
X_train, X_test, y_train, y_test = train_test_split(df["clean_text"],df["target2"])

#Word2vec
# Fit and transform
modelw = MeanEmbeddingVectorizer(w2v)
X_train_vectors_w2v = modelw.transform(X_train)
X_test_vectors_w2v = modelw.transform(X_test)
X_train_vectors_w2v.shape<jupyter_output><empty_output><jupyter_text>Build ModelWe can now use the new data to make predictions.<jupyter_code># Make predictions
lr_w2v = RandomForestClassifier()
lr_w2v.fit(X_train_vectors_w2v, y_train)  #model

#Predict y value for test dataset
y_predict = lr_w2v.predict(X_test_vectors_w2v)
y_prob = lr_w2v.predict_proba(X_test_vectors_w2v)[:,1]
 

print(classification_report(y_test,y_predict))
print('Confusion Matrix:',confusion_matrix(y_test, y_predict))
 
fpr, tpr, thresholds = roc_curve(y_test, y_prob)
roc_auc = auc(fpr, tpr)
print('AUC:', roc_auc)<jupyter_output>precision    recall  f1-score   support

           0       0.98      0.99      0.98      1203
           1       0.93      0.85      0.88       190

    accuracy                           0.97      1393
   macro avg       0.95      0.92      0.93      1393
weighted avg       0.97      0.97      0.97      1393

Confusion Matrix: [[1190   13]
 [  29  161]]
AUC: 0.9712845080281751<jupyter_text>Exercise - Word2VecUse the newsgroup data and Word2Vec to make predictions.<jupyter_code># Prepare datsets and Tokenize

# create Word2vec model

#Word2vec
# Fit and transform

# Make predictions


#Predict y value for test dataset<jupyter_output>precision    recall  f1-score   support

           0       0.55      0.62      0.58       319
           1       0.42      0.34      0.38       251

    accuracy                           0.50       570
   macro avg       0.48      0.48      0.48       570
weighted avg       0.49      0.50      0.49       570

Confusion Matrix: [[199 120]
 [165  86]]
AUC: 0.51149633441157
{'fraction_non_alphanumeric': 0.07443667443667444, 'fraction_numerical': 0.022843822843822845, 'mean_word_length': 4.285831622176591, 'pattern_counts': {'":': 2, '<': 46, '<?xml version=': 0, '>': 47, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '22749', 'license': 'MIT', 'n_tokens_mistral': 3998, 'n_tokens_neox': 3760, 'n_words': 1516, 'path': '014_nlp_part_2.ipynb', 'repo_name': 'allyryry/3950_Workbooks', 'size': '12870'}