<jupyter_start><jupyter_text>NLP Vectorizing Raw Data Count VectorizationCreate a document-term matrix where the entry of each cell will be a count of the number of times that word occured in that document.<jupyter_code># Import the dependencies
import pandas as pd
import re
import string
import nltk

# Setting the max column width to display the text data in the dataframe
pd.set_option('display.max_colwidth', 100)

# Assigning the stopwords
stopwords = nltk.corpus.stopwords.words('english')

# Create the Porter Stemmer
ps = nltk.PorterStemmer()

# Read in the text data
data = pd.read_csv('./data/SMSSpamCollection.tsv', sep='\t')
data.columns = ['label', 'body_text']
# Create function to remove punctuation, tokenize, remove stopwords, and stem
def clean_text(text):
    text = "".join([word.lower() for word in text if word not in string.punctuation])
    tokens = re.split('\W+', text)
    text = [ps.stem(word) for word in tokens if word not in stopwords]
    return text<jupyter_output><empty_output><jupyter_text>Apply CountVectorizer<jupyter_code>from sklearn.feature_extraction.text import CountVectorizer

count_vect = CountVectorizer(analyzer=clean_text)
X_counts = count_vect.fit_transform(data['body_text'])
print(X_counts.shape)
print(count_vect.get_feature_names())
### Apply CountVectorizer to a small sample
data_sample = data[0:20]

count_vect_sample = CountVectorizer(analyzer=clean_text)
X_counts_sample = count_vect_sample.fit_transform(data_sample['body_text'])
print(X_counts_sample.shape)
print(count_vect_sample.get_feature_names())<jupyter_output><empty_output><jupyter_text>Vectorizers output sparse matrices_**Sparse Matrix**: A matrix in which most entries are 0. In the interest of efficient storage, a sparse matrix will be stored by only storing the locations of the non-zero elements._<jupyter_code>X_counts_sample
X_counts_df = pd.DataFrame(X_counts_sample.toarray())
X_counts_df
X_counts_df.columns = count_vect_sample.get_feature_names()
X_counts_df<jupyter_output><empty_output><jupyter_text>N-Grams VectorizingN-Grams creates a document-term matrix where counts still occupy the cell but instead of the columns representing single terms, they represent all combinations of adjacent words of length **n** in your text. Example: "NLP is an interesting topic"| n | Name      | Tokens                                                         ||---|-----------|----------------------------------------------------------------|| 2 | bigram    | ["nlp is", "is an", "an interesting", "interesting topic"]      || 3 | trigram   | ["nlp is an", "is an interesting", "an interesting topic"] || 4 | four-gram | ["nlp is an interesting", "is an interesting topic"]    |  Apply CountVectorizer (w/ N-Grams)<jupyter_code>def clean_text(text):
    text = "".join([word.lower() for word in text if word not in string.punctuation])
    tokens = re.split('\W+', text)
    text = " ".join([ps.stem(word) for word in tokens if word not in stopwords])
    return text

data['cleaned_text'] = data['body_text'].apply(lambda x: clean_text(x))
data.head()
from sklearn.feature_extraction.text import CountVectorizer

ngram_vect = CountVectorizer(ngram_range=(2,2))
X_counts = ngram_vect.fit_transform(data['cleaned_text'])
print(X_counts.shape)
print(ngram_vect.get_feature_names())<jupyter_output><empty_output><jupyter_text>Apply CountVectorizer (w/ N-Grams) to smaller sample<jupyter_code>data_sample = data[0:20]

ngram_vect_sample = CountVectorizer(ngram_range=(2,2))
X_counts_sample = ngram_vect_sample.fit_transform(data_sample['cleaned_text'])
print(X_counts_sample.shape)
print(ngram_vect_sample.get_feature_names())
X_counts_df = pd.DataFrame(X_counts_sample.toarray())
X_counts_df.columns = ngram_vect_sample.get_feature_names()
X_counts_df<jupyter_output><empty_output><jupyter_text>Vectorizing Raw Data with TF-IDFTF-IDF, short for term frequency-inverse document frequence, is a numerical statistic that is intended to reflect how important a word is to a document in a collection or corpus. It creates a document-term matrix where the columns represent single unique terms (unigrams) but the cell represents a weighting meant to represent how important a word is to a document.![image.png](./images/image.png)  Apply TF-IDF Vectorizer<jupyter_code>from sklearn.feature_extraction.text import TfidfVectorizer

tfidf_vect = TfidfVectorizer(analyzer=clean_text)
X_tfidf = tfidf_vect.fit_transform(data['body_text'])
print(X_tfidf.shape)
print(tfidf_vect.get_feature_names())<jupyter_output><empty_output><jupyter_text>Apply TF-IDF Vectorizer on a small sample<jupyter_code>data_sample = data[0:20]

tfidf_vect_sample = TfidfVectorizer(analyzer=clean_text)
X_tfidf_sample = tfidf_vect_sample.fit_transform(data_sample['body_text'])
print(X_tfidf_sample.shape)
print(tfidf_vect_sample.get_feature_names())<jupyter_output><empty_output><jupyter_text>Vectorizers output sparse matrices_**Sparse Matrix**: A matrix in which most entries are 0. In the interest of efficient storage, a sparse matrix will be stored by only storing the locations of the non-zero elements._<jupyter_code>X_tfidf_df = pd.DataFrame(X_tfidf_sample.toarray())
X_tfidf_df.columns = tfidf_vect_sample.get_feature_names()
X_tfidf_df<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10929481087245771, 'fraction_numerical': 0.003991636570994108, 'mean_word_length': 5.703184713375796, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '163717', 'license': 'MIT', 'n_tokens_mistral': 1731, 'n_tokens_neox': 1695, 'n_words': 443, 'path': 'NLP_Vectorizing_Raw_Data.ipynb', 'repo_name': 'NormanLo4319/Natural_Language_Processing', 'size': '5261'}