<jupyter_start><jupyter_text>Vectorizing Raw Data: N-Grams  N-Grams Creates a document-term matrix where counts still occupy the cell but instead of the columns representing single terms, they represent all combinations of adjacent words of length n in your text."NLP is an interesting topic"| n | Name      | Tokens                                                         ||---|-----------|----------------------------------------------------------------|| 2 | bigram    | ["nlp is", "is an", "an interesting", "interesting topic"]      || 3 | trigram   | ["nlp is an", "is an interesting", "an interesting topic"] || 4 | four-gram | ["nlp is an interesting", "is an interesting topic"]    |  Read in text<jupyter_code>import pandas as pd
import re
import string
import nltk
pd.set_option('display.max_colwidth', 100)

stopwords = nltk.corpus.stopwords.words('english')
ps = nltk.PorterStemmer()

data = pd.read_csv("SMSSpamCollection.tsv", sep='\t')
data.columns = ['label', 'body_text']<jupyter_output><empty_output><jupyter_text>Create function to remove punctuation, tokenize, remove stopwords, and stem<jupyter_code>def clean_text(text):
    text = "".join([word.lower() for word in text if word not in string.punctuation])
    tokens = re.split('\W+', text)
    # join back to a string
    text = " ".join([ps.stem(word) for word in tokens if word not in stopwords])
    return text

data['cleaned_text'] = data['body_text'].apply(lambda x: clean_text(x))
data.head()
data.shape<jupyter_output><empty_output><jupyter_text>Apply CountVectorizer (w/ N-Grams)<jupyter_code>from sklearn.feature_extraction.text import CountVectorizer

ngram_vect = CountVectorizer(ngram_range=(10,10))
X_counts = ngram_vect.fit_transform(data['cleaned_text'])
print(X_counts.shape)
print(ngram_vect.get_feature_names())<jupyter_output>(5567, 11425)
['0089mi last four digit select receiv 350 award number match', '01223585334 cum wan 2c pic gettin shag text pix 8552', '02 user today lucki day find log onto httpwwwurawinnercom fantast', '0207 153 9153 offer end 26th juli tc appli optout', '0207 153 9996 offer end 14thmarch tc appli optout availa', '06 good luck remov send stop 87239 custom servic 08708034412', '07732584351 rodger burn msg tri call repli sm free nokia', '07734396839 ibh custom loyalti offer new nokia6600 mobil 10 txtauctiontxt', '07742676969 show 786 unredeem bonu point claim call 08719180248 identifi', '07753741225 show 800 unredeem point call 08715203677 identifi code 42478', '0776xxxxxxx uve invit xchat final attempt contact txt chat 86688', '077xxx 2000 bonu caller prize 020603 2nd attempt reach call', '07801543489 guarante latest nokia phone 40gb ipod mp3 player 500', '07808 xxxxxx show 800 unredeem point call 08719899217 identifi code', '07808247860 show 800 unredeem point call 08719899229 identif[...]<jupyter_text>Apply CountVectorizer (w/ N-Grams) to smaller sample<jupyter_code>data_sample = data[0:20]

ngram_vect_sample = CountVectorizer(ngram_range=(2,2))
X_counts_sample = ngram_vect_sample.fit_transform(data_sample['cleaned_text'])
print(X_counts_sample.shape)
print(ngram_vect_sample.get_feature_names())
X_counts_df = pd.DataFrame(X_counts_sample.toarray())
X_counts_df.columns = ngram_vect_sample.get_feature_names()
X_counts_df<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11866464339908953, 'fraction_numerical': 0.07890743550834597, 'mean_word_length': 4.712305025996534, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '610948', 'license': 'MIT', 'n_tokens_mistral': 1247, 'n_tokens_neox': 1059, 'n_words': 325, 'path': 'nlp/Ex_Files_NLP_Python_ML_EssT/Exercise Files/Ch03/03_03/End/03_03.ipynb', 'repo_name': 'joejoeyjoseph/playground', 'size': '3295'}