<jupyter_start><jupyter_text>Class solution for NLP EC question- It's good to see and learn different ways to solve the same problem- It's also a good review for midterm 2: Without running the code, can you explain what each block of code does? To facilitate this, I've added midterm review questions to each solution cell.<jupyter_code># imports and general set up

import pandas as pd
import nltk
from nltk.tokenize import RegexpTokenizer
from nltk.tokenize import word_tokenize
from nltk.stem import PorterStemmer
from nltk.corpus import stopwords

stop_words=set(stopwords.words("english"))
stemmer = PorterStemmer()
# read data and create X DataFrame with 1 column labeled 'text'

inputData = pd.read_csv('spam_short.csv', encoding='ISO-8859-1')   # Unicode characters
df = inputData[['v1','v2']]
df.columns = ('label','text')
X = df.drop(columns=['label'])
X<jupyter_output><empty_output><jupyter_text>To run a cell below, which is a correct solution that someone submitted:- select the cell and change the pull down menu of cell type from 'Raw NBConvert' to 'Code'- at the Cell pull down menu, select Run All Above so it re-runs the cells above to re-initialize the X DataFrame- when you're done with a cell, change it back to 'Raw NBConvert' so it doesn't run again when you're testing a different cell. These cells should not be run together.<jupyter_code>tokenizer = RegexpTokenizer('\w+')
only_words = [tokenizer.tokenize(text) for text in X.text] #remove punt
print(only_words)
only_words = [[word.lower() for word in low] for low in only_words] #lower
print(only_words)
only_words = [[word for word in l if word not in stopwords.words('english')] for l in only_words]
print(only_words)

ps = PorterStemmer()
only_words=[[ps.stem(w) for w in l] for l in only_words]
listSentence =[' '.join(ele) for ele in only_words]
X_processed = X
X_processed['text'] = listSentence
X_processed

# midterm review question: 
# after the code runs, does X still have the original strings?
tokenizer = RegexpTokenizer('[A-Za-z]+')   # \w [0-9A-Za-z_]    + (1 or more)
X.text = [tokenizer.tokenize(aStr) for aStr in X.text]
# change all words to lowercase
X.text = [[word.lower() for word in aList] for aList in X.text]
# remove all stop words
X.text = [[word for word in aList if word not in stop_words] for aList in X.text]
# stem all words
X.text = [" ".join([stemmer.stem(word) for word in aList]) for aList in X.text]
# end up with an X DataFrame, name it X_processed
X_processed = pd.DataFrame({0: X.text})
print(X_processed.head())
print(X_processed.shape)

# midterm review questions:
# - after the code runs, does X still have the original strings?
# - how are the strings in X_processed for this block of code different from the strings
# in X_processed of the previous block of code?
texts = X.text.str.lower()
print(texts)
texts = [tokenizer.tokenize(text) for text in texts]
print(texts)
texts = [' '.join([stemmer.stem(word) for word in text if word not in stop_words]) for text in texts]
X_processed = pd.DataFrame(texts)
X_processed.head()

# midterm review questions:
# - what data type is texts in the first line?
# - what data type is texts in the second line?
# - after the code runs, does X still have the orignal strings?
# - are the strings in X_processed the same as the strings in X_processed of the first cell
# or the second cell?
tokenizer = RegexpTokenizer('[A-Za-z]+')
stop_words = set(stopwords.words("english"))
stemmer = PorterStemmer()

def clean_text(row: pd.Series) -> str:
    """Function applied to text column in each row.
    
    Args:
        row(pandas.Series):   input rows as a series.
    
    Returns:
        str:    output string of tokens after cleaning.
    """
    
    # Get the contents of the text field.
    str_ = row.text
    
    # Remove all numbers and punctuations.
    list_ = tokenizer.tokenize(str_)

    # Change all words to lowercase.
    list_ = [w.lower() for w in list_]
    
    # Remove all stop words.
    list_ = [w for w in list_ if w not in stop_words]
    
    # Stem all words.
    list_ = [stemmer.stem(w) for w in list_]

    # How should results be packaged? 
    # - As one string of tokens per row.
    return " ".join(list_)


# Note: we need X to be a dataframe not a series!
X_processed = X.apply(clean_text, axis=1).to_frame()
print(X_processed.shape)
print(X_processed.head())
print(X.head())

# midterm review questions:
# - what is 'text' in the line: str_ = row.text ?
# - after the code runs, does X still have the original strings?
X_processed = []

tokenizer = RegexpTokenizer('\w+')
stemmer = PorterStemmer()
for e in range(len(X)):
    t = word_tokenize(X.loc[e,'text'])
    t_words = tokenizer.tokenize(str(t))
    noStop = [w.lower() for w in t_words if w not in stop_words]
    stemed = [stemmer.stem(w) for w in noStop]
    X_processed.append(' '.join(stemed))

X_processed = pd.Series(X_processed).to_frame()
X_processed

# midterm review questions:
# - one step of the for loop is not needed, which is it?<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10174880763116058, 'fraction_numerical': 0.0027821939586645467, 'mean_word_length': 4.058291457286432, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 10, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '464495', 'license': 'Apache-2.0', 'n_tokens_mistral': 1681, 'n_tokens_neox': 1619, 'n_words': 637, 'path': 'CIS009/4_Natural_Language_Processing_with_nltk/class_answers_for_EC.ipynb', 'repo_name': 'jackh423/python', 'size': '5032'}