<jupyter_start><jupyter_text>NLP Prepare ExercisesThe end result of this exercise should be a file named ```prepare.py``` that defines the requested functions.In this exercise we will be defining some functions to prepare textual data. These functions should apply equally well to both the codeup blog articles and the news articles that were previously acquired.**1. Define a function named ```basic_clean```. It should take in a string and apply some basic text cleaning to it:**- Lowercase everything- Normalize unicode characters- Replace anything that is not a letter, number, whitespace or a single quote.<jupyter_code>#imports
import unicodedata
import re
import json

import acquire
import prepare

import nltk
from nltk.tokenize.toktok import ToktokTokenizer
from nltk.corpus import stopwords

import pandas as pd
def basic_clean(text):
    '''
    This function takes in a string and normalizes it by lowercasing
    everything and replacing anything that is not a letter, number, 
    whitespace or a single quote.
    '''
    
    #lowercase all letters in the text
    text = text.lower()
    
    # normalize unicode by encoding into ASCII (ignore non-ASCII characters)
    # then decoding back into unicode 
    text = unicodedata.normalize('NFKD', text)\
    .encode('ascii', 'ignore')\
    .decode('utf-8', 'ignore')

    # remove any that is not a letter, number, single quote, or whitespace
    text = re.sub(r"[^a-z0-9'\s]", '', text)
    
    return text
text = "HERE is a sTring with lőt^^^S óf s\\##tranGe things+ go%$ing ón. I'm attempting to 'normalize' some text."
text


text = basic_clean(text)
text<jupyter_output><empty_output><jupyter_text>**2.  Define a function named ```tokenize```. It should take in a string and tokenize all the words in the string.**<jupyter_code>def tokenize(text):
    '''
    This function takes in a string and returns the string will the
    words tokenized
    '''

    # Create the tokenizer
    tokenizer = nltk.tokenize.ToktokTokenizer()

    # Use the tokenizer
    text = tokenizer.tokenize(text, return_str=True)
    
    return text

text = tokenize(text)
text<jupyter_output><empty_output><jupyter_text>**3.  Define a function named ```stem```. It should accept some text and return the text after applying stemming to all the words.**<jupyter_code>def stem(text):
    '''
    This function takes in a string and returns the string after applying
    stemming to all the words.
    '''

    # Create the porter stemmer
    ps = nltk.porter.PorterStemmer()

    # Apply the stemmer to each word in our string.
    stems = [ps.stem(word) for word in text.split()]
    
    # Join our lists of words into a string again
    text_stemmed = ' '.join(stems)

    return text_stemmed

stem(text)<jupyter_output><empty_output><jupyter_text>**4. Define a function named ```lemmatize```. It should accept some text and return the text after applying lemmatization to each word.**<jupyter_code>def lemmatize(text):
    '''
    This function takes in a string and returns the string after applying
    lemmatization to all the words.
    '''

    # Create the Lemmatizer.
    wnl = nltk.stem.WordNetLemmatizer()

    # Use the lemmatizer on each word in the list of words we created by using split.
    lemmas = [wnl.lemmatize(word) for word in text.split()]

    # Join our list of words into a string again; assign to a variable to save changes.
    text_lemmatized = ' '.join(lemmas)
    
    return text_lemmatized
text = lemmatize(text)
text<jupyter_output><empty_output><jupyter_text>**5. Define a function named ```remove_stopwords```. It should accept some text and return the text after removing all the stopwords.**This function should define two optional parameters, extra_words and exclude_words. These parameters should define any additional stop words to include, and any words that we don't want to remove.<jupyter_code># import standard English language stopwords list from nltk
from nltk.corpus import stopwords

def remove_stopwords(text, extra_words=[], exclude_words=[]):
    '''
    This function takes in a string and optional lists of extra_words and 
    words to exclude from the list and then returns the string after removing stop_words
    '''

    # Define the stop word list
    stopword_list = stopwords.words('english')

    # add extra_words (if any) to the stopwords list
    if len(extra_words) > 0:
        stopword_list = stopword_list.append(extra_words)
      
    # remove exclude_words (if any) from the stopwords list
    if len(exclude_words) > 0:
        stopword_list = stopword_list.remove(exclude_words)   

    # Split words in text.
    text = text.split()
    
    # Create a list of words from my string with stopwords removed and assign to variable.
    filtered_words = [word for word in text if word not in stopword_list]
    
    # Join words in the list back into strings; assign to a variable to keep changes.
    text_without_stopwords = ' '.join(filtered_words)

    return text_without_stopwords


remove_stopwords(text)<jupyter_output><empty_output><jupyter_text>**6. Use your data from the acquire to produce a dataframe of the news articles. Name the dataframe ```news_df```.**<jupyter_code>articles = acquire.acquire_news_articles()
articles<jupyter_output>/Users/barbmarques/codeup-data-science/natural-language-processing-exercises/acquire.py:92: GuessedAtParserWarning: No parser was explicitly specified, so I'm using the best available HTML parser for this system ("lxml"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.

The code that caused this warning is on line 92 of the file /Users/barbmarques/codeup-data-science/natural-language-processing-exercises/acquire.py. To get rid of this warning, pass the additional argument 'features="lxml"' to the BeautifulSoup constructor.

  soup = BeautifulSoup(response.text)<jupyter_text>**7.  Make another dataframe for the Codeup blog posts. Name the dataframe ```codeup_df```.**<jupyter_code># acquire the dataframe of codeup blog articles
blog = acquire.acquire_codeup_blog()
blog<jupyter_output>/Users/barbmarques/codeup-data-science/natural-language-processing-exercises/acquire.py:16: GuessedAtParserWarning: No parser was explicitly specified, so I'm using the best available HTML parser for this system ("lxml"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.

The code that caused this warning is on line 16 of the file /Users/barbmarques/codeup-data-science/natural-language-processing-exercises/acquire.py. To get rid of this warning, pass the additional argument 'features="lxml"' to the BeautifulSoup constructor.

  soup = BeautifulSoup(response.text)<jupyter_text>**8.  For each dataframe, produce the following columns:**- ```title``` to hold the title- ```original``` to hold the original article/post content- ```clean``` to hold the normalized and tokenized original with the stopwords removed.- ```stemmed``` to hold the stemmed version of the cleaned data.- ```lemmatized``` to hold the lemmatized version of the cleaned data.<jupyter_code>def prep_article_data(df, column, extra_words=[], exclude_words=[]):
    '''
    This function take in a df and the string name for a text column with 
    option to pass lists for extra_words and exclude_words and
    returns a df with the text article title, original text, stemmed text,
    lemmatized text, cleaned, tokenized, & lemmatized text with stopwords removed.
    '''
    df['clean'] = df[column].apply(basic_clean)\
                            .apply(tokenize)\
                            .apply(remove_stopwords, 
                                   extra_words=extra_words, 
                                   exclude_words=exclude_words)\
                            .apply(lemmatize)
    
    df['stemmed'] = df[column].apply(basic_clean).apply(stem)
    
    df['lemmatized'] = df[column].apply(basic_clean).apply(lemmatize)
    
    return df[['title', column, 'stemmed', 'lemmatized', 'clean']]
prep_article_data(articles,'content')
prep_article_data(blog,'content')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08397683397683398, 'fraction_numerical': 0.002533783783783784, 'mean_word_length': 3.8787522071806944, 'pattern_counts': {'":': 0, '<': 31, '<?xml version=': 0, '>': 33, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 4}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '278367', 'license': 'MIT', 'n_tokens_mistral': 2398, 'n_tokens_neox': 2323, 'n_words': 911, 'path': 'nlp-prepare-exercises.ipynb', 'repo_name': 'barbmarques/natural-language-processing-exercises', 'size': '8288'}