<jupyter_start><jupyter_text>- 1. Load Data- 2. Get to know Data- 3. Clean Data    - 3.1 remove weird characters    - 3.2 stopwords    - 3.3 popularity based prefiltering    - 3.4 lemmatization / stemming- 4. Tokenize Data- 5. Save preprocessed data  ----------------------------------------------------------------------  1. Load DataRaw data is gathered from [newsapi](https://newsapi.org/) using [this script](https://gist.github.com/davidlenz/961b5cf501b8f89a8c10cbd45306001b) Newsapi:<jupyter_code>from IPython.display import IFrame    
display(IFrame("https://newsapi.org/", width=1200, height=350))<jupyter_output><empty_output><jupyter_text>We use [pandas](https://pandas.pydata.org/) to generally handle the data. Pandas:<jupyter_code>display(IFrame("https://pandas.pydata.org/", width=1200, height=350))
import pandas as pd
df = pd.read_csv('data/sample_5000.csv')<jupyter_output><empty_output><jupyter_text>2. Get to know Data<jupyter_code>print("Shape:", df.shape)
df.head()
pd.set_option("max.colwidth",200)
df.sample(2)<jupyter_output><empty_output><jupyter_text>Datatypes<jupyter_code>display(IFrame("https://www.tutorialsteacher.com/python/python-data-types", width=1200, height=150))
df.dtypes<jupyter_output><empty_output><jupyter_text>Data distributionFirst cast the date column, which is currently of object type, to datetime type.<jupyter_code>df["date"] = pd.to_datetime(df.date)
df.dtypes<jupyter_output><empty_output><jupyter_text>Now cast the datetime object to the end/start of a period, for example end of week.<jupyter_code>df["week"] = df.date.dt.to_period("W").dt.to_timestamp('W')
print(df.dtypes)
df.head(2)
df.groupby("week").title.count().plot(title="Number of articles per Week")<jupyter_output><empty_output><jupyter_text>Number of words per article<jupyter_code>df.text.str.split().apply(len).describe()
df.text.str.split().apply(len).plot(kind="hist", bins=3330, xlim=(0,5000))<jupyter_output><empty_output><jupyter_text>3. Clean Data<jupyter_code>import re # regular expression library

# get rid of everything except letters, space and .!?
# regex explanation: https://regex101.com/r/xrfp0C/1
# remove everything except lowercase a-z, spaces and .!?
clean = lambda x : re.sub('[^a-z .!?]', "", str(x).lower())

# clean text
%time df["clean_text"] = df.text.apply(clean) 
df.head()<jupyter_output>Wall time: 443 ms<jupyter_text>Lemmatization / Stemming- Stemming -> Heuristic (fast)- Lemmatization -> Based on dictionary (not so fast)<jupyter_code>%%bash

pip3 install -U spacy
python3 -m spacy download en
import spacy
# make sure your downloaded the english model with "python -m spacy download en"
nlp = spacy.load('en')

def lemmatize(text, nlp):
    """Takes a text and a spacy nlp object as input and returns lemmatized tokens for the text using the spacy model"""
    # pass to spacy 
    doc = nlp(text, disable=["ner", "pos"])
    
    # get lemma for each token (i.e. word)
    list_of_lemmatized_words = [token.lemma_ for token in doc]
    
    # join words to string
    lemmatized_text = " ".join(list_of_lemmatized_words)
    
    # return string
    return lemmatized_text
    

# apply the lemmatize function to the first 50 texts
%time df["clean_text"].iloc[:10].apply(lambda text : lemmatize(text, nlp))<jupyter_output>Wall time: 972 ms<jupyter_text>Popularity based prefilteringIdea: Very frequent and very infrequent words are either too common or too rare to provide insights.-> removing corpus-specific stop words  Word frequency for our corpus:<jupyter_code>import matplotlib.pyplot as plt
pd.Series([j for k in df["clean_text"].str.split().values for j in k]).value_counts().rank().plot(logy=True, logx=True)<jupyter_output><empty_output><jupyter_text>https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html<jupyter_code>from sklearn.feature_extraction.text import CountVectorizer
stopwords=['of', "the", "to", "a"]
cv = CountVectorizer(min_df=0.005, # min percentage of documents a token needs to appear in 
                     max_df=0.6, # max percentage of docs a token is allowed to appear in
                     stop_words=stopwords) 

%time cv.fit(df["clean_text"].values)
len(cv.vocabulary_)<jupyter_output>Wall time: 2.93 s<jupyter_text>Get the vocabulary from the cv object and filter all words not in the vocab<jupyter_code>vocab = frozenset(cv.vocabulary_) # for speed

# split text into list of tokens then
# filter words not in the vocab then
# join words
df["clean_text"] = df["clean_text"].str.split().apply(lambda wordlist : [word for word in wordlist if word in vocab]).apply(" ".join)

df.head()<jupyter_output><empty_output><jupyter_text>Save Data for next steps<jupyter_code>df.to_csv("data/sample_5000_clean.csv")<jupyter_output><empty_output><jupyter_text>Summary- 1. Loaded Data- 2. Got to know Data    - 2.1 Distributions    - 2.2 frequent words and stuff- 3. Clean Data    - 3.1 remove weird characters    - 3.2 stopwords    - 3.3 popularity based prefiltering    - 3.4 lemmatization / stemming- 4. Tokenize Data- 5. Save preprocessed data<jupyter_code>?cv<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1360265417642467, 'fraction_numerical': 0.024004683840749413, 'mean_word_length': 4.739081746920493, 'pattern_counts': {'":': 0, '<': 54, '<?xml version=': 0, '>': 57, 'https://': 8, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '328604', 'license': 'MIT', 'n_tokens_mistral': 1833, 'n_tokens_neox': 1731, 'n_words': 436, 'path': '01_data_prep.ipynb', 'repo_name': 'davidlenz/nlpseminar', 'size': '5124'}