<jupyter_start><jupyter_text>Feature Engineering The next step is to create features from the raw text so we can train the machine learning models. The steps followed are:* **Text Cleaning and Preparation**: cleaning of special characters, downcasing, punctuation signs. possessive pronouns and stop words removal and lemmatization.* **Label coding**: creation of a dictionary to map each category to a code.* **Train-test split**: to test the models on unseen data.* **Text representation**: use of TF-IDF scores to represent text.<jupyter_code>import pickle
import pandas as pd
import re
import nltk
from nltk.corpus import stopwords
from nltk.stem import WordNetLemmatizer
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.model_selection import train_test_split
from sklearn.feature_selection import chi2
import numpy as np<jupyter_output><empty_output><jupyter_text>**First of all we'll load the dataset:**<jupyter_code>path_df = "/home/andrewpap22/Desktop/DataScience-testDir/data.pickle"

with open(path_df, 'rb') as data:
    df = pickle.load(data)
df.head()<jupyter_output><empty_output><jupyter_text>**Visualize one sample news content:**<jupyter_code>df.loc[2]['Content']<jupyter_output><empty_output><jupyter_text>1. Text cleaning and preperation  1.1. Special character cleaning We can see the following special characters:* \r* \n* \ before possessive pronouns (government's = government\'s)* \ before possessive pronouns 2 (Yukos' = Yukos\')* " when quoting text<jupyter_code># \r and \n
df['Content_Parsed_1'] = df['Content'].str.replace("\r", " ")
df['Content_Parsed_1'] = df['Content_Parsed_1'].str.replace("\n", " ")
df['Content_Parsed_1'] = df['Content_Parsed_1'].str.replace("    ", " ")
# Regarding 3rd and 4th bullet, although it seems there is a special character, it won't affect us since it is not a real character:

text = "<NAME>"
text
# " when quoting text
df['Content_Parsed_1'] = df['Content_Parsed_1'].str.replace('"', '')<jupyter_output><empty_output><jupyter_text>1.2. Upcase/DowncaseWe'll downcase the texts because we want, for example, Football and football to be the same word.<jupyter_code># Lowercasing the text
df['Content_Parsed_2'] = df['Content_Parsed_1'].str.lower()<jupyter_output><empty_output><jupyter_text>1.3. Punctuation signsPunctuation signs won't have any predicting power, so we'll just get rid of them.<jupyter_code>punctuation_signs = list("?:!.,;")
df['Content_Parsed_3'] = df['Content_Parsed_2']

for punct_sign in punctuation_signs:
    df['Content_Parsed_3'] = df['Content_Parsed_3'].str.replace(punct_sign, '')<jupyter_output><empty_output><jupyter_text>By doing this we are messing up with some numbers, but it's no problem since we aren't expecting any predicting power from them.  1.4. Possessive pronounsWe'll also remove possessive pronoun terminations:<jupyter_code>df['Content_Parsed_4'] = df['Content_Parsed_3'].str.replace("'s", "")<jupyter_output><empty_output><jupyter_text>1.5. Stemming and Lemmatization *Since stemming can produce output words that don't exist, we'll only use a lemmatization process at this moment. Lemmatization takes into consideration the morphological analysis of the words and returns words that do exist, so it will be more useful for us.*<jupyter_code># Downloading punkt and wordnet from NLTK
nltk.download('punkt')
print("------------------------------------------------------------")
nltk.download('wordnet')
# Saving the lemmatizer into an object
wordnet_lemmatizer = WordNetLemmatizer()
# IN order to lemmatize, we have to iterate through every word:

nrows = len(df)
lemmatized_text_list = []

for row in range(0, nrows):
    
    # Create an empty list containing lemmatized words
    lemmatized_list = []
    
    # Save the text and its words into an object
    text = df.loc[row]['Content_Parsed_4']
    text_words = text.split(" ")

    # Iterate through every word to lemmatize
    for word in text_words:
        lemmatized_list.append(wordnet_lemmatizer.lemmatize(word, pos="v"))
        
    # Join the list
    lemmatized_text = " ".join(lemmatized_list)
    
    # Append to the list containing the texts
    lemmatized_text_list.append(lemmatized_text)
df['Content_Parsed_5'] = lemmatized_text_list<jupyter_output><empty_output><jupyter_text>Although lemmatization doesn't work perfectly in all cases (as can be seen in the example below), it can be useful.  1.6 Stop words<jupyter_code># Downloading the stop words list
nltk.download('stopwords')
# Loading the stop words in english
stop_words = list(stopwords.words('english'))
stop_words[0:10]
#To remove the stop words, we'll handle a regular expression only detecting whole words, as seen in the following example:

example = "me eating a meal"
word = "me"

# The regular expression is:
regex = r"\b" + word + r"\b"  # we need to build it like that to work properly

re.sub(regex, "StopWord", example)
# We can now loop through all the stop words:

df['Content_Parsed_6'] = df['Content_Parsed_5']

for stop_word in stop_words:

    regex_stopword = r"\b" + stop_word + r"\b"
    df['Content_Parsed_6'] = df['Content_Parsed_6'].str.replace(regex_stopword, '')<jupyter_output><empty_output><jupyter_text>*We have some dobule/triple spaces between words because of the replacements. However, it's not a problem because we'll tokenize by the spaces later.**As an example, we'll show an original news article and its modifications throughout the process:*<jupyter_code>df.loc[22]['Content']<jupyter_output><empty_output><jupyter_text>**1. Special character cleaning**<jupyter_code>df.loc[22]['Content_Parsed_1']<jupyter_output><empty_output><jupyter_text>**1. Uppercase/Downcase :**<jupyter_code>df.loc[22]['Content_Parsed_2']<jupyter_output><empty_output><jupyter_text>**1. Punctuation signs:**<jupyter_code>df.loc[22]['Content_Parsed_3']<jupyter_output><empty_output><jupyter_text>**1. Possessive pronouns:**<jupyter_code>df.loc[22]['Content_Parsed_4']<jupyter_output><empty_output><jupyter_text>**1. Stemming and Lemmatization:**<jupyter_code>df.loc[22]['Content_Parsed_5']<jupyter_output><empty_output><jupyter_text>**1. Stop Words:**<jupyter_code>df.loc[22]['Content_Parsed_6'] 
# Now we can delete the intermediate coliumns: 

df.head(1)
list_columns = ["File_Name", "Category", "Complete_Filename", "Content", "Content_Parsed_6"]
df = df[list_columns]

df = df.rename(columns={'Content_Parsed_6': 'Content_Parsed'})
df.head()<jupyter_output><empty_output><jupyter_text>2. Label coding*We'll create a dictionary with the label codification:*<jupyter_code>category_codes = {
    'business': 0,
    'entertainment': 1,
    'politics': 2,
    'sport': 3,
    'tech': 4
}

# Category mapping
df['Category_Code'] = df['Category']
df = df.replace({'Category_Code':category_codes})

df.head()<jupyter_output><empty_output><jupyter_text>3. Train - Test split**We'll set apart a test set to prove the quality of our models. We'll do Cross Validation in the train set in order to tune the hyperparameters and then test performance on the unseen data of the test set.**<jupyter_code>X_train, X_test, y_train, y_test = train_test_split(df['Content_Parsed'], 
                                                    df['Category_Code'], 
                                                    test_size=0.20, 
                                                    random_state=8)<jupyter_output><empty_output><jupyter_text>Since we don't have much observations (only 2.225), we'll choose a test set size of 20% of the full dataset as described in the project definition document. Text representation : We have various options:* Count Vectors as features* TF-IDF Vectors as features* Word Embeddings as features* Text / NLP based features* Topic Models as featuresWe'll use **TF-IDF Vectors** as features, since it's requested from the project, classification part to be done in that particular text representationWe have to define the different parameters:* ngram_range: We want to consider both unigrams and bigrams.* max_df: When building the vocabulary ignore terms that have a document frequency strictly higher than the given threshold* min_df: When building the vocabulary ignore terms that have a document frequency strictly lower than the given threshold.* max_features: If not None, build a vocabulary that only consider the top max_features ordered by term frequency across the corpus.See TfidfVectorizer? for further detail.It needs to be mentioned that we are implicitly scaling our data when representing it as TF-IDF features with the argument norm.<jupyter_code># Parameter election
ngram_range = (1,2)
min_df = 10
max_df = 1.
max_features = 300<jupyter_output><empty_output><jupyter_text>We have chosen these values as a first approximation. Since the models that we develop later have a very good predictive power, we'll stick to these values. But it has to be mentioned that different combinations could be tried in order to improve even more the accuracy of the models.<jupyter_code>tfidf = TfidfVectorizer(encoding='utf-8',
                        ngram_range=ngram_range,
                        stop_words=None,
                        lowercase=False,
                        max_df=max_df,
                        min_df=min_df,
                        max_features=max_features,
                        norm='l2',
                        sublinear_tf=True)
                        
features_train = tfidf.fit_transform(X_train).toarray()
labels_train = y_train
print(features_train.shape)

features_test = tfidf.transform(X_test).toarray()
labels_test = y_test
print(features_test.shape)<jupyter_output>(1780, 300)
(445, 300)<jupyter_text>Please note that we have fitted and then transformed the training set, but we have **only transformed the test set**.We can use the Chi squared test in order to see what unigrams and bigrams are most correlated with each category:<jupyter_code>from sklearn.feature_selection import chi2
import numpy as np

for Product, category_id in sorted(category_codes.items()):
    features_chi2 = chi2(features_train, labels_train == category_id)
    indices = np.argsort(features_chi2[0])
    feature_names = np.array(tfidf.get_feature_names())[indices]
    unigrams = [v for v in feature_names if len(v.split(' ')) == 1]
    bigrams = [v for v in feature_names if len(v.split(' ')) == 2]
    print("# '{}' category:".format(Product))
    print("  . Most correlated unigrams:\n. {}".format('\n. '.join(unigrams[-5:])))
    print("  . Most correlated bigrams:\n. {}".format('\n. '.join(bigrams[-2:])))
    print("")<jupyter_output># 'business' category:
  . Most correlated unigrams:
. price
. market
. economy
. growth
. bank
  . Most correlated bigrams:
. last year
. year old

# 'entertainment' category:
  . Most correlated unigrams:
. music
. best
. star
. award
. film
  . Most correlated bigrams:
. mr blair
. prime minister

# 'politics' category:
  . Most correlated unigrams:
. minister
. blair
. election
. party
. labour
  . Most correlated bigrams:
. prime minister
. mr blair

# 'sport' category:
  . Most correlated unigrams:
. win
. side
. game
. team
. match
  . Most correlated bigrams:
. say mr
. year old

# 'tech' category:
  . Most correlated unigrams:
. digital
. software
. computer
. technology
. users
  . Most correlated bigrams:
. year old
. say mr<jupyter_text>As we can see, the unigrams correspond well to their category. However, bigrams do not. If we get the bigrams in our features:<jupyter_code>bigrams<jupyter_output><empty_output><jupyter_text>We can see there are only six. This means the unigrams have more correlation with the category than the bigrams, and since we're restricting the number of features to the most representative 300, only a few bigrams are being considered.We'll save our files here and move on to the next notebook, the Classification one!<jupyter_code># X_train
with open('Pickles/X_train.pickle', 'wb') as output:
    pickle.dump(X_train, output)
    
# X_test    
with open('Pickles/X_test.pickle', 'wb') as output:
    pickle.dump(X_test, output)
    
# y_train
with open('Pickles/y_train.pickle', 'wb') as output:
    pickle.dump(y_train, output)
    
# y_test
with open('Pickles/y_test.pickle', 'wb') as output:
    pickle.dump(y_test, output)
    
# df
with open('Pickles/df.pickle', 'wb') as output:
    pickle.dump(df, output)
    
# features_train
with open('Pickles/features_train.pickle', 'wb') as output:
    pickle.dump(features_train, output)

# labels_train
with open('Pickles/labels_train.pickle', 'wb') as output:
    pickle.dump(labels_train, output)

# features_test
with open('Pickles/features_test.pickle', 'wb') as output:
    pickle.dump(features_test, output)

# labels_test
with open('Pickles/labels_test.pickle', 'wb') as output:
    pickle.dump(labels_test, output)
    
# TF-IDF object
with open('Pickles/tfidf.pickle', 'wb') as output:
    pickle.dump(tfidf, output)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10853913447394906, 'fraction_numerical': 0.009754586978400558, 'mean_word_length': 4.494683113568694, 'pattern_counts': {'":': 0, '<': 92, '<?xml version=': 0, '>': 92, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 11, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '316176', 'license': 'MIT', 'n_tokens_mistral': 4186, 'n_tokens_neox': 3998, 'n_words': 1242, 'path': '02-FeatureEngineering.ipynb', 'repo_name': 'andrewpap22/BBC_NewsClassifier-Project', 'size': '12926'}