<jupyter_start><jupyter_text>Introduction to Dataset Processing by <NAME> and <NAME> This Jupyter Notebook will share more details about how to process your data. Data processing is like preparing the ingredients before cooking; if you prepare them poorly (e.g., leave things half-peeled and dirty) , the meal will taste poor no matter how skillful a chef you are. It's similarly true in machine learning. Dataset processing can be one of the most important things you can do to get your model to perform well.You can read more about dataset processing on the [course notes here](https://jennselby.github.io/MachineLearningCourseNotes/data-processing).If you haven't already, follow [the setup instructions here](https://jennselby.github.io/MachineLearningCourseNotes/setting-up-python3) to get all necessary software installed. Table of Contents  * [Setup](Setup)  * [Section 1: Converting Categorical Values to Numerical Ones](Section-1:-Converting-Categorical-Values-to-Numerical-Ones)  * [Section 2: Dealing with Text Data](Section-2:-Dealing-with-Text-Data)  * [Section 3: Dealing with Null Values](Section-3:-Dealing-with-Null-Values)  * [Section 4: Now Let's Learn How to Standardize Data](Section-4:-Now-Let's-Learn-How-to-Standardize-Data)  * [Bonus: What if we would like to split up columns?](Bonus:-What-if-we-would-like-to-split-up-columns?)  Hint: Use the `?` symbolAs you go through this notebook, as well as learn more about processing data in iPython, it will be helpful to know the `?` symbol.E.g., You can try to type the following into Python```pythonimport sklearnsklearn?```Typing the `?` symbol after a function, module or variable will bring up the documentation of that bit of code, assuming it exists. It'll tell you more about the variable, function or module.  Setup<jupyter_code>import pandas as pd
from sklearn import preprocessing<jupyter_output><empty_output><jupyter_text>Download the [student performance data](http://archive.ics.uci.edu/ml/machine-learning-databases/00320/) and change the path below to wherever you put the data.<jupyter_code>student_data = pd.read_csv('../data/student/student-mat.csv', sep=';')
student_data.head()<jupyter_output><empty_output><jupyter_text>Section 1: Converting Categorical Values to Numerical OnesLooking at the data above, we want to convert a number of the columns from categorical to numerical. Most machine learning models deal with numbers and don't know how to model data that is in text form. As a result we need to learn how to do things such as e.g., convert the values in the `school` column to numbers.  First, let's see what values there are in the `school` column<jupyter_code># This shows a list of unique values and how many times they appear
student_data['school'].value_counts()
# Converting values in the school column to text
# We are going to define a function that takes a single value and apply it to all the values
def convert_school(row):
    if row == 'GP':
        return 0
    elif row == 'MS':
        return 1
    else:
        return None<jupyter_output><empty_output><jupyter_text>Avoid for loopsNormally, we might write a for loop like the one below. But this is really slow when using Pandas. _Don't write loops like this_.<jupyter_code># Here's a slow way of using the above function
%time
converted_school = []

for row in student_data['school']:
    new_value = convert_school(row)
    converted_school.append(new_value)
converted_school


# Don't do this! It's very slow.<jupyter_output><empty_output><jupyter_text>Instead, Use `.apply`This will do the same thing as the for loop above, but _much_ faster. It'll apply a function to all the rows of a `DataFrame`.<jupyter_code>%time
converted_school = student_data['school'].apply(convert_school)
converted_school<jupyter_output>CPU times: user 3 µs, sys: 0 ns, total: 3 µs
Wall time: 5.96 µs<jupyter_text>Look how much faster that took!  Or, you can use `.map()` You can also use the `.map()` function to map certain values to other data.For example, imagine you had a column named `'colors'` that contained the values `"red"` and `"blue"` and you wanted to convert these to the numbers `1` and `2`.```pythonmappings = {    'red': 1,    'blue': 2}data['colors_mapped'] = data['colors'].map(mappings)```The above will create a new column called `colors_mapped` that now has the values `1` and `2`.  Using sklearn's built-in preprocessing module, we can do the same thing<jupyter_code>enc_school = preprocessing.LabelEncoder()
transformed_school = enc_school.fit_transform(student_data['school'])
transformed_school<jupyter_output><empty_output><jupyter_text>We can also use one-hot encoding if we have more than two values. We still need to encode it first, as we did above.See example at https://stackoverflow.com/a/43589167/2159992<jupyter_code>enc_mjob = preprocessing.LabelEncoder()
encoded_mjob = enc_mjob.fit_transform(student_data['Mjob'])
encoded_mjob
onehot_mjob = preprocessing.OneHotEncoder(sparse=False)
transformed_mjob = onehot_mjob.fit_transform(encoded_mjob.reshape(-1,1))
transformed_mjob<jupyter_output><empty_output><jupyter_text>Once we've fitted the label encoder and one-hot encoder, we can use them to transform more values.<jupyter_code>onehot_mjob.transform(enc_mjob.transform(['other', 'health']).reshape(-1,1))<jupyter_output><empty_output><jupyter_text>What if we want to apply a transform that looks at multiple values in the row?For instance, what if we want to create a new column with a 1 if both parents have the highest level of education measured?<jupyter_code>medu_index = student_data.columns.get_loc('Medu')
fedu_index = student_data.columns.get_loc('Fedu')

def both_parents_edu(row):
    if row[medu_index] > 3 and row[fedu_index] >= 4:
        return 1
    else:
        return 0
    
# axis 1 means that we will apply the function to each row
student_data['parents_high_edu'] = student_data.apply(both_parents_edu, axis=1)
student_data.head(10)<jupyter_output><empty_output><jupyter_text>Section 2: Dealing with Text Data `pandas` has a lot of built-in modules that work with text-based data. `sklearn` similarly has a lot of modules for this as well.This section gives a brief outline of the things you can try.If you want to see a fuller list, with examples, of how `pandas` deals with text data, you can look at [the documentation here](https://pandas.pydata.org/pandas-docs/stable/text.html).<jupyter_code>#### First, I'm going to make some fake data that we can work with for the rest of this section

data = pd.DataFrame(data={'text': ['apple', '%badly,formatted,data%', 'pear']})
data<jupyter_output><empty_output><jupyter_text>Removing or replacing dataOkay, we want to remove the `','` and `'%'` symbols from the data. How do we do so?<jupyter_code>data['text_removed'] = data['text'].str.replace(',', '')
data<jupyter_output><empty_output><jupyter_text>Nice. Now try and replace the `'%'` symbols.<jupyter_code>#### Your code here<jupyter_output><empty_output><jupyter_text>Checking to see if a string contains a certain valueNow, we want to see if a text contains certain values, and only get the rows that contains those values.<jupyter_code>### Again, I have to make some fake data

data = pd.DataFrame(data={'text': ['<NAME>', 'San Francisco Maverick', 'Vikings']})
data<jupyter_output><empty_output><jupyter_text>Cool, what if we only wanted to get the rows that contained the word `'Maverick'`?<jupyter_code>data['text'].str.contains('Maverick')<jupyter_output><empty_output><jupyter_text>Now we can use this `Series` of `boolean` `True` and `False` values to index into our data!<jupyter_code>condition = data['text'].str.contains('Maverick')

filtered_data = data[condition]
filtered_data<jupyter_output><empty_output><jupyter_text>There is a whole list of other things you can do with your text data. **Some are listed below:*** `str.startswith()` and `str.endswith()` - checks to see if a string starts or ends with a given argument* `str.count()` - counts the number of appearances of a certain pattern* `str.numeric()` - checks to see if the string is numeric (e.g., `23123` is a digit whereas `213123abc` is not)* `str.split()` - splits the string on some deliminter and returns a dataframe of the string, split on the characters.There's plenty more and you can see the [documentation here for more](https://pandas.pydata.org/pandas-docs/stable/text.html).  What about transforming text data into a DataFrame?<jupyter_code>raw_text = ["""This is a giant series of sentences that you want to convert into a DataFrame containing 
the raw counts for each word. There are some abbr. and some punctuations here and there that make things more complicated.
So how in the world do we turn this into something that we can build a machine learning model off of?
"""]<jupyter_output><empty_output><jupyter_text>Okay, so we want to turn the above into a DataFrame where every column is a different word, and each entry stores the number of times that word came up.We're going to use the `CountVectorizer` class in `sklearn`.A more [in-depth tutorial on how to use it, and more, can be found here](https://machinelearningmastery.com/prepare-text-data-machine-learning-scikit-learn/).<jupyter_code>from sklearn.feature_extraction.text import CountVectorizer
# Initializing an empty CountVectorizer object
count_vect = CountVectorizer()
# Now we fit the object to our actual data
counts = count_vect.fit_transform(raw_text)
# This is a `sparse matrix` class. It saves our computer space.
counts
# Let's use the `.todense()` function to turn this sparse matrix into something that can be transformed into a DataFrame
word_counts_df = pd.DataFrame(data=counts.todense())
word_counts_df<jupyter_output><empty_output><jupyter_text>Great, but what do each of the columns mean?We can inspect the `count_vect.vocabulary_` attribute to find out.<jupyter_code>count_vect.vocabulary_<jupyter_output><empty_output><jupyter_text>Great. Now we know the words for each of the datasets**Your challenge**: Your challenge is to write some code so that you end converting the columns in `word_counts_df` to each of the words in `count_vect.vocabulary_`.<jupyter_code># If you're successful it should look like the output below.
### Your code here<jupyter_output><empty_output><jupyter_text>Section 3: Dealing with Null ValuesTo show you how to deal with null values, I'm going to make some simulated data of students.<jupyter_code>import numpy as np
grades = np.random.choice(range(1, 13), 100) # chooses 100 random numbers between 1 - 12
num_friends_or_none = list(range(0, 20)) + [None] * 5
num_friends = np.random.choice(num_friends_or_none, 100)
new_data = pd.DataFrame(data={'Grade': grades, '# Friends': num_friends})
new_data.head(n=20)<jupyter_output><empty_output><jupyter_text>One way to deal with null values is to drop them<jupyter_code>new_data['# Friends'].dropna()<jupyter_output><empty_output><jupyter_text>We can also drop any rows with nulls from the entire table.<jupyter_code>new_data.dropna()<jupyter_output><empty_output><jupyter_text>Or we can replace the null values with an average<jupyter_code>average_friends = new_data['# Friends'].mean()
new_data['# Friends'].fillna(average_friends)
new_data['# Friends'] = new_data['# Friends'].fillna(average_friends)<jupyter_output><empty_output><jupyter_text>What if instead of null values, there is something else that stands for missing values?Try the replace function.<jupyter_code>grades = np.random.choice(range(1, 13), 100) # chooses 100 random numbers between 1 - 12
num_friends_or_none = list(range(0, 20)) + ["Unknown"] * 5
num_friends = np.random.choice(num_friends_or_none, 100)
unknown_data = pd.DataFrame(data={'Grade': grades, '# Friends': num_friends})
unknown_data
unknown_data.replace("Unknown", 10)<jupyter_output><empty_output><jupyter_text>Section 4: Now Let's Learn How to Standardize DataBy that I mean to transform our data so that it has a mean of 0 and a standard deviation of 1.Why would we want to do this?Well often we will have strange parameter estimates on many models models if different bits of our data are in wildly different ranges.> Many researchers have noted the importance of standardizing variables for multivariate analysis. >> Otherwise, variables measured at different scales do not contribute equally to the analysis. >For example, in boundary detection, a variable that ranges between 0 and 100 will outweigh a variable that ranges between 0 and 1. Using these variables without standardization in effect gives the variable with the larger range a weight of 100 in the analysis. >Transforming the data to comparable scales can prevent this problem. Typical data standardization procedures equalize the range and/or data variability.[Source](https://www.biomedware.com/files/documentation/Preparing_data/Why_standardize_variables.htm)<jupyter_code>from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
scaler.fit_transform(new_data)<jupyter_output><empty_output><jupyter_text>The above will transform the data so that all the columns have an average of 0 and a standard deviation of 1.You can read the full documentatio for the `StandardScaler` [here](http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html).  Bonus: What if we would like to split up columns?Maybe  you have data in a column that's a mashup between multiple values.For example, imagine if you have a column that stores values like:`'8th Grade - 13 years old'` and `'12th grade - 17 years old'` and you want to create two columns: `grade` and `age` to store the two separate bits of data.How do you do so?<jupyter_code># I'm going to generate some fake data here. Ignore the below>
grades = np.random.choice(range(1, 13), 100) # chooses 100 random numbers between 1 - 12
grades_and_ages = ['Grade {grade}  - {age} years old'.format(grade=grade, age=grade+6) for grade in grades]
num_friends_or_none = list(range(0, 20)) + ["Unknown"] * 5
num_friends = np.random.choice(num_friends_or_none, 100)
combined_data = pd.DataFrame(data={'Grade and Age': grades_and_ages, '# Friends': num_friends})
combined_data<jupyter_output><empty_output><jupyter_text>Awesome, now let's split things up. We'll use the built in `.str.split()` function with the extra input `expand=True`.The `expand=True` will convert the splitted data into a `DataFrame` instead of keeping a list of values. (Try taking out `expand=True` and seeing what happens.)<jupyter_code>combined_data['Grade and Age'].str.split(' - ', expand=True)<jupyter_output><empty_output><jupyter_text>**Your challenge**: Write some code that does the following:1. Removes the words 'Grade' and 'years old' from the data2. Takes these two columns and puts them back into the original `combined_data` DataFrame.<jupyter_code>### Your code here<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0928446301037596, 'fraction_numerical': 0.01044333647756367, 'mean_word_length': 5.7498863119599815, 'pattern_counts': {'":': 0, '<': 115, '<?xml version=': 0, '>': 123, 'https://': 7, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '28', 'dir': 'jupyter-structured-clean-dedup', 'id': '93469', 'license': 'MIT', 'n_tokens_mistral': 4594, 'n_tokens_neox': 4346, 'n_words': 1704, 'path': 'Examples/.ipynb_checkpoints/IntroductiontoDatasetProcessing-checkpoint.ipynb', 'repo_name': 'FlyN-Nick/introToML', 'size': '14856'}