<jupyter_start><jupyter_text>Welcome to day 1 of the 5-Day Data Challenge! Today, we're going to be looking at how to deal with missing values. To get started, click the blue "Fork Notebook" button in the upper, right hand corner. This will create a private copy of this notebook that you can edit and play with. Once you're finished with the exercises, you can choose to make your notebook public to share with others. :)> **Your turn!** As we work through this notebook, you'll see some notebook cells (a block of either code or text) that has "Your Turn!" written in it. These are exercises for you to do to help cement your understanding of the concepts we're talking about. Once you've written the code to answer a specific question, you can run the code by clicking inside the cell (box with code in it) with the code you want to run and then hit CTRL + ENTER (CMD + ENTER on a Mac). You can also click in a cell and then click on the right "play" arrow to the left of the code. If you want to run all the code in your notebook, you can use the double, "fast forward" arrows at the bottom of the notebook editor.Here's what we're going to do today:* [Take a first look at the data](Take-a-first-look-at-the-data)* [See how many missing data points we have](See-how-many-missing-data-points-we-have)* [Figure out why the data is missing](Figure-out-why-the-data-is-missing)* [Drop missing values](Drop-missing-values)* [Filling in missing values](Filling-in-missing-values)Let's get started!  Take a first look at the data________The first thing we'll need to do is load in the libraries and datasets we'll be using. For today, I'll be using a dataset of events that occured in American Football games for demonstration, and you'll be using a dataset of building permits issued in San Francisco.> **Important!** Make sure you run this cell yourself or the rest of your code won't work!<jupyter_code># modules we'll use
import pandas as pd
import numpy as np

# read in all our data
nfl_data = pd.read_csv("../input/nflplaybyplay2009to2016/NFL Play by Play 2009-2017 (v4).csv")
sf_permits = pd.read_csv("../input/building-permit-applications-data/Building_Permits.csv")

# set seed for reproducibility
np.random.seed(0)<jupyter_output><empty_output><jupyter_text>The first thing I do when I get a new dataset is take a look at some of it. This lets me see that it all read in correctly and get an idea of what's going on with the data. In this case, I'm looking to see if I see any missing values, which will be reprsented with `NaN` or `None`.<jupyter_code># look at a few rows of the nfl_data file. I can see a handful of missing data already!
nfl_data.sample(5)<jupyter_output><empty_output><jupyter_text>Yep, it looks like there's some missing values. What about in the sf_permits dataset?<jupyter_code># your turn! Look at a couple of rows from the sf_permits dataset. Do you notice any missing data?

# your code goes here :)

sf_permits.sample(5)<jupyter_output><empty_output><jupyter_text>See how many missing data points we have___Ok, now we know that we do have some missing values. Let's see how many we have in each column.<jupyter_code># get the number of missing data points per column
missing_values_count = nfl_data.isnull().sum()

# look at the # of missing points in the first ten columns
missing_values_count[0:10]<jupyter_output><empty_output><jupyter_text>That seems like a lot! It might be helpful to see what percentage of the values in our dataset were missing to give us a better sense of the scale of this problem:<jupyter_code># how many total missing values do we have?
total_cells = np.product(nfl_data.shape)
total_missing = missing_values_count.sum()

# percent of data that is missing
(total_missing/total_cells) * 100<jupyter_output><empty_output><jupyter_text>Wow, almost a quarter of the cells in this dataset are empty! In the next step, we're going to take a closer look at some of the columns with missing values and try to figure out what might be going on with them.<jupyter_code># your turn! Find out what percent of the sf_permits dataset is missing
# get the number of missing data points per column
missing_values_count = sf_permits.isnull().sum()

# look at the # of missing points in the first ten columns
missing_values_count[0:10]

# how many total missing values do we have?
total_cells = np.product(sf_permits.shape)
total_missing = missing_values_count.sum()

# percent of data that is missing
(total_missing/total_cells) * 100<jupyter_output><empty_output><jupyter_text>Figure out why the data is missing____ This is the point at which we get into the part of data science that I like to call "data intution", by which I mean "really looking at your data and trying to figure out why it is the way it is and how that will affect your analysis". It can be a frustrating part of data science, especially if you're newer to the field and don't have a lot of experience. For dealing with missing values, you'll need to use your intution to figure out why the value is missing. One of the most important question you can ask yourself to help figure this out is this:> **Is this value missing becuase it wasn't recorded or becuase it dosen't exist?**If a value is missing becuase it doens't exist (like the height of the oldest child of someone who doesn't have any children) then it doesn't make sense to try and guess what it might be. These values you probalby do want to keep as NaN. On the other hand, if a value is missing becuase it wasn't recorded, then you can try to guess what it might have been based on the other values in that column and row. (This is called "imputation" and we'll learn how to do it next! :)Let's work through an example. Looking at the number of missing values in the nfl_data dataframe, I notice that the column `TimesSec` has a lot of missing values in it:<jupyter_code># look at the # of missing points in the first ten columns
missing_values_count<jupyter_output><empty_output><jupyter_text>By looking at [the documentation](https://www.kaggle.com/maxhorowitz/nflplaybyplay2009to2016), I can see that this column has information on the number of seconds left in the game when the play was made. This means that these values are probably missing because they were not recorded, rather than because they don't exist. So, it would make sense for us to try and guess what they should be rather than just leaving them as NA's.On the other hand, there are other fields, like `PenalizedTeam` that also have lot of missing fields. In this case, though, the field is missing because if there was no penalty then it doesn't make sense to say *which* team was penalized. For this column, it would make more sense to either leave it empty or to add a third value like "neither" and use that to replace the NA's.> **Tip:** This is a great place to read over the dataset documentation if you haven't already! If you're working with a dataset that you've gotten from another person, you can also try reaching out to them to get more information.If you're doing very careful data analysis, this is the point at which you'd look at each column individually to figure out the best strategy for filling those missing values. For the rest of this notebook, we'll cover some "quick and dirty" techniques that can help you with missing values but will probably also end up removing some useful information or adding some noise to your data. Your turn!* Look at the columns `Street Number Suffix` and `Zipcode` from the `sf_permits` datasets. Both of these contain missing values. Which, if either, of these are missing because they don't exist? Which, if either, are missing because they weren't recorded?  Drop missing values___If you're in a hurry or don't have a reason to figure out why your values are missing, one option you have is to just remove any rows or columns that contain missing values. (Note: I don't generally recommend this approch for important projects! It's usually worth it to take the time to go through your data and really look at all the columns with missing values one-by-one to really get to know your dataset.)  If you're sure you want to drop rows with missing values, pandas does have a handy function, `dropna()` to help you do this. Let's try it out on our NFL dataset!<jupyter_code>sf_permits.dropna()
# remove all the rows that contain a missing value
nfl_data.dropna()<jupyter_output><empty_output><jupyter_text>Oh dear, it looks like that's removed all our data! 😱 This is because every row in our dataset had at least one missing value. We might have better luck removing all the *columns* that have at least one missing value instead.<jupyter_code># remove all columns with at least one missing value
columns_with_na_dropped = nfl_data.dropna(axis=1)
columns_with_na_dropped.head()
columns_with_na_dropped = sf_permits.dropna(axis=1)
columns_with_na_dropped.head()
# just how much data did we lose?
print("Columns in original dataset: %d \n" % sf_permits.shape[1])
print("Columns with na's dropped: %d" % columns_with_na_dropped.shape[1])<jupyter_output><empty_output><jupyter_text>We've lost quite a bit of data, but at this point we have successfully removed all the `NaN`'s from our data.<jupyter_code># Your turn! Try removing all the rows from the sf_permits dataset that contain missing values. How many are left?

# Now try removing all the columns with empty values. Now how much of your data is left?<jupyter_output><empty_output><jupyter_text>Filling in missing values automatically_____Another option is to try and fill in the missing values. For this next bit, I'm getting a small sub-section of the NFL data so that it will print well.<jupyter_code># get a small subset of the NFL dataset
subset_nfl_data = nfl_data.loc[:, 'EPA':'Season'].head()
subset_nfl_data<jupyter_output><empty_output><jupyter_text>We can use the Panda's fillna() function to fill in missing values in a dataframe for us. One option we have is to specify what we want the `NaN` values to be replaced with. Here, I'm saying that I would like to replace all the `NaN` values with 0.<jupyter_code># replace all NA's with 0
subset_nfl_data.fillna(0)<jupyter_output><empty_output><jupyter_text>I could also be a bit more savvy and replace missing values with whatever value comes directly after it in the same column. (This makes a lot of sense for datasets where the observations have some sort of logical order to them.)<jupyter_code># replace all NA's the value that comes directly after it in the same column, 
# then replace all the reamining na's with 0
subset_nfl_data.fillna(method = 'bfill', axis=0).fillna("0")<jupyter_output><empty_output><jupyter_text>Filling in missing values is also known as "imputation", and you can find more exercises on it [in this lesson, also linked under the "More practice!" section](https://www.kaggle.com/dansbecker/handling-missing-values). First, however, why don't you try replacing some of the missing values in the sf_permit dataset?<jupyter_code># Your turn! Try replacing all the NaN's in the sf_permits data with the one that
# comes directly after it and then 
sf_permits.fillna(method = 'bfill', axis=0)


# then replace all the reamining na's with 0
sf_permits.fillna(method = 'bfill', axis=0).fillna("0")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.061912364733014015, 'fraction_numerical': 0.004967181124711726, 'mean_word_length': 4.956154252509244, 'pattern_counts': {'":': 0, '<': 57, '<?xml version=': 0, '>': 61, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '18478', 'license': 'MIT', 'n_tokens_mistral': 3183, 'n_tokens_neox': 3049, 'n_words': 1689, 'path': 'MissingValues/DataCleaning HandlingMissingValues.ipynb', 'repo_name': 'jahnavijasti/Data-Cleaning-Practice', 'size': '11274'}