<jupyter_start><jupyter_text>**[Data Cleaning Home Page](https://www.kaggle.com/learn/data-cleaning)**--- In this exercise, you'll apply what you learned in the **Handling missing values** tutorial. SetupThe questions below will give you feedback on your work. Run the following cell to set up the feedback system.<jupyter_code>from learntools.core import binder
binder.bind(globals())
from learntools.data_cleaning.ex1 import *
print("Setup Complete")<jupyter_output>Setup Complete<jupyter_text>1) Take a first look at the dataRun the next code cell to load in the libraries and dataset you'll use to complete the exercise.<jupyter_code># modules we'll use
import pandas as pd
import numpy as np

# read in all our data
sf_permits = pd.read_csv("../input/building-permit-applications-data/Building_Permits.csv")

# set seed for reproducibility
np.random.seed(0) 
sf_permits.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 198900 entries, 0 to 198899
Data columns (total 43 columns):
 #   Column                                  Non-Null Count   Dtype  
---  ------                                  --------------   -----  
 0   Permit Number                           198900 non-null  object 
 1   Permit Type                             198900 non-null  int64  
 2   Permit Type Definition                  198900 non-null  object 
 3   Permit Creation Date                    198900 non-null  object 
 4   Block                                   198900 non-null  object 
 5   Lot                                     198900 non-null  object 
 6   Street Number                           198900 non-null  int64  
 7   Street Number Suffix                    2216 non-null    object 
 8   Street Name                             198900 non-null  object 
 9   Street Suffix                           196132 non-null  object 
 10  Unit                                    2947[...]<jupyter_text>Use the code cell below to print the first five rows of the `sf_permits` DataFrame.<jupyter_code># TODO: Your code here!
pd.options.display.max_columns = 50
sf_permits.head(5)<jupyter_output><empty_output><jupyter_text>Does the dataset have any missing values?  Once you have an answer, run the code cell below to get credit for your work.<jupyter_code># Check your answer (Run this code cell to receive credit!)
q1.check()
# Line below will give you a hint
#q1.hint()<jupyter_output><empty_output><jupyter_text>2) How many missing data points do we have?What percentage of the values in the dataset are missing?  Your answer should be a number between 0 and 100.  (If 1/4 of the values in the dataset are missing, the answer is 25.)<jupyter_code># TODO: Your code here!
missing_values_count = sf_permits.isnull().sum()
total_missing = missing_values_count.sum()
total_cells = np.product(sf_permits.shape)


percent_missing = (total_missing / total_cells) * 100
#percent_missing

# Check your answer
q2.check()
# Lines below will give you a hint or solution code
#q2.hint()
#q2.solution()<jupyter_output><empty_output><jupyter_text>3) Figure out why the data is missingLook at the columns **"Street Number Suffix"** and **"Zipcode"** from the [San Francisco Building Permits dataset](https://www.kaggle.com/aparnashastry/building-permit-applications-data). Both of these contain missing values. - Which, if either, are missing because they don't exist? - Which, if either, are missing because they weren't recorded?  Once you have an answer, run the code cell below.<jupyter_code># Check your answer (Run this code cell to receive credit!)
q3.check()
# Line below will give you a hint
#q3.hint()<jupyter_output><empty_output><jupyter_text>4) Drop missing values: rowsIf you removed all of the rows of `sf_permits` with missing values, how many rows are left?**Note**: Do not change the value of `sf_permits` when checking this.<jupyter_code># TODO: Your code here!
sf_permits.dropna()<jupyter_output><empty_output><jupyter_text>Once you have an answer, run the code cell below.<jupyter_code># Check your answer (Run this code cell to receive credit!)
q4.check()
# Line below will give you a hint
#q4.hint()<jupyter_output><empty_output><jupyter_text>5) Drop missing values: columnsNow try removing all the columns with empty values.  - Create a new DataFrame called `sf_permits_with_na_dropped` that has all of the columns with empty values removed.  - How many columns were removed from the original `sf_permits` DataFrame? Use this number to set the value of the `dropped_columns` variable below.<jupyter_code># TODO: Your code here
sf_permits_with_na_dropped = sf_permits.dropna(axis=1)
cols_in_sf_permits = sf_permits.shape[1]
cols_in_sf_permits_with_na_dropped = sf_permits_with_na_dropped.shape[1]

dropped_columns = cols_in_sf_permits - cols_in_sf_permits_with_na_dropped
#dropped_columns

# Check your answer
q5.check()
# Lines below will give you a hint or solution code
#q5.hint()
#q5.solution()<jupyter_output><empty_output><jupyter_text>6) Fill in missing values automaticallyTry replacing all the NaN's in the `sf_permits` data with the one that comes directly after it and then replacing any remaining NaN's with 0.  Set the result to a new DataFrame `sf_permits_with_na_imputed`.<jupyter_code># TODO: Your code here
sf_permits.fillna(0)
sf_permits_with_na_imputed = sf_permits.fillna(method='bfill', axis=0)

#Compat method
# sf_permits_with_na_imputed = sf_permits.fillna(method='bfill', axis=0).fillna(0)

# Check your answer
q6.check()
# Lines below will give you a hint or solution code
#q6.hint()
#q6.solution()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08284023668639054, 'fraction_numerical': 0.02474448628294782, 'mean_word_length': 3.5947281713344315, 'pattern_counts': {'":': 0, '<': 40, '<?xml version=': 0, '>': 40, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '384588', 'license': 'MIT', 'n_tokens_mistral': 1797, 'n_tokens_neox': 1668, 'n_words': 594, 'path': 'data-cleaning/1.exercise-handling-missing-values.ipynb', 'repo_name': 'adrianteri/kaggle-learn', 'size': '5577'}