<jupyter_start><jupyter_text>Frame, Acquire & Refine  Raw DataYou are provided with the following data: **loan_data.csv**  This is the historical data that the bank has provided. It has the following columns**Application Attributes**:- `years`: Number of years the applicant has been employed  - `ownership`: Whether the applicant owns a house or not  - `income`:  Annual income of the applicant  - `age`: Age of the applicant  **Behavioural Attributes**:- `grade`:  Credit grade of the applicant**Outcome Variable**:- `amount` : Amount of Loan provided to the applicant  - `default` : Whether the applicant has defaulted or not - `interest`: Interest rate charged for the applicant    Frame the Problem - What are the features?- What is the target?Discuss?  Acquire the Data<jupyter_code>#Load the libraries
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
#Defualt Variables
%matplotlib inline
plt.rcParams['figure.figsize'] = (16,9)
plt.style.use('fivethirtyeight')
pd.set_option('display.float_format', lambda x: '%.2f' % x)
#Load the dataset
df = pd.read_csv("data/loan_data.csv")
#View the first few rows of train
df.head()
#View the columns of the train dataset
df.columns
#View the data types of the train dataset
df.dtypes
#View the number of records in the data
df.shape
#View summary of raw data 
df.describe()<jupyter_output><empty_output><jupyter_text>Refine the DataLets check the dataset for quality and compeleteness1. Missing Values2. Outliers  Check for Missing Values<jupyter_code># Find if df has missing values. Hint: There is a isnull() function
df.isnull().head()<jupyter_output><empty_output><jupyter_text>One consideration we check here is the number of observations with missing values for those columns that have missing values. If a column has too many missing values, it might make sense to drop the column.<jupyter_code>#let's see how many missing values are present
df.isnull().sum()<jupyter_output><empty_output><jupyter_text>So, we see that two columns have missing values: **interest and years**. Both the columns are numeric. We have three options for dealing with this missing valuesOptions to treat Missing Values- **REMOVE** - NAN rows- **IMPUTATION** - Replace them with something??    - Mean     - Median    - Fixed Number - Domain Relevant    - High Number (999) - Issue with modelling- **BINNING** - Categorical variable and "Missing becomes a number*- **DOMAIN SPECIFIC** - Entry error, pipeline, etc.<jupyter_code>#Let's replace missing values with the median of the column
df.describe()
#there's a fillna function
df = df.fillna(df.median())
#Now, let's check if train has missing values or not
df.isnull().any()<jupyter_output><empty_output><jupyter_text>Check for Outlier ValuesLet us check first the categorical variables<jupyter_code># Which variables are Categorical?
df.dtypes
# Create a Crosstab of those variables with another variable
pd.crosstab(df.default, df.grade)
# Create a Crosstab of those variables with another variable
pd.crosstab(df.default, df.ownership)<jupyter_output><empty_output><jupyter_text>Let us check outliers in the continuous variable- Plotting    - Histogram    - Box-Plot - Measuring     - Z-score > 3    - Modified Z-score > 3.5    where modified Z-score = 0.6745 * (x - x_median) / MAD<jupyter_code># Describe the data set continuous values
df.describe()<jupyter_output><empty_output><jupyter_text>Clearly the `age` variable looks like it has an outlier - Age cannot be greater 100! Also the `income` variable looks like it may also have an outlier.<jupyter_code># Make a histogram of age
df.age.hist(bins=100)
# Make a histogram of income
df.income.hist(bins=100)
# Make Histograms for all other variables
# Make a scatter of age and income
plt.scatter(df.age, df.income)<jupyter_output><empty_output><jupyter_text>Find the observation which has age = 144 and remove it from the dataframe<jupyter_code># Find the observation 
df[df.age == 144]
df[df.age == 144].index
# Use drop to remove the observation inplace
df.drop(df[df.age == 144].index, axis=0, inplace=True)
# Find the shape of the df
df.shape
# Check again for outliers
df.describe()
# Save the new file as cleaned data
df.to_csv("data/loan_data_clean.csv", index=False)
#We are good to go to the next step<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09040204508482454, 'fraction_numerical': 0.00906344410876133, 'mean_word_length': 5.061971830985915, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 35, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '386742', 'license': 'MIT', 'n_tokens_mistral': 1366, 'n_tokens_neox': 1262, 'n_words': 517, 'path': 'reference/Module-01a-reference.ipynb', 'repo_name': 'poornimaarunp/applied-machine-learning', 'size': '4303'}