<jupyter_start><jupyter_text>EDALet's get started !!! 1. Importing the required libraries for EDA<jupyter_code>import pandas as pd 
import numpy as np                     # For mathematical calculations 
import seaborn as sns                  # For data visualization 
import matplotlib.pyplot as plt        # For plotting graphs 
%matplotlib inline 
sns.set(color_codes=True)
import warnings                        # To ignore any warnings warnings.filterwarnings("ignore")<jupyter_output><empty_output><jupyter_text>2. Loading the data into the data frame.<jupyter_code>df = pd.read_csv("example.",delimiter=' ')
# To display the top 5 rows
df.head(5)<jupyter_output><empty_output><jupyter_text>Firstly, we will check the features present in our data and then we will look at their data types.<jupyter_code>df.columns<jupyter_output><empty_output><jupyter_text>Given below is the description for each variable.| Variable | Description | |---|---||||  3. Checking the types of dataHere we check for the datatypes because sometimes ***variable** would be stored as a string or object, if in that case, we have to convert that string to the integer data only then we can plot the data via a graph.<jupyter_code># Checking the data type
df.dtypes
df.info()<jupyter_output><empty_output><jupyter_text>We can see there are three format of data types:1. object: descrip2. int64: descrip2. float64: descrip<jupyter_code>df.shape<jupyter_output><empty_output><jupyter_text>4. Dropping irrelevant columnsThis step is certainly needed in every EDA because sometimes there would be many columns that we never use in such cases dropping is the only solution.<jupyter_code># Dropping irrelevant columns
df = df.drop([ #column_names 
            ])

df.head(5)<jupyter_output><empty_output><jupyter_text>5. Renaming the columnsIn this instance, most of the column names are very confusing to read, so I just tweaked their column names. This is a good approach it improves the readability of the data set.<jupyter_code># Renaming the column names
df = df.rename(columns={}
              )
df.head(5)<jupyter_output><empty_output><jupyter_text>6. Dropping the duplicate rowsThis is often a handy thing to do because a huge data set as often have some duplicate data which might be disturbing, so here I remove all the duplicate value from the data-set.<jupyter_code># Total number of rows and columns
print(df.shape)

# Rows containing duplicate data
duplicate_rows_df = df[df.duplicated()]
print("number of duplicate rows: ", duplicate_rows_df.shape)
# Dropping the duplicates 
df = df.drop_duplicates()
df.head(5)
df.describe()<jupyter_output><empty_output><jupyter_text>***Here, notices mean, medain(represented by 50%(50th percentile)) value of each column in index column. \      notices difference between 75th %tile and max values of predictors \      observed any extreme values-Outliers in our data set.***  Next Section  7. Univariate Analysis**In this section, we will do univariate analysis. It is the simplest form of analyzing data where we examine each variable individually. For categorical features we can use frequency table or bar plots which will calculate the number of each category in a particular variable. For numerical features, probability density plots can be used to look at the distribution of the variable.** Target Variable( if categorical)<jupyter_code>df.Target.unique()
# We will first look at the target variable, i.e., Loan_Status. 
# As it is a categorical variable, let us look at its frequency table, percentage distribution and bar plot.

df['Target'].value_counts()
# Normalize can be set to True to print proportions instead of number 
df['Target'].value_counts(normalize=True)
df['Target'].value_counts().plot.bar()<jupyter_output><empty_output><jupyter_text>Target Variable( if Numerical)<jupyter_code>plt.figure(1)
plt.subplot(121) 
sns.distplot(df['Target']); 
plt.subplot(122)
df['Target'].plot.box(figsize=(16,5)) 
plt.show()<jupyter_output><empty_output><jupyter_text>Independent Variable (if Categorical)<jupyter_code>plt.figure(1) 

plt.subplot(221)
df['Variable'].value_counts(normalize=True).plot.bar(figsize=(20,10), title= 'Variable name') 
plt.subplot(222)
df['Variable'].value_counts(normalize=True).plot.bar(title= 'Variable name') 

plt.show()<jupyter_output><empty_output><jupyter_text>It can be inferred from the above bar plots that:1. 2.   Independent Variable (if Ordinal)<jupyter_code>plt.figure(1) 

plt.subplot(221)
df['Variable'].value_counts(normalize=True).plot.bar(figsize=(20,10), title= 'Variable name') 
plt.subplot(222)
df['Variable'].value_counts(normalize=True).plot.bar(title= 'Variable name') 

plt.show()<jupyter_output><empty_output><jupyter_text>It can be inferred from the above bar plots that:1. 2.   Independent Variable (if Numerical)<jupyter_code>plt.figure(1)
plt.subplot(121) 
sns.distplot(train['Variable']); 
plt.subplot(122)
train['Variable'].plot.box(figsize=(16,5)) 
plt.show()<jupyter_output><empty_output><jupyter_text>Following inferences can be made from the above dist  plots:1. Most of the data in the distribution of is towards right / left which means it is not normally distributed. We will try to make it normal in later sections as algorithms works better if the data is normally distributed.  After looking at every variable individually in univariate analysis, we will now explore them again with respect to the target variable(if Categorical). 8. Univariate Analysis(if target variable is Categorical) 8.1 Categorical Independent Variable vs Target VariableFirst of all we will find the relation between target variable and categorical independent variables. Let us look at the stacked bar plot now which will give us the proportion.<jupyter_code>Gender=pd.crosstab(train['Categorical'],train['Target']) 
Gender
Gender.div(Gender.sum(1).astype(float), axis=0).plot(kind="bar", stacked=True, figsize=(4,4))<jupyter_output><empty_output><jupyter_text>Following inferences can be made from the above proportion plots:1.  8.2 Numerical Independent Variable vs Target Variable<jupyter_code>bins=[] # bin range

group=[] # group name

train['extra_col']=pd.cut(train['variable'],bins,labels=group)

Income_bin=pd.crosstab(train['extra_col'],train['variable']) 

Income_bin.div(Income_bin.sum(1).astype(float), axis=0).plot(kind="bar", stacked=True) 

plt.xlabel('') 
P = plt.ylabel('')<jupyter_output><empty_output><jupyter_text>Following inferences can be made from the above bar plots:1.   9. Univariate Analysis(if target variable is Numerical)     Now lets look at the correlation between all the numerical variables. We will use the heat map to visualize the correlation. Heatmaps visualize data through variations in coloring.  The variables with darker color means their correlation is more.<jupyter_code># Finding the relations between the variables.
plt.figure(figsize=(20,10))
matrix = train.corr()
sns.heatmap(matrix, vmax=.8, square=True, cmap="BuPu",annot=True);<jupyter_output><empty_output><jupyter_text>To check distribution-Skewnesssns.distplot(df[variable],kde=True)   10. irrelevant columns which do not have any effect on the target variable After exploring all the variables in our data, Lets drop some variable as it do not have any effect on the target variable.  We will do the same changes to the test dataset which we did for the training dataset. df=df.drop(['col_name'],axis=1)     Next Section 11. Dropping or Replacing the missing or null values.This is mostly similar to the previous step but in here all the missing values are detected and are dropped/repalced later.\**Dropping: this is not a good approach to do so, but it is done when there is is a small number and this is negligible.\Replacing: Just replace the missing values with the mean or the median of that column.**After exploring all the variables in our data, we can now impute the missing values and treat the outliers because missing data and outliers can have adverse effect on the model performance.<jupyter_code># Finding the null values.
print(df.isnull().sum())<jupyter_output><empty_output><jupyter_text>We can consider these methods to fill the missing values:***1. For numerical variables: imputation using mean or median***\***2. For categorical variables: imputation using mode***<jupyter_code># Replacing the missing values with median.

df['categorical variables'].fillna(df['categorical variables'].mode()[0], inplace=True)
df['numerical variables'].fillna(df['numerical variables'].median(), inplace=True)
df.info()<jupyter_output><empty_output><jupyter_text>12. Outlier TreatmentAn outlier is a point or set of points that are different from other points. Sometimes they can be very high or very low. It’s often a good idea to detect and remove the outliers. Because outliers are one of the primary reasons for resulting in a less accurate model. Hence it’s a good idea to remove them. The outlier detection and removing that I am going to perform is called IQR score technique. Often outliers can be seen with visualizations using a box plot.As we saw earlier in univariate analysis, varaible can contains outliers so we have to treat them as the presence of outliers affects the distribution of the data.<jupyter_code>Q1 = df.quantile(0.25)
Q3 = df.quantile(0.75)
IQR = Q3 - Q1
print(IQR)
df = df[~((df < (Q1 - 1.5 * IQR)) | (df > (Q3 + 1.5 * IQR))).any(axis=1)]
df.shape
df['var_log'] = np.log(df['var'])<jupyter_output><empty_output><jupyter_text>13.Get_Dummies<jupyter_code>df=pd.get_dummies(df)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09254013220018886, 'fraction_numerical': 0.012590494176896443, 'mean_word_length': 5.614850798056905, 'pattern_counts': {'":': 0, '<': 82, '<?xml version=': 0, '>': 82, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '554535', 'license': 'MIT', 'n_tokens_mistral': 2957, 'n_tokens_neox': 2742, 'n_words': 1046, 'path': 'tutorial/5. EDA/EDA_template.ipynb', 'repo_name': 'rjnp2/Data-Science', 'size': '9531'}