<jupyter_start><jupyter_text>Correlations and understanding them Relationship can be either linear, non linear, monotonic and no relationship. We will also seeing how to assess strength of relationship with examples. We will get idea of polynomials and segmentationIn real world data no two relationship form a straight line. many data is curvy in nature. So let us see how different relationship looks like and get intuition on how best to model it.<jupyter_code>import warnings
import pandas as pd
import numpy as np

import matplotlib.pyplot as plt
import seaborn as sns
sns.set()
warnings.simplefilter(action="ignore")
loan_df = pd.read_csv("https://raw.githubusercontent.com/srivatsan88/YouTubeLI/master/dataset/loan_analysis_summary.csv")
loan_df.head()<jupyter_output><empty_output><jupyter_text>Statistical tests do not conclude anything. We need to apply domain knowledge and context which is related to the dataset. This will help us understand the data.<jupyter_code>loan_df.describe()<jupyter_output><empty_output><jupyter_text>Covariance vs Correlation Covariance are not standardised values. Range from -infinity to infinityCorrelation is standardise between -1 to 1.Covariance shows direction of relationship. Positive covariance shows positive relationship.Negatice covariance shows negative rrelationship.<jupyter_code>loan_df.cov()<jupyter_output><empty_output><jupyter_text>Domain knowledge for correlationFico score is like credit score. Describes how risky a customer is.Lower fico_score more risky cutomer, higher the fico_score more sincere is the customer in repaying.dti is debt to income ratio amount. How much debt you have against how much do you earn.int_rate will depend on fico_score.default rate is how many couldn't repay the loan.So we can visualize the correlations and add domain knowledge.dti inversely depends on fico_scoreint_rate inversely depends on fico_score- It is seen that we have correlation between fico_score and annual_inc.This comes from domain knowledge as well.- default rate will be higher for people with lower fico_score.- Also we will have high income with high fico_score.<jupyter_code>loan_df.corr()<jupyter_output><empty_output><jupyter_text>Pearson vs Spearman Pearson measures the relationship between linearly related variables. It makes assumption of data to be normally distributedSpearman is good when relationship is monotonic or ordinal. Monotonic is still linear but the rate of increase between x and y might not be constant.Spearman is rank_correlation coefficient. So it works better for ordinal data.It is non-parametric test. It measures relationship between continous and ordinal variables.Does not assume the underlaying data distribution.<jupyter_code>loan_df.corr(method="spearman")
loan_df.corr(method="pearson")<jupyter_output><empty_output><jupyter_text>Data Visualization<jupyter_code>sns.distplot(loan_df['median_annual_inc'], hist=True)
loan_df[['fico_score', 'median_annual_inc']].corr("spearman")
sns.regplot(x="fico_score", y="median_annual_inc", data=loan_df)
loan_df[['fico_score', 'avg_annual_inc']].corr("spearman")
sns.regplot(x="fico_score", y="avg_annual_inc", data=loan_df)
loan_df[['fico_score', 'dti']].corr("spearman")
sns.regplot(x="fico_score", y="dti", data=loan_df)
loan_df[['fico_score', 'int_rate']].corr("spearman")
sns.regplot(x="fico_score", y="int_rate", data=loan_df)
sns.distplot(loan_df["int_rate"], hist=True)
import scipy.stats as stats
stats.probplot(loan_df["int_rate"], dist="norm", plot=plt)
loan_df[['fico_score', 'loan_total']].corr("spearman")
sns.regplot(x="fico_score", y="loan_total", data=loan_df)
loan_df[['fico_score', 'default_rate']].corr("spearman")
sns.regplot(x="fico_score", y="default_rate", data=loan_df)
loan_df[['median_annual_inc', 'loan_total']].corr("spearman")
sns.regplot(x="median_annual_inc", y="loan_total", data=loan_df)
loan_df[['dti', 'loan_total']].corr("spearman")
sns.regplot(x="dti", y="loan_total", data=loan_df)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09614421632448673, 'fraction_numerical': 0.00100150225338007, 'mean_word_length': 7.647186147186147, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '284827', 'license': 'MIT', 'n_tokens_mistral': 1302, 'n_tokens_neox': 1202, 'n_words': 381, 'path': 'Learning_Applied_Statistics/Correlations.ipynb', 'repo_name': 'oke-aditya/Machine_Learning', 'size': '3994'}