<jupyter_start><jupyter_text>SLU 05 - Covariance and Correlation: Example notebook<jupyter_code>import pandas as pd 
import numpy as np 

# just to get rid of an annoying numpy warning 
import warnings
warnings.filterwarnings("ignore", message="numpy.dtype size changed")

import seaborn as sns
from matplotlib import pyplot as plt 
%matplotlib inline<jupyter_output><empty_output><jupyter_text>In this notebook, you will see examples on how to calculate:    - Covariance     - Pearson correlation    - Spearman correlation    - Correlation matrixYou will also find examples of:    - Spurious correlations    - Observational and Experimental Data  Loading and previewing the data<jupyter_code>housing_data = pd.read_csv('data/HousingData.csv')

housing_data.head()<jupyter_output><empty_output><jupyter_text>CovarianceCalculating the covariance between `crime` and `poverty`:<jupyter_code>housing_data['crime'].cov(housing_data['poverty'])<jupyter_output><empty_output><jupyter_text>Pearson correlationCalculating the Pearson correlation between `crime` and `poverty`:<jupyter_code>housing_data['crime'].corr(housing_data['poverty'], method='pearson')<jupyter_output><empty_output><jupyter_text>Spearman correlationCalculating the Spearman correlation between `crime` and `poverty`:<jupyter_code>housing_data['crime'].corr(housing_data['poverty'], method='spearman')<jupyter_output><empty_output><jupyter_text>Correlation matrix Using the standard Pandas method:<jupyter_code>housing_data.corr()<jupyter_output><empty_output><jupyter_text>With visualization:<jupyter_code>sns.heatmap(housing_data.corr(), annot=True)<jupyter_output><empty_output><jupyter_text>Spurious correlation in Observational data The Pearson correlation between `crime` and `road_quality` is high:<jupyter_code>housing_data['crime'].corr(housing_data['roads_quality'], method='pearson')<jupyter_output><empty_output><jupyter_text>However, this is likely a spurious correlation; It's unlikely that people are being directly driven to crime because of bad roads: Rather, it is poverty that results in more crime:<jupyter_code>housing_data['crime'].corr(housing_data['poverty'], method='pearson')<jupyter_output><empty_output><jupyter_text>And it is also poverty that causes less investment in road infrastructure:<jupyter_code>housing_data['poverty'].corr(housing_data['roads_quality'], method='pearson')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10062370062370063, 'fraction_numerical': 0.0008316008316008316, 'mean_word_length': 9.837837837837839, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 41, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '62472', 'license': 'MIT', 'n_tokens_mistral': 790, 'n_tokens_neox': 733, 'n_words': 164, 'path': 'S01 - Bootcamp and Binary Classification/SLU05 - Covariance and Correlation/Examples notebook.ipynb', 'repo_name': 'claury/sidecar-academy-batch2', 'size': '2405'}