<jupyter_start><jupyter_text>Correlation Correlation is a measure of the strength and direction of a relationship between 2 variables. Correlation ranges from -1 to 1. A perfect positive correlation is 1, no correlation is 0, and a perfect negative correlation is -1. The calculation involves two parts. The first part is taking 1 over n-1 where n is the number of observations. The second part is the sum of the product of the z-scores, also known as standard scores, for two variables, namely x and y.  Correlation, known as r,  is a calculation of the first part multiplied by the second part.  Computing Correlation Formula for correlation:r = (1/(n-1)) * SUM[[z-score of x]*[z-score of y]]<jupyter_code>import pandas as pd
# Let's say we have 2 variables, x and y
dict1 = {'x':[1, 2, 3, 3], 'y':[1, 2, 4, 5]}
df1 = pd.DataFrame(dict1)
df1
# Let's calculate the mean of x
df1['x'].mean()
# standard deviation of x
df1['x'].std()
# mean of y
df1['y'].mean()
# standard deviation of y
df1['y'].std()<jupyter_output><empty_output><jupyter_text>Recall how the standard score (also called z-score) is calculated.standard score = (observation - mean) / standard deviation<jupyter_code># Create a new column with z-scores for x and y
df1['zscore_x'] = (df1['x'] - df1['x'].mean())/df1['x'].std()
df1['zscore_y'] = (df1['y'] - df1['y'].mean())/df1['y'].std()
df1
# Let's create another new column with the product of the z-scores for x and y
df1['zscore_product_xy'] = df1['zscore_x']*df1['zscore_y']
df1
# Apply the formula of Correlation (r)

# n is the number of observations
n = df1.shape[0]

r = (1/(n-1))*(df1['zscore_product_xy'].sum())
r
# Now, let's look at some other examples

import numpy as np
import pandas as pd
# Read in file

sports = pd.read_csv("sports.csv", skiprows=2)
# Reset column names
col_names = ['Month', 'Golf', 'Soccer', 'Tennis', 'Hockey', 'Baseball']
sports.columns = col_names

# Set index
sports.set_index('Month', inplace=True)

sports.head()
# We can simply use the correlation function

sports['Golf'].corr(sports['Baseball'])<jupyter_output><empty_output><jupyter_text>Correlation Matrix<jupyter_code>sports_correlation = sports[["Golf", "Soccer", "Tennis", "Hockey", "Baseball"]].corr()
print(sports_correlation)<jupyter_output>Golf    Soccer    Tennis    Hockey  Baseball
Golf      1.000000  0.442875  0.694183 -0.473888  0.681249
Soccer    0.442875  1.000000  0.379443 -0.404575  0.379608
Tennis    0.694183  0.379443  1.000000 -0.416072  0.282095
Hockey   -0.473888 -0.404575 -0.416072  1.000000 -0.329349
Baseball  0.681249  0.379608  0.282095 -0.329349  1.000000<jupyter_text>Rolling Correlation<jupyter_code>sports['Golf'].rolling(12).corr(sports['Soccer'])
# end<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13386978785662035, 'fraction_numerical': 0.08156547183613752, 'mean_word_length': 4.8315565031982945, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '630402', 'license': 'MIT', 'n_tokens_mistral': 1105, 'n_tokens_neox': 967, 'n_words': 310, 'path': 'Python/3. Computational Sciences and Mathematics/Statisitcs/Fundamentals-of-Statistics-and-Visualization-in-Python/Section 2/2.5_code.ipynb', 'repo_name': 'okara83/Becoming-a-Data-Scientist', 'size': '2734'}