<jupyter_start><jupyter_text>Correlation What is correlation?Correlation is a statistical term which in common usage refers to how close two variables are to having a linear relationship with each other.For example, two variables which are linearly dependent (say, **x** and **y** which depend on each other as x = 2y) will have a higher correlation than two variables which are non-linearly dependent (say, u and v which depend on each other as u = v2) How does correlation help in feature selection?Features with high correlation are more linearly dependent and hence have almost the same effect on the dependent variable. So, when two features have high correlation, we can drop one of the two features.  Import the necessary libraries<jupyter_code>import numpy as np
import pandas as pd
import seaborn as sns<jupyter_output><empty_output><jupyter_text>Loading the dataset<jupyter_code>data = pd.read_csv('./Datasets/Data_for_Correlation.csv')
data.head()<jupyter_output><empty_output><jupyter_text>Removing the Class Label entry (Y)<jupyter_code>data = data.iloc[:,:-1]
data.head()
data.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 14 entries, 0 to 13
Data columns (total 4 columns):
 #   Column  Non-Null Count  Dtype
---  ------  --------------  -----
 0   X1      14 non-null     int64
 1   X2      14 non-null     int64
 2   X3      14 non-null     int64
 3   X4      14 non-null     int64
dtypes: int64(4)
memory usage: 576.0 bytes<jupyter_text>Selecting features based on correlationGenerating the correlation matrix<jupyter_code>corr = data.corr()
corr.head()<jupyter_output><empty_output><jupyter_text>Generating the correlation heatmap<jupyter_code>sns.heatmap(corr)<jupyter_output><empty_output><jupyter_text>Next, we compare the correlation between features and remove one of two features that have a correlation higher than 0.9<jupyter_code>columns = np.full((corr.shape[0],), True, dtype=bool)
for i in range(corr.shape[0]):
    for j in range(i+1, corr.shape[0]):
        if corr.iloc[i,j] >= 0.9:
            if columns[j]:
                columns[j] = False
selected_columns = data.columns[columns]
selected_columns.shape
data = data[selected_columns]
print(data)<jupyter_output>X1   X4
0    1   -2
1    2   -4
2    3    3
3    4    4
4    5   25
5    6   76
6    7   34
7    8  346
8    9   67
9   10    3
10  11  355
11  12   88
12  13    2
13  14    1
{'fraction_non_alphanumeric': 0.08985024958402663, 'fraction_numerical': 0.045757071547420966, 'mean_word_length': 3.5896946564885495, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '3910', 'license': 'MIT', 'n_tokens_mistral': 824, 'n_tokens_neox': 767, 'n_words': 278, 'path': 'Lab-2/4feature-selection-correlation.ipynb', 'repo_name': 'yash-a-18/002_YashAmethiya', 'size': '2404'}