<jupyter_start><jupyter_text>AUTO-DROP HIGHLY CORRELATED COLUMNS - <NAME> **Explanation** :Steps:1. First pass data frame into the function2. Get Corr() data frame using **' pearson method '**3. Filter with condition **df[ df [ columns > 0.85 ]**4. **Set the diagonal to np.nan, because diagonal of corr() is always 1.0**5. **Remove all completely empty columns and rows**, with absolute np.nan6. If corr() is of shape (0,0), it means there's no highly correlated columns7. Else, while corr() is not equal to (0,0) keep removing both the column and row with the highest correlation value in the whole corr() matrix, also remove all rows and columns with absolute np.nan. This will keep reducing the shape of correlation matrix.8. Now, remove the columns from original dataFrame and return the DF  Module Imports - Problem - 3<jupyter_code>import pandas as pd
import numpy as np
import seaborn as sns                                                           # FOR VISUALIZATION OF HEATMAP<jupyter_output>/usr/local/lib/python3.6/dist-packages/statsmodels/tools/_testing.py:19: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.
  import pandas.util.testing as tm<jupyter_text>**THE FUNCTION** - PROBLEM 3<jupyter_code>def dropHighlyCorrelatedColumns(df):

  '''
  This method removes minimum number of highly correlated columns using pearson method.
  '''  


  # INITIAL FEW STEPS
  corr = df.corr(method='pearson')                                              # CORR dataFrame consisting only correlation
  corr = corr[(corr >= 0.85)]                                                   # Filtering dateFrame with corr() >= 0.85                                               


  for column in corr.columns:                                                   # np.nan Diagonal, as corr() of diag is 1.0 always
    corr.loc[column][column] = np.nan


  corr.dropna(axis=1,how='all',inplace=True)                                    # Drop all columns with absolute NaN
  corr.dropna(axis=0,how='all',inplace=True)                                    # Drop all row with absolute NaN




  ###################### THIS IS THE IMPORTANT PART ######################
  if corr.shape!=(0,0):                                                         # If shape of the current dataFrame is not (0,0)  
    
    removed_cols = []                                                           # Stored the names of columns to be removed from original dataframe
        
    while corr.shape != (0,0):                                                  # While Correlation DF is not NONE:
      corr_dict = {}                                                            # Keep removing highly correlated columns in the descending order of
      for column in corr.columns:                                               # Correlation
        corr_dict[corr[column].max()] = column
      try:
          val = max(corr_dict)
          corr.drop(corr_dict[val],inplace=True)
          corr.drop(corr_dict[val],axis=1,inplace=True)
          corr.dropna(axis=1,how='all',inplace=True)
          corr.dropna(axis=0,how='all',inplace=True)
          removed_cols.append(corr_dict[val])
          del corr_dict[val]
      except ValueError:                                                        # When corr_dict is empty, it means all columns have been noted
          break

    df.drop(removed_cols,axis=1,inplace=True)                                   # Remove the columns from the original DF

    print("\nRemoved Columns are {}".format(removed_cols))                      # Print the removed columns

  else:
    print('There are no highly correlated columns')                             # No need of removal of columns if all corr() is less than 0.85
    

  return df                                                                     # In any case return DF<jupyter_output><empty_output><jupyter_text>- - - - - TEST YOUR DATA HERE - - - - -  PROBLEM 3<jupyter_code>############### CHANGE THE NEXT LINE TO LOAD YOUR DATA ###############
# df = pd.read_csv('health.csv')


################ DO NOT CHANGE THIS ################
# This is the resultant DATA FRAME
new_df = dropHighlyCorrelatedColumns(df)<jupyter_output>Removed Columns are ['per_capita_exp_PPP_2016', 'Specialist_surgical_per_1000_2008-18']<jupyter_text>VISUALIZE HERE FIRST CELL IS THE **HEAT MAP OF ORIGINAL DATA** SECOND CELL IS THE **HEAT MAP OF NEW DATA** THIRD CELL IS TO VIEW **NEW_DATA.corr() > 0.85**<jupyter_code>######################### THIS IS HEAT MAP OF CORR() OF ORIGINAL DATAFRAME ##############################
# VISUALIZE HERE
# UNCOMMENT THE BELOW LINE

# sns.heatmap(df.corr())
######################### THIS IS HEAT MAP OF CORR() OF NEW DATAFRAME ##############################
# VISUALIZE HERE
# UNCOMMENT THE BELOW LINE

# sns.heatmap(new_df.corr())
###################### VIEW NEW_DF.CORR() > 0.85 HERE ######################
# UNCOMMENT THE BELOW LINE

# new_df.corr() > 0.85<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13063331348024618, 'fraction_numerical': 0.013897161008536827, 'mean_word_length': 2.146783260462211, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 22, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 31, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '31856', 'license': 'MIT', 'n_tokens_mistral': 1433, 'n_tokens_neox': 1308, 'n_words': 441, 'path': 'DROP HIGHLY CORRELATED COLUMNS - GANESH RAM GURURAJAN.ipynb', 'repo_name': 'ganeshramg/Automatic-Removal-of-Highly-Correlated-Columns-for-AutoML', 'size': '5051'}