<jupyter_start><jupyter_text>Correlated Filter Example The `CorrelatedFilter` class is used to keep only those rules which are uncorrelated.  Requirements To run, you'll need the following:* A rule set (specifically the binary columns of the rules as applied to a dataset). ----  Import packages<jupyter_code>from iguanas.rule_selection import CorrelatedFilter
from iguanas.correlation_reduction import AgglomerativeClusteringReducer
from iguanas.metrics.pairwise import JaccardSimilarity
from iguanas.metrics.classification import FScore

import pandas as pd<jupyter_output><empty_output><jupyter_text>Read in data Let's read in some dummy rules (stored as binary columns) and their corresponding performance metric dataframes:<jupyter_code>X_rules_train = pd.read_csv(
    'dummy_data/X_rules_train.csv', 
    index_col='eid'
)
X_rules_test = pd.read_csv(
    'dummy_data/X_rules_test.csv', 
    index_col='eid'
)
y_train = pd.read_csv(
    'dummy_data/y_train.csv', 
    index_col='eid'
).squeeze()
y_test = pd.read_csv(
    'dummy_data/y_test.csv', 
    index_col='eid'
).squeeze()<jupyter_output><empty_output><jupyter_text>----  Calculate uncorrelated rules Firstly, we need to instantiate the class which will perform the correlation reduction. See the `correlation_reduction` module for more information. We'll be using the `AgglomerativeClusteringReducer` class from that module. To instantiate the `AgglomerativeClusteringReducer` class, we need to first choose our similarity function. See the `metrics.pairwise` module for more information. In this example, we'll use the Jaccard similarity:<jupyter_code>js = JaccardSimilarity()<jupyter_output><empty_output><jupyter_text>Now we can instantiate the `AgglomerativeClusteringReducer` class with the necessary parameters. See the class docstring for more information regarding these:<jupyter_code>f1 = FScore(1)
params = {
    'threshold': 0.1,
    'strategy': 'bottom_up', 
    'similarity_function': js.fit, 
    'metric': f1.fit
}
agg_clust = AgglomerativeClusteringReducer(**params)<jupyter_output><empty_output><jupyter_text>Finally, we can instantiate the `CorrelatedFilter` class and run the `fit` method:<jupyter_code>fcr = CorrelatedFilter(correlation_reduction_class=agg_clust)
fcr.fit(
    X_rules=X_rules_train, 
    y=y_train
)<jupyter_output><empty_output><jupyter_text>Outputs The `fit` method does not return anything. See the `Attributes` section in the class docstring for a description of each attribute generated:<jupyter_code>fcr.rules_to_keep<jupyter_output><empty_output><jupyter_text>----  Drop rules which are correlated Use the `transform` method to drop the rules which are correlated from a given dataset:<jupyter_code>X_rules_test_uncorr = fcr.transform(X_rules=X_rules_test)<jupyter_output><empty_output><jupyter_text>Outputs The `transform` method returns a dataframe with the correlated rules dropped:<jupyter_code>X_rules_test_uncorr.head()<jupyter_output><empty_output><jupyter_text>----  Calculate correlated rules and drop them from a dataset (in one step) You can also use the `fit_transform` method to calculate correlated rules and drop them from a dataset (in one step):<jupyter_code>agg_clust = AgglomerativeClusteringReducer(**params)
fcr = CorrelatedFilter(correlation_reduction_class=agg_clust)
X_rules_train_uncorr = fcr.fit_transform(
    X_rules=X_rules_train, 
    y=y_train
)<jupyter_output><empty_output><jupyter_text>Outputs The `fit_transform` method returns a dataframe with the correlated rules dropped. See the `Attributes` section in the class docstring for a description of each attribute generated:<jupyter_code>fcr.rules_to_keep
X_rules_train_uncorr.head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08097165991902834, 'fraction_numerical': 0.001349527665317139, 'mean_word_length': 6.785714285714286, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 41, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '247358', 'license': 'Apache-2.0', 'n_tokens_mistral': 1223, 'n_tokens_neox': 1149, 'n_words': 323, 'path': 'iguanas/rule_selection/examples/correlated_filter_example.ipynb', 'repo_name': 'Aditya-Kapadiya/Iguanas', 'size': '3705'}