<jupyter_start><jupyter_text>Pairwise Metrics Example Pairwise metrics are used to calculate distances or similarity of a set of samples. This example shows how they can be applied and how to create your own.  Requirements To run, you'll need the following:* A dataset or rule set (in the case of a rule set, you need to provide the binary columns of the rules as applied to a dataset) ----  Import packages<jupyter_code>from iguanas.correlation_reduction import AgglomerativeClusteringReducer
from iguanas.metrics.pairwise import CosineSimilarity, JaccardSimilarity

import pandas as pd<jupyter_output><empty_output><jupyter_text>Read in data Let's read in some dummy data.<jupyter_code>X_train = pd.read_csv(
    'dummy_data/X_train.csv', 
    index_col='eid'
)
X_test = pd.read_csv(
    'dummy_data/X_test.csv', 
    index_col='eid'
)<jupyter_output><empty_output><jupyter_text>----  Apply pairwise functions The example applies two pairwise functions:* Cosine Similarity* Jaccard Similarity  Instantiate class and run fit method We can run the `fit` method to calculate the pairwise metrics for the dataset.  Cosine Similarity<jupyter_code>cs = CosineSimilarity()
cs_sim_X = cs.fit(X_train)<jupyter_output><empty_output><jupyter_text>Jaccard Similarity<jupyter_code>js = JaccardSimilarity()
js_sim_X = js.fit(X_train)<jupyter_output><empty_output><jupyter_text>Outputs The `fit` method returns the similarity matrix (stored as a dataframe) for the given pairwise function.<jupyter_code>cs_sim_X.head()
js_sim_X.head()<jupyter_output><empty_output><jupyter_text>----  Creating your own similarity function You can also create your own similarity function to use in Iguanas.Say we want to create a class which calculates the absolute (between 0 and 1) Spearman's correlation for a dataset. The main class structure involves having a `fit` method which has one argument - the dataset on which the similarity metric should be applied. This method should return a dataframe containing the similarity matrix calculated (stored as a dataframe).<jupyter_code>from scipy.stats import spearmanr

class SpearmansCorrelation:
    
    def fit(self, X: pd.DataFrame) -> pd.DataFrame:
        # Calculate pairwise Spearman's Correlation
        spearman_sim_matrix = abs(spearmanr(X_train)[0])
        # Return dataframe of the Spearman's Correlation matrix
        return pd.DataFrame(spearman_sim_matrix, index=X_train.columns, columns=X_train.columns)<jupyter_output><empty_output><jupyter_text>We can then apply the `fit` method to the dataset to check it works:<jupyter_code>sc = SpearmansCorrelation()
sc_sim_X = sc.fit(X_train)
sc_sim_X.head()<jupyter_output><empty_output><jupyter_text>Now that we have our similarity function defined, say we want to use it in the `AgglomerativeClusteringReducer` class. We just feed our `fit` method to the *similarity_function* parameter. Note that we don't call the `fit` method when passing it:<jupyter_code>params = {
    'threshold': 0.75,
    'strategy': 'bottom_up',
    'similarity_function': sc.fit,
    'metric': None,
}
fr = AgglomerativeClusteringReducer(**params)
X_train_reduced = fr.fit_transform(X_train)
X_train_reduced.head()
X_train.shape, X_train_reduced.shape<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0786308973172988, 'fraction_numerical': 0.0018501387604070306, 'mean_word_length': 5.758333333333334, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 34, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '277054', 'license': 'Apache-2.0', 'n_tokens_mistral': 1054, 'n_tokens_neox': 986, 'n_words': 325, 'path': 'docs/examples/metrics/pairwise_example.ipynb', 'repo_name': 'Aditya-Kapadiya/Iguanas', 'size': '3243'}