<jupyter_start><jupyter_text>Complex Datatype DistanceWe implement various distance metrics suitable for complex datatype. The work bases on the the work of <NAME> Martinez:><NAME>. & <NAME>. (1997). Improved Heterogeneous Distance Functions. Journal of Artificial Intelligence Research (JAIR), 6, 1-34.The authors propose various metrics for computing distances between heterogenous vectors, where components consist of numerical as well as categorial datatypes. The metrics are:* HEOM* VDM* IVDM* WVDM**Application on Complex Datatypes Instances**From the metrics above the VDM, IVDM and WVDM distance measures require labeled data instances. 1. Compute pairwise HEOM distances between instances of complex datatypes1. Cluster distances; find groups of instances which are to each other, but far from others1. Use cluster centers to label instances of complex datatypes1. Apply the VDM distance measure to labeled instances.  Heterogeneous Euclidean-Overlap Metric (HEOM)The metric is defined as follows:$$d_a(x,y) = \begin{cases}1 & \mbox{if x or y is unknown} \\overlap(x,y) & \mbox{if a is nominal} \\\text{rn_diff}_a(x,y)\end{cases}$$where `overlap` and `rn_diff` are defined as follows:$$overlap(x,y) = \begin{cases}0 & \mbox{if } x=y \\1 & \mbox{otherwise}\end{cases}$$and the difference of a numerical attribute $a$ is  $$\text{rn_diff}_a(x,y) = \frac{\mid x -y \mid}{\max_a - \min_a}$$Latter is minmax normalized. Another approach is to normalize by the standard deviation. Finally, the distance is the length of the vector of all components $d_a(x,y)$.$$HEOM(x,y) = \sqrt{\sum_a {d_a(x,y)}^2}$$<jupyter_code>import numpy as np
import pandas as pd

class HEOM():
    """Heterogeneous Euclidean-Overlap Metric (HEOM)
    
    The class works with dataframes and 1-dim series slices as input to keep
    the nominal attributes in their original form. Internally, the vectors
    are converted into numpy arrays for processing.    
    """
    def __init__(self, df, cat_idx = None):
        """Init HEOM calculations initializing relevant parameters
        
        If you do not provide an array of indices for categorical variables,
        the function will derive them using the assumption that categorical
        variables are non-numerical variables.
        """
        # we don't handle null or na
        assert df.notnull().values.all()
        assert df.notna().values.all()
        # dataframe 2dim 
        assert df.ndim == 2
        
        self.df = df
        # indices of categorical and numerical columns
        self.cat_idx = cat_idx
        self.num_idx = [self.df.columns.get_loc(c) for c in self.df._get_numeric_data().columns if c in df]

        # detect categorical indices
        if self.cat_idx is None:
            # list of indices for categorical vars
            # Assumption: categorical vars are non-numerical vars
            col_names = self.df.columns
            numeric_col_names = self.df._get_numeric_data().columns
            cat_col_names = list(set(col_names) - set(numeric_col_names))
            self.cat_idx = [self.df.columns.get_loc(c) for c in cat_col_names if c in self.df]
        
    def heom(self, x_row_idx: int, y_row_idx: int) -> np.float64:
        """Computes HEOM distance using indices of two rows from the dataframe.
        """
        assert 0 <= x_row_idx < self.df.shape[0], 'x_row_idx is out of range'
        assert 0 <= y_row_idx < self.df.shape[0], 'y_row_idx is out of range'
        
        d_vec = self.heom_vec(self.df.loc[x_row_idx], self.df.loc[y_row_idx])
        return np.sqrt(np.sum(np.square(d_vec))) # length of d_vec
    
    def heom_vec(self, x: pd.core.series.Series, y: pd.core.series.Series) -> np.ndarray:
        """HEOM distance vector between two vectors x and y.
        
        The vector contains the HEOM distance for each attribute.
        Input vectors x, y shall be supplied as 1-dim series of same length.
        """
        
        # expect x, y to be row vectors
        assert x.ndim == 1, 'x must be a 1-dim series'
        assert y.ndim == 1, 'y must be a 1-dim series'
        assert len(x) == len(y)
        assert isinstance(x, pd.core.series.Series), 'x must be of 1-dim Series type'
        assert isinstance(y, pd.core.series.Series), 'y must be of 1-dim Series type'
        
        # self.cat_idx and self.num_idx identify the same attributes
        # for *all* vectors x and y. 
        # nan_idx is value based, i.e. it may identify a nan value in the x vector
        # for a different attribute than in the y vector. As a consequence, nan values 
        # in the x vector and in the y vector *together* affect cat_idx and num_idx.
        
        # nan / null values in x *and* y vector. Duplicates possible.
        nan_xy = x.isnull().append(x.isna()).append(y.isnull()).append(y.isna())
        nan_xy_names = [i for i,v in nan_xy.iteritems() if v is True]
        nan_xy_idx = [self.df.columns.get_loc(n) for n in nan_xy_names if n in self.df]
        nan_idx = list(set(nan_xy_idx)) # remove duplicates
        cat_idx = list(set(self.cat_idx) - set(nan_xy_idx))
        num_idx = list(set(self.num_idx) - set(nan_xy_idx))
        
        assert (len(nan_idx) + len(cat_idx) + len(num_idx)) == len(self.df.columns), 'Indices do not sum up to number of columns in df'
        assert len(nan_idx) <= len(x), 'More NaNs than elements in vector'
        
        # convert x, y series into numpy arrays
        x_vec = x.to_numpy()
        y_vec = y.to_numpy()
        # result distance vector
        d_vec = np.full(shape=x_vec.shape, fill_value=-1., dtype=np.float64)
        
        # Not necessary to call with this params, 
        # but I wanted to keep the fun(x, y) format
        d_vec[nan_idx] = self.unknown(x_vec[nan_idx], y_vec[nan_idx])
        
        # normalization scaler, this is a vector
        self.scaler = self.normalization_scaler()
        
        # distance for categorical columns
        d_vec[cat_idx] = self.overlap(x[cat_idx], y[cat_idx]) 
        # distance for numeric columns
        d_vec[num_idx] = self.rn_diff(x[num_idx], y[num_idx]) 

        assert (d_vec >= 0).all(), 'd_vec must not contain negative values'
        return d_vec
    
        
    def normalization_scaler(self)-> np.ndarray:
        _max = self.df.iloc[:,self.num_idx].max(axis=0,numeric_only=True).to_numpy()
        _min = self.df.iloc[:,self.num_idx].min(axis=0,numeric_only=True).to_numpy()
        _range = _max - _min
        return _range

    def unknown(self, x: np.ndarray, y: np.ndarray) -> np.ndarray:
        # works for nan values only
        return np.ones(len(x))         
    
    def overlap(self, x: np.ndarray, y: np.ndarray) -> np.ndarray:
        # works for categorical values only
        return np.not_equal(x, y).astype(int)
    
    def rn_diff(self, x: np.ndarray, y: np.ndarray) -> np.ndarray:
        # works for numerical values only
        return np.abs(x - y) / self.scaler
    
    def heom_matrix(self) -> np.ndarray:
        matrix_dim = self.df.shape[0]
        matrix = np.full(shape=[matrix_dim, matrix_dim], fill_value=-1., dtype=np.float64)
        # upper/lower triangle matrix
        for row_idx in range(matrix_dim):
            for col_idx in range(row_idx+1, matrix_dim):
                matrix[row_idx, col_idx] = self.heom(row_idx, col_idx)
                matrix[col_idx, row_idx] = matrix[row_idx, col_idx]
        # diagonal elementes
        for row_idx in range(matrix_dim):
            matrix[row_idx, row_idx] = self.heom(row_idx, row_idx)
        # test
        assert (matrix >= 0).all(), 'matrix must not contain negative values'
        return matrix<jupyter_output><empty_output><jupyter_text>Compute HEOM Distance Matrix<jupyter_code># load data and initialize HEOM object
df_sm = pd.read_pickle('/tmp/df_sm.pkl')
heom_metric = HEOM(df_sm)
# test all heom distances between the same vectors equals 0
d_diag = [heom_metric.heom(i, i) for i in range(df_sm.shape[0])]
assert all(diag == 0 for diag in d_diag)
# test all diagonal elements in heom_matrix equals 0
heom_matrix = heom_metric.heom_matrix()
assert all(heom_matrix[i,i] == 0 for i in range(heom_matrix.shape[0]))
heom_matrix<jupyter_output><empty_output><jupyter_text>Cluster Distances1. Compute the the similarity matrix from the distance matrix1. Run spectral clustering<jupyter_code># example
X = np.array([[1, 1], [2, 1], [1, 0], [4, 7], [3, 5], [3, 6]])
X<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10262163438828531, 'fraction_numerical': 0.007557864903164856, 'mean_word_length': 2.860072926162261, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 26, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '56785', 'license': 'MIT', 'n_tokens_mistral': 2742, 'n_tokens_neox': 2652, 'n_words': 863, 'path': 'notebooks/complex_datatype_distance.ipynb', 'repo_name': 'cdeck3r/DRL4REST', 'size': '8486'}