<jupyter_start><jupyter_text>String similarity: comparing Levenshtein distance and Tf-Idf Vectorizer methods String similarity (Comparing 'hello world' and 'hello wolrd') is an important component of deduplication.    This workbook makes uses of the flexible structure of the *Suricate* package to compare two methods:- One using the classic Levenshtein distance- The other one using a tokenization of the words, (either by character or word), using n-grams, and then using the cosine similarity  1. Prepare the environment  1.1. Libraries needed<jupyter_code># Standard libraries for data science + data visualization
import pandas as pd
import numpy as np<jupyter_output><empty_output><jupyter_text>1.2. Load the two tables to be comparedAs an example, the *Suricate* package already has some datasets about companies incorporated.   This data consists of two tables, called "*left*" and "*right*"<jupyter_code>n_lines = 100
from suricate.data.companies import getXst
df_X = getXst() # df_X is a list containing two datasets to compare [left, right]
left = df_X[0]
left.sample(5)
right = df_X[1]
right.sample(5)
n_possible_pairs= left.shape[0]*right.shape[0]
print('Two datasets of size {} and {}  rows yield:\n {} possible pairs to scan --> manually exhausting'.format(left.shape[0], right.shape[0], n_possible_pairs))<jupyter_output>Two datasets of size 100 and 100  rows yield:
 10000 possible pairs to scan --> manually exhausting<jupyter_text>2. How does the string comparator performs?  2.1. Using Levenshtein distanceThe Levenshtein distance, and its derivative from the awesome *fuzzywuzzy* package can be used to compare each row of *left* against each row of *right*<jupyter_code>from suricate.lrdftransformers import FuzzyConnector
t3 = FuzzyConnector(on='name', kind='simple')
simple_levenshtein = t3.fit_transform(X=df_X)<jupyter_output><empty_output><jupyter_text>The results are more readable using the CartesianDataPasser from suricate<jupyter_code>from suricate.lrdftransformers.cartesian import CartesianDataPasser
X_sbs = CartesianDataPasser().fit_transform(df_X).set_index(['ix_source', 'ix_target'])[['name_source', 'name_target']]
X_sbs['levenshteinsimplescore'] =  simple_levenshtein
X_sbs.sample(3)<jupyter_output><empty_output><jupyter_text>Let's look at those "close matches" which are neither obvious mistakes nor identical strings<jupyter_code>X_sbs.loc[(X_sbs['levenshteinsimplescore']>0.7) & (X_sbs['levenshteinsimplescore']<1.0)].sample(5)<jupyter_output><empty_output><jupyter_text>The framework is fully compatible with open-source Scikit-Learn Machine Learning libraries<jupyter_code>t1 = VectorizerConnector(on='name', analyzer='word', ngram_range=(1,2))
t2 = VectorizerConnector(on='name', analyzer='char', ngram_range=(1,2))

%%timeit
y1=t1.fit_transform(X=df_X)
%%timeit
y2 = t2.fit_transform(X=df_X)
%%timeit
y3 = t3.fit_transform(X=df_X)<jupyter_output><empty_output><jupyter_text>Make prediction using training data<jupyter_code>p1 = PipeLrClf(transformer=t1, classifier=Classifier())
y1_pred = pd.Series(
    data = p1.fit_predict(X=df_X, y=y_true),
    index=createmultiindex(X=df_X)
)
print(scores(y_true=y_true, y_pred=y1_pred))
p2 = PipeLrClf(transformer=t2, classifier=Classifier())
y2_pred = pd.Series(
    data = p2.fit_predict(X=df_X, y=y_true),
    index=createmultiindex(X=df_X)
)
print(scores(y_true=y_true, y_pred=y2_pred))
p3 = PipeLrClf(transformer=t3, classifier=Classifier())
y3_pred = pd.Series(
    data = p3.fit_predict(X=df_X, y=y_true),
    index=createmultiindex(X=df_X)
)
print(scores(y_true=y_true, y_pred=y3_pred))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10794353722668143, 'fraction_numerical': 0.01799058953778024, 'mean_word_length': 7.424242424242424, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 31, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '584791', 'license': 'MIT', 'n_tokens_mistral': 1295, 'n_tokens_neox': 1245, 'n_words': 310, 'path': 'tutorial/.ipynb_checkpoints/Levenshtein vs Tf-Idf-checkpoint.ipynb', 'repo_name': 'ogierpaul/suricate', 'size': '3613'}