<jupyter_start><jupyter_text>Exploring convergence problems In this notebook I There are a few conclusions - some of these may be specific to this dataset:- The EM algorithms does not always converge to the 'correct answer'.- Where the EM algorithm converges to different parameters than the data generating mechanism, the log likelihood of the input data given these 'incorrect' parameters is _the same_ as the log likelihood of the data given the correct parameters- If you generate data according to the 'incorrect' parameters it is statistically indistinguishable from data generated according to the correct answerConsider the following data generating process:<jupyter_code>settings = {
    "link_type": "dedupe_only",
    "proportion_of_matches" :0.2,
    "comparison_columns": [
        {
            "col_name": "col_1",
            "m_probabilities": [0.3, 0.7],  # Probability of typo
            "u_probabilities": [0.9, 0.1],  # Probability of collision
        },
        {
            "col_name": "col_2",
            "m_probabilities": [0.1, 0.9],  # Probability of typo
            "u_probabilities": [0.975, 0.025],  # Probability of collision
        }
    ],
    "max_iterations": 200,
    "em_convergence": 0.0001,
    "additional_columns_to_retain": [
        "true_match", "true_match_probability", "true_log_likelihood"
    ]
}<jupyter_output><empty_output><jupyter_text>Let's generate some data according to these parameters<jupyter_code>from splink_data_generation.generate_data_exact import generate_df_gammas_exact
from splink_data_generation.match_prob import add_match_prob
from splink_data_generation.log_likelihood import add_log_likelihood
df = generate_df_gammas_exact(settings)
df = add_match_prob(df, settings)
df = add_log_likelihood(df, settings)
cols = [c for c in df.columns if "_r" not in c]
df[cols].head()<jupyter_output>/Users/robinlinacre/anaconda3/lib/python3.8/site-packages/splink_data_generation/generate_data_exact.py:130: UserWarning: Note that the proportion_of_matches setting is ignored by this generator. Only the m_probabilities and u_probabilities are observed. The proportion of matches in the generated dataset was 0.200
  warnings.warn(<jupyter_text>First we will use Splink to estimate the parameters, setting the true parameters as the starting values.<jupyter_code>import logging 
logging.basicConfig()  # Means logs will print in Jupyter Lab

# Set to DEBUG if you want splink to log the SQL statements it's executing under the hood
logging.getLogger("splink").setLevel(logging.INFO)

from pyspark.context import SparkContext
from pyspark.sql import SparkSession
sc = SparkContext.getOrCreate()
spark = SparkSession(sc)

# Now use Splink to estimate the params from the data

from splink_data_generation.estimate_splink import estimate

df_e, linker = estimate(df, settings ,spark)
df_e.toPandas().head(5)<jupyter_output>/Users/robinlinacre/anaconda3/lib/python3.8/site-packages/splink/case_statements.py:17: UserWarning: The jaro_winkler_sim user definined function is not available in Spark Or you did not pass 'spark' (the SparkSession) into 'Params' Falling back to using levenshtein in the default string comparison functions You can import these functions using the scala-udf-similarity-0.0.6.jar provided with Splink
  warnings.warn(f"The jaro_winkler_sim user definined function is not available in Spark "
INFO:splink.expectation_step:Log likelihood for iteration 0:  -458.2802386038024
INFO:splink.iterate:Iteration 0 complete
INFO:splink.params:The maximum change in parameters was 2.384185793236071e-08 for key π_gamma_col_1_prob_dist_non_match_level_0_probability
INFO:splink.iterate:EM algorithm has converged
INFO:splink.expectation_step:Log likelihood for iteration 1:  -458.2802400939189<jupyter_text>As expected, Splink converges immediately because it realises that it's already at an optima  What happens if we use Splink default starting values?<jupyter_code>settings_2 = {
    "link_type": "dedupe_only",
    "comparison_columns": [
        {
            "col_name": "col_1",
        },
        {
            "col_name": "col_2",
        }
    ],
    "max_iterations": 200,
    "em_convergence": 0.0001,
    "additional_columns_to_retain": [
        "true_match", "true_match_probability", "true_log_likelihood"
    ]
}
df_e, linker = estimate(df, deepcopy(settings_2) ,spark)
linker.model<jupyter_output><empty_output><jupyter_text>Generating data according to the 'alternative' data generating mechanismWe see we get the _same_ log likelihood with _different_ parameters.What happens if we generate a dataset using the alternative parameters?<jupyter_code>from splink.params import get_or_update_settings
settings_alternative = get_or_update_settings(linker.model, settings_2)
settings_alternative<jupyter_output><empty_output><jupyter_text>Note that we have to use `generate_df_gammas_random` becaues to use `generate_df_gammas_exact` requires the `m` and `u` probabilities to have a 'reasonably small' lowest common multiplier (see [here](https://github.com/moj-analytical-services/splink_data_generation/blob/a2ab256f6cd25899c4c84cfa1b58bca615249a15/splink_data_generation/generate_data_exact.pyL56))<jupyter_code>from splink_data_generation.generate_data_random import generate_df_gammas_random
# Increase number of rows for higher accuracy!
df_alt = generate_df_gammas_random(1000000, settings_alternative)  
df_alt.head()<jupyter_output><empty_output><jupyter_text>In a real-world linking situation, the only information we have is the comparison vectors (the gamma columns).Is there any difference between the gammas columns in this alternative dataset `df_alt`, and the original dataset `df`?<jupyter_code>display(df[["gamma_col_1", "gamma_col_2"]].corr())
display(df_alt[["gamma_col_1", "gamma_col_2"]].corr())
display(df[["gamma_col_1", "gamma_col_2"]].mean())
display(df_alt[["gamma_col_1", "gamma_col_2"]].mean())<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08194514554938583, 'fraction_numerical': 0.02692242974928487, 'mean_word_length': 5.482006543075245, 'pattern_counts': {'":': 18, '<': 27, '<?xml version=': 0, '>': 27, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '664281', 'license': 'MIT', 'n_tokens_mistral': 1965, 'n_tokens_neox': 1808, 'n_words': 520, 'path': 'notebooks/02_convergence_problems.ipynb', 'repo_name': 'moj-analytical-services/splink_data_generation', 'size': '5943'}