<jupyter_start><jupyter_text>Phi_K spark tutorialThis notebook shows you how to obtain the Phi_K correlation matrix for a spark dataframe.Calculating the Phi_K matrix consists of two steps:- Obtain the 2d contingency tables for all variable pairs. To make these we use the [`histogrammar` package](https://github.com/histogrammar/histogrammar-python).- Calculate the Phi_K value for each variable pair from its contingency table.Make sure you install the histogrammar package to make the 2d histograms, that are then used to calculate phik.<jupyter_code>%%capture
# install histogrammar (if not installed yet)
import sys

!"{sys.executable}" -m pip install histogrammar
import itertools

import pandas as pd
import histogrammar as hg
from histogrammar.plot.hist_numpy import get_2dgrid

import phik
from phik import resources
from phik.phik import spark_phik_matrix_from_hist2d_dict<jupyter_output><empty_output><jupyter_text>histogramming is done using the histogrammar library<jupyter_code>from pyspark.sql import SparkSession
from pyspark import __version__ as pyspark_version

scala = '2.12' if int(pyspark_version[0]) >= 3 else '2.11'
hist_jar = f'io.github.histogrammar:histogrammar_{scala}:1.0.20'
hist_spark_jar = f'io.github.histogrammar:histogrammar-sparksql_{scala}:1.0.20'

spark = SparkSession.builder.config(
    "spark.jars.packages", f'{hist_spark_jar},{hist_jar}'
).getOrCreate()

spark = SparkSession.builder.config(
    "spark.jars.packages", f'{hist_spark_jar},{hist_jar}'
).getOrCreate()

sc = spark.sparkContext<jupyter_output><empty_output><jupyter_text>Load dataA simulated dataset is part of the phik-package. The dataset concerns fake car insurance data. Load the dataset here:<jupyter_code>data = pd.read_csv( resources.fixture('fake_insurance_data.csv.gz') )
sdf = spark.createDataFrame(data)
sdf.show()
combis = itertools.combinations_with_replacement(sdf.columns, 2)
combis = [list(c) for c in combis]
print(combis)<jupyter_output><empty_output><jupyter_text>step 1: create histograms (this runs spark histogrammar in the background)<jupyter_code># see the doc-string of hg_make_histograms() for binning options.
hists = sdf.hg_make_histograms(combis)
# collect the numpy contingency tables into a dict
grids = {k:(get_2dgrid(h)[2]) for k,h in hists.items()}
print(grids)
# we can store the histograms if we want to
if False:
    import pickle

    with open('grids.pkl', 'wb') as outfile:
        pickle.dump(grids, outfile)

    with open('grids.pkl', 'rb') as handle:
        grids = pickle.load(handle)<jupyter_output><empty_output><jupyter_text>step 2: calculate phik matrix (runs rdd parallellization over all 2d histograms)<jupyter_code>phik_matrix = spark_phik_matrix_from_hist2d_dict(sc, grids)
phik_matrix<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09261261261261261, 'fraction_numerical': 0.00972972972972973, 'mean_word_length': 6.3052631578947365, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 22, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '448977', 'license': 'Apache-2.0', 'n_tokens_mistral': 965, 'n_tokens_neox': 908, 'n_words': 256, 'path': 'phik/notebooks/phik_tutorial_spark.ipynb', 'repo_name': 'ionicsolutions/PhiK', 'size': '2775'}