<jupyter_start><jupyter_text>IntroductionThis IPython notebook illustrates how to perform blocking using rule-based blocker.First, we need to import *py_entitymatching* package and other libraries as follows:<jupyter_code># Import py_entitymatching package
import py_entitymatching as em
import os
import pandas as pd<jupyter_output>/Users/pradap/miniconda3/lib/python3.5/site-packages/sklearn/cross_validation.py:44: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.
  "This module will be removed in 0.20.", DeprecationWarning)<jupyter_text>Then, read the (sample) input tables for blocking purposes.<jupyter_code># Get the datasets directory
datasets_dir = em.get_install_path() + os.sep + 'datasets'

# Get the paths of the input tables
path_A = datasets_dir + os.sep + 'person_table_A.csv'
path_B = datasets_dir + os.sep + 'person_table_B.csv'
# Read the CSV files and set 'ID' as the key attribute
A = em.read_csv_metadata(path_A, key='ID')
B = em.read_csv_metadata(path_B, key='ID')<jupyter_output><empty_output><jupyter_text>Different Ways to Block Using Blackbox Based BlockerThere are three different ways to do overlap blocking:1. Block two tables to produce a `candidate set` of tuple pairs.2. Block a `candidate set` of tuple pairs to typically produce a reduced candidate set of tuple pairs.3. Block two tuples to check if a tuple pair would get blocked.  Block Tables to Produce a Candidate Set of Tuple Pairs First, define a blackbox function<jupyter_code>def address_address_function(x, y):
    # x, y will be of type pandas series
    
    # get name attribute
    x_address = x['address']
    y_address = y['address']
    # get the city
    x_split, y_split = x_address.split(','), y_address.split(',')
    x_city = x_split[len(x_split) - 1]
    y_city = y_split[len(y_split) - 1]
    # check if the cities match
    if x_city != y_city:
        return True
    else:
        return False
# Instantiate blackbox blocker
bb = em.BlackBoxBlocker()
# Set the black box function
bb.set_black_box_function(address_address_function)
C = bb.block_tables(A, B, l_output_attrs=['name', 'address'], r_output_attrs=['name', 'address'])
C<jupyter_output><empty_output><jupyter_text>Block Candidate SetFirst, define a blackbox function<jupyter_code>def name_name_function(x, y):
    # x, y will be of type pandas series
    
    # get name attribute
    x_name = x['name']
    y_name = y['name']
    # get last names
    x_name = x_name.split(' ')[1]
    y_name = y_name.split(' ')[1]
    # check if last names match
    if x_name != y_name:
        return True
    else:
        return False
# Instantiate blackbox blocker
bb = em.BlackBoxBlocker()
# Set the black box function
bb.set_black_box_function(name_name_function)
D = bb.block_candset(C)
D<jupyter_output><empty_output><jupyter_text>Block Two tuples To Check If a Tuple Pair Would Get Blocked First, define the black box function first<jupyter_code>def address_address_function(x, y):
    # x, y will be of type pandas series
    
    # get name attribute
    x_address = x['address']
    y_address = y['address']
    # get the city
    x_split, y_split = x_address.split(','), y_address.split(',')
    x_city = x_split[len(x_split) - 1]
    y_city = y_split[len(y_split) - 1]
    # check if the cities match
    if x_city != y_city:
        return True
    else:
        return False
# Instantiate blackabox blocker
bb = em.BlackBoxBlocker()
# Set the blackbox function 
bb.set_black_box_function(address_address_function)
A.ix[[0]]
B.ix[[0]]
status = bb.block_tuples(A.ix[0], B.ix[0])

print(status)<jupyter_output>False
{'fraction_non_alphanumeric': 0.09076682316118936, 'fraction_numerical': 0.007042253521126761, 'mean_word_length': 4.326388888888889, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 19, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '300476', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 1269, 'n_tokens_neox': 1233, 'n_words': 407, 'path': 'notebooks/guides/step_wise_em_guides/Performing Blocking Using Blackbox Blocker.ipynb', 'repo_name': 'kvpradap/py_entitymatching', 'size': '3834'}