<jupyter_start><jupyter_text>How to Generate BISG ProxiesThe purpose of this notebook is to illustrate how to use `BISGWrapper`, a wrapper class for BISG, to generate race & ethnicity predictions<jupyter_code>%load_ext autoreload
%autoreload 2
%config Completer.use_jedi=False
from os.path import join, expanduser, dirname
import pandas as pd
import sys
import os
import re
import warnings
warnings.filterwarnings(action='ignore')
home = expanduser('~')

src_path = '{}/zrp'.format(home)
sys.path.append(src_path)
from zrp.modeling.predict import BISGWrapper
from zrp.prepare import ProcessStrings
from zrp.prepare.utils import *<jupyter_output><empty_output><jupyter_text>Load sample data for predictionLoad processed list of New Jersey Mayors downloaded from https://www.nj.gov/dca/home/2022mayors.csv<jupyter_code>nj_mayors = load_file("../2022-nj-mayors-sample.csv")
nj_mayors.shape
nj_mayors<jupyter_output><empty_output><jupyter_text>Data Processing (optional)  When generating BISG proxies, we use data output from ZRP_Prepare. Since it is not a requirement for the BISGWrapper we will use our most basic processing module to clean up the data in preparation for BISG predictions.<jupyter_code>%%time
process = ProcessStrings()
process.fit(nj_mayors)
prepared = process.transform(nj_mayors)<jupyter_output>[Start] Validating input data
     Number of observations: 462
     Is key unique: True
       (Warning!!) middle_name is 68.3982683982684% missing, this may impact the ability to return race approximations
Directory already exists
   [Completed] Validating input data

   Formatting P1
   Formatting P2
   reduce whitespace
CPU times: user 76.8 ms, sys: 4.91 ms, total: 81.7 ms
Wall time: 82.2 ms<jupyter_text>*The warning message above can be ignored since middle name is commonly missing.   Invoke the BISGWrapper on the sample dataTo generate BISG proxies, you are required to have `last_name` and `zip_code` in your dataframe otherwise.<jupyter_code>bisg = BISGWrapper()
bisg.fit(prepared)
bisg_output = bisg.transform(prepared)<jupyter_output><empty_output><jupyter_text>Inspect the output<jupyter_code>bisg_output<jupyter_output><empty_output><jupyter_text>Check CoverageA quick glance at the BISG output we can see high missing proxy rates. BISG has some limitations that contribute to missing values:- last names are required to be in the Census Surname List (ref: [https://www.census.gov/topics/population/genealogy/data/2010_surnames.html](https://www.census.gov/topics/population/genealogy/data/2010_surnames.html))    - est. 162K out of 6.3MM last names from the 2010 Census are on the list- zip codes being BISG invalid<jupyter_code>bisg_output.isna().mean()<jupyter_output><empty_output><jupyter_text>Checking the distribution of predicted race & ethnicity<jupyter_code>bisg_output.race_proxy.value_counts(normalize=True, dropna=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07950224680262703, 'fraction_numerical': 0.020048392671966817, 'mean_word_length': 6.216957605985037, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 28, 'https://': 3, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '474579', 'license': 'Apache-2.0', 'n_tokens_mistral': 971, 'n_tokens_neox': 899, 'n_words': 283, 'path': 'examples/modeling/zsub_predict_BISG.ipynb', 'repo_name': 'zestai/zrp', 'size': '2893'}