<jupyter_start><jupyter_text>Application: 2000/2010 Political Campaign Contributions by RaceUsing ethnicolr, we look to answer three basic questions:What proportion of contributions were made by blacks, whites, Hispanics, and Asians? What proportion of unique contributors were blacks, whites, Hispanics, and Asians?What proportion of total donations were given by blacks, whites, Hispanics, and Asians?<jupyter_code>import pandas as pd
df = pd.read_csv('/opt/names/fec_contrib/contribDB_2000.csv', nrows=100)
df.columns
from ethnicolr import pred_fl_reg_name, pred_fl_reg_ln<jupyter_output>/opt/venv/local/lib/python2.7/site-packages/h5py/__init__.py:34: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
Using TensorFlow backend.<jupyter_text>**Load and Subset on Individual Contributors**<jupyter_code>df = pd.read_csv('/opt/names/fec_contrib/contribDB_2000.csv', usecols=['amount', 'contributor_type', 'contributor_lname', 'contributor_fname', 'contributor_name'])
sdf = df[df.contributor_type=='I'].copy()
sdf.fillna('', inplace=True)
rdf2000 = pred_fl_reg_name(sdf, 'contributor_lname', 'contributor_fname')
rdf2000['year'] = 2000

df = pd.read_csv('/opt/names/fec_contrib/contribDB_2010.csv.zip', usecols=['amount', 'contributor_type', 'contributor_lname', 'contributor_fname', 'contributor_name'])
sdf = df[df.contributor_type=='I'].copy()
sdf.fillna('', inplace=True)
rdf2010 = pred_fl_reg_name(sdf, 'contributor_lname', 'contributor_fname')
rdf2010['year'] = 2010

rdf = pd.concat([rdf2000, rdf2010])
rdf.head(20)<jupyter_output>/opt/venv/local/lib/python2.7/site-packages/IPython/core/interactiveshell.py:2718: DtypeWarning: Columns (8,7) have mixed types. Specify dtype option on import or set low_memory=False.
  interactivity=interactivity, compiler=compiler, result=result)<jupyter_text>What proportion of contributons were by blacks, whites, Hispanics, and Asians?<jupyter_code>adf = rdf.groupby(['year', 'race']).agg({'contributor_lname': 'count'})
adf.unstack().apply(lambda r: r / r.sum(), axis=1).style.format("{:.2%}")<jupyter_output><empty_output><jupyter_text>What proportion of the donors were blacks, whites, Hispanics, and Asians?<jupyter_code>udf = rdf.drop_duplicates(subset=['contributor_name']).copy()
gdf = udf.groupby(['year', 'race']).agg({'contributor_name': 'count'})
gdf.unstack().apply(lambda r: r / r.sum(), axis=1).style.format("{:.2%}")<jupyter_output><empty_output><jupyter_text>What proportion of the total donation was given by blacks, whites, Hispanics, and Asians?<jupyter_code>bdf = rdf.groupby(['year', 'race']).agg({'amount': 'sum'})
bdf.unstack().apply(lambda r: r / r.sum(), axis=1).style.format("{:.2%}")<jupyter_output><empty_output><jupyter_text>What if we estimated by using probabilities for race rather than labels?  What proportion of contributons were by blacks, whites, Hispanics, and Asians?<jupyter_code>rdf['white_count'] = rdf.nh_white
rdf['black_count'] = rdf.nh_black
rdf['asian_count'] = rdf.asian
rdf['hispanic_count'] = rdf.hispanic
gdf = rdf.groupby(['year']).agg({'white_count': 'sum', 'black_count': 'sum', 'asian_count': 'sum', 'hispanic_count': 'sum'})
gdf.apply(lambda r: r / r.sum(), axis=1).style.format("{:.2%}")<jupyter_output><empty_output><jupyter_text>What proportion of the donors were blacks, whites, Hispanics, and Asians?<jupyter_code>udf['white_count'] = udf.nh_white
udf['black_count'] = udf.nh_black
udf['asian_count'] = udf.asian
udf['hispanic_count'] = udf.hispanic
gdf = udf.groupby(['year']).agg({'white_count': 'sum', 'black_count': 'sum', 'asian_count': 'sum', 'hispanic_count': 'sum'})
gdf.apply(lambda r: r / r.sum(), axis=1).style.format("{:.2%}")<jupyter_output><empty_output><jupyter_text>What proportion of the total donation was given by blacks, whites, Hispanics, and Asians?<jupyter_code>rdf['white_amount'] = rdf.amount * rdf.nh_white
rdf['black_amount'] = rdf.amount * rdf.nh_black
rdf['api_amount'] = rdf.amount * rdf.asian
rdf['hispanic_amount'] = rdf.amount * rdf.hispanic
gdf = rdf.groupby(['year']).agg({'white_amount': 'sum', 'black_amount': 'sum', 'api_amount': 'sum', 'hispanic_amount': 'sum'}) / 10e6
gdf.style.format("{:0.2f}")
gdf.apply(lambda r: r / r.sum(), axis=1).style.format("{:.2%}")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.16343115124153498, 'fraction_numerical': 0.020090293453724605, 'mean_word_length': 7.72244094488189, 'pattern_counts': {'":': 0, '<': 31, '<?xml version=': 0, '>': 31, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '131318', 'license': 'MIT', 'n_tokens_mistral': 1691, 'n_tokens_neox': 1601, 'n_words': 325, 'path': 'ethnicolr/examples/ethnicolr_app_contrib20xx-fl_reg.ipynb', 'repo_name': 'AnanthVivekanand/ethnicolr', 'size': '4430'}