<jupyter_start><jupyter_text>Implicit Bias v Demographics<jupyter_code># ! pip install pyreadstat

import bs4
from IPython.core.interactiveshell import InteractiveShell
import numpy as np
import pandas as pd
import requests

InteractiveShell.ast_node_interactivity = "all"<jupyter_output><empty_output><jupyter_text>_0. Download bias dataset to `iat.sav` and pickle to `iat.pkl` (already done)_ _This step is already done for you; no need to do it again._Grab dataset `iat.sav` from 2018 from Harvard Project Implicit     ! curl -L https://osf.io/z4bd2/download > iat.savGrab documentation for dataset columns `iat.xlsx`    ! curl -L https://osf.io/szwuf/download > iat.xlsx    pd.read_excel('iat.xlsx')See also the Harvard Project Implicit Dataset homepage https://osf.io/y9hiq/  and this [Blog post](https://app-prod-03.implicit.harvard.edu/implicit/user/jaxt/blogposts/piblogpost005.html)Then we convert SPSS `iat.sav` to pickle `iat.pkl`  It takes ~1h to read in the sav file    df = pd.read_spss('iat.sav')    df.to_pickle('iat.pkl')  1. Unpickle and clean bias dataset to `Iat`<jupyter_code>Iat = pd.read_pickle('iat.pkl')[['raceomb_002', 'ethnicityomb', 'D_biep.White_Good_all', 'STATE']]
Iat.columns = ['Race','Ethnicity','Iat','State']

# Iat.head()
# Iat.describe(include='all').T

Iat = Iat[(Iat.Race=='White') & (Iat.Ethnicity=='Not Hispanic or Latino')]
Iat = Iat[['State','Iat']]

# Iat.head()
# np.array(Iat.State.unique())

Iat = pd.pivot_table(Iat,index='State',aggfunc=np.mean)
Iat = Iat.reset_index()

Iat.Iat = Iat.Iat*100 # Convert IAT to percentage

Iat.head()<jupyter_output><empty_output><jupyter_text>2. Scrape and clean demograpic data to `df.Percent_Black`<jupyter_code>link = 'https://en.wikipedia.org/wiki/List_of_U.S._state_abbreviations'
response = requests.get(link)
soup1 = bs4.BeautifulSoup(response.text, 'html.parser')

link = 'https://en.wikipedia.org/wiki/List_of_U.S._states_and_territories_by_African-American_population'
response = requests.get(link)
soup2 = bs4.BeautifulSoup(response.text, 'html.parser')

table = soup1.find_all('table')[0] 
State = pd.read_html(str(table))[0]
State = State.iloc[11:11+51,[0,5]]
State = State.reset_index(drop=True)
State.columns = ['State','USPS']

# State.head(10).T

table = soup2.find_all('table')[1]
Demographics = pd.read_html(str(table))[0]
Demographics = Demographics.iloc[:, [0,2]]
Demographics.columns = ['Percent_Black', 'State']

# Demographics.head(10).T

df = pd.merge(State, Demographics, how='left', on='State')
df.Percent_Black = df.Percent_Black.str.strip('%')
df.Percent_Black = df.Percent_Black.astype(float)

df.head(10).T<jupyter_output><empty_output><jupyter_text>3. Merge bias data `Iat` into demographic data `df`<jupyter_code>df = pd.merge(df, Iat, how='left', on='State')
df.head()
df = df.drop(df.index[df.USPS=='DC'])<jupyter_output><empty_output><jupyter_text>4. Regression<jupyter_code>from sklearn import linear_model
model = linear_model.LinearRegression()
model.fit(df[['Percent_Black']], df[['Iat']])

print(f'''

    Correlation between Percent_Black and IAT Racial Bias score of Whites is
    
        {df.Percent_Black.corr(df.Iat):.4f}
    
    Regression line is
    
        y = {model.coef_[0][0]:.2f}x + {model.intercept_[0]:.2f}

''')

pd.set_option('float_format', '{:.1f}'.format)

df['Predict_Iat'] = model.predict(df[['Percent_Black']])
df['Difference'] = df.Iat - df.Predict_Iat
df = df.sort_values(['Difference'], ascending=False)
df

df = df.sort_values(['Iat'], ascending=False)
df
df.plot(x='Percent_Black', y='Iat', kind='scatter');<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1625382688561091, 'fraction_numerical': 0.016142499304202616, 'mean_word_length': 5.806818181818182, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 23, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '433866', 'license': 'MIT', 'n_tokens_mistral': 1437, 'n_tokens_neox': 1373, 'n_words': 251, 'path': 'iat.ipynb', 'repo_name': 'gatewaycat/whimsy', 'size': '3593'}