<jupyter_start><jupyter_text>Data 512 Final Project:  Discrimination In the United States: Experiences of Asian Ethnic Group in the City of Austin Author: <NAME>  I. Motivation Asian American is the fastest-growing racial group in the United States according to the research by Pew Research Center. The City of Austin in Texas is no exception. According to the data released by the government of Austin, the number of Asians in Austin is doubling every 10 years compared with the general population which doubles every 20 to 25 years. Yet, the majority of the group is still immigrants. Once here, Asian immigrant families face many obstacles, such as language barriers, culture shock, and ongoing adjustments to new lives. One of the most prevalent issues that most immigrants face is racism and discrimination. Many studies have found that racism has been linked with increased risk of health and lower satisfaction of life. The survey conducted by the City of Austin collects data from 230 different questions from the Asian immigration group to help us better understand their life experience and the basis of a personal characteristic that causes the immigration group treated unfairly in the United States.  II. Selected Data for analysis The data contains more than 2000 rows and 230 attributes collecting from the surveys the city government sent out in 2020. The questions accurately measure the respondents' opinions, behaviors, and experiences. The demographic data regarding age, gender, and marital status are also retrieved. Few ethical considerations will also be addressed in our research.The dataset can be collected through API here: https://data.austintexas.gov/City-Government/Final-Report-of-the-Asian-American-Quality-of-Life/hc5t-p62zFor API documentation, please visit here: https://dev.socrata.com/foundry/data.austintexas.gov/hc5t-p62zTerms of Service:https://austintexas.gov/page/city-austin-open-data-terms-use  III. Research questions  1. How does discriminatory experience across age have an effect on the quality of life in the City of Austin?  2. Does limited English proficiency lead to racial discrimination?  3. Does the income class correlate with discriminatory experience in the City of Austin?  IV. Related Work One of the most prevalent issues that most immigrants face today is racism and discrimination. In the study by Yoo, Gee, and Takeuchi investigating discrimination within an Asian American community, 21% of the respondents reported being racially discriminated against because of their race, language, and accent. The other study researched by the same group of researchers indicates that racial discrimination has been recognized as the key determinant of mental disorders and depressive symptoms. The project will examine several personal characteristics, including income level, marital status, and age to identify different factors behind the discrimination in the Asian immigration group in the City of Austin, Texas. The outcome of the analysis will help the city improve race relations and ensure equal treatment to its people.  V. Methodology The methodology for this project focuses on the mix of descriptive and regression analysis to understand the basis of a personal characteristic that causes racial discrimination among Asian immigrant families.The survey data is a two-dimensional data structure with organized rows and columns that is well-suited for both methods. Each entry of which represents a participant's answer to the survey questions, making it efficient to manipulate and perform analysis at different levels. We will begin with a descriptive analysis to identify outliers and detect typos in the survey data. Then, we will use the same method to summarize the characteristics of the data in a meaningful way. For example, we will calculate the average number of discrimination per person across different age groups as well as including the average quality of life in the analysis. We will also refine our calculation to compute the percentage of racial discrimination experienced across gender and marital status. Lastly, we will introduce the linear regression model to the project. Linear regression is a powerful way of discovering the relationship between one or more predictor variables and the response. The method provides important evaluation metrics such as regression coefficients and p-values. We will be interpreting the outputs carefully to understand what factors are significant predictors of the outcome variable. (i.e., the quality of life and the percentage of racial discrimination among the Asian immigration group)  VI. Analysis  There are 2 ways to retrieve data of the Asian American Quality of Life (AAQoL) provided by the City of Austin  1. Communicate with API through HTTPS<jupyter_code>client = Socrata("data.austintexas.gov", None)

results = client.get("hc5t-p62z", limit=3000)

# Convert to pandas DataFrame
df_api = pd.DataFrame.from_records(results)<jupyter_output>WARNING:root:Requests made without an app_token will be subject to strict throttling limits.<jupyter_text>2. (Recommended) Directly read files from the Github<jupyter_code>url = 'https://raw.githubusercontent.com/jeffrey25706660/data-512-final/main/Final_Report_of_the_Asian_American_Quality_of_Life__AAQoL_.csv'
df = pd.read_csv(url, error_bad_lines=False)<jupyter_output><empty_output><jupyter_text>Clean the data  Select columns required for analysis, 'Age','Gender', 'Ethnicity', 'Marital Status','Education Completed', 'Household Size', 'English Speaking', 'Income', 'Quality of Life' and drop rows with  null values.<jupyter_code>result_df =  df[['Age','Gender', 'Ethnicity', 'Marital Status',
       'Education Completed', 'Household Size', 'English Speaking', 
    'Income', 'Quality of Life','Discrimination ' 
   ]].dropna(axis=0,how='all')<jupyter_output><empty_output><jupyter_text>Check null values for each column. We see that there is some missing information in each attribute, which contributes between .03% to 12%.<jupyter_code>result_df.isna().sum()/result_df.shape[0]<jupyter_output><empty_output><jupyter_text>Research Question 1:   How does discriminatory experience across age have an effect on the quality of life in the City of Austin? Let's begin by calculating the summary statistics on age. The survey data indicates that the age ranges from 18 - 98, showing there is no unusual data in the age column.<jupyter_code>result_df['Age'].describe()<jupyter_output><empty_output><jupyter_text>Calculate average the discrimination and quality of life against age and visualize the result<jupyter_code>result_discrim = result_df.groupby(['Age']).agg({'Discrimination ': 'mean','Quality of Life': 'mean'}).reset_index()

# create figure and axis objects with subplots()
fig,ax = plt.subplots(figsize=(8,5))
# make a plot
ax.plot(result_discrim["Age"],result_discrim["Discrimination "], color="pink")
# set x-axis label
ax.set_xlabel("Age",fontsize=14)
# set y-axis label
ax.set_ylabel("% of Discrimination",fontsize=14)
plt.legend()
ax2=ax.twinx()
# make a plot with different y-axis using second axis object
ax2.plot(result_discrim["Age"],result_discrim["Quality of Life"])
ax2.set_ylabel("Quality of Life",fontsize=14)
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>Let's categorize age into different groups to observe if there is any pattern across age groups. The bin size of each age group is 10.<jupyter_code>bins= [18,28,38,48,58,68,78,88,98]
labels = ['18-28','29-38','39-48','49-58','59-68','69-78','79-88','89-98']
result_df['age goup(10)'] = pd.cut(result_df['Age'], bins=bins, labels=labels, right=True)
result_discrim_gp = result_df.groupby(['age goup(10)']).agg({'Discrimination ': 'mean','Quality of Life': 'mean'}).reset_index()
fig,ax = plt.subplots()
# make a plot
ax.plot(result_discrim_gp.index,result_discrim_gp["Discrimination "], color="pink")
# set x-axis label
ax.set_xlabel("age goup(10)",fontsize=14)
# set y-axis label
ax.set_ylabel("% of Discrimination",fontsize=14)
plt.legend()
positions = (0, 1, 2, 3, 4, 5, 6, 7)

plt.xticks(positions, labels)

ax2=ax.twinx()
# make a plot with different y-axis using second axis object
ax2.plot(result_discrim_gp.index,result_discrim_gp["Quality of Life"])
ax2.set_ylabel("Quality of Life",fontsize=14)
#plt.legend()
plt.show()
plt.savefig('age_group_vs_discrimination.png')<jupyter_output><empty_output><jupyter_text>We can further understand the relationship between discrimination and quality of life. By simply plotting the scatter plot, the relationship between 2 variables seems to be weak. We will perform a linear regression to validate our assumption<jupyter_code>sns.regplot(x='Discrimination ',y='Quality of Life',data=result_discrim, scatter_kws={"color": "black"}, line_kws={"color": "red"})
plt.savefig("regression analysis")<jupyter_output><empty_output><jupyter_text>Model Interpretation: The two-sided p-value for a hypothesis test whose null hypothesis states the slope is .1856. The result shows that it fails to reject the null hypothesis, showing that there is no relationship between discriminatory experience and the response variable, quality of life in the City of Texas.<jupyter_code>slope, intercept, r_value, p_value, std_err = stats.linregress(result_discrim['Discrimination '],result_discrim['Quality of Life'])
print('Standard error:', std_err)
print('P-value:', p_value)
print('Slope:', slope)
print('R-value:', r_value)<jupyter_output>R-value: 0.1566641349449716<jupyter_text>Discussion/Implications: Prior to conducting the research, we assume a higher percentage of discriminatory experience will lead to a lower quality of life in the City of Austin. The result from the regression analysis indicates that there is no significant relationship between two variables across age. The descriptive analysis of visualizing discriminatory experience against the quality of life by age and age groups shows that both variables fluctuate in the same direction, meaning that even though the respondents did experience racial discrimination in the City of Austin, the concern may not have a significant impact on the quality of life. The reasons can be that the government and non-profit organizations in the state offer accessible personal and professional resources for anti-racism work on government websites. For example, the Texas Workforce Commission offers channels to file complaints to protect against employment discrimination based on color and TexasLawHelp provides free legal information and court forms for civil legal problems.  Limitations:The bias in the sampling data may raise the concern. After running multiple descriptive analyses, we observe that there exists an imbalance in one of the attributes. The images below suggest that the survey data is dominated by married participants, which comprises 67% of the data collection. This phenomenon will predetermined ideas or influences in certain direction. For example, the percentage of discriminatory experience by married respondents is much lower than an unmarried person. If we were to train a model to predict the ratio of discriminatory experience in the City of Austin, the model's outcome will be skewed toward marrieds. The researcher must ensure that collected data can reflect the demographics in society in future research.<jupyter_code>result_gen_df = result_df[ result_df['Marital Status'] != 'Other'].groupby(['Marital Status']).agg({'Discrimination ': 'mean','Quality of Life': 'count'}).sort_values(by='Quality of Life', ascending=False).rename(columns={'Quality of Life': 'Records'})<jupyter_output><empty_output><jupyter_text>Percentage of respondents by marital status<jupyter_code>result_df_marital = result_df[ result_df['Marital Status'] != 'Other'].groupby(['Marital Status']).agg({'Discrimination ': 'mean','Quality of Life': 'count'}).sort_values(by='Quality of Life', ascending=False).rename(columns={'Quality of Life': 'Records'})

result_df_marital['% of Records'] = result_df_marital['Records']/result_df_marital['Records'].sum()

result_df_marital
fig = plt.figure() # Create matplotlib figure

ax = fig.add_subplot(111) # Create matplotlib axes
ax2 = ax.twinx() # Create another axes that shares the same x-axis as ax.
ax.set_ylim((0,.5))
ax.legend()
width = 0.2
result_gen_df['Records'].plot(kind='bar', color='blue', ax=ax2, width=width, position=0)
result_gen_df['Discrimination '].plot(kind='bar', color='red', ax=ax, width=width, position=1)


ax.set_ylabel('% of Discrimination')
ax2.set_ylabel('Frequency')
ax2.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>Research Question 2 Does limited English proficiency lead to racial discrimination?  Let's first look at the overall discriminatory experience by English proficiency. Interesting enough, 35% of the respondents who are fluent in English report prior experience with racial discrimination.<jupyter_code>ability_df = result_df[result_df['English Speaking'].notnull()].groupby(['English Speaking']).agg({'Discrimination ':'mean'}).sort_values(by=['Discrimination '],ascending=False).reset_index()
ability_df
gx = sns.barplot(x='English Speaking' ,y = 'Discrimination ',data=ability_df)
plt.savefig("English_Speaking_vs_Discrimination.png")<jupyter_output><empty_output><jupyter_text>Let's break down the graph by ethnicity. The result suggests that regardless of the participant's ethnicity, strong English speaking ability does show the highest ratio of discriminatory experience.<jupyter_code>ability_eth_df = result_df[(result_df['English Speaking'].notnull()) & (result_df['Ethnicity'] != 'Other') ].groupby(['Ethnicity','English Speaking']).agg({'Discrimination ':'mean'}).sort_values(by=['Ethnicity','Discrimination '],ascending=False).reset_index()
ability_eth_df
ability_eth_df.set_index([ 'Ethnicity','English Speaking']).unstack().plot(kind='bar',figsize=(8,5))
plt.legend(loc='upper left')
plt.show()<jupyter_output><empty_output><jupyter_text>Discussion/Implications:The analysis shows that fluent English speakers and limited English speakers are two groups that have a high percentage of experiencing racial inequality in the City of Austin, which indicates that regardless of English speaking ability, a person from the Asian immigration group may have been discriminated against in any way. One implication appears to be that individuals are treated differently based on their physical appearance and traits. The other implication can be of linguistic bias. Asian ethnic group comprises different origin groups and cultures. Each of the origins has a unique use of language and characteristics of speech, which is profoundly different from western culture. Because of the existing discrepancy, it will implicitly lead to unfair treatment of immigrant Asian Americans due to their first language, accent, and size of the vocabulary.  Limitations:The data is sourced from several countries of origin, comprising of Chinese, Asian Indian, Vietnamese, Korean, and Filipino ethnic groups. However, the collected survey data does not reflect the demographic trends provided by the Commission on Immigrant Affairs of Austin. According to the data, the Chinese stand about 12% and the Indian made up 18% of the Asian ethnic group.The survey data overrepresents these two groups as they are contributing 24% and 22% respectively. Moreover, Chinese and Asian Indian ethnic groups represent two peaks of the discriminatory experience in the data, where one reports the highest percentage of discrimination and the other show the lowest percentage in the city. The unintentional biases in the data will not accurately represent the discriminatory experience against other underrepresented groups.<jupyter_code>ex = sns.barplot(x= 'Ethnicity', y ='% of total', data=(result_df.groupby(['Ethnicity']).size().sort_values(ascending=False)/result_df.shape[0]).reset_index().rename(columns={0:'% of total'}))<jupyter_output><empty_output><jupyter_text>Research Question 3: Does the income class correlate with discriminatory experience in the City of Austin?  Let's first remove any unreported income from the data.<jupyter_code>result_ic_temp = result_df[(result_df['Income'].notnull()) & (result_df['Discrimination '].notnull())].rename(columns={'Discrimination ': 'Discrimination'})<jupyter_output><empty_output><jupyter_text>Then, let's aggregate the percentage of discriminatory experience by income level<jupyter_code>result_ic_df = result_ic_temp[['Income','Discrimination']].groupby(['Income']).agg({'Discrimination': 'mean'}).reset_index().rename(columns={'Discrimination': '% of Discrimination'})
icx = sns.barplot(x= 'Income', y ='% of Discrimination', data=result_ic_df)
plt.xticks(rotation=70)
plt.savefig("Income_Level_vs_Discrimination.png")<jupyter_output><empty_output><jupyter_text>To further validate our observation, we perform a regression analysis on the categorical variable, income level.<jupyter_code>result_ic_cat = result_ic_temp[['Income','Discrimination']].sort_values(by='Discrimination',ascending=True)<jupyter_output><empty_output><jupyter_text>Encode categorical variables and run logistic regression. The result of the logit regression shows that P-value is less than .05. We reject the null hypothesis, meaning that the coefficient is significant. The coefficient indicates as we move away from the baseline, which is the income group of 0 to 9,999 dollars, we will expect the odds of discriminatory experience to increase by about 6%.<jupyter_code>result_ic_cat['Income_cat'] = result_ic_cat['Income'].astype('category').cat.codes
y, X = dmatrices('Discrimination ~ Income_cat', result_ic_cat, return_type = 'dataframe')
logit = sm.Logit(y, X)
logit.fit().params
logit.fit().summary()
print('The odd ratio as we move away from the baseline group: ', np.exp(0.0558)-1)<jupyter_output>The odd ratio as we move away from the baseline group:  0.057386185350407626<jupyter_text>Discussion/Implications:Before the research, we expect the income class may have a negative relationship with discriminatory experience in the Asian ethnic group. The result, however, disproves our assumption. The descriptive analysis and the statistical inference for logistic regression suggest that the income class has a positive relationship with the prior experience of racial discrimination, meaning that as the income level increases, we will expect the discriminatory experience raises as well. The phenomenon indicates that even the middle class of immigrant Asian Americans still have to handle numerous personal experiences of unfair treatment across many areas of life and workspace such as ethnic slurs and offensive comments, pay or compensation inequality, promotion discrimination, and unjustified firings.   Limitations:The survey data shows a negatively-skewed distribution of participants' income levels. We can observe more than 40% of the respondents earn more than 70,000 in income in the City of Austin compared with 8% from the rest of the groups. The overrepresentation of the specific income level creates repercussions. The insidious outcome includes inaccurately measuring the overall quality of life experience of the residents as well as performing unreliable predictions of various metrics for lower-income students and lower-class families since the bias data causes the analysis to lose its objectivity in the first place. The survey researchers must reconsider data sampling techniques, in which income groups are over or underrepresented in the survey data.<jupyter_code>result_ic_agg_df = result_ic_temp[['Income','Discrimination']].groupby(['Income']).agg({'Discrimination': 'count'}).reset_index().rename(columns={'Discrimination': 'records'})
result_ic_agg_df['% of Records'] = result_ic_agg_df['records']/result_ic_agg_df['records'].sum()
result_ic_agg_df
icx = sns.barplot(x= 'Income', y ='records', data=result_ic_agg_df)
plt.xticks(rotation=70)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07993974391162441, 'fraction_numerical': 0.012553351744915892, 'mean_word_length': 6.484404359263435, 'pattern_counts': {'":': 2, '<': 75, '<?xml version=': 0, '>': 75, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '446690', 'license': 'MIT', 'n_tokens_mistral': 5466, 'n_tokens_neox': 4997, 'n_words': 2230, 'path': 'data_512_final_project.ipynb', 'repo_name': 'jeffrey25706660/data-512-final', 'size': '19920'}