<jupyter_start><jupyter_text>**Murder Rates**  Murder rates are calculated as follows:murder rate for state X in year Y = 100000 * (number of murders in state X in year Y / population in state X in year Y)(Murder is rare, so we multiply by 100,000 just to avoid dealing with tiny numbers.)<jupyter_code># Run this cell to set up the notebook, but please don't change it.
import numpy as np
from datascience import *

# These lines do some fancy plotting magic.
import matplotlib
%matplotlib inline
import matplotlib.pyplot as plt
plt.style.use('fivethirtyeight')
import warnings
warnings.simplefilter('ignore', FutureWarning)

#from client.api.assignment import load_assignment
#tests = load_assignment('project2.ok')
import pandas as pd

df = pd.read_csv (r'/content/crime_rates.csv')
#df.to_csv (r'/content/crime_rates.csv', index=None)
df
state_abb = {
'Alabama': 'AL', 'Alaska': 'AK', 'Arizona': 'AZ', 'Arkansas': 'AR', 'California': 'CA', 'Colorado': 'CO',
'Connecticut': 'CT', 'Delaware': 'DE', 'Florida': 'FL', 'Georgia': 'GA', 'Hawaii': 'HI', 'Idaho': 'ID',
'Illinois': 'IL', 'Indiana': 'IN', 'Iowa': 'IA', 'Kansas': 'KS', 'Kentucky': 'KY', 'Louisiana': 'LA',
'Maine': 'ME', 'Maryland': 'MD', 'Massachusetts': 'MA', 'Michigan': 'MI', 'Minnesota': 'MN', 'Mississippi': 'MS',
'Missouri': 'MO', 'Montana': 'MT', 'Nebraska': 'NE', 'Nevada': 'NV', 'New Hampshire': 'NH', 'New Jersey': 'NJ',
'New Mexico': 'NM', 'New York': 'NY', 'North Carolina': 'NC', 'North Dakota': 'ND', 'Ohio': 'OH', 'Oklahoma': 'OK',
'Oregon': 'OR', 'Pennsylvania': 'PA', 'Rhode Island': 'RI', 'South Carolina': 'SC', 'South Dakota': 'SD',
'Tennessee': 'TN', 'Texas': 'TX', 'Utah': 'UT', 'Vermont': 'VT', 'Virginia': 'VA', 'Washington': 'WA',
'West Virginia': 'WV', 'Wisconsin': 'WI', 'Wyoming': 'WY'}

df['state_abbrev']=df['State'].map(state_abb)
df
state_list_in=df['state_abbrev'].to_list()
state_list=set(state_list_in)
state_list<jupyter_output><empty_output><jupyter_text>Question 1.1. What additional information will we need before we can check for that association?Murder rates vary over time, and different states exhibit different trends. The rates in some states change dramatically from year to year, while others are quite stable. Let’s plot a couple, just to see the variety.<jupyter_code>import seaborn as sns

mur_rate= df.pivot("Year", "State", "Murder Rate")
mur_rate.head()


#sns.lmplot(x='Year', y='Murder Rate', hue='State', 
           #data=df.loc[df['State'].isin([state_list])], 
           #fit_reg=False)
sns.lineplot(data=mur_rate)
sns.lineplot(data=df, x="Year", y="Murder Rate", hue="State")
sns.lineplot(data=mur_rate[['Texas', 'Louisiana', 'Missouri','Utah','Connecticut', 'Idaho']])<jupyter_output><empty_output><jupyter_text>Question 1.2. Draw a line plot with years on the horizontal axis and murder rates on the vertical axis. Include two lines: one for Alaska murder rates and one for Minnesota murder rates. Create this plot using a single call: ak mn.plot(’Year’).<jupyter_code>sns.lineplot(data=mur_rate[['Alaska', 'Minnesota']])
mur_rate_table=Table().with_columns(
    'State',df['State'].to_numpy(),
    'state_abbrev',df['state_abbrev'].to_numpy(),
    'Year',df['Year'].to_numpy(),
    'Population',df['Population'].to_numpy(),
    'Murder Rate',df['Murder Rate'].to_numpy()
)
mur_rate_table.set_format("Population", NumberFormatter)
mur_table=mur_rate_table.where('state_abbrev','AK')
mur_table
ak = mur_rate_table.where('state_abbrev', 'AK').drop('State','state_abbrev', 'Population').relabeled(1, 'Murder rate in Alaska')
mn = mur_rate_table.where('state_abbrev', 'MN').drop('State','state_abbrev', 'Population').relabeled(1, 'Murder rate in Minnesota')
ak
mn
ak_mn =Table().with_columns(
    'Year',ak['Year'],
    'Murder rate in Alaska',ak['Murder rate in Alaska'],
    'Murder rate in Minnesota',mn['Murder rate in Minnesota']
)
ak_mn
ak_mn.plot('Year')<jupyter_output><empty_output><jupyter_text>Question 1.3. Implement the function most murderous, which takes a year (an integer) as its argument. It does two things:1.	It draws a horizontal bar chart of the 5 states that had the highest murder rate in that year.2.	It returns an array of the names of these states in order of increasing murder rate.<jupyter_code>def most_murderous(year):
    # Fill in this line so that the next 2 lines do what the function
    # is supposed to do most should be a table.
    most =mur_rate_table.where('Year', are.equal_to(year)).sort('Murder Rate',descending=True).take(np.arange(5))
    most.barh('State', 'Murder Rate')
    return most.column('State')

most_murderous(1990)<jupyter_output><empty_output><jupyter_text>Question 1.4.  How many more people were murdered in California in 1988 than in 1975?  Assignca change to the answer.<jupyter_code>#there is no difference between murder rate in both years
#however, the population in these two years are different, so number of the people would be different
ca = mur_rate_table.where('State', are.equal_to('California'))
ca_change = ca.where('Year', are.equal_to(1988)).column('Murder Rate') - ca.where('Year', are.equal_to(1975)).column('Murder Rate')
ca_change
np.round(ca_change)    
ca = mur_rate_table.where('State', are.equal_to('California'))
num_people_1988=(ca.where('Year',are.equal_to(1988)).column('Murder Rate') * ca.where('Year', 1988).column('Population'))/100000
num_people_1975=(ca.where('Year', 1975).column('Murder Rate') * ca.where('Year', 1975).column('Population'))/100000
ca_change = num_people_1988 - num_people_1975
np.round(ca_change)<jupyter_output><empty_output><jupyter_text>**2. Changes in Murder Rates**<jupyter_code>def diff_n(values, n):            #example of what we expected
    return np.array(values)[n:] - np.array(values)[:-n]

diff_n(make_array(1, 10, 100, 1000, 10000), 2)<jupyter_output><empty_output><jupyter_text>Question 2.1. Implement the function two year changes that takes an array of murder rates for a state, ordered by increasing year. For all two-year periods (e.g., from 1960 to 1962), it computes and returns the number of increases minus the number of decreases.<jupyter_code>"Return the number of increases minus the number of decreases after two years."
def two_year_changes(rates):
    return np.count_nonzero((diff_n(rates,2))>0) - np.count_nonzero((diff_n(rates,2))<0)
print('Alaska:',    two_year_changes(ak.column('Murder rate in Alaska')))
print('Minnesota:', two_year_changes(mn.column('Murder rate in Minnesota')))<jupyter_output>Alaska: -5
Minnesota: 6<jupyter_text>Question 2.2. Assign changes by state to a table with one row per state that has two columns: the State name and the Murder Rate two year changes statistic computed across all years in our data set for that state.<jupyter_code>changes_by_state = mur_rate_table.drop('state_abbrev','Population','Year').group('State', two_year_changes)
changes_by_state 
changes_by_state.hist("Murder Rate two_year_changes", bins=np.arange(-11, 12, 2))<jupyter_output><empty_output><jupyter_text>Question 2.3. Assign total changes to the total increases minus the total decreases for all two-year periods and all states in our data set.<jupyter_code>#total_changes = sum(changes_by_state.column('Murder Rate two_year_changes'))  #Also can make it short
total_changes = sum((mur_rate_table.drop('state_abbrev','Population','Year').group('State', two_year_changes)).column('Murder Rate two_year_changes'))
print('Total increases minus total decreases, across all states and years:', total_changes)<jupyter_output>Total increases minus total decreases, across all states and years: 45<jupyter_text>Question 2.4. Set num changes to the number of different two-year periods in the entire data set that could result in a change of a state’s murder rate. Include both those periods where a change occurred and the periods where a state’s rate happened to stay the same.We now have enough information to perform a hypothesis test.<jupyter_code>all_states=mur_rate_table.column('State')
change_mur_rate=make_array()
for i in np.arange(50):
  mur_rates=mur_rate_table.where('State',all_states.item(i)).column('Murder Rate')
  mur_occurance=len(diff_n(mur_rates,2))
  change_mur_rate=np.append(change_mur_rate, mur_occurance)
num_changes = sum(change_mur_rate)
num_changes<jupyter_output><empty_output><jupyter_text>Question 2.5. Given these null and alternative hypotheses, define a good test statistic.Important requirements for your test statistic: Choose a test statistic for which large positive values are evidence in favor of the alternative hypothesis, and other values are evidence in favor of the null hypothesis. Your test statistic should depend only on whether murder rates increased or decreased, not on the size of any change.<jupyter_code>uniform = Table().with_columns(
    "Change", make_array('Increase', 'Decrease'),
    "Chance", make_array(0.5,0.5))
uniform.sample_from_distribution('Chance', 100)<jupyter_output><empty_output><jupyter_text>Question 2.6. Complete the simulation below, which samples num changes increases/decreases at random many times and forms an empirical distribution of your test statistic under the null hypothesis. Your job is to•	fill in the function simulate under null, which simulates a single sample under the null hypothesis, and•	fill in its argument when it’s called below<jupyter_code>def simulate_under_null(num_chances_to_change):
    """Simulates some number changing several times, with an equal
    chance to increase or decrease.  Returns the value of your
    test statistic for these simulated changes.
    
    num_chances_to_change is the number of times the number changes.
    """
    inc_occurance=uniform.sample_from_distribution('Chance', 200).column('Chance sample').item(0)
    dec_occurance=uniform.sample_from_distribution('Chance', 200).column('Chance sample').item(1)
    return inc_occurance - dec_occurance

uniform_samples = make_array()
for i in np.arange(5000):
    uniform_samples = np.append(uniform_samples, simulate_under_null(num_changes))

# Feel free to change the bins if they don't make sense for your test statistic.    
Table().with_column('Test statistic under null', uniform_samples).hist(0, bins=np.arange(-100, 400+25, 5))<jupyter_output><empty_output><jupyter_text>**3. The death penalty** **A Natural Experiment**<jupyter_code>non_death_penalty_states = make_array('Alaska', 'Hawaii', 'Maine', 'Michigan', 'Wisconsin', 'Minnesota')
def had_death_penalty_in_1971(state):
    """Returns True if the argument is the name of a state that had the death penalty in 1971."""
    # The implementation of this function uses a bit of syntax
    # we haven't seen before.  Just trust that it behaves as its
    # documentation claims.
    return state not in non_death_penalty_states

states = mur_rate_table.group('State').select('State')
death_penalty = states.with_column('Death Penalty', states.apply(had_death_penalty_in_1971, 0))
death_penalty
num_death_penalty_states = death_penalty.where("Death Penalty", are.equal_to(True)).num_rows
num_death_penalty_states<jupyter_output><empty_output><jupyter_text>Question 3.3. Assign death_penalty_murder_rates to a table with the same columns and data as murder_rates, but that has only the rows for states that had the death penalty in 1971.<jupyter_code>death_penalty_murder_rate=mur_rate_table.join('State', death_penalty,'State').where('Death Penalty', are.equal_to(True)).drop('Death Penalty','state_abbrev')
death_penalty_murder_rate<jupyter_output><empty_output><jupyter_text>Question 3.4. Assign changes_72 to the value of the test statistic for the years 1971 to 1973 and the states in death_penalty_murder_rates.<jupyter_code># The staff solution took 5 lines of code.
dp_mr=death_penalty_murder_rate.where('Year',are.between(1971,1974))
id_change=make_array()
state_true=dp_mr.sort('State',distinct = True).column(0)
for i in np.arange(len(state_true)):
  chosen=state_true.item(i)
  mur_change=diff_n(dp_mr.where('State', chosen).column(3),2)
  id_change=np.append(id_change, mur_change)
test_state_72 = np.count_nonzero(id_change>0) - np.count_nonzero(id_change<0)
print('Increases minus decreases from 1971 to 1973:', test_state_72)<jupyter_output>Increases minus decreases from 1971 to 1973: 22<jupyter_text>Question 3.5.: Draw an empirical histogram of the statistic under the null hypothesis by simulating the test statistic 5,000 times<jupyter_code>samples = make_array()    #5000 times simulating based on the assignment     
for i in np.arange(5000):
    samples =np.append(samples, simulate_under_null(num_death_penalty_states))
# Feel free to change the bins if they don't make sense for your test statistic.
Table().with_column('Test statistic under null', samples).hist(bins=np.arange(-4, 28+2, 2))
samples = make_array()        #10000 times simulating based on the our reference GitHub assignment   
for i in np.arange(10000):
    samples =np.append(samples, simulate_under_null(num_death_penalty_states))
# Feel free to change the bins if they don't make sense for your test statistic.
Table().with_column('Test statistic under null', samples).hist(bins=np.arange(-4, 28+2, 2))
Table().with_column('Test statistic under null', samples)<jupyter_output><empty_output><jupyter_text>Question 3.6. Complete the analysis as follows:Compute a P-value.Draw a conclusion about the null and alternative hypotheses.Describe your findings using simple, non-technical language. Be careful not to claim that the statistical analysis has established more than it really has.<jupyter_code>repetitions=5000
observed_statistic=22
p_value=np.count_nonzero(samples >= observed_statistic)/repetitions
p_value
repetitions=10000
observed_statistic=22
p_value=np.count_nonzero(samples >= observed_statistic)/repetitions
p_value<jupyter_output><empty_output><jupyter_text>**4. Further evidence** Question 4.1. Implement run_test, which takes the following arguments:A table of murder rates for certain states, sorted by state and year like murder_rates, andthe year when the analysis starts. (The comparison group is two years later.)<jupyter_code>def run_test(rates, start_year):
    """Return a P-value for the observed difference between increases and decreases."""
    end_year = start_year + 2
    all_state=rates.sort('State',distinct =True).column('State')
    two_years= rates.where('Year',are.between(start_year,end_year+1))
    i_and_d=make_array()
    
    for i in np.arange(len(all_state)):
      differ=diff_n(two_years.where('State', all_state[i]).column(3),2)
      i_and_d=np.append(i_and_d, differ)
    observed_test_statistic =np.count_nonzero(i_and_d > 0)-np.count_nonzero(i_and_d <0)
    print('Test statistic', start_year, 'to', end_year, ':', observed_test_statistic)
    
    n_states=rates.group('State').num_rows
    samples = make_array()
    for i in np.arange(5000):
        samples =np.append(samples, simulate_under_null(len(all_state)))
    p_value_above=(np.count_nonzero(samples >= observed_test_statistic))/5000
    p_value_below=(np.count_nonzero(samples < observed_test_statistic))/5000
    if observed_test_statistic < 0:
      p_value=p_value_below
      return p_value
    elif observed_test_statistic > 0:
      p_value=p_value_above 
      return p_value
   
run_test(death_penalty_murder_rate, 1971)<jupyter_output>Test statistic 1971 to 1973 : 22<jupyter_text>Question 4.2. Create a table called non_death_penalty_murder_rates with the same columns as murder_rates but only containing rows for the six states without the death penalty in 1971. Perform the same test on this table. Then, in one sentence, conclude whether their murder rates were also more likely to increase from 1971 to 1973.<jupyter_code>non_death_penalty_murder_rates = mur_rate_table.where('State', are.contained_in(non_death_penalty_states))
non_death_penalty_murder_rates
run_test(non_death_penalty_murder_rates, 1971)<jupyter_output>Test statistic 1971 to 1973 : 6<jupyter_text>**The death penalty reinstated**<jupyter_code>print("Increases minus decreases from 1975 to 1977 (when the death penalty was reinstated) among death penalty states:",
      sum(death_penalty_murder_rate.where('Year', are.between_or_equal_to(1975, 1977))
                                    .group('State', two_year_changes)
                                    .column("Murder Rate two_year_changes")))
run_test(death_penalty_murder_rate, 1975)<jupyter_output>Increases minus decreases from 1975 to 1977 (when the death penalty was reinstated) among death penalty states: -18
Test statistic 1975 to 1977 : -18<jupyter_text>Question 5.1. Create a table called average_murder_rates with 1 row for each year in murder_rates. It should have 3 columns:Year, the year,Death penalty states, the average murder rate of the states that had the death penalty in 1971, andNo death penalty states, the average murder rate of the other states.<jupyter_code>states = dp_mr.sort('State', distinct = True).column(0)
average_murder_rates= Table().with_columns(
    'Year',mur_rate_table.sort('Year',distinct = True).column('Year'),
    'Death penalty states',mur_rate_table.drop('state_abbrev','Population').where('State', are.contained_in(states)).group('Year',np.mean).column(2),
    'No death penalty states',mur_rate_table.drop('state_abbrev','Population').where('State', are.not_contained_in(states)).group('Year',np.mean).column(2)
)
average_murder_rates
average_murder_rates.plot('Year')
df = pd.read_csv (r'/content/canada.txt')
df.head(5)
df.to_csv (r'/content/canada.csv', index=None)
canada = Table.read_table('/content/canada.csv')
murder_rates_with_canada = average_murder_rates.join("Year", canada.select("Year", "Homicide").relabeled("Homicide", "Canada"))
murder_rates_with_canada.plot('Year')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11691387964581806, 'fraction_numerical': 0.022954148102193896, 'mean_word_length': 6.181855002025111, 'pattern_counts': {'":': 0, '<': 84, '<?xml version=': 0, '>': 86, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '21', 'dir': 'jupyter-structured-clean-dedup', 'id': '110460', 'license': 'MIT', 'n_tokens_mistral': 6180, 'n_tokens_neox': 5509, 'n_words': 1639, 'path': 'INFO_5502_Assign5_Murder_Rates.ipynb', 'repo_name': 'Kate5-7-2021/INFO-5502-SU2021', 'size': '17731'}