<jupyter_start><jupyter_text>SSL Data OverviewThe Chicago Police Department's Strategic Subject List (SSL) was created to assign individual's a score that identifies how likely they are to be a "party to violence" with a score ranging from 1 to 500. This score attempts to predict not only the perpetrators, but also potential victims of violent crimes. This information [released on the Chicago city data portal](https://data.cityofchicago.org/Public-Safety/Strategic-Subject-List/4aki-r3np) is the output of the SSL algorithm, as well as the inputs and some contextual information, run on all individuals arrested by CPD from August 1, 2012 through July 31, 2016.The creators have stated that no potentially discriminatory variables are included in the analysis [source: New York Times article, May 23, 2016](https://www.nytimes.com/2016/05/24/us/armed-with-data-chicago-police-try-to-predict-who-may-shoot-or-be-shot.html), and the algorithm has gone through at least 5 versions, with the 5th being in use as recently as August of 2016 [source: CPD press release, August 17, 2016](http://4abpn833c0nr1zvwp7447f2b.wpengine.netdna-cdn.com/wp-content/uploads/2016/08/RAND_Response-1.pdf). The eight variables being used to generate scores on arrest information in this version are (in no particular order):* Age at Latest Arrest: Likely an actual age in the real data, but grouped into ranges from "Less than 20" to "70-80" here* Number of Times a Victim of a Shooting: Integer count of instances* Number of Times a Victim of Aggravated Battery or Assault: Integer count of instances* Number of Times Arrested for Violent Offenses: Integer count of instances* Number of Times Arrested for a Narcotics Offense: Integer count of instances* Number of Times Arrested for Unlawful Use of Weapons: Integer count of instances* Indicator if the Individual is Confirmed to be a Member of a Criminal Street Gang: Boolean yes/no represented as 0 and 1* Trend of Individual's Recent Criminal Activity: Floating point number ranging from ~-8.2 to ~7.3. The least clear of the variables on how it's actually generated, but appears to be pulling from arrests and interactions not related to the mentioned categories.When model changes took place, and what the variables previously were is not clear, though the New York Times article above says 10 variables were used at the time it was published (May 23, 2016). Regarding the specific variables, the creator of the model, Dr. <NAME>, is described as feeling that "the model intentionally avoided using variables that could discriminate in some way, like race, gender, ethnicity and geography." However, a CPD press release issued on February 29, 2016 about a gang enforcement raid refers to the SSL and states "the score is based upon previous contact with police, known affiliations with gang members, and geographic location." ([source](http://home.chicagopolice.org/wp-content/uploads/2016/02/29-FEB-16-CPD-RELEASE-Gang-Enforcement-Missions.pdf)) Whether the model did incorporate geographic location at that time is unclear.<jupyter_code>ssl_df = pd.read_csv('data/Strategic_Subject_List.csv')
ssl_df.info(verbose=True)
ssl_df.head()
# Basic summary statistics on the SSL score and input variables 
ssl_df[['SSL_SCORE', 'PREDICTOR_RAT_AGE_AT_LATEST_ARREST', 'PREDICTOR_RAT_VICTIM_SHOOTING_INCIDENTS',
        'PREDICTOR_RAT_VICTIM_BATTERY_OR_ASSAULT', 'PREDICTOR_RAT_ARRESTS_VIOLENT_OFFENSES',
        'PREDICTOR_RAT_GANG_AFFILIATION', 'PREDICTOR_RAT_NARCOTIC_ARRESTS', 'PREDICTOR_RAT_TREND_IN_CRIMINAL_ACTIVITY',
        'PREDICTOR_RAT_UUW_ARRESTS']].describe()<jupyter_output><empty_output><jupyter_text>Number of Individuals Included in Arrest DataIn total, there are 398,684 individuals in the arrest information processed by the SSL model. Even though this is more than just those "on the list", almost any way of subdividing it appears drastically different from the numbers that have previously been cited on the size of the list, including  [30,000 in a Chicago Sun-Times on September 4, 2015](http://chicago.suntimes.com/news/four-of-eight-men-slain-wednesday-on-police-at-risk-list/), [1,500 in a CBS Local story on May 31, 2016](http://chicago.cbslocal.com/2016/05/31/going-inside-the-chicago-police-departments-strategic-subject-list/), and [1,400 in a Chicago Tribune story on July 22, 2016](http://www.chicagotribune.com/news/ct-chicago-police-violence-strategy-met-20160722-story.html). The reason for the discrepancy seems to be that the department has a number of individuals who are focused on which is a subset of the overall list, with the July 2016 Tribune article saying "The department has not said how many people in total are on its strategic subject list, but it said it is focusing on the 1,400 people on the list whose overall scores are between about 400 to more than 500." The counts of individuals overall and counts above certain thresholds are below.<jupyter_code>print('{:,} total individuals on SSL'.format(ssl_df['SSL_SCORE'].count()))

score_counts = [100, 150, 200, 250, 300, 350, 400, 450]
for score in score_counts:
    print('{:,} individuals with scores >= {}'.format(ssl_df.loc[ssl_df['SSL_SCORE'] >= score, 'SSL_SCORE'].count(), score))
plt.rcParams["figure.figsize"] = [15, 5]

ssl_df.hist('SSL_SCORE')<jupyter_output><empty_output><jupyter_text>Subdividing by CutoffsBecause multiple articles have described cutoff scores for inclusion on the SSL, it seems relevant to take a look at trends inside of those cutoffs. The most commonly cited ones are scores in the "upper 200s", scores above 400, and scores in the "upper 400s". In this case, three groups make sense, with all individuals with scores greater than or equal to 250, 400, and 450.<jupyter_code>ssl_gte_250 = ssl_df.loc[ssl_df['SSL_SCORE'] >= 250].copy()
ssl_gte_400 = ssl_df.loc[ssl_df['SSL_SCORE'] >= 400].copy()
ssl_gte_450 = ssl_df.loc[ssl_df['SSL_SCORE'] >= 450].copy()<jupyter_output><empty_output><jupyter_text>Input Variable DetailsOut of all the input variables, "PREDICTOR_RAT_TREND_IN_CRIMINAL_ACTIVITY" is the least transparent. While most individuals with SSL scores at or over 400 have positive values in some of the clear predictor variables, 60.79% of those with SSL scores at or over 250 have no incidents in any of 6 of the 8 predictor variables.Additionally, 48.98% of individuals with SSL scores over 250, are listed with zeroes or a "N" flag in all of the explicitly stated predictors, and also have no arrests in the past ten years in the additionally included categories of weapons, narcotics, or domestic violenceThese are the additional categories mentioned:* WEAPONS_ARR_CNT: Integer count of weapons (UUW) arrests in past 10 years* NARCOTICS_ARR_CNT: Integer count of drugs arrests in past 10 years* DOMESTIC_ARR_CNT: Integer count of domestic violence arrests in past 10 yearsA combination of these, and the "LATEST_X_DATE" for each arrest category seem to factor into creating the trend score. However, other information must be included in generating the names on the list given that a majority have no arrests in the past ten years in any of those categories.<jupyter_code>zero_cat_arrests = ssl_df.loc[(ssl_df['PREDICTOR_RAT_VICTIM_SHOOTING_INCIDENTS'] == 0) &
                              (ssl_df['PREDICTOR_RAT_VICTIM_BATTERY_OR_ASSAULT'] == 0) &
                              (ssl_df['PREDICTOR_RAT_ARRESTS_VIOLENT_OFFENSES'] == 0) &
                              (ssl_df['PREDICTOR_RAT_GANG_AFFILIATION'] == 0) &
                              (ssl_df['PREDICTOR_RAT_NARCOTIC_ARRESTS'] == 0) &
                              (ssl_df['PREDICTOR_RAT_UUW_ARRESTS'] == 0)].copy()

for cutoff in [250, 400, 450]:
    print('''
      {:,} of {:,} individuals with SSL scores greater than or equal to {}, {:.2f}%, have zeros or no flags for the 6 predictors other than trend
      '''.format(zero_cat_arrests.loc[zero_cat_arrests['SSL_SCORE'] >= cutoff, 'SSL_SCORE'].count(), 
                 ssl_df.loc[ssl_df['SSL_SCORE'] >= cutoff, 'SSL_SCORE'].count(), 
                 cutoff,
                 (zero_cat_arrests.loc[zero_cat_arrests['SSL_SCORE'] >= cutoff, 'SSL_SCORE'].count()/ssl_df.loc[ssl_df['SSL_SCORE'] >= cutoff, 'SSL_SCORE'].count())*100)
    )

total_ssl_count = ssl_df['SSL_SCORE'].count()
zero_cat_arrests = ssl_df.loc[(ssl_df['PREDICTOR_RAT_VICTIM_SHOOTING_INCIDENTS'] == 0) &
                              (ssl_df['PREDICTOR_RAT_VICTIM_BATTERY_OR_ASSAULT'] == 0) &
                              (ssl_df['PREDICTOR_RAT_ARRESTS_VIOLENT_OFFENSES'] == 0) &
                              (ssl_df['PREDICTOR_RAT_GANG_AFFILIATION'] == 0) &
                              (ssl_df['PREDICTOR_RAT_NARCOTIC_ARRESTS'] == 0) &
                              (ssl_df['PREDICTOR_RAT_UUW_ARRESTS'] == 0) &
                              (ssl_df['WEAPONS_ARR_CNT'].isnull()) &
                              (ssl_df['NARCOTICS_ARR_CNT'].isnull()) &
                              (ssl_df['DOMESTIC_ARR_CNT'].isnull())].copy()


for cutoff in [250, 400, 450]:
    print('''
      {:,} of {:,} individuals with SSL scores greater than or equal to {}, {:.2f}%, have no records of arrests in the past ten years for any of the stated categories
      '''.format(zero_cat_arrests.loc[zero_cat_arrests['SSL_SCORE'] >= cutoff, 'SSL_SCORE'].count(), 
                 ssl_df.loc[ssl_df['SSL_SCORE'] >= cutoff, 'SSL_SCORE'].count(), 
                 cutoff,
                 (zero_cat_arrests.loc[zero_cat_arrests['SSL_SCORE'] >= cutoff, 'SSL_SCORE'].count()/ssl_df.loc[ssl_df['SSL_SCORE'] >= cutoff, 'SSL_SCORE'].count())*100)
    )<jupyter_output>140,762 of 287,404 individuals with SSL scores greater than or equal to 250, 48.98%, have no records of arrests in the past ten years for any of the stated categories
      

      79 of 3,721 individuals with SSL scores greater than or equal to 400, 2.12%, have no records of arrests in the past ten years for any of the stated categories
      

      14 of 709 individuals with SSL scores greater than or equal to 450, 1.97%, have no records of arrests in the past ten years for any of the stated categories<jupyter_text>Demographic BreakdownGiven that age is intentionally included in the model as an input variable, it's understandable that it results in higher scores, but more concerning is the fact that race and sex seem extremely skewed towards Black and Latino men.While the pattern is extremely apparent in scores greater than or equal to 400, it's still clear among all those with scores >= 250.**Note**: Race codes are converted using the CPD's standard race codes as found in [Investigatory Stop Report data](http://home.chicagopolice.org/isr-data/)<jupyter_code># Breakdown by age
fig, axs = plt.subplots(1,3)

count_by_age_250 = ssl_gte_250.groupby('AGE_GROUP')['SSL_SCORE'].count()
count_by_age_250.sort_values(ascending=False, inplace=True)

count_by_age_400 = ssl_gte_400.groupby('AGE_GROUP')['SSL_SCORE'].count()
count_by_age_400.sort_values(ascending=False, inplace=True)

count_by_age_450 = ssl_gte_450.groupby('AGE_GROUP')['SSL_SCORE'].count()
count_by_age_450.sort_values(ascending=False, inplace=True)

count_by_age_250.plot(kind='bar', ax=axs[0], title='SSL Score >= 250 by Age Group')
count_by_age_400.plot(kind='bar', ax=axs[1], title='SSL Score >= 400 by Age Group')
count_by_age_450.plot(kind='bar', ax=axs[2], title='SSL Score >= 450 by Age Group')
# Converting CPD race codes to clear meanings
cpd_race_map = {
    'BLK': 'BLACK',
    'WHI': 'WHITE',
    'API': 'ASIAN/PACIFIC ISLANDER',
    'WBH': 'BLACK HISPANIC',
    'WWH': 'WHITE HISPANIC',
    'I': 'AMER IND/ALASKAN NATIVE',
    'U': 'UKNOWN'
}

ssl_df['RACE'] = ssl_df['RACE_CODE_CD'].apply(lambda x: cpd_race_map.get(x, 'U'))
ssl_gte_250['RACE'] = ssl_gte_250['RACE_CODE_CD'].apply(lambda x: cpd_race_map.get(x, 'U'))
ssl_gte_400['RACE'] = ssl_gte_400['RACE_CODE_CD'].apply(lambda x: cpd_race_map.get(x, 'U'))
ssl_gte_450['RACE'] = ssl_gte_450['RACE_CODE_CD'].apply(lambda x: cpd_race_map.get(x, 'U'))
fig, axs = plt.subplots(1,3)

count_by_race_250 = ssl_gte_250.groupby('RACE')['SSL_SCORE'].count()
count_by_race_250.sort_values(ascending=False, inplace=True)

count_by_race_400 = ssl_gte_400.groupby('RACE')['SSL_SCORE'].count()
count_by_race_400.sort_values(ascending=False, inplace=True)

count_by_race_450 = ssl_gte_450.groupby('RACE')['SSL_SCORE'].count()
count_by_race_450.sort_values(ascending=False, inplace=True)

count_by_race_250.plot(kind='bar', ax=axs[0], title='SSL Score >= 250 by Race')
count_by_race_400.plot(kind='bar', ax=axs[1], title='SSL Score >= 400 by Race')
count_by_race_450.plot(kind='bar', ax=axs[2], title='SSL Score >= 450 by Race')
fig, axs = plt.subplots(1,3)

count_by_sex_250 = ssl_gte_250.groupby('SEX_CODE_CD')['SSL_SCORE'].count()
count_by_sex_250.sort_values(ascending=False, inplace=True)

count_by_sex_400 = ssl_gte_400.groupby('SEX_CODE_CD')['SSL_SCORE'].count()
count_by_sex_400.sort_values(ascending=False, inplace=True)

count_by_sex_450 = ssl_gte_450.groupby('SEX_CODE_CD')['SSL_SCORE'].count()
count_by_sex_450.sort_values(ascending=False, inplace=True)

count_by_sex_250.plot(kind='bar', ax=axs[0], title='SSL Score >= 250 by Sex')
count_by_sex_400.plot(kind='bar', ax=axs[1], title='SSL Score >= 400 by Sex')
count_by_sex_450.plot(kind='bar', ax=axs[2], title='SSL Score >= 450 by Sex')
count_by_race_sex_250 = ssl_gte_250.groupby(['RACE', 'SEX_CODE_CD'])['SSL_SCORE'].count()
count_by_race_sex_250.sort_values(ascending=False, inplace=True)
count_by_race_sex_250.plot(kind='bar', title='SSL Score >= 250 by Race and Sex')
count_by_race_sex_400 = ssl_gte_400.groupby(['RACE', 'SEX_CODE_CD'])['SSL_SCORE'].count()
count_by_race_sex_400.sort_values(ascending=False, inplace=True)
count_by_race_sex_400.plot(kind='bar', title='SSL Score >= 400 by Race and Sex')
count_by_race_sex_450 = ssl_gte_450.groupby(['RACE', 'SEX_CODE_CD'])['SSL_SCORE'].count()
count_by_race_sex_450.sort_values(ascending=False, inplace=True)
count_by_race_sex_450.plot(kind='bar', title='SSL Score >= 450 by Race and Sex')<jupyter_output><empty_output><jupyter_text>Focus on Black and Latino IndividualsThere is a clear racial and gender disparity to who is on the list, whether the methodology is including potentially discriminatory variables or not. Given the size of this list compared to the overall population of Black and Latino residents, the pattern is concerning.For a particular focus on one demographic group, the 2015 American Community Survey found that Chicago has an estimated 72,739 Black men between the ages of 18 and 29. **Note:** This is assuming that the list would not include juveniles, so the "less than 20" category would be made up of 18-19, and that the "20-30" category represents ages 20-29 (given that the lowest category does not include the upper bound, and there is also a "30-40" category). Particular Focus on Young Black Men**Alarmingly, this shows that 33,810 out of 60,112 Black men ages 20-29 in Chicago--56.25%--have SSL scores >= 250. In contrast, 5,185 out of 87,842 white men (5.9%) have SSL scores >= 250.**Going with the assumption that juveniles are not included, 52,279 out of 72,739 Black men between the ages of 18 and 29 in Chicago--71.87%--have scores of 250 or greater on the SSL. Additionally, 45,474--62.52%--have scores >= 300.<jupyter_code>total_chi_bm_20_29 = 60112
bm_20_29_250_df = ssl_gte_250.loc[(ssl_gte_250['RACE_CODE_CD'] == 'BLK') & 
                                  (ssl_gte_250['SEX_CODE_CD'] == 'M') & 
                                  (ssl_gte_250['AGE_GROUP'] == '20-30') &
                                  (ssl_gte_250['LATEST_DIST_RES'] != 31)].copy()
print('{:,} out of {:,} Black men 20-29 in Chicago, or {:.2f}% have SSL scores >= 250'.format(
    bm_20_29_250_df['SSL_SCORE'].count(), total_chi_bm_20_29, (bm_20_29_250_df['SSL_SCORE'].count()/total_chi_bm_20_29)*100
))
total_chi_wm_20_29 = 87842
wm_20_29_250_df = ssl_gte_250.loc[(ssl_gte_250['RACE_CODE_CD'] == 'WHI') & 
                                  (ssl_gte_250['SEX_CODE_CD'] == 'M') & 
                                  (ssl_gte_250['AGE_GROUP'] == '20-30') &
                                  (ssl_gte_250['LATEST_DIST_RES'] != 31)].copy()
print('{:,} out of {:,} white men 20-29 in Chicago, or {:.2f}% have SSL scores >= 250'.format(
    wm_20_29_250_df['SSL_SCORE'].count(), total_chi_wm_20_29, (wm_20_29_250_df['SSL_SCORE'].count()/total_chi_wm_20_29)*100
))
total_chi_ybm = 72739
ybm_250_df = ssl_gte_250.loc[(ssl_gte_250['RACE_CODE_CD'] == 'BLK') & 
                             (ssl_gte_250['SEX_CODE_CD'] == 'M') & 
                             (ssl_gte_250['AGE_GROUP'].isin(['less than 20', '20-30'])) &
                             (ssl_gte_250['LATEST_DIST_RES'] != 31)].copy()
print('{:,} out of {:,} young Black men in Chicago, or {:.2f}% have SSL scores >= 250'.format(
    ybm_250_df['SSL_SCORE'].count(), total_chi_ybm, (ybm_250_df['SSL_SCORE'].count()/total_chi_ybm)*100
))
ssl_gte_300 = ssl_df.loc[ssl_df['SSL_SCORE'] >= 300].copy()
ybm_300_df = ssl_gte_300.loc[(ssl_gte_300['RACE_CODE_CD'] == 'BLK') & 
                             (ssl_gte_300['SEX_CODE_CD'] == 'M') & 
                             (ssl_gte_300['AGE_GROUP'].isin(['less than 20', '20-30'])) &
                             (ssl_gte_300['LATEST_DIST_RES'] != 31)].copy()
print('{:,} out of {:,} young Black men in Chicago, or {:.2f}% have SSL scores >= 300'.format(
    ybm_300_df['SSL_SCORE'].count(), total_chi_ybm, (ybm_300_df['SSL_SCORE'].count()/total_chi_ybm)*100
))<jupyter_output>45,474 out of 72,739 young Black men in Chicago, or 62.52% have SSL scores >= 300
{'fraction_non_alphanumeric': 0.11128135285438656, 'fraction_numerical': 0.05084553399160141, 'mean_word_length': 4.439197530864197, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 59, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '570969', 'license': 'MIT', 'n_tokens_mistral': 6590, 'n_tokens_neox': 5701, 'n_words': 1721, 'path': 'SSL Data Overview.ipynb', 'repo_name': 'pjsier/strategic-subject-list', 'size': '17629'}