<jupyter_start><jupyter_text>"DH140 Project: Exploring the Surveillance Dataset"> "Digital Humanities 140 Project: Exploring the Surveillance Dataset"- toc: true- branch: master- badges: true- comments: true- author: JH- categories: [fastpages, jupyter]  Digital Humanities 140 Project: Exploring the Surveillance Dataset  Project Introduction  Research Question: How is law-enforcing surveillance technology implemented in today's world? The concept of surveillance seems to be an enticing topic by nature – humans are easily interested in knowing how they are being watched. Surveillance technology implemented by governmental agencies can be a double-edged sword: on the one hand, it helps departments like the police greatly in increasing their efficiency and effectiveness in resolving crimes and protecting other citizens; on the other hand, its presence constantly incites discussions and concerns for privacy. I am really interested in learning how surveillance technologies were adopted in the context of law enforcement in these years in the United States. What are the popular surveillance technologies used? What are the primary context in which they are used? What are their effectiveness and drawback?This topic is tied to both sides, the implementers and the ones surveillance is implemented upon, who are the majority of people living in the U.S. Through exploring datasets with analyses and visualizations, this project could suggest how surveillance technologies were implemented and whether it has been an effective means for regulation. This could potentially give insight into issues like whether the benefit of surveillance technology outweighs the privacy concerns, or vice versa.  Data SourcesThe main dataset I will be using for this project is the [Atlas of Surveillance dataset](https://atlasofsurveillance.org/), which holds information on surveillance technologies deployed by law enforcement in communities across the United States. The dataset includes instances of the adoption of various surveillance technology by law enforcement agencies. I might also use supplemental datasets to dig deeper into various kinds of surveillance methods, such as using drones or face recognition technology. For example, the [Who Has Your Face?](https://whohasyourface.org/) dataset provides information on which governmental agencies can access face recognition images, which is helpful in understanding how these agencies use this surveillance technology and the accessibities to these privacy-concerning databases.  ScopeIn the analysis portion of the project, I will explore the variety of Law Enforcement Administration and Organizations (LEA) that use surveillance technologies, the variety of types of surveillance technologies used in different agencies, how many of the law enforcement agencies of each state has implemented surveillance technologies, etc. I will use a variety of visualizations to visualize the outcome of the datasets, such as bar plots, scatterplots, bubble plots, etc. An example would be to visualize the type of surveillance technology implemented categorized by the type of LEA. Another instance would be to visualize the use of a surveillance technology regionally across the country.  ConclusionBy the end of this project, I hope to gain a deeper understanding of how surveillance technologies are used today in the context of law enforcement. Surveillance is an evolving and convoluted topic in the modern era, as technological development is happening at an increasingly rapid speed. Questions arise along with the advancements: where to draw the line between protection and privacy? This could allow us to gain insight into the effectiveness of law-enforcing surveillance, and hence, to gain an idea of how to best protect citizens living in this society.  Introduction of the Data<jupyter_code>import pandas as pd
import matplotlib.pyplot as plt
#
# import Atlas of Surveillance csv file and turn into dataframe
aos_raw_df = pd.read_csv("Atlas of Surveillance.csv")
#
# select useful columns for dataframe
aos_df = aos_raw_df.iloc[:,1:10]
#collapse-hide
# clean dataset and resolve inconsistencies
aos_df['State'] = aos_df['State'].replace(['NB'],'NE')
aos_df['State'] = aos_df['State'].replace(['Montana'],'MT')
aos_df['State'] = aos_df['State'].replace(['Idaho'],'ID')
aos_df['State'] = aos_df['State'].replace(['US Virgin Islands'],'VI')
aos_df['State'] = aos_df['State'].replace(['Puerto Rico'],'PR')
aos_df['State'] = aos_df['State'].replace(['FL '],'FL')
aos_df['State'] = aos_df['State'].replace(['WI '],'WI')
aos_df['State'] = aos_df['State'].replace(['GA '],'GA')
aos_df['State'] = aos_df['State'].replace(['OH '],'OH')
aos_df['State'] = aos_df['State'].replace(['AL '],'AL')
aos_df['State'] = aos_df['State'].replace(['IL '],'IL')
aos_df['Type of LEA'] = aos_df['Type of LEA'].replace(['Police ','Poilice'],'Police')
aos_df['Type of LEA'] = aos_df['Type of LEA'].replace(['Security '],'Security')
aos_df['Type of LEA'] = aos_df['Type of LEA'].replace(['Wildlife '],'Wildlife')
aos_df['Type of LEA'] = aos_df['Type of LEA'].replace(['State Police '],'State Police')
aos_df['Type of Juris'] = aos_df['Type of Juris'].replace(['University '],'University')
aos_df['Type of Juris'] = aos_df['Type of Juris'].replace(['Municipal '],'Municipal')
aos_df['Type of Juris'] = aos_df['Type of Juris'].replace(['School '],'School')
#
# look at dataframe
aos_df.head()
#
# get number of columns and rows
aos_df.shape
#
# get column names
aos_df.columns
#
# get summary information of the dataset
aos_df.describe()<jupyter_output><empty_output><jupyter_text>Type of LEA<jupyter_code>#
# get frequency of types of LEA
aos_lea_freq = aos_df['Type of LEA'].value_counts()
#
# plot bar graph
aos_lea_freq.plot(kind='bar', figsize=(15,5), title='Frequency of Types of LEA', color='steelblue')
#
# plot bar graph without the two highest frequency LEAs for visability
aos_lea_freq_trimmed = aos_lea_freq[2:]
aos_lea_freq_trimmed.plot(kind='bar', figsize=(15,5), title='Frequency of Types of LEA (trimmed)', color='steelblue')<jupyter_output><empty_output><jupyter_text>Type of Jurisdiction<jupyter_code>#
# get frequency of types of juris
aos_juris_freq = aos_df['Type of Juris'].value_counts()
print(aos_juris_freq)
#
# plot bar graph
aos_juris_freq.plot(kind='bar', figsize=(10,5), title='Frequency of Types of Juris', color='mediumseagreen')<jupyter_output><empty_output><jupyter_text>Technology<jupyter_code>#
# get frequency of technology
aos_tech_freq = aos_df['Technology'].value_counts()
print(aos_tech_freq)
#
# plot bar graph
aos_tech_freq.plot(kind='bar', figsize=(8,5), title='Frequency of Technology', color='slateblue')<jupyter_output><empty_output><jupyter_text>Vendor<jupyter_code>#
# get frequency of vendors
aos_vendor_freq = aos_df['Vendor'].value_counts()
#
# filter vendors with frequencies no less than 5 times
aos_vendor_trimmed_freq = aos_vendor_freq[aos_vendor_freq >= 5]
#
print(aos_vendor_trimmed_freq)
#
# plot bar graph
aos_vendor_trimmed_freq.plot(kind='bar', figsize=(12,5), title='Frequency of Selected Vendor', color='yellowgreen')<jupyter_output><empty_output><jupyter_text>Analysis<jupyter_code># determine the frequency of states
aos_state_freq = aos_df['State'].value_counts()
# create bar graph
aos_state_freq.plot(kind='bar', figsize=(20,5), title='No. of Instances in States')<jupyter_output><empty_output><jupyter_text>The top three states are California, Florida, and New Jersey.Note that this data is not normalized; i.e. the size and population difference between the states are not accounted for.<jupyter_code>#collapse-output
aos_state_freq<jupyter_output><empty_output><jupyter_text>Explore LEA in States<jupyter_code>#
# find the frequency of the most frequent type of LEA
aos_df['Type of LEA'].value_counts()[0]
#
# find the most frequent type of LEA
aos_df['Type of LEA'].value_counts().reset_index()['index'][0]
#
# determine no. of cases with police LEA for each state in the US
state_leaPolice_count = aos_df[aos_df['Type of LEA'] == 'Police']['State'].value_counts()
#
# plot bar graph
state_leaPolice_count.plot(kind='bar', figsize=(12,5), title='Number of Cases with Police LEA Per State')<jupyter_output><empty_output><jupyter_text>Explore Type of Technology in States<jupyter_code>#
# create dataframe that stores the count of kinds of technology of each state
state_tech_count = aos_df.groupby('Technology')['State'].value_counts().reset_index(name='counts')
#
# create bubble plot (bubble size with "s")
plt.scatter('State', 'Technology', 
             s='counts', 
             alpha=0.8, 
             data=state_tech_count)
plt.xlabel("States", size=16)
plt.ylabel("Technology", size=16)
plt.title("No. of Usage of Technology in Each State", size=18)
plt.gcf().set_size_inches((20, 6))<jupyter_output><empty_output><jupyter_text>Some most common technology uses are ring/neighbors partnership, body-worn cameras, automated license plate readers, and drones. From this graph, we can also see that some technology is extensively used in particular areas. For example, face recognition is used heavily in Florida but less in other states. In New Jersey, most cases are done with body-worn cameras while other kinds of technology are not as commonly used. What percentage of the cases in New Jersey are with body-worn cameras instead of other technology?<jupyter_code>#
# isolate dataframe of technology in New Jersey
stateNJ_tech_count = state_tech_count[state_tech_count['State'] == 'NJ']
# find percentage of usage of body-worn camera in NJ
int(stateNJ_tech_count[stateNJ_tech_count['Technology'] == 'Body-worn Cameras']['counts'])/sum(stateNJ_tech_count['counts'])
#
# find the general percentage of body-worn cameras in all the states
aos_df['Technology'].value_counts()['Body-worn Cameras']/sum(aos_df['Technology'].value_counts())<jupyter_output><empty_output><jupyter_text>Nearly 3/4th (72%) of the cases in New Jersey use body-worn cameras. However, among all the states, the percentage of body-worn cameras is only 34%.<jupyter_code>#
# find percentage of body-worn cameras among all technology in each state
state_techBWC_percentage = state_tech_count[state_tech_count['Technology'] == 'Body-worn Cameras'].groupby('State').sum() / state_tech_count.groupby('State').sum()
#
# sort values from high to low
state_techBWC_percentage = state_techBWC_percentage.sort_values('counts', ascending=False)
# turn NaN to 0
state_techBWC_percentage['counts'] = state_techBWC_percentage['counts'].replace([float('NaN')],0)
#collapse-output
state_techBWC_percentage['counts']
#
# create bar plot
state_techBWC_percentage.plot(kind='bar', figsize=(12,6))<jupyter_output><empty_output><jupyter_text>Now, let's explore the top types of technology use. The top five are Body-worn Cameras, Ring/Neighbors Partnership, Drones, Automated License Plate Readers, and Camera Registry.What is the relationship of the number of technology use in each state?<jupyter_code>#
# create state list
state_list = list(state_tech_count['State'].unique())

# create empty lists for technology types
bwc_list = []
rnp_list = []
drone_list = []
alpr_list = []
cr_list = []

for i in state_list:
    try:
        rnp_list.append(state_tech_count.loc[(state_tech_count['State'] == i) & 
                                (state_tech_count['Technology'] == 'Ring/Neighbors Partnership'),'counts'].values[0])
    except:
        rnp_list.append(0)
    try:
        bwc_list.append(state_tech_count.loc[(state_tech_count['State'] == i) & 
                                (state_tech_count['Technology'] == 'Body-worn Cameras'),'counts'].values[0])
    except:
        bwc_list.append(0)
    try:
        drone_list.append(state_tech_count.loc[(state_tech_count['State'] == i) & 
                                (state_tech_count['Technology'] == 'Drones'),'counts'].values[0])
    except:
        drone_list.append(0)
    try:
        alpr_list.append(state_tech_count.loc[(state_tech_count['State'] == i) & 
                                (state_tech_count['Technology'] == 'Automated License Plate Readers'),'counts'].values[0])
    except:
        alpr_list.append(0)
    try:
        cr_list.append(state_tech_count.loc[(state_tech_count['State'] == i) & 
                                (state_tech_count['Technology'] == 'Camera Registry'),'counts'].values[0])
    except:
        cr_list.append(0)
state_tech_df = pd.DataFrame({'State':state_list, 'RNP':rnp_list, 'BWC':bwc_list,
                              'Drones':drone_list, 'ALPR':alpr_list, 'CR':cr_list})
state_tech_df.head()<jupyter_output><empty_output><jupyter_text>As we can notice, the most prevalent technology use in surveillance is camera-related technology. [Body-worn cameras](https://bja.ojp.gov/sites/g/files/xyckuh186/files/media/document/BWC_FAQs.pdf) are devices that record interactions between community members (e.g., the public, suspects, and victims) and officers for law enforcement purposes. Ring/Neighborhood partnership is an app-based surveillance service which uses doorbell cameras and other home surveillance devices to record video footages, alert, and report incidents. The service is provided by [Ring Inc.](https://ring.com/), which has signed agreements with more than 1,300 law enforcement agencies to promote its products.With them both being camera-related technology, do we see a similar pattern in the number of use cases in each state?What is the correlation between the number of cases of body-worn cameras and of ring/neighbor partnership in each state?<jupyter_code>import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score
np.corrcoef(state_tech_df['RNP'], state_tech_df['BWC'])[1][0]<jupyter_output><empty_output><jupyter_text>The correlation coefficient is approximately 0.64. Therefore, we have found a moderately positive correlation between the number of body-worn cameras and Ring's neighbor partnerships.<jupyter_code>#
# create scatter plot
state_tech_df.plot.scatter(x='RNP', y='BWC')
#
# plot linear regression
X = state_tech_df[['RNP']]
y = state_tech_df[['BWC']]
reg = LinearRegression().fit(X, y)
reg.coef_
reg.intercept_
ytrain = reg.intercept_ + reg.coef_ * X
plt.plot(X, y, 'ro', X, ytrain);<jupyter_output><empty_output><jupyter_text>Now, determine the linear correlation between each of the top types of technology.<jupyter_code>np.corrcoef(state_tech_df['RNP'], state_tech_df['Drones'])[1][0]
np.corrcoef(state_tech_df['RNP'], state_tech_df['ALPR'])[1][0]
np.corrcoef(state_tech_df['RNP'], state_tech_df['CR'])[1][0]
np.corrcoef(state_tech_df['BWC'], state_tech_df['Drones'])[1][0]
np.corrcoef(state_tech_df['BWC'], state_tech_df['ALPR'])[1][0]
np.corrcoef(state_tech_df['BWC'], state_tech_df['CR'])[1][0]
np.corrcoef(state_tech_df['Drones'], state_tech_df['ALPR'])[1][0]
np.corrcoef(state_tech_df['Drones'], state_tech_df['CR'])[1][0]
np.corrcoef(state_tech_df['ALPR'], state_tech_df['CR'])[1][0]<jupyter_output><empty_output><jupyter_text>The pair with the least correlation is body-worn cameras and automated license plate readers. The correlation coefficient is approximately 0.27.<jupyter_code>state_tech_df.plot.scatter(x='BWC', y='ALPR')
X = state_tech_df[['BWC']]
y = state_tech_df[['ALPR']]
reg = LinearRegression().fit(X, y)
reg.coef_
reg.intercept_
ytrain = reg.intercept_ + reg.coef_ * X
plt.plot(X, y, 'ro', X, ytrain);<jupyter_output><empty_output><jupyter_text>The pair with the greatest correlation is automated license plate readers and camera registry. The correlation coefficient is approximately 0.89.<jupyter_code>state_tech_df.plot.scatter(x='ALPR', y='CR')
X = state_tech_df[['ALPR']]
y = state_tech_df[['CR']]
reg = LinearRegression().fit(X, y)
reg.coef_
reg.intercept_
ytrain = reg.intercept_ + reg.coef_ * X
plt.plot(X, y, 'ro', X, ytrain);
#
# calculate mean squared error
mean_squared_error(y, ytrain)
#
# calculate R-squared value
r2_score(y, ytrain)<jupyter_output><empty_output><jupyter_text>Since most data points are rather clustered in the bottom left corner of the scatter plot, we can omit data points that are far from the cluster to better observe the pattern.<jupyter_code>state_tech_trimmed_df = state_tech_df[state_tech_df['ALPR'] < 200]
np.corrcoef(state_tech_trimmed_df['ALPR'], state_tech_trimmed_df['CR'])[1][0]<jupyter_output><empty_output><jupyter_text>The correlation coefficient has become smaller after the dataset is trimmed; however, there is still a positive relationship.<jupyter_code>reg = LinearRegression().fit(state_tech_trimmed_df[['ALPR']], state_tech_trimmed_df[['CR']])
ytrain = reg.intercept_ + reg.coef_ * state_tech_trimmed_df[['ALPR']]
plt.plot(state_tech_trimmed_df[['ALPR']], state_tech_trimmed_df[['CR']], 'ro', state_tech_trimmed_df[['ALPR']], ytrain);<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11202283267927221, 'fraction_numerical': 0.006659531454394102, 'mean_word_length': 6.141825902335457, 'pattern_counts': {'":': 0, '<': 78, '<?xml version=': 0, '>': 79, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '307297', 'license': 'Apache-2.0', 'n_tokens_mistral': 5340, 'n_tokens_neox': 5100, 'n_words': 1594, 'path': '_notebooks/2022-03-14-project.ipynb', 'repo_name': 'jhuang00/dh140-blog', 'size': '16818'}