<jupyter_start><jupyter_text>--- 
 
    Preparing the Police Activity data for analysis (Part - 1)
     


---<jupyter_code>!git clone https://github.com/mohd-faizy/CAREER-TRACK-Data-Scientist-with-Python.git<jupyter_output>Cloning into 'CAREER-TRACK-Data-Scientist-with-Python'...
remote: Enumerating objects: 440, done.[K
remote: Counting objects: 100% (440/440), done.[K
remote: Compressing objects: 100% (375/375), done.[K
remote: Total 1811 (delta 131), reused 367 (delta 60), pack-reused 1371[K
Receiving objects: 100% (1811/1811), 188.37 MiB | 18.41 MiB/s, done.
Resolving deltas: 100% (623/623), done.
Checking out files: 100% (802/802), done.<jupyter_text>__Change the current working directory__<jupyter_code># import os module 
import os 
   
# to specified path 
os.chdir('/content/CAREER-TRACK-Data-Scientist-with-Python/21_Analyzing Police Activity with pandas/_dataset') 
  
# varify the path using getcwd() 
cwd = os.getcwd() 
  
# print the current directory 
print("Current working directory is:", cwd) 
ls<jupyter_output>police.csv  weather.csv<jupyter_text>$\color{green}{\textbf{Dataset_1_police.csv:}}$ 

[Stanford Open Policing Project dataset](https://openpolicing.stanford.edu/)


On a typical day in the United States, police officers make more than 50,000 traffic stops. __THE STANFORD OPEN POLICING PROJECT__ gathers, analyse, and release the records from millions of traffic stops by law enforcement agencies across the __US__.


  
    
  


  __01 Examining the dataset__

Before beginning your analysis, it's important that you familiarize yourself with the dataset. In this exercise, we'll read the dataset into pandas, examine the first few rows, and then count the number of missing values.<jupyter_code># Import the pandas library as pd
import pandas as pd

# Read 'police.csv' into a DataFrame named ri
ri = pd.read_csv('police.csv')

# Examine the head of the DataFrame
ri.head()
ri.isna().sum()
ri.isnull()
# Count the number of missing values in each column
ri.isnull().sum()<jupyter_output><empty_output><jupyter_text>__02 Dropping columns__

Often, a DataFrame will contain columns that are not useful to our analysis. Such columns should be dropped from the DataFrame, to make it easier for us to focus on the remaining columns.

In this exercise, we'll drop the `'county_name'` column because it only contains missing values, and we'll drop the `'state'` column because all of the traffic stops took place in one state (__Rhode Island__).

Thus, these columns can be dropped because **they contain no useful information**.<jupyter_code># Examine the shape of the DataFrame
print(ri.shape)

# Drop the 'county_name' and 'state' columns
ri.drop(['county_name', 'state'], axis='columns', inplace=True)

# Examine the shape of the DataFrame (again)
print(ri.shape)<jupyter_output>(91741, 15)
(91741, 13)<jupyter_text>__03 Dropping rows__

When we know that a **specific column** will be **critical to our analysis**, and only a small fraction of rows are missing a value in that column, *it often makes sense to remove those rows from the dataset.*

the `'driver_gender'` column will be critical to many of our analyses. Because only a small fraction of rows are missing `'driver_gender'`, we'll drop those rows from the dataset.<jupyter_code># Count the number of missing values in each column
print(ri.isnull().sum())
# Drop all rows that are missing 'driver_gender'
ri.dropna(subset=['driver_gender'], inplace=True)

# Count the number of missing values in each column (again)
print(ri.isnull().sum())
# Examine the shape of the DataFrame
print(ri.shape)<jupyter_output>(86536, 13)<jupyter_text>__04 Finding an incorrect data type__<jupyter_code>ri.dtypes<jupyter_output><empty_output><jupyter_text>$\color{green}{\textbf{Note:}} $ $\Rightarrow$ `is_arrested` should have a data type of __bool__  __05 Fixing a data type__

- `is_arrested column` currently has the __object__ data type. 

- we have to change the data type to __bool__, which is the most suitable type for a column containing **True** and **False** values.

>Fixing the data type will enable us to use __mathematical operations__ on the `is_arrested` column that would not be possible otherwise.<jupyter_code># Examine the head of the 'is_arrested' column
print(ri.is_arrested.head())
# Change the data type of 'is_arrested' to 'bool'
ri['is_arrested'] = ri.is_arrested.astype('bool')

# Check the data type of 'is_arrested' 
print(ri.is_arrested.dtype)<jupyter_output>bool<jupyter_text>__06 Combining object columns (datetime format)__ - Currently, the date and time of each traffic stop are stored in separate object columns: **stop_date** and **stop_time**.

- we have to **combine** these two columns into a **single column**, and then convert it to **datetime format**. 

- This will be beneficial because unlike object columns, datetime columns provide date-based attributes that will make our analysis easier.<jupyter_code># Concatenate 'stop_date' and 'stop_time' (separated by a space)
combined = ri.stop_date.str.cat(ri.stop_time, sep=' ')

# Convert 'combined' to datetime format
ri['stop_datetime'] = pd.to_datetime(combined)

# Examine the data types of the DataFrame
print(ri.dtypes)<jupyter_output>stop_date                     object
stop_time                     object
driver_gender                 object
driver_race                   object
violation_raw                 object
violation                     object
search_conducted                bool
search_type                   object
stop_outcome                  object
is_arrested                     bool
stop_duration                 object
drugs_related_stop              bool
district                      object
stop_datetime         datetime64[ns]
dtype: object<jupyter_text>__07 Setting the index__

The last step is to set the `stop_datetime` column as the DataFrame's **index**. By **replacing** the **default index** with a **DatetimeIndex**, this will make it easier to analyze the dataset by date and time, which will come in handy later.<jupyter_code># Set 'stop_datetime' as the index
ri.set_index('stop_datetime', inplace=True)

# Examine the index
ri.index
# Examine the columns
ri.columns
ri.head()<jupyter_output><empty_output><jupyter_text>--- 
 
    Exploring the relationship between gender and policing (Part - 2)
     


---  __08 Examining traffic violations__ 

Before comparing the violations being committed by each gender, we should examine the **violations** committed by all drivers to get a baseline understanding of the data.

In this exercise, we'll count the **unique values** in the `violation` column, and then separately express those counts as **proportions**.<jupyter_code>ri['violation'].value_counts()
# dot method
# Count the unique values in 'violation'
ri.violation.value_counts()
# Counting unique values (2)
print(ri.violation.value_counts().sum()) 
print(ri.shape)
48423/86536 # Speeding `55.95%`
# Express the counts as proportions
ri.violation.value_counts(normalize=True)<jupyter_output><empty_output><jupyter_text>More than half of all violations are for **speeding**, followed by other moving violations and equipment violations.  __09 Comparing violations by gender__ The question we're trying to answer is whether male and female drivers tend to commit different types of traffic violations.

In this exercise, we'll first create a DataFrame for each gender, and then analyze the violations in each DataFrame separately.<jupyter_code># Create a DataFrame of male drivers
male = ri[ri.driver_gender == 'M']

# Create a DataFrame of female drivers
female = ri[ri.driver_gender == 'F']

# Compute the violations by male drivers (as proportions)
print(male.violation.value_counts(normalize=True))
# Compute the violations by female drivers (as proportions)
print(female.violation.value_counts(normalize=True))<jupyter_output>Speeding               0.658114
Moving violation       0.138218
Equipment              0.105199
Registration/plates    0.044418
Other                  0.029738
Seat belt              0.024312
Name: violation, dtype: float64<jupyter_text>__10 Filtering by multiple conditions__ Which one of these commands would filter the `ri` DataFrame to only include female drivers **who were stopped for a speeding violation**?<jupyter_code>female_and_speeding = ri[(ri.driver_gender == 'F') & (ri.violation == 'Speeding')]
female_and_speeding<jupyter_output><empty_output><jupyter_text>__11 Comparing speeding outcomes by gender__ When a driver is pulled over for `speeding`, **many people believe that gender has an impact on whether the driver will receive a ticket or a warning**. Can you find evidence of this in the dataset?

First, you'll create two DataFrames of drivers who were stopped for speeding: one containing females and the other containing males.

Then, for each gender, you'll use the `stop_outcome` column to calculate what percentage of stops resulted in a "Citation" (meaning a ticket) versus a "Warning".<jupyter_code># Create a DataFrame of female drivers stopped for speeding
female_and_speeding = ri[(ri.driver_gender == 'F') & (ri.violation == 'Speeding')]

# Compute the stop outcomes for female drivers (as proportions)
print(female_and_speeding.stop_outcome.value_counts(normalize=True))
# Create a DataFrame of male drivers stopped for speeding
male_and_speeding = ri[(ri.driver_gender == 'M') & (ri.violation == 'Speeding')]

# Compute the stop outcomes for male drivers (as proportions)
print(male_and_speeding.stop_outcome.value_counts(normalize=True))<jupyter_output>Citation            0.944595
Warning             0.036184
Arrest Driver       0.015895
Arrest Passenger    0.001281
No Action           0.001068
N/D                 0.000976
Name: stop_outcome, dtype: float64<jupyter_text>$\color{red}{\textbf{Interpretation:}}$

>The numbers are similar for **males** and **females**: about **95%** of stops for speeding result in a ticket. Thus, __the data fails to show that gender has an impact on who gets a ticket for speeding__.<jupyter_code># Filtering by multiple conditions (1)
female = ri[ri.driver_gender == 'F']
female.shape
# Filtering by multiple conditions (2)
# Only includes female drivers who were arrested
female_and_arrested = ri[(ri.driver_gender == 'F') &(ri.is_arrested == True)]
female_and_arrested.shape
# Filtering by multiple conditions (3)
female_or_arrested = ri[(ri.driver_gender == 'F') | (ri.is_arrested == True)]
female_or_arrested.shape<jupyter_output><empty_output><jupyter_text>- Includes all females
- Includes all drivers who were arrested  __12 Comparing stop outcomes for two groups__<jupyter_code># driver race --> White
white = ri[ri.driver_race == 'White']
white.stop_outcome.value_counts(normalize=True)
# driver race --> Black
black = ri[ri.driver_race =='Black']
black.stop_outcome.value_counts(normalize=True)
# driver race --> Asian
asian = ri[ri.driver_race =='Asian']
asian.stop_outcome.value_counts(normalize=True)<jupyter_output><empty_output><jupyter_text>__13 Does gender affect whose vehicle is searched?__ **Mean** of **Boolean Series** represents percentage of True values<jupyter_code>ri.isnull().sum()
# Taking the mean of a Boolean Series
print(ri.is_arrested.value_counts(normalize=True))
print(ri.is_arrested.mean())
print(ri.is_arrested.dtype)<jupyter_output>False    0.964431
True     0.035569
Name: is_arrested, dtype: float64
0.0355690117407784
bool<jupyter_text>__Comparing groups using groupby (1)__<jupyter_code># Study the arrest rate by police district
print(ri.district.unique())

# Mean
print(ri[ri.district == 'Zone K1'].is_arrested.mean())<jupyter_output>['Zone X4' 'Zone K3' 'Zone X1' 'Zone X3' 'Zone K1' 'Zone K2']
0.024349083895853423<jupyter_text>__Comparing groups using groupby (2)__<jupyter_code>ri[ri.district == 'Zone K2'].is_arrested.mean()
ri.groupby('district').is_arrested.mean()<jupyter_output><empty_output><jupyter_text>__Grouping by multiple categories__<jupyter_code>ri.groupby(['district', 'driver_gender']).is_arrested.mean()
ri.groupby(['driver_gender', 'district']).is_arrested.mean()<jupyter_output><empty_output><jupyter_text>__14 Calculating the search rate__

During a traffic stop, the police officer sometimes conducts a search of the vehicle.

In this exercise, you'll calculate the percentage of all stops in the ri DataFrame that result in a vehicle search, also known as the search rate.<jupyter_code># Check the data type of 'search_conducted'
print(ri.search_conducted.dtype)

# Calculate the search rate by counting the values
print(ri.search_conducted.value_counts(normalize=True))

# Calculate the search rate by taking the mean
print(ri.search_conducted.mean())<jupyter_output>bool
False    0.961785
True     0.038215
Name: search_conducted, dtype: float64
0.0382153092354627<jupyter_text>$\color{red}{\textbf{Interpretation:}}$

>It looks like the search rate is about __3.8%__.  __Comparing search rates by gender__ Remember that the vehicle **search rate **across all stops is about __3.8%.__

First, we'll filter the DataFrame by gender and calculate the **search rate** for each group separately. Then, you'll perform the same calculation for both genders at once using a `.groupby()`. __Instructions:__

- Filter the DataFrame to only include female drivers, and then calculate the search rate by taking the mean of search_conducted.

- Filter the DataFrame to only include male drivers, and then repeat the search rate calculation.

- Group by driver gender to calculate the search rate for both groups simultaneously. (It should match the previous results.)<jupyter_code># Calculate the search rate for female drivers
print(ri[ri.driver_gender == 'F'].search_conducted.mean())
# Calculate the search rate for male drivers
print(ri[ri.driver_gender == 'M'].search_conducted.mean())
# Calculate the search rate for both groups simultaneously
print(ri.groupby('driver_gender').search_conducted.mean())<jupyter_output>driver_gender
F    0.019181
M    0.045426
Name: search_conducted, dtype: float64<jupyter_text>$\color{red}{\textbf{Interpretation:}}$

>Male drivers are searched more than twice as often as female drivers. Why might this be?
  __15 Adding a second factor to the analysis__ Even though the **search rate** for **males is much higher than for females**, *it's possible that the difference is mostly due to a second factor.*

>For example, we might **hypothesize** that **the search rate varies by violation type**, and the difference in search rate between males and females is because they tend to commit different violations.

we can test this hypothesis by examining the **search rate** for **each combination of gender and violation**. If the hypothesis was true, you would find that males and females are searched at about the same rate for each violation. Find out below if that's the case! __Instructions__

- Use a `.groupby()` to calculate the search rate for each combination of gender and violation. Are males and females searched at about the same rate for each violation?

- Reverse the ordering to group by violation before gender. The results may be easier to compare when presented this way.<jupyter_code># Calculate the search rate for each combination of gender and violation
ri.groupby(['driver_gender', 'violation']).search_conducted.mean()
# Reverse the ordering to group by violation before gender
ri.groupby(['violation', 'driver_gender']).search_conducted.mean()<jupyter_output><empty_output><jupyter_text>$\color{red}{\textbf{Interpretation:}}$

>For all types of violations, the search rate is higher for males than for females, disproving our hypothesis  __16 Does gender affect who is frisked during a search?__<jupyter_code>ri.search_conducted.value_counts()<jupyter_output><empty_output><jupyter_text>`.value_counts()`
__excludes missing
values by default__<jupyter_code>ri.search_type.value_counts(dropna=False)<jupyter_output><empty_output><jupyter_text>- `dropna=False`
 **displays missing
values** **Examining the search types**<jupyter_code>ri.search_type.value_counts()<jupyter_output><empty_output><jupyter_text>- Multiple values are separated by commas.

- 219 searches in which **"Inventory"** was the only search type.

- Locate **"Inventory"** among multiple search types. __Searching for a string (1)__<jupyter_code>ri['inventory'] = ri.search_type.str.contains('Inventory', na=False)<jupyter_output><empty_output><jupyter_text>- `str.contains()` returns
    - True if string is found
    - False if not found.

- `na=False` returns `False` when it ,finds a missing value __Searching for a string (2)__<jupyter_code>ri.inventory.dtype<jupyter_output><empty_output><jupyter_text>**True** means inventory was done, **False** means it was not<jupyter_code>ri.inventory.sum()<jupyter_output><empty_output><jupyter_text>__Calculating the inventory rate__<jupyter_code>ri.inventory.mean()<jupyter_output><empty_output><jupyter_text>**0.5%** of all traffic stops resulted in an inventory.<jupyter_code>searched = ri[ri.search_conducted == True]
searched.inventory.mean()<jupyter_output><empty_output><jupyter_text>__13.3% of searches included an inventory__  __17 Counting protective frisks__ During a vehicle search, the police officer may pat down the driver to check if they have a weapon. This is known as a "protective frisk."

In this exercise, you'll first check to see how many times "Protective Frisk" was the only search type. Then, you'll use a string method to locate all instances in which the driver was frisked. __Instructions__

- Count the `search_type` values in the `ri` DataFrame to see how many times "Protective Frisk" was the only search type.

- Create a new column, `frisk`, that is `True` if search_type contains the string "Protective Frisk" and `False` otherwise.

- Check the data type of `frisk` to confirm that it's a Boolean Series.

- Take the sum of `frisk` to count the total number of frisks.<jupyter_code># Count the 'search_type' values
print(ri.search_type.value_counts())

# Check if 'search_type' contains the string 'Protective Frisk'
ri['frisk'] = ri.search_type.str.contains('Protective Frisk', na=False)

# Check the data type of 'frisk'
print(ri['frisk'].dtype)

# Take the sum of 'frisk'
print(ri['frisk'].sum())<jupyter_output>Incident to Arrest                                          1290
Probable Cause                                               924
Inventory                                                    219
Reasonable Suspicion                                         214
Protective Frisk                                             164
Incident to Arrest,Inventory                                 123
Incident to Arrest,Probable Cause                            100
Probable Cause,Reasonable Suspicion                           54
Incident to Arrest,Inventory,Probable Cause                   35
Probable Cause,Protective Frisk                               35
Incident to Arrest,Protective Frisk                           33
Inventory,Probable Cause                                      25
Protective Frisk,Reasonable Suspicion                         19
Incident to Arrest,Inventory,Protective Frisk                 18
Incident to Arrest,Probable Cause,Protective Frisk            13
Inventory,Protective Fris[...]<jupyter_text>$\color{red}{\textbf{Interpretation:}}$

>It looks like there were **303 drivers** who were **frisked**. Next, you'll examine whether gender affects who is frisked.  __18 Comparing frisk rates by gender__ In this exercise, we'll compare the rates at which **female** and **male** drivers are **frisked during a search**.

>Are males frisked more often than females, perhaps because police officers consider them to be higher risk?

Before doing any calculations, it's important to filter the DataFrame to only include the relevant subset of data, namely stops in which a search was conducted. __Instructions__

- Create a DataFrame, **searched**, that only contains rows in which `search_conducted` is `True`.

- Take the mean of the `frisk` column to find out what percentage of searches included a frisk.

- Calculate the frisk rate for each gender using a `.groupby()`.<jupyter_code># Create a DataFrame of stops in which a search was conducted
searched = ri[ri.search_conducted == True]

# Calculate the overall frisk rate by taking the mean of 'frisk'
print(searched.frisk.mean())

# Calculate the frisk rate for each gender
print(searched.groupby('driver_gender').frisk.mean())<jupyter_output>0.09162382824312065
driver_gender
F    0.074561
M    0.094353
Name: frisk, dtype: float64<jupyter_text>$\color{red}{\textbf{Interpretation:}}$

>The **frisk rate** is **higher for males than for females**, though we **can't** conclude that this difference is caused by the driver's gender. --- 
 
    Does time of day affect arrest rate?(Part - 3)
     


---  __19 Calculating the hourly arrest rate__ When a police officer stops a driver, a small percentage of those stops ends in an arrest. This is known as the arrest rate. In this exercise, you'll find out whether the arrest rate varies by time of day.

First, you'll calculate the arrest rate across all stops in the **ri** DataFrame. Then, you'll calculate the hourly arrest rate by using the **hour** attribute of the index. The **hour** ranges from 0 to 23, in which:

- *0 = midnight*
- *12 = noon*
- *23 = 11 PM* __Instructions__

- Take the mean of the `is_arrested` column to calculate the overall arrest rate.

- Group by the `hour` attribute of the DataFrame index to calculate the hourly arrest rate.

- Save the hourly arrest rate Series as a new object, `hourly_arrest_rate`.<jupyter_code># Calculate the overall arrest rate
print(ri.is_arrested.mean())

# Calculate the hourly arrest rate
print(ri.groupby(ri.index.hour).is_arrested.mean())

# Save the hourly arrest rate
hourly_arrest_rate = ri.groupby(ri.index.hour).is_arrested.mean()<jupyter_output>0.0355690117407784
stop_datetime
0     0.051431
1     0.064932
2     0.060798
3     0.060549
4     0.048000
5     0.042781
6     0.013813
7     0.013032
8     0.021854
9     0.025206
10    0.028213
11    0.028897
12    0.037399
13    0.030776
14    0.030605
15    0.030679
16    0.035281
17    0.040619
18    0.038204
19    0.032245
20    0.038107
21    0.064541
22    0.048666
23    0.047592
Name: is_arrested, dtype: float64<jupyter_text>__20 Plotting the hourly arrest rate__ In this exercise, we'll create a line plot from the `hourly_arrest_rate` object. A line plot is appropriate in this case because you're showing how a quantity changes over time.

This plot should help you to spot some trends that may not have been obvious when examining the raw numbers!<jupyter_code># Import matplotlib.pyplot as plt
import matplotlib.pyplot as plt

# Create a line plot of 'hourly_arrest_rate'
hourly_arrest_rate.plot()

# Add the xlabel, ylabel, and title
plt.xlabel('Hour')
plt.ylabel('Arrest Rate')
plt.title('Arrest Rate by Time of Day')

# Display the plot
plt.show()<jupyter_output><empty_output><jupyter_text>__21 Plotting drug-related stops__

__Are drug-related
stops on the rise?__ In a small portion of traffic stops, drugs are found in the vehicle during a search. In this exercise, you'll assess whether these drug-related stops are becoming more common over time.

The Boolean column `drugs_related_stop` indicates whether drugs were found during a given stop. You'll calculate the annual drug rate by resampling this column, and then you'll use a line plot to visualize how the rate has changed over time. __Instructions__

- Calculate the annual rate of drug-related stops by resampling the `drugs_related_stop` column (on the `'A'` frequency) and taking the mean.

- Save the annual drug rate Series as a new object, `annual_drug_rate`.

- Create a line plot of `annual_drug_rate` using the `.plot()` method.

- Display the plot using the `.show()` function.<jupyter_code># Calculate the annual rate of drug-related stops
# resampling `drugs_related_stop` represented by 'A' for Annual rate
# & chain with mean at end
print(ri.drugs_related_stop.resample('A').mean())

# Save the annual rate of drug-related stops
annual_drug_rate = ri.drugs_related_stop.resample('A').mean()

# Create a line plot of 'annual_drug_rate'
annual_drug_rate.plot()

# Display the plot
plt.xlabel('year')
plt.ylabel('Annual drug rate')
plt.title('drugs_related_stop')
plt.show()<jupyter_output>stop_datetime
2005-12-31    0.006501
2006-12-31    0.007258
2007-12-31    0.007970
2008-12-31    0.007505
2009-12-31    0.009889
2010-12-31    0.010081
2011-12-31    0.009731
2012-12-31    0.009921
2013-12-31    0.013094
2014-12-31    0.013826
2015-12-31    0.012266
Freq: A-DEC, Name: drugs_related_stop, dtype: float64<jupyter_text>**Resampling**
 
Resampling is when we change the frequncy of our __time-series__ obervation.

Most commonly used time series frequency are –
- **W** : weekly frequency
- **M** : month end frequency
- **SM** : semi-month end frequency -(15th and end of month)
- **Q** : quarter end frequency
- **A** : Annual

A time series is a series of data points indexed (or listed or graphed) in time order. Most commonly, a time series is a sequence taken at successive equally spaced points in time. 

It is a Convenience method for **frequency conversion** and resampling of **time serie**s. Object must have a datetime-like index (**DatetimeIndex**, **PeriodIndex**, or **TimedeltaIndex**), or pass datetime-like values to the on or level keyword.  __22 Comparing drug and search rates__ >From the above plot its evident that rate of `drug-related` stops increased significantly between **2005** and **2015**. 

we might **hypothesize** that the **rate of vehicle searches** was also **increasing**, which would have led to an **increase** in **drug-related stops** even if more drivers were not carrying drugs.

You can test this **hypothesis** by calculating the **annual search rate**, and then plotting it against the **annual drug rate**.

*If the hypothesis is true, then you'll see both rates increasing over time.*<jupyter_code># Calculate and save the annual search rate
annual_search_rate = ri.search_conducted.resample('A').mean()

# Concatenate 'annual_drug_rate' and 'annual_search_rate'
annual = pd.concat([annual_drug_rate, annual_search_rate], axis='columns')

# Create subplots from 'annual'
annual.plot(subplots=True)

# Display the subplots
plt.show()<jupyter_output><empty_output><jupyter_text>$\color{red}{\textbf{Interpretation:}}$

>The rate of **drug-related** stops **increased** even though the **search rate decreased**, disproving our hypothesis  __23 What violations are caught in each district?__<jupyter_code># Computing a frequency table
table = pd.crosstab(ri.driver_race, ri.driver_gender)
table
# Driver Asian and Gender is Female 
ri[(ri.driver_race == 'Asian') & (ri.driver_gender == 'F')].shape
# Selecting a DataFrame slice
table1 = table.loc['Asian':'Hispanic']
table1
# Line Plot
table1.plot()
plt.show()<jupyter_output><empty_output><jupyter_text>**line plot** is not appropriate in this case because it implies a change in time along the **x-axis**, whereas the **x-axis** actually represents **three** distinct categories.<jupyter_code># Creating a bar plot
table.plot(kind='bar')
plt.show()
# Creating a bar plot
table1.plot(kind='bar')
plt.show()
# Stacking the bars for table1
table1.plot(kind='bar', stacked=True)
plt.show()<jupyter_output><empty_output><jupyter_text>__Tallying violations by district__

The state of Rhode Island is broken into **six police districts**, also known as zones. How do the zones compare in terms of what violations are caught by police?

In this exercise, we'll create a **frequency table** to determine **how many violations of each type took place in each of the six zones**. Then, you'll filter the table to focus on the `"K" zones`, which you'll examine further in the next exercise. __Instructions:__

- Create a frequency table from the `ri` DataFrame's `district` and `violation` columns using the `pd.crosstab()` function.

- Save the frequency table as a new object, `all_zones`.

- Select rows `'Zone K1'` through `'Zone K3'` from `all_zones` using the `.loc[]` accessor.
Save the smaller table as a new object, `k_zones`.<jupyter_code># Create a frequency table of districts and violations
print(pd.crosstab(ri.district, ri.violation))

# Save the frequency table as 'all_zones'
all_zones = pd.crosstab(ri.district, ri.violation)

# Select rows 'Zone K1' through 'Zone K3'
print(all_zones.loc['Zone K1':'Zone K3'])

# Save the smaller table as 'k_zones'
k_zones = all_zones.loc['Zone K1':'Zone K3']<jupyter_output>violation  Equipment  Moving violation  ...  Seat belt  Speeding
district                                ...                     
Zone K1          672              1254  ...          0      5960
Zone K2         2061              2962  ...        481     10448
Zone K3         2302              2898  ...        638     12322
Zone X1          296               671  ...         74      1119
Zone X3         2049              3086  ...        820      8779
Zone X4         3541              5353  ...        843      9795

[6 rows x 6 columns]
violation  Equipment  Moving violation  ...  Seat belt  Speeding
district                                ...                     
Zone K1          672              1254  ...          0      5960
Zone K2         2061              2962  ...        481     10448
Zone K3         2302              2898  ...        638     12322

[3 rows x 6 columns]<jupyter_text>__24 Plotting violations by district__ we've created a frequency table focused on the `"K"` zones, **visualize** the data to help you compare what **violations** are being caught in each zone.

>**First** we'll create a **bar plot**, which is an appropriate plot type since we're **comparing categorical data**. 

>Then we'll create a **stacked bar** plot in order to get a slightly different look at the data to find which plot is more insightful?<jupyter_code># Creating a bar plot
k_zones.plot(kind='bar', figsize=(12, 7))

# Display the plot
plt.show()
# Create a stacked bar plot of 'k_zones'
k_zones.plot(kind='bar', stacked=True, figsize=(12, 7))

# Display the plot
plt.show()<jupyter_output><empty_output><jupyter_text>The vast majority of traffic stops in **Zone K1** are for speeding, and **Zones K2** and **K3** are remarkably similar to one another in terms of violations.  __25 Converting stop durations to numbers__ In the traffic stops dataset, the `stop_duration` column tells you approximately how long the driver was detained by the officer. Unfortunately, the durations are stored as strings, such as `'0-15 Min'`. How can you make this data easier to analyze?

In this exercise, you'll convert the stop durations to integers. Because the precise durations are not available, you'll have to estimate the numbers using reasonable values:

- Convert `'0-15 Min'` to `8`
- Convert `'16-30 Min'` to `23`
- Convert `'30+ Min'` to `45` $\color{red}{\textbf{Note:}}$

>`astype()` method to convert strings to numbers or Booleans. 

However, `astype()` only works when pandas can infer how the conversion should be done, and that's not the case here.<jupyter_code># Print the unique values in 'stop_duration'
print(ri.stop_duration.unique())

# Create a dictionary that maps strings to integers
mapping = {'0-15 Min':8, 
            '16-30 Min':23,
            '30+ Min':45
            }

# Convert the 'stop_duration' strings to integers using the 'mapping'
ri['stop_minutes'] = ri.stop_duration.map(mapping)

# Print the unique values in 'stop_minutes'
print(ri.stop_minutes.unique())<jupyter_output>['0-15 Min' '16-30 Min' '30+ Min']
[ 8 23 45]<jupyter_text>__26 Plotting stop length__

>If we were **stopped** for a particular violation, how long might we expect to be detained?

In this exercise, we'll visualize the average length of time drivers are stopped for each type of violation. Rather than using the `violation` column in this exercise, you'll use `violation_raw` since it contains more detailed descriptions of the violations. __Instructions__

- For each value in the ri DataFrame's **violation_raw** column, calculate the mean number of **stop_minutes** that a driver is detained.

- Save the resulting Series as a new object, **stop_length**.

- Sort **stop_length** by its values, and then visualize it using a horizontal bar plot.

- Display the plot.<jupyter_code># Calculate the mean 'stop_minutes' for each value in 'violation_raw'
print(ri.groupby('violation_raw').stop_minutes.mean())

# Save the resulting Series as 'stop_length'
stop_length = ri.groupby('violation_raw').stop_minutes.mean()

# Sort 'stop_length' by its values and create a horizontal bar plot
stop_length.sort_values().plot(kind='barh')

# Display the plot
plt.show()<jupyter_output>violation_raw
APB                                 17.967033
Call for Service                    22.124371
Equipment/Inspection Violation      11.445655
Motorist Assist/Courtesy            17.741463
Other Traffic Violation             13.844490
Registration Violation              13.736970
Seatbelt Violation                   9.662815
Special Detail/Directed Patrol      15.123632
Speeding                            10.581562
Suspicious Person                   14.910714
Violation of City/Town Ordinance    13.254144
Warrant                             24.055556
Name: stop_minutes, dtype: float64<jupyter_text>__Calculating the search rate__

- **Visualizing** how often searches were done after each **violation** type<jupyter_code>search_rate = ri.groupby('violation').search_conducted.mean()
search_rate.sort_values().plot(kind='barh')
plt.show()<jupyter_output><empty_output><jupyter_text>--- 
 
    Analyzing the effect of weather on policing(Part - 4)
     


---<jupyter_code>ls<jupyter_output>police.csv  weather.csv<jupyter_text>$\color{green}{\textbf{Dataset_2_weather.csv:}}$ 

[National Centers for Environmental Information](https://www.ncei.noaa.gov/)<jupyter_code># Import the pandas library as pd
import pandas as pd

# Read 'weather.csv' into a DataFrame named 'weather'
weather = pd.read_csv('weather.csv')

# Examine the head of the DataFrame
weather.head()<jupyter_output><empty_output><jupyter_text>- __TAVG , TMIN , TMAX__ : Temperature
- __AWND , WSF2 :__ Wind speed
- __WT01 ... WT22 :__ Bad weather conditions __The difference between isnull () and isna ()?__

`.isnull()` and `isna()` are the same functions (an alias), so we can choose either one. `df.isnull().sum()` or `df.isna().sum()`.

Finding which index (or row number) contains missing values can be done analogously to the previous example, simply by adding axis=1.<jupyter_code># Count the number of missing values in each column
print(weather.isnull().sum())

# Columns
print(weather.columns)

# Shape
print(weather.shape)<jupyter_output>STATION       0
DATE          0
TAVG       2800
TMIN          0
TMAX          0
AWND          0
WSF2          0
WT01       2250
WT02       3796
WT03       3793
WT04       3900
WT05       3657
WT06       3992
WT07       3938
WT08       3613
WT09       3948
WT10       4015
WT11       4016
WT13       2842
WT14       3442
WT15       4011
WT16       2691
WT17       4005
WT18       3672
WT19       4013
WT21       3999
WT22       3985
dtype: int64
Index(['STATION', 'DATE', 'TAVG', 'TMIN', 'TMAX', 'AWND', 'WSF2', 'WT01',
       'WT02', 'WT03', 'WT04', 'WT05', 'WT06', 'WT07', 'WT08', 'WT09', 'WT10',
       'WT11', 'WT13', 'WT14', 'WT15', 'WT16', 'WT17', 'WT18', 'WT19', 'WT21',
       'WT22'],
      dtype='object')
(4017, 27)<jupyter_text>__27 Plotting the temperature__<jupyter_code># Describe the temperature columns
print(weather[['TMIN', 'TAVG', 'TMAX']].describe())

# Create a box plot of the temperature columns
weather[['TMIN', 'TAVG', 'TMAX']].plot(kind='box')

# Display the plot
plt.show()<jupyter_output>TMIN         TAVG         TMAX
count  4017.000000  1217.000000  4017.000000
mean     43.484441    52.493016    61.268608
std      17.020298    17.830714    18.199517
min      -5.000000     6.000000    15.000000
25%      30.000000    39.000000    47.000000
50%      44.000000    54.000000    62.000000
75%      58.000000    68.000000    77.000000
max      77.000000    86.000000   102.000000<jupyter_text>__Examining the wind speed__<jupyter_code>print(weather[['AWND', 'WSF2']].head())
print(weather[['AWND', 'WSF2']].describe())

# Creating a box plot
weather[['AWND', 'WSF2']].plot(kind='box', figsize=(7, 7))
plt.figure()
plt.show()
# Creating a histogram (1)
weather['WDIFF'] = weather.WSF2 - weather.AWND
weather.WDIFF.plot(kind='hist')
plt.show()
# Creating a histogram (2)
weather.WDIFF.plot(kind='hist', bins=20)
plt.show()<jupyter_output><empty_output><jupyter_text>__28 Plotting the temperature difference__<jupyter_code># Create a 'TDIFF' column that represents temperature difference
weather['TDIFF'] = weather.TMAX - weather.TMIN

# Describe the 'TDIFF' column
print(weather.TDIFF.describe())

# Create a histogram with 20 bins to visualize 'TDIFF'
weather.TDIFF.plot(kind='hist', bins=20)

# Display the plot
plt.show()<jupyter_output>count    4017.000000
mean       17.784167
std         6.350720
min         2.000000
25%        14.000000
50%        18.000000
75%        22.000000
max        43.000000
Name: TDIFF, dtype: float64<jupyter_text>__29 Categorizing the weather__<jupyter_code># Printing the shape and columns
print(weather.shape)
print(weather.columns)
# Selecting a DataFrame slice
temp = weather.loc[:, 'TAVG':'TMAX']
print(temp.shape)
print(temp.columns)<jupyter_output>(4017, 3)
Index(['TAVG', 'TMIN', 'TMAX'], dtype='object')<jupyter_text>__Mapping one set of values to another__<jupyter_code>print(ri.stop_duration.unique())

mapping = {'0-15 Min':'short',
           '16-30 Min':'medium',
           '30+ Min':'long'
           }

ri['stop_length'] = ri.stop_duration.map(mapping)
print(ri.stop_length.dtype)
print(ri.stop_length.unique())<jupyter_output>['0-15 Min' '16-30 Min' '30+ Min']
object
['short' 'medium' 'long']<jupyter_text>- Category type stores the data more efficiently.
- Allows you to specify a logical order for the categories.<jupyter_code>ri.stop_length.memory_usage(deep=True)<jupyter_output><empty_output><jupyter_text>`pandas.DataFrame.memory_usage` __function__

Return the **memory usage** of each column in bytes.

- The **memory usage** can optionally include the contribution of the **index** and **elements** of object dtype.

- `deep` bool, __default False__
If **True**, introspect the data deeply by interrogating object dtypes for system-level memory consumption, and include it in the returned values.
 __Changing data type from object to category__<jupyter_code>from pandas.api.types import CategoricalDtype 

# Changing data type from object to category
cats = ['short', 'medium', 'long']

cat_type = CategoricalDtype(categories=cats, ordered=True)

ri['stop_length'] = ri['stop_length'].astype(cat_type)

print(ri.stop_length.memory_usage(deep=True))
# Using ordered categories
print(ri.stop_length.value_counts())
ri[ri.stop_length > 'short'].shape
ri.groupby('stop_length').is_arrested.mean()<jupyter_output><empty_output><jupyter_text>__30 Counting bad weather conditions__ The `weather` DataFrame contains 20 columns that start with `'WT'`, each of which represents a bad weather condition. For example:

- `WT05` indicates "Hail"
- `WT11` indicates "High or damaging winds"
- `WT17` indicates "Freezing rain"

For every row in the dataset, each `WT` column contains either a 1 (meaning the condition was present that day) or `NaN` (meaning the condition was not present).

In this exercise, you'll quantify "how bad" the weather was each day by counting the number of `1` values in each row.<jupyter_code># Copy 'WT01' through 'WT22' to a new DataFrame
WT = weather.loc[:, 'WT01':'WT22']

# Calculate the sum of each row in 'WT'
weather['bad_conditions'] = WT.sum(axis='columns')

# Replace missing values in 'bad_conditions' with '0'
weather['bad_conditions'] = weather.bad_conditions.fillna(0).astype('int')

# Create a histogram to visualize 'bad_conditions'
weather.bad_conditions.plot(kind='hist')

# Display the plot
plt.show()<jupyter_output><empty_output><jupyter_text>$\color{red}{\textbf{Interpretation:}}$

>It looks like many days didn't have any bad weather conditions, and only a small portion of days had more than four bad weather conditions.  __31 Rating the weather conditions__

In the previous exercise, we have counted the number of bad weather conditions each day. In this exercise, we'll use the counts to create a rating system for the weather.

The counts range from 0 to 9, and should be converted to ratings as follows:

- Convert 0 to `'good'`
- Convert 1 through 4 to `'bad'`
- Convert 5 through 9 to `'worse'`<jupyter_code># Count the unique values in 'bad_conditions' and sort the index
print(weather.bad_conditions.value_counts().sort_index())

# Create a dictionary that maps integers to strings
mapping = {0:'good',
           1:'bad', 2:'bad', 3:'bad', 4:'bad', 
           5:'worse', 6:'worse', 7:'worse', 8:'worse', 9:'worse'}

# Convert the 'bad_conditions' integers to strings using the 'mapping'
weather['rating'] = weather.bad_conditions.map(mapping)

# Count the unique values in 'rating'
print(weather.rating.value_counts())<jupyter_output>0    1749
1     613
2     367
3     380
4     476
5     282
6     101
7      41
8       4
9       4
Name: bad_conditions, dtype: int64
bad      1836
good     1749
worse     432
Name: rating, dtype: int64<jupyter_text>This rating system should make the weather condition data easier to understand.  __32 Changing the data type to category__ Since the `rating` column only has a few possible values, we have to change its **data type to category** in order to store the data more efficiently & then specify a logical order for the categories.<jupyter_code>from pandas.api.types import CategoricalDtype 

# Create a list of weather ratings in logical order
cats = ['good', 'bad', 'worse']

# Change the data type of 'rating' to category
cat_types= CategoricalDtype(categories=cats, ordered=True)
weather['rating'] = weather['rating'].astype(cat_types)

# Examine the head of 'rating'
print(weather.rating.head())
print(weather.rating.memory_usage(deep=True))
# Using ordered categories
print(weather.rating.value_counts())
weather.rating.value_counts().plot(kind='bar')
plt.show()<jupyter_output><empty_output><jupyter_text>__33 Merging datasets(Preparing the DataFrames)__ In this exercise, we'll prepare the traffic stop and weather rating DataFrames so that they're ready to be merged:

1. With the `ri` DataFrame, you'll move the `stop_datetime` index to a column since the index will be lost during the merge.

2. With the `weather` DataFrame, you'll select the `DATE` and `rating` columns and put them in a new DataFrame. __Instructions__

- Reset the index of the ri DataFrame.

- Examine the head of ri to verify that `stop_datetime` is now a DataFrame column, and the index is now the default integer index.

- Create a new DataFrame named `weather_rating` that contains only the `DATE` and rating columns from the weather DataFrame.

- Examine the head of `weather_rating` to verify that it contains the proper columns.<jupyter_code># Reset the index of 'ri'
ri.reset_index(inplace=True)

# Examine the head of 'ri'
print(ri.head())

# Create a DataFrame from the 'DATE' and 'rating' columns
weather_rating = weather[['DATE','rating']]

# Examine the head of 'weather_rating'
print(weather_rating.head())<jupyter_output>stop_datetime   stop_date stop_time  ...  frisk stop_minutes stop_length
0 2005-01-04 12:55:00  2005-01-04     12:55  ...  False            8       short
1 2005-01-23 23:15:00  2005-01-23     23:15  ...  False            8       short
2 2005-02-17 04:15:00  2005-02-17     04:15  ...  False            8       short
3 2005-02-20 17:15:00  2005-02-20     17:15  ...  False           23      medium
4 2005-02-24 01:20:00  2005-02-24     01:20  ...  False            8       short

[5 rows x 18 columns]
         DATE rating
0  2005-01-01    bad
1  2005-01-02    bad
2  2005-01-03    bad
3  2005-01-04    bad
4  2005-01-05    bad<jupyter_text>The **ri** and **weather_rating** DataFrames are now ready to be **merged**. __Merging the DataFrames__

In this exercise, we'll merge the `ri` and `weather_rating` DataFrames into a new DataFrame, ri_weather.

The DataFrames will be joined using the `stop_date` column from `ri` and the `DATE` column from `weather_rating`. Thankfully the date formatting matches exactly, which is not always the case!

Once the merge is complete, you'll set `stop_datetime` as the index, which is the column you saved in the previous exercise.<jupyter_code># Examine the shape of 'ri'
print(ri.shape)

# Merge 'ri' and 'weather_rating' using a left join
ri_weather = pd.merge(left=ri, right=weather_rating, left_on='stop_date', right_on='DATE', how='left')

# Examine the shape of 'ri_weather'
print(ri_weather.shape)

# Set 'stop_datetime' as the index of 'ri_weather'
ri_weather.set_index('stop_datetime', inplace=True)<jupyter_output>(86536, 18)
(86536, 20)<jupyter_text>__34 Does weather affect the arrest rate?__<jupyter_code># Driver gender and vehicle searches
print(ri.search_conducted.mean())
print(ri.groupby('driver_gender').search_conducted.mean())
search_rate = ri.groupby(['violation','driver_gender']).search_conducted.mean()
search_rate
print(type(search_rate))
print(type(search_rate.index))
search_rate.loc['Equipment']
search_rate.loc['Equipment', 'M']
search_rate.unstack()
type(search_rate.unstack())<jupyter_output><empty_output><jupyter_text>__Converting a multi-indexed Series to a DataFrame__<jupyter_code>ri.pivot_table(index='violation',
               columns='driver_gender',
               values='search_conducted')<jupyter_output><empty_output><jupyter_text>__Comparing arrest rates by weather rating__ Do police officers arrest drivers more often when the weather is bad? Find out below!

- First, you'll calculate the overall arrest rate.

- Then, you'll calculate the arrest rate for each of the weather ratings you previously assigned.

- Finally, you'll add violation type as a second factor in the analysis, to see if that accounts for any differences in the arrest rate.

Since you previously defined a logical order for the weather categories, `good < bad < worse`, they will be sorted that way in the results. __Calculate the overall arrest rate by taking the mean of the `is_arrested` Series__<jupyter_code># Calculate the overall arrest rate
print(ri_weather.is_arrested.mean())<jupyter_output>0.0355690117407784<jupyter_text>__Calculate the arrest rate for each weather `rating` using a `.groupby()`.__<jupyter_code># Calculate the arrest rate for each 'rating'
print(ri_weather.groupby('rating').is_arrested.mean())<jupyter_output>rating
good     0.033715
bad      0.036261
worse    0.041667
Name: is_arrested, dtype: float64<jupyter_text>__Calculate the arrest rate for each combination of `violation` and `rating`. How do the arrest rates differ by group?__<jupyter_code># Calculate the arrest rate for each 'violation' and 'rating'
print(ri_weather.groupby(['violation', 'rating']).is_arrested.mean())<jupyter_output>violation            rating
Equipment            good      0.059007
                     bad       0.066311
                     worse     0.097357
Moving violation     good      0.056227
                     bad       0.058050
                     worse     0.065860
Other                good      0.076966
                     bad       0.087443
                     worse     0.062893
Registration/plates  good      0.081574
                     bad       0.098160
                     worse     0.115625
Seat belt            good      0.028587
                     bad       0.022493
                     worse     0.000000
Speeding             good      0.013405
                     bad       0.013314
                     worse     0.016886
Name: is_arrested, dtype: float64<jupyter_text>$\color{red}{\textbf{Interpretation:}}$

>The arrest rate **increases as the weather gets worse**, and that trend persists across many of the violation types. This doesn't prove a causal link, but it's quite an interesting result  __35 Selecting from a multi-indexed Series__

The output of a single `.groupby()` operation on multiple columns is a Series with a MultiIndex. Working with this type of object is similar to working with a DataFrame:

- The outer index level is like the DataFrame rows.
- The inner index level is like the DataFrame columns.

In this exercise, you'll practice accessing data from a multi-indexed Series using the `.loc[]` accessor.<jupyter_code># Save the output of the groupby operation from the last exercise
arrest_rate = ri_weather.groupby(['violation', 'rating']).is_arrested.mean()

# Print the 'arrest_rate' Series
print(arrest_rate)

# Print the arrest rate for moving violations in bad weather
print(arrest_rate.loc['Moving violation', 'bad'])

# Print the arrest rates for speeding violations in all three weather conditions
print(arrest_rate.loc['Speeding'])<jupyter_output>violation            rating
Equipment            good      0.059007
                     bad       0.066311
                     worse     0.097357
Moving violation     good      0.056227
                     bad       0.058050
                     worse     0.065860
Other                good      0.076966
                     bad       0.087443
                     worse     0.062893
Registration/plates  good      0.081574
                     bad       0.098160
                     worse     0.115625
Seat belt            good      0.028587
                     bad       0.022493
                     worse     0.000000
Speeding             good      0.013405
                     bad       0.013314
                     worse     0.016886
Name: is_arrested, dtype: float64
0.05804964058049641
rating
good     0.013405
bad      0.013314
worse    0.016886
Name: is_arrested, dtype: float64<jupyter_text>__36 Reshaping the arrest rate data__

In this exercise, we'll the `arrest_rate` Series into a DataFrame. This is a useful step when working with any **multi-indexed Series**, since it enables us to access the *full range of DataFrame methods*.

Then, we'll create the exact same DataFrame using a pivot table. This is a great example of how pandas often gives you more than one way to reach the same result!<jupyter_code># Unstack the 'arrest_rate' Series into a DataFrame
print(arrest_rate.unstack())

# Create the same DataFrame using a pivot table
print(ri_weather.pivot_table(index='violation', 
                             columns='rating', 
                             values='is_arrested'))<jupyter_output>rating                   good       bad     worse
violation                                        
Equipment            0.059007  0.066311  0.097357
Moving violation     0.056227  0.058050  0.065860
Other                0.076966  0.087443  0.062893
Registration/plates  0.081574  0.098160  0.115625
Seat belt            0.028587  0.022493  0.000000
Speeding             0.013405  0.013314  0.016886
rating                   good       bad     worse
violation                                        
Equipment            0.059007  0.066311  0.097357
Moving violation     0.056227  0.058050  0.065860
Other                0.076966  0.087443  0.062893
Registration/plates  0.081574  0.098160  0.115625
Seat belt            0.028587  0.022493  0.000000
Speeding             0.013405  0.013314  0.016886
{'fraction_non_alphanumeric': 0.09124186028590144, 'fraction_numerical': 0.049917157939351905, 'mean_word_length': 3.6565892168296403, 'pattern_counts': {'":': 0, '<': 230, '<?xml version=': 0, '>': 249, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 18, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '65', 'dir': 'jupyter-structured-clean-dedup', 'id': '4556', 'license': 'MIT', 'n_tokens_mistral': 17465, 'n_tokens_neox': 15474, 'n_words': 5596, 'path': '21_Analyzing Police Activity with pandas/Police_Activity_data_for_analysis.ipynb', 'repo_name': 'mohd-faizy/DataScience-With-Python', 'size': '51906'}