<jupyter_start><jupyter_text>Analyzing NYC's 311 Street Flooding Complaints from 2010 to 2020 *Data Inspection*<NAME>  GoalIn this notebook, we inspect and preview our data before we begin analysis. Specifically, I attempt to inspect the following items:- the shape of the data- number of null/na values in the dataset- identify columns and data types- descriptive or summary statistics- correlation of columns Main Python Libraries[pandas](https://pandas.pydata.org/) is a fast, powerful, flexible and easy to use open source data analysis and manipulation tool, built on top of the Python programming language.[Seaborn](https://seaborn.pydata.org/index.html) is a Python data visualization library based on matplotlib. It provides a high-level interface for drawing attractive and informative statistical graphics.  Datasets 311 Service Requests from 2010 to Present  Website: https://data.cityofnewyork.us/Social-Services/311-Service-Requests-from-2010-to-Present/erm2-nwe9311's data dictionary can be found in the data folder. ![311](../images/nyc-311-screenshot.png) Table of Contents=================   * [1. Importing Libraries](1-Importing-Libraries)                 * [2. Loading Data](2-Loading-Data)                 * [3. Data Inspection](3-Data-Inspection)       * [3.1 Basic Data Inspection](31-Basic-Data-Inspection)       * [3.2 Calculating Summary Statistics](32-Calculating-Summary-Statistics)       * [3.3 Identifying Null/NA Values](33-Identifying-Null/NA-Values)       * [3.4 Inspecting Items In Each Column](34-Inspecting-Items-In-Each-Column)  1. Importing Libraries<jupyter_code>import os
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

plt.rcParams['savefig.facecolor'] = 'white'
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Printing versions of Python modules and packages with **watermark** - the IPython magic extension.<jupyter_code>%load_ext watermark
%watermark -v -p numpy,pandas,seaborn<jupyter_output>CPython 3.7.1
IPython 7.20.0

numpy 1.19.2
pandas 1.2.1
seaborn 0.11.1<jupyter_text>Documention for installing watermark: https://github.com/rasbt/watermark  2. Loading Data<jupyter_code># list items in data folder
%ls ../data-raw/
# read data as a dataframe
flooding_311 = pd.read_csv('../data-raw/raw_street_flooding_complaints.csv', 
                           low_memory=False)

# previewing first five rows in data
flooding_311.head()
# previewing the last five rows of our dataframe
flooding_311.tail()
print(flooding_311.shape)<jupyter_output>(27428, 34)<jupyter_text>3. Data Inspection  3.1 Basic Data Inspection<jupyter_code># the object's type
type(flooding_311)
# printing the columns of our dataframe
flooding_311.columns
# printing the data types of our columns
flooding_311.dtypes
# printing the column names, non-null counts, and data types of our columns
flooding_311.info()
# counts of unique values of our datatypes 
flooding_311.dtypes.value_counts()
# printing index type
flooding_311.index
cols = flooding_311.columns

print('Inspecting data for unique columns (True/False)\n' + \
     ('-' * 47))

for i in cols:
    print(str(i) + ':', flooding_311[i].is_unique)
# printing True/False if column is unique on our unique key (DOITT_ID)
flooding_311['unique_key'].is_unique
# printing descriptive statistics of our numeric columns in our data
all_columns = flooding_311.columns
flooding_311[all_columns].astype(str).describe(include='all')[:2].T.sort_values(by='unique', 
                                                                                ascending=False)
flooding_311['descriptor'].value_counts()
flooding_311['agency'].value_counts()
flooding_311['agency_name'].value_counts()
flooding_311['complaint_type'].value_counts()
flooding_311['status'].value_counts()
flooding_311['resolution_description'].value_counts().reset_index().head()
print('Top 10 Resolution Descriptions in Descending Order:\n')

for i in range(10):
    print(str(i + 1) + '.', \
          flooding_311['resolution_description'].value_counts().index[i], \
          '\nCount: ' \
          f"{flooding_311['resolution_description'].value_counts()[i]:,}" + \
          '\n')
flooding_311.created_date.min()
flooding_311.created_date.max()
print('This dataset is from', \
      flooding_311.created_date.min()[0:10], \
      'to', \
      flooding_311.created_date.max()[0:10] + \
      '.')<jupyter_output>This dataset is from 2010-01-02 to 2020-12-31.<jupyter_text>3.2 Calcualting Summary Statistics<jupyter_code># display floats up to three decimals
pd.set_option('display.float_format', lambda x: '%.2f' % x)
# printing descriptive statistics of our numeric columns in our data
flooding_311.describe()
flooding_311.describe(include=['O']).T
# printing descriptive statistics of our numeric columns in our data
flooding_311.describe(include='all').T
# returning the mode of our numeric columns (excluding our unique key = 'DOITT_ID') as 'mode'
mode = flooding_311.loc[:, flooding_311.columns != 'unique_key'].mode(numeric_only=False)

# renaming index to 'mode'
mode.rename(index={0: "mode"}, inplace=True)

# transposing and printing the mode ofour numeric columns (excluding our unique key = 'DOITT_ID')
mode.T
# compute pairwise correlation of columns, excluding NA/null values
flooding_311.corr()
fig, ax = plt.subplots(figsize=(15,10))

sns.heatmap(flooding_311.corr(), 
            annot=True, 
            ax=ax)
flooding_311.info()
flooding_311.columns
heatmap = flooding_311[['unique_key', 'created_date', 'closed_date', 'agency', 'agency_name',
                       'complaint_type', 'descriptor', 'incident_zip', 'incident_address',
                       'street_name', 'cross_street_1', 'cross_street_2', 'address_type',
                       'city', 'status', 'resolution_description',
                       'resolution_action_updated_date', 'community_board', 'borough',
                       'x_coordinate_state_plane', 'y_coordinate_state_plane',
                       'open_data_channel_type', 'park_facility_name', 'park_borough',
                       'latitude', 'longitude', 'location', 'intersection_street_1',
                       'intersection_street_2', 'bbl']]
fig, ax = plt.subplots(figsize=(15,10))

sns.heatmap(heatmap.corr(), 
            annot=True, 
            ax=ax)<jupyter_output><empty_output><jupyter_text>3.3 Identifying Null/NA Values<jupyter_code>print('Sum of nulls in column:')
flooding_311.isna().sum().sort_values(ascending=False)
print('Sum of nulls in all columns:')
flooding_311.isna().sum().sum()
print('Identifying nulls in column (True/False):')
flooding_311.isna().head()
print('Identifying any null values in column (True/False):')
flooding_311.isna().any().sort_values(ascending=False)
# return a boolean same-sized object indicating if any of the values are NA
flooding_311.isna().any().value_counts()
# return descriptive statistics of boolean indicating if any of the values are NA
flooding_311.isna().any().describe()
# calculating a percentage of the number of nulls to total number of records of each column
missing_data = (flooding_311.isna().sum() / len(flooding_311)) * 100

# creating a dataframe
missing_data = pd.DataFrame({'Missing Ratio (%)' :missing_data}) 
missing_data.sort_values(by='Missing Ratio (%)', 
                         ascending=False, 
                         inplace=True)

missing_data.head(len(missing_data))<jupyter_output><empty_output><jupyter_text>3.4 Inspecting Items In Each Column<jupyter_code>len(flooding_311.columns)
flooding_311.iloc[:,:15].head()
flooding_311.iloc[:,15:30].head()
flooding_311.iloc[:,30:].head()
print('Returning a dataframe containing counts and normalized counts of unique rows in each column (sorted desc).',
     '\n' + '-' * 60, '\n')

for col in flooding_311.columns:
    series_1 = flooding_311[col].value_counts().head()
    series_2 = flooding_311[col].value_counts(normalize=True).head()
    series_2 = series_2.rename(str(col) +  ' (normalized)')
    value_counts = pd.concat([series_1, series_2], axis=1)
    
    print(value_counts, '\n' \
         'Column Name:', col + ',', 'dtype:', flooding_311[col].dtype, '\n')<jupyter_output>Returning a dataframe containing counts and normalized counts of unique rows in each column (sorted desc). 
------------------------------------------------------------ 

          unique_key  unique_key (normalized)
16912386           1                     0.00
34671374           1                     0.00
18551703           1                     0.00
28195057           1                     0.00
21377939           1                     0.00 
Column Name: unique_key, dtype: int64 

                         created_date  created_date (normalized)
2011-08-28T00:00:00.000            13                       0.00
2019-10-27T13:22:00.000             7                       0.00
2011-08-29T00:00:00.000             6                       0.00
2017-05-05T13:49:00.000             6                       0.00
2018-11-26T18:33:00.000             5                       0.00 
Column Name: created_date, dtype: object 

                         closed_date  closed_date (normalized)
2017-05-06T08:3[...]
{'fraction_non_alphanumeric': 0.12091503267973856, 'fraction_numerical': 0.05403050108932462, 'mean_word_length': 3.351184834123223, 'pattern_counts': {'":': 0, '<': 26, '<?xml version=': 0, '>': 26, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 13, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '123986', 'license': 'MIT', 'n_tokens_mistral': 3213, 'n_tokens_neox': 2812, 'n_words': 709, 'path': 'data-wrangling/data_inspection.ipynb', 'repo_name': 'datalifenyc/nyc-311-street-flooding', 'size': '9184'}