<jupyter_start><jupyter_text>Project Assignment B: Social Data Analysis: NYPD Vehicle Accident  Motivation:Vehicular accidents are common. In fact they are so common that a study has actually stated that there is 1 in 606 chance of dying in vehicle accident in a lifetime. Compared to say with 174,426 chance of dying from being hit with lightning this is indeed quite a high probability. There are many different reasons and types of accidents but in the end the loss of life is always inexcusable. Our motivation for this project is therefore to analyse and show the major reasons of fatalities and injuries from vehicle accidents in a major city. We hope that our analysis and visualisation will therefore be able to show the major reasons of accidents and locations, so that in the long run people can be saved and harm be reduced. The dataset it the NewYork City Vehicle Collisions from 2012 till now. The reason why we choose this particular dataset was because of the dataset has lots of information in details regarding the vehicle collisions happened in Newyork City with the timespan, location, and the types of the vehicles and what kind of reason it cause the collisions. It can give a very good overview of from what kind of the vehicle collisions that have been occured from 2012, along with the vehicle type, in which period and how that is occured. Basic Stats:As of 2008, the The New York State Department of Motor Vehicles  has over 10 millions vehicle registrations in force. This is practically a portion of vehicles and cars travelling within the state borders. In New York city alone of the 300000 crashes that happened in 2013, over 1000 were fatal crashes that involved vehicles, drivers, pedestrians and bicyclists.Therefore, We decided to do our project using the New York Police Department Motor Vehicle Collisions dataset. This dataset is 200MB in size and has approximately 1 million rows and 28 columns. Therefore a lot of training data is available and with the major elements of the set being zone of the accident, the type of accident, the type of vehicle, the hour and the date, we will be able to have a look at the most dangerous streets, hours and cause of accidents and if there were changes over the years, signifying that the authorities tried doing something about it.Since the basic crux of our project is the analysis of the vehicle collisions happening in New York and find out which vehicles are the safest ones, at what hour is the safest, we hope as a final result of the project to have some impact in form of visualizing the analysed data and presenting it in a format that would be understandable to the common people in order for them to understand the major risks related with traffic and that safety is always the priority since human lives cannot be replaced Theory:One of the machine learning tools that we use is K-means clustering algorithm, which is used to solve clustering problem. The main idea of it is to define k centers, one for each cluster. These centers should be placed as much as possible  far away from each other, and thus to take each point belonging to our dataset and associate it to the nearest center. We used it for displaying April.2017 for passenger vehicles and 2016 for bicycles of different places in NewYork City. KNN is an non parametric algorithm, which means that it does not make any assumptions on the underlying data distribution. We are given some data points for training and also a new unlabelled data for testing. The algorithm has different behavior based on k.In our case, we are using k = 5,10,30. Visualizations:We filtered the dataset with the different type of the factors for occuring the vehicle collisions, and displayed them in one heatmap for one week to check which factors have more amount and in which time period does the vehicle collisions happen more with the different type of the factors (e.g. Driver Inanttention-Distraction, Failure to Yield Right-of-Way, Fatigued or Drowsy or others).We have also filtered the dataset by different facotrs/streets and boroughs to have an overview of the percentage of the different factors/streets cause the vehicle collisions in different boroughs in NewYork. And over all, there is also K clusters for k=2-6 showing for the vehicle type of Passenger Vehicles and the bicycles and also KNN for different factors.By using them, we can have our idea well presented.<jupyter_code>import pandas as pd
df = pd.read_csv("nypd.csv")<jupyter_output><empty_output><jupyter_text>Dropping empty columns values<jupyter_code>df.dropna(subset=['LATITUDE'], inplace=True)
df.dropna(subset=['LONGITUDE'], inplace=True)
df.dropna(subset=['CONTRIBUTING FACTOR VEHICLE 1'], inplace=True)
df = df.drop(df[df['CONTRIBUTING FACTOR VEHICLE 1'] =='Unspecified'].index)
inattetion =  df.loc[df['CONTRIBUTING FACTOR VEHICLE 1'] == "Driver Inattention/Distraction"]
failure = df.loc[df['CONTRIBUTING FACTOR VEHICLE 1'] == "Failure to Yield Right-of-Way"]
otherVehicle = df.loc[df['CONTRIBUTING FACTOR VEHICLE 1'] == "Other Vehicular"]
fatigued =  df.loc[df['CONTRIBUTING FACTOR VEHICLE 1'] == "Fatigued/Drowsy"]
inattention25k = inattetion.iloc[-10000:]
otherVehicle25k = otherVehicle.iloc[-10000:]
failure25k = failure.iloc[-10000:]
fatigued25k = fatigued.iloc[-10000:]
minLat, minLon = 40.499186, -74.251609
maxLat, maxLon = 40.977385, -73.648929
minLat =  min(df[df['LATITUDE']>39]['LATITUDE'])
maxLat = max(df[df['LATITUDE']>39]['LATITUDE'])
minLon =  min(df[df['LONGITUDE']>-175]['LONGITUDE'])
maxLon =  max(df[df['LONGITUDE']<-72]['LONGITUDE'])
latIncrease = (maxLat- minLat)/50.0
lonIncrease = (maxLon - minLon)/50.0
import numpy as np
Y = np.arange(minLat,maxLat, latIncrease)
X = np.arange(minLon+0.04, maxLon+0.04, lonIncrease)<jupyter_output><empty_output><jupyter_text>Calculating the knn for Driver Inattention/Distraction, Failure to Yield Right-of-Way,Other Vehicular, Fatigued/Drowsy<jupyter_code>### needed to calculate KNN later, took from the book

from __future__ import division
from collections import Counter
import math

import matplotlib.pyplot as plt
def vector_subtract(v, w):
    """subtracts two vectors componentwise"""
    return [v_i - w_i for v_i, w_i in zip(v,w)]

def dot(v, w):
    """v_1 * w_1 + ... + v_n * w_n"""
    return sum(v_i * w_i for v_i, w_i in zip(v, w))

def sum_of_squares(v):
    """v_1 * v_1 + ... + v_n * v_n"""
    return dot(v, v)

def magnitude(v):
    return math.sqrt(sum_of_squares(v))

def squared_distance(v, w):
    return sum_of_squares(vector_subtract(v, w))

def distance(v, w):
   return math.sqrt(squared_distance(v, w))

def majority_vote(labels):
    """assumes that labels are ordered from nearest to farthest"""
    vote_counts = Counter(labels)
    winner, winner_count = vote_counts.most_common(1)[0]
    num_winners = len([count 
                       for count in vote_counts.values()
                       if count == winner_count])

    if num_winners == 1:
        return winner                     # unique winner, so return it
    else:
        return majority_vote(labels[:-1]) # try again without the farthest


def knn_classify(k, labeled_points, new_point):
    """each labeled point should be a pair (point, label)"""
    
    # order the labeled points from nearest to farthest
    by_distance = sorted(labeled_points,
                         key=lambda (point, _): distance(point, new_point))

    # find the labels for the k closest
    k_nearest_labels = [label for _, label in by_distance[:k]]

    # and let them vote
    return majority_vote(k_nearest_labels)
allCrimes = []
for index,row in inattention25k.iterrows():
    allCrimes.append(([row['LATITUDE'], row['LONGITUDE']],'inattention'))
for index,row in fatigued25k.iterrows():
    allCrimes.append(([row['LATITUDE'], row['LONGITUDE']],'fatigued'))
for index,row in failure25k.iterrows():
    allCrimes.append(([row['LATITUDE'], row['LONGITUDE']],'failure'))
for index,row in otherVehicle25k.iterrows():
    allCrimes.append(([row['LATITUDE'], row['LONGITUDE']],'otherVehicle'))
#creating the csv for knn=5
inattention5 = open("inattention5.csv",'w')
inattention5.write("name,lat,lon\n")
fatigued5 = open("fatigued5.csv",'w')
fatigued5.write("fatigued5,lat,lon\n")
otherVehicle5 = open("otherVehicle5.csv",'w')
otherVehicle5.write("name,lat,lon\n")
failure5 = open("failure5.csv",'w')
failure5.write("name,lat,lon\n")



for y in Y:
    for x in X: 
        c = knn_classify(5, allCrimes, [y,x])
        if c=='inattention':
            inattention5.write('inattention, %f, %f \n' %( y, x))
        if c=='fatigued':
            fatigued5.write('fatigued5, %f, %f \n' %( y, x))
        if c=='otherVehicle':
            otherVehicle5.write('otherVehicle, %f, %f \n' %( y, x))
        if c=='failure':
            failure5.write('failure, %f, %f \n' %( y, x))
            
        
inattention5.close()
fatigued5.close()
otherVehicle5.close()
failure5.close()
#creating the csv for knn=10

inattention10 = open("inattention10.csv",'w')
inattention10.write("name,lat,lon\n")
fatigued10 = open("fatigued10.csv",'w')
fatigued10.write("fatigued10,lat,lon\n")
otherVehicle10 = open("otherVehicle10.csv",'w')
otherVehicle10.write("name,lat,lon\n")
failure10 = open("failure10.csv",'w')
failure10.write("name,lat,lon\n")

for y in Y:
    for x in X: 
        c = knn_classify(10, allCrimes, [y,x])
        if c=='inattention':
            inattention10.write('inattention, %f, %f \n' %( y, x))
        if c=='fatigued':
            fatigued10.write('fatigued10, %f, %f \n' %( y, x))
        if c=='otherVehicle':
            otherVehicle10.write('otherVehicle, %f, %f \n' %( y, x))
        if c=='failure':
            failure10.write('failure, %f, %f \n' %( y, x))
            
        
inattention10.close()
fatigued10.close()
otherVehicle10.close()
failure10.close()

#creating the csv for knn=30
inattention30 = open("inattention30.csv",'w')
inattention30.write("name,lat,lon\n")
fatigued30 = open("fatigued30.csv",'w')
fatigued30.write("fatigued30,lat,lon\n")
otherVehicle30 = open("otherVehicle30.csv",'w')
otherVehicle30.write("name,lat,lon\n")
failure30 = open("failure30.csv",'w')
failure30.write("name,lat,lon\n")

for y in Y:
    for x in X: 
        c = knn_classify(30, allCrimes, [y,x])
        if c=='inattention':
            inattention30.write('inattention, %f, %f \n' %( y, x))
        if c=='fatigued':
            fatigued30.write('fatigued30, %f, %f \n' %( y, x))
        if c=='otherVehicle':
            otherVehicle30.write('otherVehicle, %f, %f \n' %( y, x))
        if c=='failure':
            failure30.write('failure, %f, %f \n' %( y, x))
            
        
inattention30.close()
fatigued30.close()
otherVehicle30.close()
failure30.close()<jupyter_output><empty_output><jupyter_text>Getting the intersections where there were the most crashes by years: 2012,2013,2014,2015<jupyter_code>dfCount = df.groupby(['ON STREET NAME','CROSS STREET NAME']).size().reset_index().rename(columns={0:'count'})
failureCount = failure.groupby(['ON STREET NAME','CROSS STREET NAME']).size().reset_index().rename(columns={0:'count'})
fatiguedCount = fatigued.groupby(['ON STREET NAME','CROSS STREET NAME']).size().reset_index().rename(columns={0:'count'})
inattetionCount = inattetion.groupby(['ON STREET NAME','CROSS STREET NAME']).size().reset_index().rename(columns={0:'count'})
otherVehicleCount = otherVehicle.groupby(['ON STREET NAME','CROSS STREET NAME']).size().reset_index().rename(columns={0:'count'})
dfCount = dfCount.sort_values('count', ascending=False)
inattetionCount =  inattetionCount.sort_values('count', ascending=False)
failureCount = failureCount.sort_values('count', ascending=False)
fatiguedCount = fatiguedCount.sort_values('count', ascending=False)
otherVehicleCount = otherVehicleCount.sort_values('count', ascending=False)
#creating masks for each year
mask2012 = ( pd.to_datetime(df['DATE'])  >= '2012-1-1') & ( pd.to_datetime(df['DATE'])  < '2013-1-1')
mask2013 = ( pd.to_datetime(df['DATE'])  >= '2013-1-1') & ( pd.to_datetime(df['DATE'])  < '2014-1-1')
mask2014 = ( pd.to_datetime(df['DATE'])  >= '2014-1-1') & ( pd.to_datetime(df['DATE'])  < '2015-1-1')
mask2015 = ( pd.to_datetime(df['DATE'])  >= '2015-1-1') & ( pd.to_datetime(df['DATE'])  < '2016-1-1')
mask2016 = ( pd.to_datetime(df['DATE'])  >= '2016-1-1') & ( pd.to_datetime(df['DATE'])  < '2017-1-1')
#creating small pandas for each year
df2012 = df.loc[mask2012]
df2013 = df.loc[mask2013]
df2014 = df.loc[mask2014]
df2015 = df.loc[mask2015]
df2016 = df.loc[mask2016]
#getting the intersections for each year and sort them descending 
dfCount2012 = df2012.groupby(['ON STREET NAME','CROSS STREET NAME']).size().reset_index().rename(columns={0:'count'})
dfCount2012 = dfCount2012.sort_values('count', ascending=False)
dfCount2013 = df2013.groupby(['ON STREET NAME','CROSS STREET NAME']).size().reset_index().rename(columns={0:'count'})
dfCount2013 = dfCount2013.sort_values('count', ascending=False)
dfCount2014 = df2014.groupby(['ON STREET NAME','CROSS STREET NAME']).size().reset_index().rename(columns={0:'count'})
dfCount2014 = dfCount2014.sort_values('count', ascending=False)
dfCount2015 = df2015.groupby(['ON STREET NAME','CROSS STREET NAME']).size().reset_index().rename(columns={0:'count'})
dfCount2015 = dfCount2015.sort_values('count', ascending=False)
dfCount2016 = df2016.groupby(['ON STREET NAME','CROSS STREET NAME']).size().reset_index().rename(columns={0:'count'})
dfCount2016 = dfCount2016.sort_values('count', ascending=False)

#creating the csv files with the intersection data
streetNames2012 = []
crossStreetNames2012 = []
count2012 = []
counter =0 
for i in  dfCount2012['ON STREET NAME']:
    streetNames2012.append(i)
    counter = counter +1 
    if counter==10:
        break
counter =0 
for i in  dfCount2012['CROSS STREET NAME']:
    crossStreetNames2012.append(i)
    counter = counter +1 
    if counter==10:
        break      
counter =0 
for i in  dfCount2012['count']:
    count2012.append(i)
    counter = counter +1 
    if counter==10:
        break      
dfs = [df2013,df2014,df2015,df2016]
count2013= []
count2014= []
count2015= []
count2016= []
countAll = [count2013,count2014,count2015,count2016]

x =0 
for dfX in dfs:
    for i in range(0,10):
        aMask = ( dfX['ON STREET NAME']  == streetNames2012[i]) & ( dfX['CROSS STREET NAME']  == crossStreetNames2012[i])
        countAll[x].append( len(dfX.loc[aMask]))
    x = x+1

    
f = open('intersections.csv', 'w')
for i in range(0,10):
    f.write('date,%s With %s,'%(streetNames2012[i],crossStreetNames2012[i]))
f.write('\n')
years =['1-Jan-2013','1-Jan-2014','1-Jan-2015','1-Jan-2016']
year= 0
for i in countAll:
    f.write('%s,'%years[year])
    for j in range(0,10):
        f.write('%i,'%countAll[year][j])
    year = year + 1
    f.write('\n')
f.write('1-Jan-2012,')
for i in count2012: 
     f.write('%i,'%i)
f.write("\n")
f.close()<jupyter_output><empty_output><jupyter_text>Getting the data necessary for day hour heatmap<jupyter_code>alltheyears = pd.date_range(start=pd.datetime(2012, 1, 1), periods=365*5, freq='D')
w, h = 7, 24;
Matrix = [[0 for x in range(h)] for y in range(w)] 
weekA_actualResults = []
for i in range(0,365*5):
    k = '{:%m/%d/%Y}'.format(alltheyears[i])
    weekday = alltheyears[i].weekday()
    dfPerDay  = df.loc[df['DATE']==k]
    for j in range(0,24):
        
        if j<9:
            maskH  = (dfPerDay['TIME']>= '%i:00'%j) & (dfPerDay['TIME']<='%i:00'%(j+1))
        else:
            if j==9:
                maskH  = (dfPerDay['TIME']>= '%i:00'%j) & (dfPerDay['TIME']<='%i:00'%(j+1))
            else:
                maskH  = (dfPerDay['TIME']>= '%i:00'%j) & (dfPerDay['TIME']<='%i:00'%(j+1))
        dfPerHour = dfPerDay.loc[maskH]
        Matrix[weekday][j] =  Matrix[weekday][j] + len(dfPerHour)
f = open('data1.tsv', 'w')
f.write("day hour value\n")
for i in range(0,7):
    for j in range(0,24):
            s = '%i %i %i\n'%(i,j, Matrix[i][j])
            f.write(s)
f.close()
MatrixFatigued = [[0 for x in range(h)] for y in range(w)] 
for i in range(0,365*5):
    k = '{:%m/%d/%Y}'.format(alltheyears[i])
    weekday = alltheyears[i].weekday()
    fatiguedPerDay  = fatigued.loc[fatigued['DATE']==k]
    for j in range(0,24):
        if j+1!=24:
            fatiguedPerHour = fatiguedPerDay.between_time( '%i:00:00'%j, '%i:00:00'%((j+1)) )
        else:
            fatiguedPerHour = fatiguedPerDay.between_time( '%i:00:00'%j, '%i:00:00'%(0 ))
        MatrixFatigued[weekday][j] =  MatrixFatigued[weekday][j] + len(fatiguedPerHour)
        
f = open('dataFatigued.tsv', 'w')
f.write("day hour value\n")
for i in range(0,7):
    for j in range(0,24):
        if(j>=2):
            s = '%i %i %i\n'%(i,j, MatrixFatigued[i][j])
            f.write(s)
f.close()
MatrixFailure = [[0 for x in range(h)] for y in range(w)] 
for i in range(0,365*5):
    k = '{:%m/%d/%Y}'.format(alltheyears[i])
    weekday = alltheyears[i].weekday()
    failurePerDay  = failure.loc[failure['DATE']==k]
    for j in range(0,24):
        if j+1!=24:
            failurePerHour = failurePerDay.between_time( '%i:00:00'%j, '%i:00:00'%((j+1)) )
        else:
            failurePerHour = failurePerDay.between_time( '%i:00:00'%j, '%i:00:00'%((0) ))
        MatrixFailure[weekday][j] =  MatrixFailure[weekday][j] + len(failurePerHour)
f = open('dataFailure.tsv', 'w')
f.write("day hour value\n")
for i in range(0,7):
    for j in range(0,24):
        if(j>=2):
            s = '%i %i %i\n'%(i,j, MatrixFailure[i][j])
            f.write(s)
f.close()
MatrixotherVehicle = [[0 for x in range(h)] for y in range(w)] 
for i in range(0,365*5):
    k = '{:%m/%d/%Y}'.format(alltheyears[i])
    weekday = alltheyears[i].weekday()
    otherVehiclePerDay  = otherVehicle.loc[otherVehicle['DATE']==k]
    for j in range(0,24):
        if j+1!=24:
            otherVehiclePerHour = otherVehiclePerDay.between_time( '%i:00:00'%j, '%i:00:00'%((j+1)) )
        else: 
            otherVehiclePerHour = otherVehiclePerDay.between_time( '%i:00:00'%j, '%i:00:00'%((0)) )
        MatrixotherVehicle[weekday][j] =  MatrixotherVehicle[weekday][j] + len(otherVehiclePerHour)
f = open('dataotherVehicle.tsv', 'w')
f.write("day hour value\n")
for i in range(0,7):
    for j in range(0,24):
        if(j>=2):
            s = '%i %i %i\n'%(i,j, MatrixotherVehicle[i][j])
            f.write(s)
f.close()
Matrixinattetion = [[0 for x in range(h)] for y in range(w)] 
for i in range(0,365*5):
    k = '{:%m/%d/%Y}'.format(alltheyears[i])
    weekday = alltheyears[i].weekday()
    inattetionPerDay  = inattetion.loc[inattetion['DATE']==k]
    for j in range(0,24):
        
        if j+1!=24:
            inattetionPerHour = inattetionPerDay.between_time( '%i:00:00'%j, '%i:00:00'%((j+1)) )
        else: 
            inattetionPerHour = inattetionPerDay.between_time( '%i:00:00'%j, '%i:00:00'%((0)) )
        Matrixinattetion[weekday][j] =  Matrixinattetion[weekday][j] + len(inattetionPerHour)
f = open('datainattetion.tsv', 'w')
f.write("day hour value\n")
for i in range(0,7):
    for j in range(0,24):
        if(j>=2):
            s = '%i %i %i\n'%(i,j, Matrixinattetion[i][j])
            f.write(s)
f.close()<jupyter_output><empty_output><jupyter_text>Splitting the data into boroughs and calculating the percentage of the most contributing factors for each borrow, as well the street with the most crashes for each borough<jupyter_code>#getting the boroughs
staten = df.loc[df['BOROUGH'] == "STATEN ISLAND"]
bronx = df.loc[df['BOROUGH'] == "BRONX"]
manhattan  =  df.loc[df['BOROUGH'] == "MANHATTAN"]
queens =   df.loc[df['BOROUGH'] == "QUEENS"]
brooklyn = df.loc[df['BOROUGH'] == "BROOKLYN"]
#getting contributings factors for each borrow
statenContributing = pd.value_counts(staten['CONTRIBUTING FACTOR VEHICLE 1'].values, sort=True)
statenContributing = statenContributing[1:8]
bronxContributing = pd.value_counts(bronx['CONTRIBUTING FACTOR VEHICLE 1'].values, sort=True)
bronxContributing = bronxContributing[1:8]
manhattanContributing = pd.value_counts(manhattan['CONTRIBUTING FACTOR VEHICLE 1'].values, sort=True)
manhattanContributing = manhattanContributing[1:8]
queensContributing = pd.value_counts(queens['CONTRIBUTING FACTOR VEHICLE 1'].values, sort=True)
queensContributing = queensContributing[1:8]
brooklynContributing = pd.value_counts(brooklyn['CONTRIBUTING FACTOR VEHICLE 1'].values, sort=True)
brooklynContributing = brooklynContributing[1:8]
#getting the streets for each borough
statenStreets = pd.value_counts(staten['ON STREET NAME'].values, sort=True)
statenStreets = statenStreets[0:8]
bronxStreets = pd.value_counts(bronx['ON STREET NAME'].values, sort=True)
bronxStreets = bronxStreets[1:9]
manhattanStreets = pd.value_counts(manhattan['ON STREET NAME'].values, sort=True)
manhattanStreets = manhattanStreets[0:9]
queensStreets = pd.value_counts(queens['ON STREET NAME'].values, sort=True)
queensStreets = queensStreets[0:9]
brooklynStreets = pd.value_counts(brooklyn['ON STREET NAME'].values, sort=True)
brooklynStreets = brooklynStreets[0:9]
#writting the contributing factors to the csv
from random import randint
f = open("contributing.csv", "w")
f.write("id,value,color\n")
f.write("NewYork,\n")
s= "NewYork.Staten Island"
f.write(s)
f.write(",\n")
for row in statenContributing.iteritems():
    if row[0]!="":
        f.write('%s.%s,%f,%s\n'%(s,row[0],row[1]*100.0/len(staten),'#%06X' % randint(0, 0xFFFFFF)))
        
s= "NewYork.Bronx"
f.write(s)
f.write(",\n")
for row in bronxContributing.iteritems():
    if row[0]!="":
        f.write('%s.%s,%f,%s\n'%(s,row[0],row[1]*100.0/len(bronx),'#%06X' % randint(0, 0xFFFFFF)))

s= "NewYork.Manhattan"
f.write(s)
f.write(",\n")
for row in manhattanContributing.iteritems():
    if row[0]!="":
        f.write('%s.%s,%f,%s\n'%(s,row[0],row[1]*100.0/len(manhattan),'#%06X' % randint(0, 0xFFFFFF)))

s= "NewYork.Queens"
f.write(s)
f.write(",\n")
for row in queensContributing.iteritems():
    if row[0]!="":
        f.write('%s.%s,%f,%s\n'%(s,row[0],row[1]*100.0/len(queens),'#%06X' % randint(0, 0xFFFFFF)))

s= "NewYork.Brooklyn"
f.write(s)
f.write(",\n")
for row in brooklynContributing.iteritems():
    if row[0]!="":
        f.write('%s.%s,%f,%s\n'%(s,row[0],row[1]*100.0/len(brooklyn),'#%06X' % randint(0, 0xFFFFFF)))
f.close()
#writting the street names for each borough on csv
from random import randint
f = open("streets.csv", "w")
f.write("id,value,color\n")
f.write("NewYork,\n")
s= "NewYork.Staten Island"
f.write(s)
f.write(",\n")
for row in statenStreets.iteritems():
    if row[0]!="":
        f.write('%s.%s,%f,%s\n'%(s,row[0],row[1]*100.0/len(staten),'#%06X' % randint(0, 0xFFFFFF)))
        
s= "NewYork.Bronx"
f.write(s)
f.write(",\n")
for row in bronxStreets.iteritems():
    if row[0]!="":
        f.write('%s.%s,%f,%s\n'%(s,row[0],row[1]*100.0/len(bronx),'#%06X' % randint(0, 0xFFFFFF)))

s= "NewYork.Manhattan"
f.write(s)
f.write(",\n")
for row in manhattanStreets.iteritems():
    if row[0]!="":
        f.write('%s.%s,%f,%s\n'%(s,row[0],row[1]*100.0/len(manhattan),'#%06X' % randint(0, 0xFFFFFF)))

s= "NewYork.Queens"
f.write(s)
f.write(",\n")
for row in queensStreets.iteritems():
    if row[0]!="":
        f.write('%s.%s,%f,%s\n'%(s,row[0],row[1]*100.0/len(queens),'#%06X' % randint(0, 0xFFFFFF)))

s= "NewYork.Brooklyn"
f.write(s)
f.write(",\n")
for row in brooklynStreets.iteritems():
    if row[0]!="":
        f.write('%s.%s,%f,%s\n'%(s,row[0],row[1]*100.0/len(brooklyn),'#%06X' % randint(0, 0xFFFFFF)))
f.close()<jupyter_output><empty_output><jupyter_text>We will make predictions for the interval 8am-8pm of the accident type of driver inattention/distraction within 2016<jupyter_code># Get only type of driver inattention in the type of first factor within 2016
inattetion =  df.loc[df['CONTRIBUTING FACTOR VEHICLE 1'] == "Driver Inattention/Distraction"]
timespan = ( pd.to_datetime(inattetion['DATE'])  >= '2016-1-1') & ( pd.to_datetime(inattetion['DATE'])  < '2017-1-1')
inattention2016 = inattetion.loc[timespan]
days = pd.date_range(start=pd.datetime(2016, 1, 1), periods=366, freq='D')


x=[]
y=[]
# 2016 is leap year, i = 366 days, j from 8am-8pm 
#preparing the X data 
for i in range(0,366):
    for j in range(8,20):
        x.append(j)
        
#getting the Y data /target data
for i in range(0,366):
    k = '{:%m/%d/%Y}'.format(days[i])
#getting hours in the right format    
    tmp  = inattention2016.loc[inattention2016['DATE']==k]
    for j in range(8,20):
        if j<9:
            hours  = (tmp['TIME']>= '0%i:00'%j) & (tmp['TIME']<='0%i:00'%(j+1))
        else:
            if j==9:
                hours  = (tmp['TIME']>= '0%i:00'%j) & (tmp['TIME']<='%i:00'%(j+1))
            else:
                hours  = (tmp['TIME']>= '%i:00'%j) & (tmp['TIME']<='%i:00'%(j+1))
        tmpH = tmp.loc[hours]
        y.append(len(tmpH))<jupyter_output><empty_output><jupyter_text>Go over the training data and bin and observed number of inattention per hour. There are 12 data-points per day of training data<jupyter_code>##creating bins for the Y data
data = []
for i in range(0,12):
    data.append([])

idx = 0 
for j in y:
    data[idx].append(j)
    idx = (idx+1) % 12
#plotting the data + the line 
import numpy as np
from sklearn import datasets, linear_model
import matplotlib.pyplot as plt
%matplotlib inline
line = plt.figure()
xarray = np.asarray(x)
xarray = xarray.reshape(4392,1)
lin = linear_model.LinearRegression()
lin.fit(xarray,y)
alpha = lin.intercept_
beta = lin.coef_[0]

for i in range(0,12):
    a = [i+8 for j in range(0,366)]
    b = data[i]
    plt.scatter(a, b, color="black")

plt.plot(xarray, lin.predict(xarray), color='red'
         ,linewidth=3)
plt.savefig('figure1')
plt.show()<jupyter_output><empty_output><jupyter_text>The idea is that if there is a lot going on a certain day, then more vehicle collisions in general will be happening - including more of the vehicle collisions we're looking at (e.g. INATTENTION). In order for the multiple regression to work well, it is important that the input variables have approximately the same size.<jupyter_code>## getting all the inattention vehicle collisions that happened in 2016
timespan = ( pd.to_datetime(inattetion['DATE'])  >= '2016-1-1') & ( pd.to_datetime(inattetion['DATE'])  < '2017-1-1')
inattention2016 = inattetion.loc[timespan]
##getting the averages for all year for each hour between 8 and 20
mean = []
for i in range(8,20):
    if i<9:
        hours  = (inattention2016['TIME']>= '0%i:00'%i) & (inattention2016['TIME']<='0%i:00'%(i+1))
    else:
        if i==9:
            hours  = (inattention2016['TIME']>= '0%i:00'%i) & (inattention2016['TIME']<='%i:00'%(i+1))
        else:
            hours  = (inattention2016['TIME']>= '%i:00'%i) & (inattention2016['TIME']<='%i:00'%(i+1))
    everyH = inattention2016.loc[hours]
    mean.append(len(everyH)/366)
print "Average inattention in each hours : ", mean
### getting the average for each hour and each day. (basically the xi2) and saving into list xi2
xi2 = []
for i in range(0,366):
    k = '{:%m/%d/%Y}'.format(days[i])
    
    tmpDays  = inattention2016.loc[inattention2016['DATE']==k]
    for j in range(8,20):
        if j<9:
            hours  = (tmpDays['TIME']>= '0%i:00'%j) & (tmpDays['TIME']<='0%i:00'%(j+1))
        else:
            if j==9:
                hours  = (tmpDays['TIME']>= '0%i:00'%j) & (tmpDays['TIME']<='%i:00'%(j+1))
            else:
                hours  = (tmpDays['TIME']>= '%i:00'%j) & (tmpDays['TIME']<='%i:00'%(j+1))
        tmpHours = tmpDays.loc[hours]
        xi2.append(len(tmpHours) - mean[j-8])
#adding data X (xi1) from linear regression to the calculated xi2 (xi2) into an array 'tmp'
xi1 = np.array(x).reshape(4392,1)
xi2 = np.array(xi2).reshape(4392,1)
tmp = np.column_stack((xi1,xi2))
#creating the model from xi1 and x2i and Y 
regrMultiple = linear_model.LinearRegression()
regrMultiple.fit(tmp,y)<jupyter_output><empty_output><jupyter_text>Trying to plot the multiple regression with red line<jupyter_code>%matplotlib inline
line = plt.figure()
for i in range(0,12):
    a = [i+8 for j in range(0,366)]
    b = data[i]
    plt.scatter(a, b, color="black")
plt.plot(xarray, regrMultiple.predict(tmp), color='red',
        linewidth=3)
plt.savefig('figure2')
plt.show()<jupyter_output><empty_output><jupyter_text>Get only the type of at least 2 cars (passenger vehicle) vehicle collisions, drop nan for the latitude and longitude, borough, on street name, vehicle 2 factor<jupyter_code>car =  df.loc[df['VEHICLE TYPE CODE 1'] == "PASSENGER VEHICLE"]
car =  car.loc[car['VEHICLE TYPE CODE 2'] == "PASSENGER VEHICLE"]
car.dropna(subset=['LATITUDE'], inplace=True)
car.dropna(subset=['LONGITUDE'], inplace=True)
car.dropna(subset=['BOROUGH'], inplace=True)
car.dropna(subset=['ON STREET NAME'], inplace=True)
car.dropna(subset=['CONTRIBUTING FACTOR VEHICLE 2'], inplace=True)

car
# get only for April 2017
timespan = ( pd.to_datetime(car['DATE'])  >= '2017-4-1') & ( pd.to_datetime(car['DATE'])  < '2017-5-1')
car = car.loc[timespan]<jupyter_output><empty_output><jupyter_text>Get latitude and logitude ready for creating CSV files<jupyter_code>X = []
Y = []
for i in car['LONGITUDE']:
    X.append(i)
for i in car['LATITUDE']:
    Y.append(i)
    
# get x and y value, if the value is out of the range, set it to the max.
import numpy 
points = numpy.zeros(shape=(len(Y),2))
for i in range(0,len(Y)):
    if Y[i] <41 and X[i]<0: 
        points[i][0] = X[i]
        points[i][1] = Y[i]
    else:
        points[i][0] = -74.251609
        points[i][1]= 40.499186
# get the functions from the textbook
import numpy as np
import random
def cluster_points(X, mu):
    clusters  = {}
    for x in X:
        bestmukey = min([(i[0], np.linalg.norm(x-mu[i[0]])) \
                    for i in enumerate(mu)], key=lambda t:t[1])[0]
        try:
            clusters[bestmukey].append(x)
        except KeyError:
            clusters[bestmukey] = [x]
    return clusters
 
def reevaluate_centers(mu, clusters):
    newmu = []
    keys = sorted(clusters.keys())
    for k in keys:
        newmu.append(np.mean(clusters[k], axis = 0))
    return newmu
def has_converged(mu, oldmu):
    return (set([tuple(a) for a in mu]) == set([tuple(a) for a in oldmu]))

def find_centers(X, K):
    # Initialize to K random centers
    oldmu = random.sample(X, K)
    mu = random.sample(X, K)
    while not has_converged(mu, oldmu):
        oldmu = mu
        # Assign all points in X to clusters
        clusters = cluster_points(X, mu)
        # Reevaluate centers
        mu = reevaluate_centers(oldmu, clusters)
    return(mu, clusters)<jupyter_output><empty_output><jupyter_text>Creating CSV files for k=2,3,4,5,6<jupyter_code>ck2 = find_centers(points,2)
ck3 = find_centers(points,3)
ck4 = find_centers(points,4)
ck5 = find_centers(points,5)
ck6 = find_centers(points,6)
f =  open('ck2.csv', 'w')
f.write("lat,lon,type,\n")
counter = 1


# all the points besides the clusters   
for i in range(0, len(ck2)):
    for j in ck2[1][i]:
        s= ""
        s = s + str(j[0]) +  ','+  str(j[1]) + "," + str(i+1) + '\n'
        f.write(s)
# the clusters points
for i in ck2[0]:
    s= ""
    s = s + str(i[0]) +  ','+  str(i[1]) + "," + str(counter*10) + '\n'
    f.write(s )
    counter=  counter +1 
f.close()
f =  open('ck3.csv', 'w')
f.write("lat,lon,type,\n")
counter = 1


    
for i in range(0, len(ck3)):
    for j in ck3[1][i]:
        s= ""
        s = s + str(j[0]) +  ','+  str(j[1]) + "," + str(i+1) + '\n'
        f.write(s)
for i in ck3[0]:
    s= ""
    s = s + str(i[0]) +  ','+  str(i[1]) + "," + str(counter*10) + '\n'
    f.write(s )
    counter=  counter +1   
f.close()
k = ck4

f =  open('ck4.csv', 'w')
f.write("lat,lon,type,\n")
counter = 1
for i in k[0]:
    s= ""
    s = s + str(i[0]) +  ','+  str(i[1]) + "," + str(counter*10) + '\n'
    f.write(s )
    counter=  counter +1 

    
for i in range(0, len(k[1])):
    for j in k[1][i]:
        s= ""
        s = s + str(j[0]) +  ','+  str(j[1]) + "," + str(i+1) + '\n'
        f.write(s)
    
f.close()
k = ck5

f =  open('ck5.csv', 'w')
f.write("lat,lon,type,\n")
counter = 1
for i in k[0]:
    s= ""
    s = s + str(i[0]) +  ','+  str(i[1]) + "," + str(counter*10) + '\n'
    f.write(s )
    counter=  counter +1 

    
for i in range(0, len(k[1])):
    for j in k[1][i]:
        s= ""
        s = s + str(j[0]) +  ','+  str(j[1]) + "," + str(i+1) + '\n'
        f.write(s)
    
f.close()<jupyter_output><empty_output><jupyter_text>Dataset for Vehicle Type Bicycle(Drop nan for latitude and longitude)<jupyter_code>k = ck5

f =  open('ck5.csv', 'w')
f.write("lat,lon,type,\n")
counter = 1
for i in k[0]:
    s= ""
    s = s + str(i[0]) +  ','+  str(i[1]) + "," + str(counter*10) + '\n'
    f.write(s )
    counter=  counter +1 

    
for i in range(0, len(k[1])):
    for j in k[1][i]:
        s= ""
        s = s + str(j[0]) +  ','+  str(j[1]) + "," + str(i+1) + '\n'
        f.write(s)
    
f.close()<jupyter_output><empty_output><jupyter_text>Get the longitude and latitude ready for creating csv files<jupyter_code>X = []
Y = []
for i in car['LONGITUDE']:
    X.append(i)
for i in car['LATITUDE']:
    Y.append(i)
    
# get x and y value, if the value is out of the range, set it to the max.
import numpy 
points = numpy.zeros(shape=(len(Y),2))
for i in range(0,len(Y)):
    if Y[i] <41 and X[i]<0: 
        points[i][0] = X[i]
        points[i][1] = Y[i]
    else:
        points[i][0] = -74.251609
        points[i][1]= 40.499186
# get the functions from the textbook
import numpy as np
import random
def cluster_points(X, mu):
    clusters  = {}
    for x in X:
        bestmukey = min([(i[0], np.linalg.norm(x-mu[i[0]])) \
                    for i in enumerate(mu)], key=lambda t:t[1])[0]
        try:
            clusters[bestmukey].append(x)
        except KeyError:
            clusters[bestmukey] = [x]
    return clusters
 
def reevaluate_centers(mu, clusters):
    newmu = []
    keys = sorted(clusters.keys())
    for k in keys:
        newmu.append(np.mean(clusters[k], axis = 0))
    return newmu
def has_converged(mu, oldmu):
    return (set([tuple(a) for a in mu]) == set([tuple(a) for a in oldmu]))

def find_centers(X, K):
    # Initialize to K random centers
    oldmu = random.sample(X, K)
    mu = random.sample(X, K)
    while not has_converged(mu, oldmu):
        oldmu = mu
        # Assign all points in X to clusters
        clusters = cluster_points(X, mu)
        # Reevaluate centers
        mu = reevaluate_centers(oldmu, clusters)
    return(mu, clusters)<jupyter_output><empty_output><jupyter_text>Create csv files for k=2-6<jupyter_code>ck2 = find_centers(points,2)
ck3 = find_centers(points,3)
ck4 = find_centers(points,4)
ck5 = find_centers(points,5)
ck6 = find_centers(points,6)
f =  open('ck2.csv', 'w')
f.write("lat,lon,type,\n")
counter = 1


# all the points besides the clusters   
for i in range(0, len(ck2)):
    for j in ck2[1][i]:
        s= ""
        s = s + str(j[0]) +  ','+  str(j[1]) + "," + str(i+1) + '\n'
        f.write(s)
# the clusters points
for i in ck2[0]:
    s= ""
    s = s + str(i[0]) +  ','+  str(i[1]) + "," + str(counter*10) + '\n'
    f.write(s )
    counter=  counter +1 
f.close()
f =  open('ck3.csv', 'w')
f.write("lat,lon,type,\n")
counter = 1


    
for i in range(0, len(ck3)):
    for j in ck3[1][i]:
        s= ""
        s = s + str(j[0]) +  ','+  str(j[1]) + "," + str(i+1) + '\n'
        f.write(s)
for i in ck3[0]:
    s= ""
    s = s + str(i[0]) +  ','+  str(i[1]) + "," + str(counter*10) + '\n'
    f.write(s )
    counter=  counter +1   
f.close()
k = ck4

f =  open('ck4.csv', 'w')
f.write("lat,lon,type,\n")
counter = 1
for i in k[0]:
    s= ""
    s = s + str(i[0]) +  ','+  str(i[1]) + "," + str(counter*10) + '\n'
    f.write(s )
    counter=  counter +1 

    
for i in range(0, len(k[1])):
    for j in k[1][i]:
        s= ""
        s = s + str(j[0]) +  ','+  str(j[1]) + "," + str(i+1) + '\n'
        f.write(s)
    
f.close()
k = ck5

f =  open('ck5.csv', 'w')
f.write("lat,lon,type,\n")
counter = 1
for i in k[0]:
    s= ""
    s = s + str(i[0]) +  ','+  str(i[1]) + "," + str(counter*10) + '\n'
    f.write(s )
    counter=  counter +1 

    
for i in range(0, len(k[1])):
    for j in k[1][i]:
        s= ""
        s = s + str(j[0]) +  ','+  str(j[1]) + "," + str(i+1) + '\n'
        f.write(s)
    
f.close()
k = ck6

f =  open('ck6.csv', 'w')
f.write("lat,lon,type,\n")
counter = 1
for i in k[0]:
    s= ""
    s = s + str(i[0]) +  ','+  str(i[1]) + "," + str(counter*10) + '\n'
    f.write(s )
    counter=  counter +1 

    
for i in range(0, len(k[1])):
    for j in k[1][i]:
        s= ""
        s = s + str(j[0]) +  ','+  str(j[1]) + "," + str(i+1) + '\n'
        f.write(s)
    
f.close()<jupyter_output><empty_output><jupyter_text>Dataset for vehicle type bicycle (drop nan for latitude and longitude)<jupyter_code>bike =  df.loc[df['VEHICLE TYPE CODE 1'] == "BICYCLE"]
bike.dropna(subset=['LATITUDE'], inplace=True)
bike.dropna(subset=['LONGITUDE'], inplace=True)
bike<jupyter_output><empty_output><jupyter_text>Get the longitude and latitude ready for creating csv files<jupyter_code>X = []
Y = []
for i in bike['LONGITUDE']:
    X.append(i)
for i in bike['LATITUDE']:
    Y.append(i)
    
# get x and y value, if the value is out of the range, set it to the max.
import numpy 
points = numpy.zeros(shape=(len(Y),2))
for i in range(0,len(Y)):
    if Y[i] <41 and X[i]<0: 
        points[i][0] = X[i]
        points[i][1] = Y[i]
    else:
        points[i][0] = -74.251609
        points[i][1]= 40.499186<jupyter_output><empty_output><jupyter_text>Create csv files for k=2-6<jupyter_code>bk2 = find_centers(points,2)
bk3 = find_centers(points,3)
bk4 = find_centers(points,4)
bk5 = find_centers(points,5)
bk6 = find_centers(points,6)
f =  open('bk2.csv', 'w')
f.write("lat,lon,type,\n")
counter = 1


# all the points besides the clusters   
for i in range(0, len(bk2)):
    for j in bk2[1][i]:
        s= ""
        s = s + str(j[0]) +  ','+  str(j[1]) + "," + str(i+1) + '\n'
        f.write(s)
#the clusters points
for i in bk2[0]:
    s= ""
    s = s + str(i[0]) +  ','+  str(i[1]) + "," + str(counter*10) + '\n'
    f.write(s )
    counter=  counter +1 
f.close()
f =  open('bk3.csv', 'w')
f.write("lat,lon,type,\n")
counter = 1


    
for i in range(0, len(bk3)):
    for j in bk3[1][i]:
        s= ""
        s = s + str(j[0]) +  ','+  str(j[1]) + "," + str(i+1) + '\n'
        f.write(s)
for i in bk3[0]:
    s= ""
    s = s + str(i[0]) +  ','+  str(i[1]) + "," + str(counter*10) + '\n'
    f.write(s )
    counter=  counter +1   
f.close()
k = bk4

f =  open('bk4.csv', 'w')
f.write("lat,lon,type,\n")
counter = 1
for i in k[0]:
    s= ""
    s = s + str(i[0]) +  ','+  str(i[1]) + "," + str(counter*10) + '\n'
    f.write(s )
    counter=  counter +1 

    
for i in range(0, len(k[1])):
    for j in k[1][i]:
        s= ""
        s = s + str(j[0]) +  ','+  str(j[1]) + "," + str(i+1) + '\n'
        f.write(s)
    
f.close()
k = bk5

f =  open('bk5.csv', 'w')
f.write("lat,lon,type,\n")
counter = 1
for i in k[0]:
    s= ""
    s = s + str(i[0]) +  ','+  str(i[1]) + "," + str(counter*10) + '\n'
    f.write(s )
    counter=  counter +1 

    
for i in range(0, len(k[1])):
    for j in k[1][i]:
        s= ""
        s = s + str(j[0]) +  ','+  str(j[1]) + "," + str(i+1) + '\n'
        f.write(s)
    
f.close()
k = bk6

f =  open('bk6.csv', 'w')
f.write("lat,lon,type,\n")
counter = 1
for i in k[0]:
    s= ""
    s = s + str(i[0]) +  ','+  str(i[1]) + "," + str(counter*10) + '\n'
    f.write(s )
    counter=  counter +1 

    
for i in range(0, len(k[1])):
    for j in k[1][i]:
        s= ""
        s = s + str(j[0]) +  ','+  str(j[1]) + "," + str(i+1) + '\n'
        f.write(s)
    
f.close()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1708701768488746, 'fraction_numerical': 0.04074557877813505, 'mean_word_length': 3.6446155641115388, 'pattern_counts': {'":': 10, '<': 108, '<?xml version=': 0, '>': 104, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '282363', 'license': 'CC-BY-3.0', 'n_tokens_mistral': 16305, 'n_tokens_neox': 15078, 'n_words': 3741, 'path': 'Notebook.ipynb', 'repo_name': 'dancoconetu/dancoconetu.github.io', 'size': '39808'}