<jupyter_start><jupyter_text>Safety Recommenders Project:  Project Description The objective of the project is create a data product that helps people that are traveling to Washington DC to determine how safe are the most popular places in the city.  Dataset Description This Project will use two datasets. The first dataset is a csv file that contains all the crimes that have occurred in the past two and a half years around Washington DC. This dataset does not contain information about locations where there was no crime.The second dataset is a csv file that contains all the adresses in Washington DC. The main use of this dataset is to use in conjunction with the first file in order determine location where no crime occurred.Finally, after the first analyses, some of the features of the crime dataset will be removed to make the data product simpler to use. Features like date, time, location and address will be kept, and features like offense type and location codes used by the police department will also be removed.  Importing Necessary Libraries:<jupyter_code>%matplotlib notebook
import IPython
from IPython.display import display
from sqlalchemy import create_engine
import psycopg2
import psycopg2.extras
import pandas as pd
import csv
from numpy import nan as NA
from datetime import datetime
import re
import sys
import numpy as np
import matplotlib.pyplot as plt 
import scipy as sp
import sklearn
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.preprocessing import MinMaxScaler
from sklearn.preprocessing import Normalizer
from sklearn import model_selection
from sklearn.model_selection import cross_val_score
from sklearn.metrics import classification_report
from pandas import *
import pickle<jupyter_output><empty_output><jupyter_text>Data Ingestion:Here We are creating the module that will take care of our csv file. One class is designed to deal with csv files and the other class to deal with a postgres database.<jupyter_code>class Ingestion(object):
    """This is the ingestion class to deal with csv directly from the same directory where the module is"""

    def __init__(self, file, sep = ",", header = 0 ):
        self.file = file
        self.delimiter = sep
        self.df = pd.read_csv(file, sep= sep, header=header, engine='python')

    def file_csv(self):
        return self.df

class IngestionDatabase(object):

    """ This is the ingestion class to deal with postgress database """
    def __init__(self, database, query):
        self.engine = create_engine(database)
        self.table_names = self.engine.table_names()
        self.con = self.engine.connect()
        self.rs = self.con.execute(query)
        self.df = pd.DataFrame(self.rs.fetchmany(size=15))

    def cols(self):
        self.df.columns = self.rs.keys()
        return self.df<jupyter_output><empty_output><jupyter_text>Creating the Ingestion instances and exploring the raw dataset:<jupyter_code>ingest = Ingestion('DC_Crime_Official.csv')
data = ingest.file_csv()
with open('file1.pickle', 'wb') as object1:
    
    pickle.dump([ingest,data] , object1)<jupyter_output><empty_output><jupyter_text>Loading the objects(only use when running time is decreased)!!<jupyter_code>with open('file1.pickle', 'rb') as objects:
    
    object_list = pickle.load(objects)
    
data = object_list[1]
data.head(2)<jupyter_output><empty_output><jupyter_text>Initial Data Wrangling of Crime Dataset<jupyter_code>class Wrangling(object):

    def __init__(self, data = data):
        self.df = data
# drop empty rows
    def dropNA(self):
        self.df = self.df.dropna(how='all') # this only drop rows with 100% NA
        return self.df

    def __offense_column(self, text1 ='theft/other', text2 ='theft f/auto', text3 = 'assault w/dangerous weapon',
                       repl1 = 'theft', repl2 = 'auto theft', repl3 = 'assault with weapon' ):

        """There are 9 categories of offenses here:
        This function will transform the caterogies into more readable text
        for example : assault w/dangerous weapon = assault with dangerous weapon"""

        self.df['offense_text'] = self.df['offense_text'].replace([text1, text2, # add the column name to the arguments.
        text3], [repl1, repl2, repl3])
        return self.df

    def date_time_transformer(self, time = 'start_date', second_date = 'report_date', third_date = 'end_date'):
        ''' transform into datetime 64 object and eliminate the second date column'''
        self.df[time] = pd.to_datetime(self.df['start_date'])
        self.df.drop([second_date, third_date], axis = 1, inplace = True)
        return self.df

    def __latlong_cutter(self):
        """ Reduce the presition of the lat long data by cutting them."""
        self.newlat = []
        self.newlon = []
        for item in self.df['latitude']:
            item = str(item)
            item = float(item[0:6])
            self.newlat.append(item)

        self.df['latitude'] = self.newlat

        for item in self.df['longitude']:
            item = str(item)
            item = float(item[0:7])
            self.newlon.append(item)
        self.df['longitude'] = self.newlon

        return self.df

    def lat_long_rounder(self, decimals = 3):
        """ Reduce the presition of the lat long data by rounging decimals"""
        self.df['latitude'] = self.df['latitude'].round(decimals = decimals)
        self.df['longitude'] = self.df['longitude'].round(decimals = decimals)
        return self.df

    def adress_format_modifier(self):
        """This columns replace some of the content from the block columns to it is easy to parse it"""

        self.splitted = []

        # creating the splited column
        # this is working. it cannot be transformed into pandas' .replace because it is using the split method
        # Note that the built in .replace it does not work properly with integers and neither with large amounts of
        # things to change.. This works but it is not very wise to use.
        for row in self.df['block']:
            row = row.replace("block of ", "")
            row = row.replace("street", "St")
            row = row.replace("-", "")
            row = row.split(' ', 1)
            self.splitted.append(row)
        self.df['splitted'] = self.splitted
        return self.df

    def block_parser(self):
        """ This is the block parser that separate block in start and en blocks"""

        self.startblock = []
        self.endblock_1 = []
        self.endblock = []
        #  create column 'startblock'
        for row in self.df['splitted']:
            row = row[0]
            self.startblock.append(row)
        self.df['startblock'] = self.startblock
        # create column  'endblock_1'
        for row in self.df['splitted']:
            row = row[-1].lstrip() # enblock_1
            row = row.split(' ',1)
            self.endblock_1.append(row)
        self.df['endblock_1'] = self.endblock_1
        # create column  'endblock'
        for row in self.df['endblock_1']:
            row = row[0]
            self.endblock.append(row)
        self.df['endblock'] = self.endblock
        return self.df

    def street_parser(self):
        self.street = []
        #creating column 'street'
        for row in self.df['endblock_1']:
            row = row[1]
            self.street.append(row)
        self.df['street'] = self.street
        return self.df<jupyter_output><empty_output><jupyter_text>Creating the wrangling instances:<jupyter_code>Wrangled = Wrangling()
Wrangled.dropNA()
Wrangled.date_time_transformer()
Wrangled.lat_long_rounder()
Wrangled.adress_format_modifier()
Wrangled.block_parser()
df = Wrangled.street_parser()
with open('file2.pickle', 'wb') as object2:
    
    pickle.dump(df , object2)<jupyter_output><empty_output><jupyter_text>Loading the objects(only use when running time is decreased)!!<jupyter_code>with open('file2.pickle', 'rb') as objects2:
    
    object_ = pickle.load(objects2)
    
df = object_<jupyter_output><empty_output><jupyter_text>Dropping repeated columns created during the first wrangling process:<jupyter_code>df = df.drop(columns = ['location', 'endblock_1', 'splitted'])
df.info()
from datetime import datetime
df["start_date"] = pd.to_datetime(df["start_date"])
df["year"] =df["start_date"].dt.year
df["month"] =df["start_date"].dt.month
df["day"] =df["start_date"].dt.day
df["hour"] =df["start_date"].dt.hour
df["minute"] =df["start_date"].dt.minute
df["second"] =df["start_date"].dt.second


# Now, Eliminate the start_date column.
df = df.drop(columns = 'start_date')<jupyter_output><empty_output><jupyter_text>Exploration of the dataset and preparation of the dataset for Predictions:  Description of the dataset:<jupyter_code>df.describe()<jupyter_output><empty_output><jupyter_text>Preparing X and y sets. Dropping Remaining na values:<jupyter_code>df = df.dropna()

X = df.drop(columns = ['ucrrank'])

y = df['ucrrank']<jupyter_output><empty_output><jupyter_text>Encoding the Categorical variables:<jupyter_code>from sklearn.preprocessing import LabelEncoder
encoder = LabelEncoder() # it only support one dimentional columns.. 

for colname,col in X.iteritems(): # adapted from stack overflow
    if col is not float:
        X[colname] = LabelEncoder().fit_transform(col)<jupyter_output><empty_output><jupyter_text>Studying  the importance of the features using principal component analysis:<jupyter_code>X.columns
from sklearn.decomposition import PCA
pca = PCA(n_components = 10) # input a number for feature extraction

features = X

X_ = pca.fit_transform(features)

explained_var = pca.explained_variance_ratio_
explained_var<jupyter_output><empty_output><jupyter_text>It can be appreciated here that there is one variable that explains most of the variability of the dataset.<jupyter_code>plt.plot(np.cumsum(pca.explained_variance_ratio_))
plt.xlabel('number of components')
plt.ylabel('cumulative explained variance');<jupyter_output><empty_output><jupyter_text>Creating the Training and test set:<jupyter_code>X_train, X_test, y_train, y_test = model_selection.train_test_split(X, y, test_size = 0.2, random_state = 0)<jupyter_output><empty_output><jupyter_text>Visual Exploration<jupyter_code>cols = pd.DataFrame(features)
cols = list(cols.columns)

boxplot = X.boxplot(column=cols, rot=90, fontsize=10)<jupyter_output><empty_output><jupyter_text>As it can be appreciated here, there is one variable that has the greatest variability. Since we do not now what that means, it will be temporary eliminated in order to study the dispersion of the other features<jupyter_code>from sklearn.decomposition import PCA
pca = PCA(n_components = 10) # input a number for feature extraction

features2 = X.drop(columns = 'cnn')

X_ = pca.fit_transform(features)

explained_var = pca.explained_variance_ratio_
explained_var
cols2 = pd.DataFrame(features2)
cols2 = list(cols2.columns)

boxplot = X.boxplot(column=cols2, rot=90, fontsize=10)<jupyter_output><empty_output><jupyter_text>We can see here that location present the greatest dispersion of data. I the last section we well select from the dataframe only the features that can be easily provided by the user: time, street lat, long, etc.  Standarization of the dataset:Since this two graphs shows that there is a great dispersion of the data. They will be standarized:<jupyter_code>scaler = StandardScaler()
#scaler =  MinMaxScaler()
#scaler = Normalizer()
X_train = scaler.fit(X_train).transform(X_train)
X_test = scaler.fit(X_test).transform(X_test)<jupyter_output><empty_output><jupyter_text>Transforming the y labels into numpy arrays to increase computation performance:<jupyter_code>y_train = y_train.values
y_test = y_test.values


cols = pd.DataFrame(features)
cols = list(cols.columns)
X_train_df = pd.DataFrame(X_train)
boxplot = X_train_df.boxplot( rot=90, fontsize=10)<jupyter_output><empty_output><jupyter_text>Now all the features have been standarized.  Fitting models and performing predictions over the crime dataset:  KNN:<jupyter_code>knn = KNeighborsClassifier(n_neighbors = 10, metric = 'manhattan', weights = 'uniform', algorithm = 'auto')
knn.fit(X_train, y_train)
predicted_knn = knn.predict(X_test)
print("Predictions: {}".format(predicted_knn))<jupyter_output>Predictions: [6 7 6 ... 6 6 7]<jupyter_text>Cross Validation:<jupyter_code>scores = cross_val_score(knn, X = X_train, y = y_train)
print ("Cross Validation Scores: {}".format(scores))
report = classification_report(y_test, predicted_knn)
print (report)<jupyter_output>precision    recall  f1-score   support

          1       0.00      0.00      0.00         2
          2       1.00      0.13      0.24        15
          3       0.99      0.99      0.99        77
          4       0.88      0.99      0.93       103
          5       0.97      0.90      0.93        70
          6       0.94      0.99      0.96      1438
          7       0.92      0.84      0.88       591
          8       0.91      0.67      0.77        93

avg / total       0.93      0.93      0.93      2389<jupyter_text>Finding the best parameters for KnnIn this case, gridsearchCV and a simple loop will be used in order to find the optimal hyperparameters for KNN<jupyter_code>from sklearn.model_selection import GridSearchCV
from sklearn.model_selection import RandomizedSearchCV

params2 = [{'n_neighbors': [1,10,50,100], 'algorithm': ['auto','ball_tree','kd_tree' ], 
            'weights': ['uniform', 'distance'], 'metric': ['minkowski', 'manhattan']}]
            
grid_search = GridSearchCV(estimator = knn, param_grid = params2, scoring = 'accuracy', cv = 5, n_jobs = -1)
grid_search = grid_search.fit(X_train, y_train)
accuracy = grid_search.best_score_
best_params = grid_search.best_params_
print(accuracy)
print(best_params)
knn = [params2, grid_search, accuracy, best_params]
with open('file3.pickle', 'wb') as object3:
    pickle.dump(knn , object3)<jupyter_output><empty_output><jupyter_text>Loading the objects(to avoid recalculations)!!<jupyter_code>with open('file3.pickle', 'rb') as objects3:
    
    knn_obj = pickle.load(objects3)
    
knn_obj
train_accuracy = []
test_accuracy = []

neighbors = range(1,100,10)
algorithms = ['auto', 'ball_tree', 'kd_tree']
weights = ['uniform', 'distance']

for i in neighbors:
    knn = KNeighborsClassifier(n_neighbors = i, metric = 'manhattan', weights = 'distance', algorithm = 'auto')
    knn.fit(X_train, y_train)
    train_accuracy.append(knn.score(X_train, y_train))
    test_accuracy.append(knn.score(X_test, y_test))
figure1 = plt.plot(neighbors, train_accuracy, label = 'Train set accuracy')
figure2 = plt.plot(neighbors, test_accuracy, label = 'Test set accuracy')
plt.ylabel("Accuracy")
plt.xlabel("Number of neighbors")
plt.legend()
plt.show()
fig = [figure1, figure2]
with open('figure1.pickle', 'wb') as figure:
    pickle.dump(fig , figure)<jupyter_output><empty_output><jupyter_text>Loading the objects(to avoid recalculations)!!<jupyter_code>with open('figure1.pickle', 'rb') as figures:
    
    figs = pickle.load(figures)
    
figs<jupyter_output><empty_output><jupyter_text>For this model, the level of accuracy is very high. It will probably be possible to increase even more using other models.  Kernel SVC:<jupyter_code>from sklearn.svm import SVC
svm = SVC(C = 1000, kernel = 'rbf', gamma = 1)
svm.fit(X_train, y_train)
predicted = svm.predict(X_test)

#print("Predictions: {}".format(predicted))

scores = cross_val_score(svm, X = X_train, y = y_train)
report = classification_report(y_test, predicted)
print (report)

# print ("Cross Validation Scores: {}".format(scores))
ksvc = [svm, predicted, scores, report]
with open('file4.pickle', 'wb') as object4:
    pickle.dump(ksvc , object4)
with open('file4.pickle', 'rb') as objects4:
    
    Ksvc = pickle.load(objects4)
    
print(Ksvc[-1])<jupyter_output><empty_output><jupyter_text>Finding the best parameters for Kernel SVC:<jupyter_code>params = [{'C': [1, 10, 30, 100], 'kernel': ['rbf'], 'gamma': [1, 0.1, 0.01, 0.001]}]
     
                          
grid_search = GridSearchCV(estimator = svm, param_grid = params, scoring = 'accuracy', cv = 5, n_jobs = -1)
grid_search = grid_search.fit(X_train, y_train)
accuracySVC = grid_search.best_score_
best_paramsSVC = grid_search.best_params_
print(accuracySVC)
print(best_paramsSVC)
params = [grid_search, accuracySVC, best_paramsSVC]
with open('file5.pickle', 'wb') as object5:
    pickle.dump(params , object5)
with open('file5.pickle', 'rb') as objects5:
    
    best_params = pickle.load(objects5)
    
print(best_params[-1])
train_accuracy = []
test_accuracy = []

Ci = [1,10, 50, 100]

for i in Ci:
    svm = SVC(C = i, kernel = 'rbf', gamma = 0.001) # try rbf, linear and poly
    svm.fit(X_train, y_train)
    train_accuracy.append(svm.score(X_train, y_train))
    test_accuracy.append(svm.score(X_test, y_test))
figure3 = plt.plot(Ci, train_accuracy, label = 'Train set accuracy')
figure4 = plt.plot(Ci, test_accuracy, label = 'Test set accuracy')
plt.ylabel("Accuracy")
plt.xlabel("C")
plt.legend()
plt.show()
fig1 = [figure3, figure4]
with open('figure2.pickle', 'wb') as figure2:
    pickle.dump(fig1 , figure2)
with open('figure2.pickle', 'rb') as figures2:
    
    figs2 = pickle.load(figures2)
    
figs2<jupyter_output><empty_output><jupyter_text>Merging two datasets  In this section we will add information from another dataset. This dataset contains all the adress in Washington DC. Since the crime dataset does not contain information when there was no crime, we need a way to add the location when no crime happened.<jupyter_code>df1 = data[['latitude', 'longitude', 'ucrrank']] # selecting the relevant labels from the wrangled crime dataset (df).<jupyter_output><empty_output><jupyter_text>Since here there are no missing values, we can use the whole rows from the crime dataset.<jupyter_code># ingesting new dataset
ingest2 = Ingestion('Address_Points.csv', sep = ',', header = 0)
data2 = ingest2.file_csv()
data2.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 147181 entries, 0 to 147180
Data columns (total 52 columns):
X                             147181 non-null float64
Y                             147181 non-null float64
OBJECTID_12                   147181 non-null int64
SITE_ADDRESS_PK               147181 non-null int64
ADDRESS_ID                    147181 non-null int64
STATUS                        147181 non-null object
SSL                           146816 non-null object
TYPE_                         147181 non-null object
ENTRANCETYPE                  147181 non-null object
ADDRNUM                       145278 non-null float64
ADDRNUMSUFFIX                 2258 non-null object
STNAME                        147181 non-null object
STREET_TYPE                   147181 non-null object
QUADRANT                      147181 non-null object
CITY                          147181 non-null object
STATE                         147181 non-null object
FULLADDRESS                   145278 non-nu[...]<jupyter_text>Selecting relevant columns and transforming labels into lowercase for standarization:<jupyter_code>df2 = data2[['LATITUDE', 'LONGITUDE']] # selecting the relevant columns. 

df2.columns = df2.columns.str.lower() # transforming labels to lowercase

df2.head(5)

df1 = data[['latitude', 'longitude', 'ucrrank']] # selecting the relevant labels from the wrangled crime dataset (df).
df1.ucrrank.describe()
Wrangled2 = Wrangling(df2)
# Wrangled.dropNA()
df2 = Wrangled2.lat_long_rounder()<jupyter_output>/home/franco/.local/lib/python3.6/site-packages/ipykernel_launcher.py:48: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
/home/franco/.local/lib/python3.6/site-packages/ipykernel_launcher.py:49: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy<jupyter_text>Before merging. In the Adress_point dataset, we need a way to eliminate all the rows from the locations that match the crime dataset.<jupyter_code>lat = []
long = []


for i, j, k, l  in zip(df1['latitude'], df2['latitude'], df1['longitude'], df2['longitude']):
    if str(j) in str(i) and str(l) in str(k):
        lat.append(j)
        long.append(l)<jupyter_output><empty_output><jupyter_text>This means that we only have 27 blocks when no crime ocurred in Washington DC during the last 2 years.It would be interesting to see where are those blocks:<jupyter_code># Creating the new dataframe:


second_dataframe = {'latitude': lat, 'longitude': long}
second_dataframe = pd.DataFrame(second_dataframe)
second_dataframe['ucrrank'] = 10
second_dataframe
frames = [df1, second_dataframe]
df_merged = pd.concat(frames, sort = False)
df_merged.head(5)
merged = df_merged
with open('file6.pickle', 'wb') as file6:
    pickle.dump(merged , file6)
with open('file6.pickle', 'rb') as file_6:
    
    merg = pickle.load(file_6)
    
merg<jupyter_output><empty_output><jupyter_text>Performing predictions the new dataset:<jupyter_code>X = df_merged.drop(columns = ['ucrrank'])

y = df_merged['ucrrank']<jupyter_output><empty_output><jupyter_text>no need to encode the labels in this case... no categorical variables: Creating the Training and test set:<jupyter_code>X_train, X_test, y_train, y_test = model_selection.train_test_split(X, y, test_size = 0.2, random_state = 0)<jupyter_output><empty_output><jupyter_text>Scaling the features:<jupyter_code>scaler = StandardScaler()
#scaler =  MinMaxScaler()
#scaler = Normalizer()
X_train = scaler.fit(X_train).transform(X_train)
X_test = scaler.fit(X_test).transform(X_test)<jupyter_output><empty_output><jupyter_text>Transforming to numpy array to increase performance:<jupyter_code>y_test = y_test.values
y_train = y_train.values<jupyter_output><empty_output><jupyter_text>Performing simple Knn on merged dataset<jupyter_code>knn = KNeighborsClassifier(n_neighbors = 10, metric = 'manhattan', weights = 'uniform', algorithm = 'auto')
knn.fit(X_train, y_train)
predicted_knn = knn.predict(X_test)
print("Predictions: {}".format(predicted_knn))
scores = cross_val_score(knn, X = X_train, y = y_train)
print ("Cross Validation Scores: {}".format(scores))
report = classification_report(y_test, predicted_knn)
print (report)<jupyter_output>precision    recall  f1-score   support

          1       0.00      0.00      0.00        54
          2       0.09      0.01      0.01       131
          3       0.20      0.16      0.17       810
          4       0.15      0.06      0.08       997
          5       0.17      0.05      0.08       698
          6       0.55      0.69      0.61      5683
          7       0.49      0.58      0.53      4449
          8       0.21      0.05      0.08      1019
         10       0.00      0.00      0.00         6

avg / total       0.43      0.49      0.45     13847<jupyter_text>Finding the best parameters for Knn on merged dataset:<jupyter_code>from sklearn.model_selection import GridSearchCV
from sklearn.model_selection import RandomizedSearchCV

params2 = [{'n_neighbors': [1,10,50,100], 'algorithm': ['auto','ball_tree','kd_tree' ], 
            'weights': ['uniform', 'distance'], 'metric': ['minkowski', 'manhattan']}]
            
grid_search = GridSearchCV(estimator = knn, param_grid = params2, scoring = 'accuracy', cv = 5, n_jobs = -1)
grid_search = grid_search.fit(X_train, y_train)
accuracy = grid_search.best_score_
best_params = grid_search.best_params_
print(accuracy)
print(best_params)
knn_merged = [grid_search, accuracy, best_params]
with open('file7.pickle', 'wb') as file7:
    pickle.dump(knn_merged , file7)
with open('file7.pickle', 'rb') as file_7:
    
    knnmerg = pickle.load(file_7)
    
knnmerg<jupyter_output><empty_output><jupyter_text>Kernel SVC on merged dataset:<jupyter_code>svm = SVC(C = 10, kernel = 'rbf', gamma = 0.1)
svm.fit(X_train, y_train)
predicted = svm.predict(X_test)

#print("Predictions: {}".format(predicted))

scores = cross_val_score(svm, X = X_train, y = y_train)
report = classification_report(y_test, predicted)
print (report)

# print ("Cross Validation Scores: {}".format(scores))
svc_merged = [predicted, scores, report]
with open('file8.pickle', 'wb') as file8:
    pickle.dump(svc_merged , file8)
with open('file8.pickle', 'rb') as file_8:
    
    svcmerg = pickle.load(file_8)
    
svcmerg<jupyter_output><empty_output><jupyter_text>Finding the best parameters for Kernel SVC on merged dataset:<jupyter_code>params = [{'C': [1, 10, 30, 100], 'kernel': ['rbf'], 'gamma': [1, 0.1, 0.01, 0.001]}]
     
                          
grid_search = GridSearchCV(estimator = svm, param_grid = params, scoring = 'accuracy', cv = 5, n_jobs = -1)
grid_search = grid_search.fit(X_train, y_train)
accuracySVC = grid_search.best_score_
best_paramsSVC = grid_search.best_params_
print(accuracySVC)
print(best_paramsSVC)
svc_merged2 = [predicted, scores, report]
with open('file9.pickle', 'wb') as file9:
    pickle.dump(svc_merged , file9)
with open('file9.pickle', 'rb') as file_9:
    
    svcmerg2 = pickle.load(file_9)
    
svcmerg2<jupyter_output><empty_output><jupyter_text>Multilayer Perceptron test on merged dataset<jupyter_code>from sklearn.neural_network import MLPClassifier
clf = MLPClassifier(solver='lbfgs', alpha=1e-5,
                     hidden_layer_sizes=(200,), random_state=1, max_iter = 200)
clf.fit(X_train, y_train)  
predicted = clf.predict(X_test)
report = classification_report(y_test, predicted)
print (report)<jupyter_output>precision    recall  f1-score   support

          1       0.00      0.00      0.00        54
          2       0.00      0.00      0.00       131
          3       0.00      0.00      0.00       810
          4       0.00      0.00      0.00       997
          5       0.00      0.00      0.00       698
          6       0.46      0.74      0.57      5683
          7       0.46      0.48      0.47      4449
          8       0.00      0.00      0.00      1019
         10       0.00      0.00      0.00         6

avg / total       0.34      0.46      0.38     13847<jupyter_text>The score it lower than using the traditional models. It will be worth triying more layers and iteration if the computation power allows it.<jupyter_code>mlp = [predicted, report]
with open('file10.pickle', 'wb') as file10:
    pickle.dump(mlp , file10)
with open('file10.pickle', 'rb') as file_10:
    
    Mlp = pickle.load(file_10)
    
Mlp<jupyter_output><empty_output><jupyter_text>Other models:<jupyter_code>from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification

clf = RandomForestClassifier(max_depth= None, min_samples_split = 2, random_state=0, criterion='entropy')

clf.fit(X_train, y_train)

predicted = clf.predict(X_test)
report = classification_report(y_test, predicted)
print (report)<jupyter_output>precision    recall  f1-score   support

          1       0.00      0.00      0.00        54
          2       0.00      0.00      0.00       131
          3       0.19      0.09      0.12       810
          4       0.17      0.05      0.08       997
          5       0.10      0.03      0.05       698
          6       0.53      0.56      0.54      5683
          7       0.44      0.62      0.51      4449
          8       0.13      0.07      0.09      1019
         10       0.00      0.00      0.00         6

avg / total       0.40      0.44      0.41     13847<jupyter_text>Knn and Random forest classifier reported the highest accuracy. This dataset contains only 2 independent features:Latitude and longitude. This indicates that there are more features tha directly influence the dependent variable:  User Friendly Version:This analysis will try to predict the most likely crime based on features that are easy to understand and user friendly: We will start by selecting those features from the crime dataset:<jupyter_code>df_friendly = df[['latitude', 'longitude', 'shift', 'year', 'month', 'day', 'hour', 'minute', 'street', 'startblock', 'endblock', 'ucrrank']]
X = df_friendly.drop(columns = ['ucrrank'])

y = df_friendly['ucrrank']<jupyter_output><empty_output><jupyter_text>Encoding the Categorical variables:<jupyter_code>from sklearn.preprocessing import LabelEncoder
encoder = LabelEncoder() # it only support one dimentional columns.. 

for colname,col in X.iteritems(): # adapted from stack overflow
    if col is not float:
        X[colname] = LabelEncoder().fit_transform(col)<jupyter_output><empty_output><jupyter_text>Studying  the importance of the features :<jupyter_code>from sklearn.decomposition import PCA
pca = PCA(n_components = 10) # input a number for feature extraction

features = X

X_ = pca.fit_transform(features)

explained_var = pca.explained_variance_ratio_
explained_var

plt.plot(np.cumsum(pca.explained_variance_ratio_))
plt.xlabel('number of components')
plt.ylabel('cumulative explained variance');<jupyter_output><empty_output><jupyter_text>Creating the Training and test set:<jupyter_code>X_train, X_test, y_train, y_test = model_selection.train_test_split(X, y, test_size = 0.2, random_state = 0)<jupyter_output><empty_output><jupyter_text>Standarization of the dataset:<jupyter_code>scaler = StandardScaler()
#scaler =  MinMaxScaler()
#scaler = Normalizer()
X_train = scaler.fit(X_train).transform(X_train)
X_test = scaler.fit(X_test).transform(X_test)
y_train = y_train.values
y_test = y_test.values<jupyter_output><empty_output><jupyter_text>KNN(User Friendly Version):<jupyter_code>knn = KNeighborsClassifier(n_neighbors = 10, metric = 'manhattan', weights = 'uniform', algorithm = 'auto')
knn.fit(X_train, y_train)
predicted_knn = knn.predict(X_test)
print("Predictions: {}".format(predicted_knn))<jupyter_output>Predictions: [6 6 6 ... 6 6 6]<jupyter_text>Cross Validation:<jupyter_code>scores = cross_val_score(knn, X = X_train, y = y_train)
print ("Cross Validation Scores: {}".format(scores))
report = classification_report(y_test, predicted_knn)
print (report)
train_accuracy = []
test_accuracy = []

neighbors = range(1,100,10)
algorithms = ['auto', 'ball_tree', 'kd_tree']
weights = ['uniform', 'distance']

for i in neighbors:
    knn = KNeighborsClassifier(n_neighbors = i, metric = 'manhattan', weights = 'distance', algorithm = 'auto')
    knn.fit(X_train, y_train)
    train_accuracy.append(knn.score(X_train, y_train))
    test_accuracy.append(knn.score(X_test, y_test))
figure1 = plt.plot(neighbors, train_accuracy, label = 'Train set accuracy')
figure2 = plt.plot(neighbors, test_accuracy, label = 'Test set accuracy')
plt.ylabel("Accuracy")
plt.xlabel("Number of neighbors")
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>Kernel SVC(User Friendly Version):<jupyter_code>from sklearn.svm import SVC
svm = SVC(C = 1000, kernel = 'rbf', gamma = 1)
svm.fit(X_train, y_train)
predicted = svm.predict(X_test)

#print("Predictions: {}".format(predicted))

scores = cross_val_score(svm, X = X_train, y = y_train)
report = classification_report(y_test, predicted)
print (report)

# print ("Cross Validation Scores: {}".format(scores))<jupyter_output>precision    recall  f1-score   support

          1       0.00      0.00      0.00         2
          2       0.00      0.00      0.00        15
          3       0.12      0.04      0.06        77
          4       0.11      0.06      0.08       103
          5       0.20      0.06      0.09        70
          6       0.64      0.84      0.73      1438
          7       0.41      0.27      0.33       591
          8       0.10      0.02      0.04        93

avg / total       0.51      0.58      0.53      2389<jupyter_text>Finding the best parameters for Kernel SVC(User Friendly Version):<jupyter_code>params = [{'C': [1, 10, 30, 100], 'kernel': ['rbf'], 'gamma': [1, 0.1, 0.01, 0.001]}]
     
                          
grid_search = GridSearchCV(estimator = svm, param_grid = params, scoring = 'accuracy', cv = 5, n_jobs = -1)
grid_search = grid_search.fit(X_train, y_train)
accuracySVC = grid_search.best_score_
best_paramsSVC = grid_search.best_params_
print(accuracySVC)
print(best_paramsSVC)<jupyter_output>0.6304438860971524
{'C': 10, 'gamma': 0.1, 'kernel': 'rbf'}<jupyter_text>There is definitely less accuracy that when using all the features of the dataset. It would be important to discover the meaning of all those features in order to study if those can be included in our user friendly model.<jupyter_code>reports_list = [predicted_knn, scores, report, grid_search, accuracySVC, best_paramsSVC]
with open('file11.pickle', 'wb') as file11:
    pickle.dump(mlp , file11)
with open('file11.pickle', 'rb') as file_11:
    
    rl = pickle.load(file_11)
    
rl<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1054825555052107, 'fraction_numerical': 0.034647334239540854, 'mean_word_length': 3.523913637605903, 'pattern_counts': {'":': 0, '<': 196, '<?xml version=': 0, '>': 196, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 43, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '51', 'dir': 'jupyter-structured-clean-dedup', 'id': '50230', 'license': 'MIT', 'n_tokens_mistral': 11530, 'n_tokens_neox': 10803, 'n_words': 2905, 'path': 'franco/MachineLearning/Notebook1.ipynb', 'repo_name': 'georgetown-analytics/Safety-Recommenders', 'size': '33105'}