<jupyter_start><jupyter_text>Property Maintenance Fines **Predicting the probability that a set of blight tickets will be paid on time****Supervised Learning. Classification**Source: [Applied Machine Learning in Python | Coursera](https://www.coursera.org/learn/python-machine-learning). Solved with classical machine learning classifiers [here](https://github.com/angelmtenor/applied-ML-coursera/blob/master/4_property_maintenance_fines.ipynb)Data provided by Michigan Data Science Team ([MDST](http://midas.umich.edu/mdst/)), the Michigan Student Symposium for Interdisciplinary Statistical Sciences ([MSSISS](https://sites.lsa.umich.edu/mssiss/)) and the City of Detroit [Detroit Open Data Portal](https://data.detroitmi.gov/). Each row of the dataset   corresponds to a single blight ticket, and includes information about when, why, and to whom each ticket was issued. The target variable is compliance, which is True if the ticket was paid early, on time, or within one month of the hearing data, False if the ticket was paid after the hearing date or not at all, and Null if the violator was found not responsible. **Features**        ticket_id - unique identifier for tickets    agency_name - Agency that issued the ticket    inspector_name - Name of inspector that issued the ticket    violator_name - Name of the person/organization that the ticket was issued to    violation_street_number, violation_street_name, violation_zip_code - Address where the violation occurred    mailing_address_str_number, mailing_address_str_name, city, state, zip_code, non_us_str_code, country - Mailing address of the violator    ticket_issued_date - Date and time the ticket was issued    hearing_date - Date and time the violator's hearing was scheduled    violation_code, violation_description - Type of violation    disposition - Judgment and judgement type    fine_amount - Violation fine amount, excluding fees    admin_fee - $20 fee assigned to responsible judgmentsstate_fee - $10 fee assigned to responsible judgments    late_fee - 10% fee assigned to responsible judgments    discount_amount - discount applied, if any    clean_up_cost - DPW clean-up or graffiti removal cost    judgment_amount - Sum of all fines and fees    grafitti_status - Flag for graffiti violations    ** Labels **    payment_amount - Amount paid, if any    payment_date - Date payment was made, if it was received    payment_status - Current payment status as of Feb 1 2017    balance_due - Fines and fees still owed    collection_status - Flag for payments in collections    compliance [target variable for prediction]     Null = Not responsible     0 = Responsible, non-compliant     1 = Responsible, compliant    compliance_detail - More information on why each ticket was marked compliant or non-compliant<jupyter_code>import os
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import helper
import keras

helper.info_gpu()
#sns.set_palette("GnBu_d")
#helper.reproducible(seed=0) # Setup reproducible results from run to run using Keras

%matplotlib inline<jupyter_output>Using TensorFlow backend.<jupyter_text>1. Data Processing<jupyter_code>data_path = 'data/property_maintenance_fines_data.csv'
target = ['compliance']

df_original = pd.read_csv(data_path, encoding='iso-8859-1', dtype='unicode')
print("{} rows \n{} columns \ntarget: {}".format(*df_original.shape, target))<jupyter_output>250306 rows 
34 columns 
target: ['compliance']<jupyter_text>Explore and Clean the target<jupyter_code>print(df_original[target].squeeze().value_counts(dropna=False))
# Remove rows with NULL targets

df_original = df_original.dropna(subset=target)

print(df_original[target].squeeze().value_counts())
print(df_original.shape)<jupyter_output>0.0    148283
1.0     11597
Name: compliance, dtype: int64
(159880, 34)<jupyter_text>** Imbalanced target: the evaluation metric used in this problem is the Area Under the ROC Curve **   Split original data into training and validation test set<jupyter_code>from sklearn.model_selection import train_test_split

df, df_test = train_test_split(
    df_original, test_size=0.2, stratify=df_original[target], random_state=0)<jupyter_output><empty_output><jupyter_text>** To avoid data leakage, only the training dataframe, df, will be explored and processed here**   Show the training data<jupyter_code>df.head(2)<jupyter_output><empty_output><jupyter_text>Missing values<jupyter_code>helper.missing(df)<jupyter_output><empty_output><jupyter_text>Transform Data  Remove irrelevant features<jupyter_code>def remove_features(df):

    relevant_col = ['agency_name', 'violation_street_name', 'city', 'state', 'violator_name',
        'violation_code', 'late_fee', 'discount_amount', 'judgment_amount', 'disposition',
        'fine_amount', 'compliance']

    df = df[relevant_col]

    return df


df = remove_features(df)

print(df.shape)<jupyter_output>(127904, 12)<jupyter_text>Classify variables<jupyter_code>num = ['late_fee', 'discount_amount', 'judgment_amount', 'fine_amount']

df = helper.classify_data(df, target, numerical=num)

pd.DataFrame(dict(df.dtypes), index=["Type"])[df.columns].head()  # show data types<jupyter_output>numerical features:   4
categorical features: 7
target 'compliance': category<jupyter_text>Remove low-frequency categorical values<jupyter_code>df, dict_categories = helper.remove_categories(df, target=target, ratio=0.001, show=False)<jupyter_output><empty_output><jupyter_text>Fill missing valuesMissing categorical values filled by 'Other'There are no numerical missing values<jupyter_code>df = helper.fill_simple(df, target, missing_categorical='Other') 
helper.missing(df);<jupyter_output>No missing values found<jupyter_text>Visualize the data  Categorical features<jupyter_code>for i in ['state', 'disposition']:
    helper.show_categorical(df[[i]])<jupyter_output><empty_output><jupyter_text>Target vs Categorical features<jupyter_code>for i in ['state', 'disposition']:
    helper.show_target_vs_categorical(df[[i, target[0]]], target)<jupyter_output><empty_output><jupyter_text>Numerical features<jupyter_code>helper.show_numerical(df, kde=True)<jupyter_output><empty_output><jupyter_text>Target vs Numerical features<jupyter_code>helper.show_target_vs_numerical(df, target, point_size=10 ,jitter=0.3, fit_reg=True)
plt.ylim(ymin=-0.2, ymax=1.2)<jupyter_output><empty_output><jupyter_text>Correlation between numerical features and target<jupyter_code>helper.show_correlation(df, target, figsize=(6,3))<jupyter_output><empty_output><jupyter_text>2. Neural Network Model  Select the features<jupyter_code>droplist = []  # features to drop

# For the model 'data' instead of 'df'
data = df.copy()
# del(df)
data.drop(droplist, axis='columns', inplace=True)
data.head(2)<jupyter_output><empty_output><jupyter_text>Scale numerical variablesShift and scale numerical variables to a standard normal distribution. The scaling factors are saved to be used for predictions.<jupyter_code>data, scale_param = helper.scale(data)<jupyter_output><empty_output><jupyter_text>Create dummy featuresReplace categorical features (no target) with dummy features<jupyter_code>data, dict_dummies = helper.replace_by_dummies(data, target)

model_features = [f for f in data if f not in target]  # sorted neural network inputs

data.head(3)<jupyter_output><empty_output><jupyter_text>Split the data into training and validation sets<jupyter_code>val_size = 0.2
random_state = 0


def validation_split(data, val_size=0.25):

    train, test = train_test_split(
        data, test_size=val_size, random_state=random_state, stratify=data[target])

    # Separate the data into features and target (x=features, y=target)
    x_train, y_train = train.drop(target, axis=1).values, train[target].values
    x_val, y_val = test.drop(target, axis=1).values, test[target].values
    # _nc: non-categorical yet (needs one-hot encoding)

    return x_train, y_train, x_val, y_val


x_train, y_train, x_val, y_val = validation_split(data, val_size=val_size)

# x_train = x_train.astype(np.float16)
y_train = y_train.astype(np.float16)
# X_val = x_val.astype(np.float16)
y_val = y_val.astype(np.float16)<jupyter_output><empty_output><jupyter_text>Encode the output<jupyter_code>def one_hot_output(y_train, y_val):
    num_classes = len(np.unique(y_train))
    y_train = keras.utils.to_categorical(y_train, num_classes)
    y_val = keras.utils.to_categorical(y_val, num_classes)
    return y_train, y_val


y_train, y_val = one_hot_output(y_train, y_val)
print("train size \t X:{} \t Y:{}".format(x_train.shape, y_train.shape))
print("val size \t X:{} \t Y:{}".format(x_val.shape, y_val.shape))<jupyter_output>train size 	 X:(102323, 445) 	 Y:(102323, 2)
val size 	 X:(25581, 445) 	 Y:(25581, 2)<jupyter_text>Build a dummy classifier<jupyter_code>from sklearn.dummy import DummyClassifier

clf = DummyClassifier(strategy='most_frequent').fit(x_train, np.ravel(y_train))
# The dummy 'most_frequent' classifier always predicts class 0
y_pred = clf.predict(x_val).reshape([-1, 1])

helper.binary_classification_scores(y_val[:, 1], y_pred);<jupyter_output>Scores:
-----------
Log_Loss: 	2.5059
Accuracy: 	0.93
Precision: 	0.00
Recall: 	0.00
ROC AUC: 	0.00
F1-score: 	0.00

Confusion matrix: 
 [[23725     0]
 [ 1856     0]]<jupyter_text>Build a random forest classifier (best of grid search)<jupyter_code>from sklearn.ensemble import RandomForestClassifier


%time clf_random_forest_opt = RandomForestClassifier(n_estimators = 30, max_features=150, \
                                max_depth=13, class_weight='balanced', n_jobs=-1, \
                                   random_state=0).fit(x_train, np.ravel(y_train[:,1]))
y_pred = clf_random_forest_opt.predict(x_val).reshape([-1, 1])
helper.binary_classification_scores(y_val[:, 1], y_pred);<jupyter_output>Scores:
-----------
Log_Loss: 	4.3881
Accuracy: 	0.87
Precision: 	0.31
Recall: 	0.60
ROC AUC: 	0.75
F1-score: 	0.41

Confusion matrix: 
 [[21209  2516]
 [  734  1122]]<jupyter_text>Build the Neural Network for Binary Classification<jupyter_code>cw = helper.get_class_weight(y_train[:, 1])  # class weight (imbalanced target)

import keras
from keras.models import Sequential
from keras.layers.core import Dense, Dropout


def build_nn(input_size, output_size, summary=False):

    input_nodes = input_size // 8

    model = Sequential()
    model.add(Dense(input_nodes, input_dim=input_size, activation='relu'))

    model.add(Dense(output_size, activation='softmax'))

    if summary:
        model.summary()

    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

    return model


model = build_nn(x_train.shape[1], y_train.shape[1], summary=True)<jupyter_output>_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_1 (Dense)              (None, 55)                24530     
_________________________________________________________________
dense_2 (Dense)              (None, 2)                 112       
=================================================================
Total params: 24,642
Trainable params: 24,642
Non-trainable params: 0
_________________________________________________________________<jupyter_text>Train the Neural Network<jupyter_code>import os
from time import time
model_path = os.path.join("models", "detroit.h5")


def train_nn(model, x_train, y_train, validation_data=None, path=False, show=True):
    """ 
    Train the neural network model. If no validation_datais provided, a split for validation
    will be used
    """
    
    if show:
        print('Training ....')
    
    callbacks = [keras.callbacks.EarlyStopping(monitor='val_loss', patience=0, verbose=1)]
    t0 = time()

    history = model.fit(
        x_train,
        y_train,
        epochs=100,
        batch_size=2048,
        
        class_weight = cw,
        
        verbose=1,
        validation_split=0.3,
        validation_data = validation_data,
        callbacks=callbacks)

    if show:
        print("time: \t {:.1f} s".format(time() - t0))
        helper.show_training(history)

    if path:
        model.save(path)
        print("\nModel saved at", path)
    
    return history
        

model = None
model = build_nn(x_train.shape[1], y_train.shape[1], summary=False)
train_nn(model, x_train, y_train, path=None);


from sklearn.metrics import roc_auc_score

y_pred_train = model.predict(x_train, verbose=1)
print('\n\n ROC_AUC train:\t{:.2f} \n'.format(roc_auc_score(y_train, y_pred_train)))
y_pred_val = model.predict(x_val, verbose=1)
print('\n\n ROC_AUC val:\t{:.2f}'.format(roc_auc_score(y_val, y_pred_val)))<jupyter_output>Training ....
Train on 71626 samples, validate on 30697 samples
Epoch 1/100
71626/71626 [==============================] - 6s 84us/step - loss: 0.6311 - acc: 0.7677 - val_loss: 0.5909 - val_acc: 0.7567
Epoch 2/100
71626/71626 [==============================] - 5s 76us/step - loss: 0.5471 - acc: 0.7890 - val_loss: 0.5538 - val_acc: 0.8086
Epoch 3/100
71626/71626 [==============================] - 5s 69us/step - loss: 0.5232 - acc: 0.8325 - val_loss: 0.5477 - val_acc: 0.8383
Epoch 4/100
71626/71626 [==============================] - 5s 65us/step - loss: 0.5144 - acc: 0.8364 - val_loss: 0.5460 - val_acc: 0.8433
Epoch 5/100
71626/71626 [==============================] - 5s 64us/step - loss: 0.5092 - acc: 0.8353 - val_loss: 0.5432 - val_acc: 0.8311
Epoch 6/100
71626/71626 [==============================] - 5s 64us/step - loss: 0.5048 - acc: 0.8282 - val_loss: 0.5427 - val_acc: 0.8280
Epoch 7/100
71626/71626 [==============================] - 5s 67us/step - loss: 0.5014 - acc: 0.8279 - val_l[...]<jupyter_text>Validate the model (validation set)<jupyter_code>helper.binary_classification_scores(y_val[:, 1], y_pred_val[:, 1]);<jupyter_output>Scores:
-----------
Log_Loss: 	nan
Accuracy: 	0.83
Precision: 	0.24
Recall: 	0.63
ROC AUC: 	0.81
F1-score: 	0.35

Confusion matrix: 
 [[20094  3631]
 [  678  1178]]<jupyter_text>Evaluate the final model (test set)<jupyter_code>df_test.head(2)<jupyter_output><empty_output><jupyter_text>Process test data with training set parameters (no data leakage)<jupyter_code>df_test = remove_features(df_test)

df_test = helper.classify_data(df_test, target, numerical=num)

df_test, _ = helper.remove_categories(
    df_test, target=target, show=False, dict_categories=dict_categories)

df_test = helper.fill_simple(df_test, target, missing_categorical='Other') 

df_test, _ = helper.scale(df_test, scale_param)
df_test, _ = helper.replace_by_dummies(df_test, target, dict_dummies)
df_test = df_test[model_features+target] # sort columns to match training features order
def separate_x_y(data):
    """ Separate the data into features and target (x=features, y=target) """

    x, y = data.drop(target, axis=1).values, data[target].values
    x = x.astype(np.float16)
    y = y.astype(np.float16)
   
    return x, y

x_test, y_test = separate_x_y(df_test)

y_test = keras.utils.to_categorical(y_test, 2)<jupyter_output><empty_output><jupyter_text>Random Forest model<jupyter_code>y_pred = clf_random_forest_opt.predict_proba(x_test)[:,1]
helper.binary_classification_scores(y_test[:,1], y_pred);
helper.show_feature_importances(model_features, clf_random_forest_opt)<jupyter_output>Top contributing features:
 --------------------------
disposition_Responsible by Default          0.32
late_fee                                    0.26
judgment_amount                             0.09
disposition_Responsible by Admission        0.05
disposition_Responsible by Determination    0.04
fine_amount                                 0.03
discount_amount                             0.03
violation_code_9-1-36(a)                    0.01
violation_code_22-2-61                      0.01
violation_code_9-1-81(a)                    0.01
dtype: float64<jupyter_text>Neural Network model<jupyter_code>y_pred = model.predict(x_test, verbose=1)[:,1]
helper.binary_classification_scores(y_test[:,1], y_pred);<jupyter_output>31976/31976 [==============================] - 1s 22us/step
Scores:
-----------
Log_Loss: 	nan
Accuracy: 	0.83
Precision: 	0.24
Recall: 	0.62
ROC AUC: 	0.80
F1-score: 	0.35

Confusion matrix: 
 [[25076  4581]
 [  875  1444]]<jupyter_text>Compare with other non-neural ML models<jupyter_code>helper.ml_classification(x_train, y_train[:, 1], x_test, y_test[:, 1])<jupyter_output>Naive Bayes
AdaBoost
Decision Tree
Random Forest
Extremely Randomized Trees
{'fraction_non_alphanumeric': 0.12869134609577873, 'fraction_numerical': 0.04015942991726554, 'mean_word_length': 4.746009715475364, 'pattern_counts': {'":': 0, '<': 106, '<?xml version=': 0, '>': 106, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '30', 'dir': 'jupyter-structured-clean-dedup', 'id': '397146', 'license': 'MIT', 'n_tokens_mistral': 6023, 'n_tokens_neox': 5423, 'n_words': 1291, 'path': 'property_maintenance_fines.ipynb', 'repo_name': 'omrantoujani/Data-Science-Keras', 'size': '16559'}