<jupyter_start><jupyter_text>Logistic Regression Data Mining Classification Project<NAME>, MSDA/SV@Home for HCD/CORE/MidPen/Allied/RCD/Charities  Question  Are CA AFH State Tools Speeding Up Approvals? What is the Optimal Mix?  Methods Data collected from affordable and market rate developers as well as opensources.Test data validated using 3 methods to classify data (Naives Bayes, Decision Tree/Random Forest, Logisitic Regression).Dimenions reduced to most relevant 21 features.  Analysis  Libraries<jupyter_code>import numpy as np
import pandas as pd
import pandas.testing as tm
import matplotlib.pyplot as plt
import scikitplot as skplt

from sklearn.preprocessing import StandardScaler       # Standardization of data
from sklearn.preprocessing import LabelEncoder         # Label > Number for analysis

from sklearn.metrics import plot_confusion_matrix      # pretty plot confusion matrix
from sklearn.model_selection import train_test_split   # Training/Test split

from sklearn.linear_model import LogisticRegression    # Logistic regression
import statsmodels.api as sm                           # Step model

from sklearn.metrics import confusion_matrix, classification_report 
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, make_scorer
from sklearn.metrics import roc_curve<jupyter_output><empty_output><jupyter_text>Load data and preview<jupyter_code>df = pd.read_csv('DATA_ML_Project_MFR_FINAL.csv', parse_dates=['Submitted','Approved'])
#df.head()<jupyter_output><empty_output><jupyter_text>Clean and prepare data<jupyter_code># drop names

df.drop(columns=['Dev', 'Name','Address'], inplace=True)
# Part_AFH (> 15% inclusionary AFH), All_AFH (Y/N)
df['All_AFH']  = df.AFH >= df.DU - 2
df['Part_AFH'] = df.AFH/df.DU >= 0.15
# calculate days and cut-off, if < 180 days '1', else '0'
df['Days']         = df.Approved - df.Submitted
df['Streamlined']  = df['Days'].dt.days <= 180
# drop unused columns
df.drop(columns=['Submitted', 'Approved','Days'], inplace=True)
df.head()
# filling in NaN with 0
df["SB35"].fillna(0, inplace = True)
df["SB330"].fillna(0, inplace = True)
df["AB1763"].fillna(0, inplace = True)
df["AB2162"].fillna(0, inplace = True)
df["AB2345"].fillna(0, inplace = True)
df["AB2372"].fillna(0, inplace = True)
df["AB3194"].fillna(0, inplace = True)
df.head()
# convert 'AB2372' from float to int
df['AB2372'] = df['AB2372'].astype(int)
# converting 'Streamlined' boolean to int
df['All_AFH']     = df.apply(lambda row: row['All_AFH'] == True, axis=1).astype(int)
df['Part_AFH']    = df.apply(lambda row: row['Part_AFH'] == True, axis=1).astype(int)
df['Streamlined'] = df.apply(lambda row: row['Streamlined'] == True, axis=1).astype(int)
# convert Y and N to 1 and 0 (binary)
df = df.replace(['Y'],1)
df = df.replace(['N'],0)
df.head()<jupyter_output><empty_output><jupyter_text>2. Split input and output<jupyter_code># 17 input features (data attributes). Remove city; NOT numeric.
X = df.loc[:,['DU', 'MKT', 'AFH', 'GP_COMPLY', 'Z_COMPLY', 'TOD', 'PLAN_AREA', 'SB35',
       'SB330', 'AB1763', 'AB2162', 'AB2345', 'AB2372', 'AB3194', 'All_AFH',
       'Part_AFH']]
X.keys()
# outcome/output
y=df['Streamlined']
class_le = LabelEncoder()
y = class_le.fit_transform(y)
y
# normalize all non-binary features (e.g.'DU', 'MKT','AFH')

stdsc = StandardScaler()
X.iloc[:,[0,1,2]] = stdsc.fit_transform(X.iloc[:,[0,1,2]])
#X['Intercept'] = 1     # if set intercept before, removes warning, but not recommended, changes logistic regression!!!
X.head()<jupyter_output><empty_output><jupyter_text>Check dataframe for null values<jupyter_code>df.isnull().values.any()<jupyter_output><empty_output><jupyter_text>3. Split into test and training NOTE: * Tested setting test_size @ 15% and 20%. 15% worked better with smaller dataset (dataset  100 data points and this is expected to improve with data.* Tested L1 vs L2 (default) penalty. L1 generated low accuracy and random_state affected ROC AUC.<jupyter_code>X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.2)  # increase training size can improve accuracy of smaller datasets. 
logistic = LogisticRegression(solver='liblinear', random_state=1) # good optimizer for smaller datasets
# logistic = LogisticRegression() 
logistic.fit(X_train, y_train)
y_pred = logistic.predict(X_test) 
print(f'Model Accuracy Score: {accuracy_score(y_test,y_pred)}')
print(classification_report(y_test, y_pred))<jupyter_output>precision    recall  f1-score   support

           0       0.80      0.89      0.84        18
           1       0.33      0.20      0.25         5

    accuracy                           0.74        23
   macro avg       0.57      0.54      0.55        23
weighted avg       0.70      0.74      0.71        23<jupyter_text>4. Confusion Matrix<jupyter_code>confmat = pd.DataFrame(confusion_matrix(y_test, y_pred),
                      index  =['True[0]','True[1]'],
                      columns=['Predict[0]','Predict[1]'])
confmat<jupyter_output><empty_output><jupyter_text>5. Draw ROC Curve and cumulative gain chart<jupyter_code>fpr, tpr, thresholds = roc_curve(y_test, logistic.predict_proba(X_test)[:,1])
plt.plot(fpr, tpr, '--', label="Logistic Regression")
plt.plot([0,1],[0,1],'k--', label='Random Guess')
plt.plot([fpr],[tpr], 'r-', ms=10)
plt.xlabel('FPR')
plt.ylabel('TPR')
plt.title('ROC')
plt.show()
skplt.metrics.plot_roc(y_test, logistic.predict_proba(X_test), plot_micro=False)
plt.show()
skplt.metrics.plot_cumulative_gain(y_test, logistic.predict_proba(X_test))
plt.show()<jupyter_output><empty_output><jupyter_text>6. Use sm.logit with "intercept=1" to check coefficients<jupyter_code># Set intercept after split
X['Intercept'] = 1  
# Ensure no singularity matrix (1:1 correlations between columns except with self)
n = len(X)
X = X+0.001*np.random.rand(n, 17)
# Check correlations
X.corr()
logistic_ml   = sm.Logit(y, X)
logistic_coef = logistic_ml.fit()
logistic_coef.summary2()<jupyter_output>Optimization terminated successfully.
         Current function value: 0.328838
         Iterations 13<jupyter_text>7. Calculate Odds Ratio (OR) for each input feature for unit change=1<jupyter_code>np.exp(logistic_coef.params)<jupyter_output>/opt/anaconda3/lib/python3.8/site-packages/pandas/core/arraylike.py:358: RuntimeWarning: overflow encountered in exp
  result = getattr(ufunc, method)(*inputs, **kwargs)<jupyter_text>8. Choose input features based on 90% confidence level (P < 0.1); Eliminate 'No Impact' Features; Apply Domain Redo classification via LogisticRegression in sckitlearn with same parameter set.  Selected Conf. Lvl based on domain laws significant for streamlining.  Upon visual inspection; the following are 4 features are statistically significant. One feature is noises AB2372. One feature is a density bonus law which are commonly used for affordable housing developments AB 2345, but should not affect streamlining; however, happen to coincide with streamline eligible projects. One feature is a streamlining law AB2162, that based on training data seems to have not been included, but we will include due to having domain knowledge. The final list of features is:* PLAN_AREA* SB35* SB330* AB2162* AB2345<jupyter_code># X = Input feature (features: Income, Family, Education, CDAccount, CreditCard)
X = X.drop(['DU','MKT','AFH','GP_COMPLY','Z_COMPLY','TOD','AB1763','AB2372','AB3194','All_AFH','Part_AFH'], axis=1)
X.head()
X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.2)
logistic = LogisticRegression(solver='liblinear', random_state=1) # good optimizer for smaller datasets
# logistic = LogisticRegression() 
logistic.fit(X_train, y_train)
y_pred = logistic.predict(X_test)
print(f'Model Accuracy Score: {accuracy_score(y_test,y_pred)}')
print(classification_report(y_test, y_pred))
confmat = pd.DataFrame(confusion_matrix(y_test, y_pred),
                      index  =['True[0]','True[1]'],
                      columns=['Predict[0]','Predict[1]'])
confmat
skplt.metrics.plot_roc(y_test, logistic.predict_proba(X_test), plot_micro=False)
plt.show()
skplt.metrics.plot_cumulative_gain(y_test, logistic.predict_proba(X_test))
plt.show()
logistic_ml   = sm.Logit(y_train, X_train)
logistic_coef = logistic_ml.fit()
logistic_coef.summary2()
np.exp(logistic_coef.params)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12584215591915304, 'fraction_numerical': 0.03200192492781521, 'mean_word_length': 5.1125, 'pattern_counts': {'":': 0, '<': 46, '<?xml version=': 0, '>': 47, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '160375', 'license': 'MIT', 'n_tokens_mistral': 3012, 'n_tokens_neox': 2796, 'n_words': 716, 'path': 'rick_gosalvez_03_LogisticRegression.ipynb', 'repo_name': 'jrgosalvez/CA_State_Tool_Machine_Learning_Models', 'size': '8319'}