<jupyter_start><jupyter_text>Read in training/testing data<jupyter_code>#Read in train/test data
train_X = pd.read_pickle('Data/train_test/train_X.pkl')
test_X = pd.read_pickle('Data/train_test/test_X.pkl')
train_y = pd.read_pickle('Data/train_test/train_y.pkl')
test_y = pd.read_pickle('Data/train_test/test_y.pkl')<jupyter_output><empty_output><jupyter_text>Logistic Regression<jupyter_code>from sklearn.linear_model import LogisticRegression
#Let's try an out of the box Logisitic Regression
logi = LogisticRegression(verbose=True)
with joblib.parallel_backend('dask'):
    logi.fit(train_X, np.ravel(train_y))
logi.score(test_X, test_y)<jupyter_output>[Parallel(n_jobs=-1)]: Using backend DaskDistributedBackend with 6 concurrent workers.
[Parallel(n_jobs=-1)]: Done   1 out of   1 | elapsed:   23.4s finished<jupyter_text>Not terrible. We can make some adjustments to the hyperparameters to see if they increases our accuracy. We'll save the score and parameters for review.<jupyter_code>#Create dataframe to store metrics and parameters
logi_deets = pd.DataFrame(logi.get_params(), index=['LR_OBO'])
logi_deets.insert(0, value=logi.score(test_X, test_y), column='Accuracy')
logi_deets
#First let's see if we do better with balanced class weights
logi_2 = LogisticRegression(class_weight='balanced', verbose=True)
with joblib.parallel_backend('dask'):
    logi_2.fit(train_X, np.ravel(train_y))
logi_2.score(test_X, test_y)
#store metrics and parameters
a = logi_2.get_params()
a['Accuracy'] = logi_2.score(test_X, test_y)
logi_deets = pd.concat([logi_deets, pd.DataFrame(a, index=['LR_BAL'])], axis=0, join='outer')
logi_deets<jupyter_output><empty_output><jupyter_text>Using class weights does not improve the accuracy, in fact it decreases it. Let's try each of the penalty options for improvement and some different values for C for each. We will need to use the liblinear solver to access the L1 penalty option.<jupyter_code>#Run through for loop for hyperparamter tuning of C and penalty
C = [0.1, 0.5, 1, 5, 10]
penalty = ['l1', 'l2']
count=1
for p in penalty:
    for c in C:
        logi_pen = LogisticRegression(solver='liblinear', C=c, penalty=p, verbose=True)
        logi_pen.fit(train_X, np.ravel(train_y))
        deets = logi_pen.get_params()
        deets['Accuracy']=logi_pen.score(test_X, test_y)
        logi_deets = pd.concat([logi_deets, pd.DataFrame(deets, index=['LR_PEN{}'.format(count)])], axis=0, join='outer')
        count+=1

logi_deets<jupyter_output><empty_output><jupyter_text>The adjustments to the penalty and regularization strength show that, though slight, a higher regularization strength increases the accuracy and that the L2 penalty also perfomrs better than L1.As a final check we will increase the number of iterations the model is allowed to run through for minimzation and run through a cross validation check to make sure we are still abel to generalize well.<jupyter_code>#Run model with more iterations to see if we can improve accuracy
logi_iter = LogisticRegression(penalty='l2', C=10, max_iter=250, verbose=True, solver='liblinear')
logi_iter.fit(train_X, np.ravel(train_y))
logi_iter.score(test_X, test_y)<jupyter_output>[LibLinear]<jupyter_text>No great improvement, we will stick with the default 100 iterations max.<jupyter_code>a = logi_iter.get_params()
a['Accuracy'] = logi_iter.score(test_X, test_y)
logi_deets = pd.concat([logi_deets, pd.DataFrame(a, index=['LR_ITER_250'])], axis=0, join='outer')
logi_deets
#Re-run best model
logi_f = LogisticRegression(penalty='l2', C=10, verbose=True, solver='liblinear')
logi_f.fit(train_X, np.ravel(train_y))
logi_f.score(test_X, test_y)
#Confirm generalizability via 5-fold cross validation
print(cross_val_score(logi_f, train_X, train_y, cv=5))<jupyter_output>[LibLinear][LibLinear][LibLinear][LibLinear][LibLinear][0.75544332 0.75521886 0.75140292 0.75611672 0.74994388]<jupyter_text>Cross Validation shows a pretty consistent result. We can be confident that this model will generally relativeley well. Let's take a look at a Confusion matrix to see which classes our best Logistic Regression model is doing best with.<jupyter_code>#Visual class metrics using confusion matrix and classification report
y_pred = logi_f.predict(test_X)
cm=confusion_matrix(test_y, y_pred, labels=logi_f.classes_)
disp = ConfusionMatrixDisplay(cm, display_labels=logi_f.classes_)
plt.figure(figsize=(40,8))
disp.plot()
plt.tight_layout()
plt.xticks(rotation=90)
plt.show()
print(classification_report(test_y, y_pred, labels=logi_f.classes_))<jupyter_output><empty_output><jupyter_text>We can see that the model performed best on the functional label with non functional just behind it and functional needs repair well below the other two. This makes sense as there are many less cases of functional needs repair so the model doesn't have as much data to learn from.<jupyter_code>#output for comparision
logi_deets.to_pickle('Data/scores/LR.pkl')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10249697946033025, 'fraction_numerical': 0.019935561820378575, 'mean_word_length': 6.688854489164087, 'pattern_counts': {'":': 0, '<': 30, '<?xml version=': 0, '>': 30, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '117367', 'license': 'MIT', 'n_tokens_mistral': 1775, 'n_tokens_neox': 1668, 'n_words': 464, 'path': '3D Modeling-Logistic Regression.ipynb', 'repo_name': 'Bwils171/Pump-it-up', 'size': '4966'}