<jupyter_start><jupyter_text>Binary classificationLet's take another look at the target variable for this project.<jupyter_code># import raw dataset (before any cleanup)
raw_df = pd.read_csv('employee_data.csv')

print(raw_df['status'].unique())
raw_df.head()<jupyter_output>['Left' 'Employed']<jupyter_text>As above shows, we have two possible classes for status:  * 'Left' * 'Employed'So this is a **binary** classification problem.<jupyter_code># in the ABT the target variable was converted to 1 ('Left') and 0 ('Employed').
# Scikit-Learn requires numeric variables
abt_df = pd.read_csv('analytical_base_table.csv')

print(abt_df['status'].unique())
abt_df.head()<jupyter_output>[1 0]<jupyter_text>Additional terminologyIn binary classification: * 1 ('Left') is also known as the **"positive"** class. * 0 ('Employed') is also known as the **"negative"** class.  This is because we are trying to predict which employees will leave, therefore those who left are mapped as positive (1). In other words, the **positive class** is simply the primary class you're trying to identify.  For **binary classification** the predicted probability of the positive class and that of the negative class will **sum to 1**.  Toy Example<jupyter_code># Input feature
x = np.linspace(0, 1, 100)

# Noise
np.random.seed(555)
noise = np.random.uniform(-0.2, 0.2, 100)

# Target variable
y = ((x + noise) > 0.5).astype(int)<jupyter_output><empty_output><jupyter_text>Let's break down the above code: * **Input feature x:** First, we created an evenly spaced list of 100 numbers from 0 to 1.           * **Random noise:** Next, we generated 100 random numbers from a uniform distribution between -0.2 and 0.2.  We set a random seed so we can replicate our results.       * **Noisy conditional:** Finally, we generated 100 values of the target variable, y, based on the noisy conditional mapping function.  These operations are elementwise.       Reshape our input featureScikit-learn algorithms expect input features with 2 axes.  Right now x only has one.<jupyter_code>x.shape
# Reshape x into X
X = x.reshape(100,1)
# we can see the difference
print(x[:5])
# X with 2 axes
print(X[:5])<jupyter_output>[[ 0.        ]
 [ 0.01010101]
 [ 0.02020202]
 [ 0.03030303]
 [ 0.04040404]]<jupyter_text>Plot the synthetic dataset<jupyter_code># scatterplot
plt.figure(figsize=(8,6))
plt.scatter(X,y)
plt.show()<jupyter_output><empty_output><jupyter_text>As you can see, observations with **x > 0.5** should have **y=1**  and those with **x<0.5** should have **y=0**. However, as you get closer to **x=0.5**, the noise plays a bigger factor, and you get some overlap.Therefore, here's what we generally want out of a good model: * If x>0.5, we want our model to predict higher probability for y=1. * If x≤0.5, we want our model to predict higher probability for y=0. * If x is closer to 0.5, we want **less confidence** in the prediction. * If x is farther from 0.5, we want **more confidence** in the prediction.Finally, there's two more important nuances that we need from a good model: 1. We don't want probability predictions above 1.0 or below 0.0, as those don't make sense. 2. We want **very high confidence** by the time we pass 0.3 or 0.7, because that's when noise is no longer a factor.  Logistic Regression<jupyter_code>from sklearn.linear_model import LinearRegression, LogisticRegression
# fit linear model, make predictions, plot them
model = LinearRegression()
model.fit(X,y)

# plot dataset and predictions
plt.scatter(X, y)
plt.plot(X, model.predict(X), 'k--')
plt.show()<jupyter_output><empty_output><jupyter_text>Right away, we can see 2 flaws with linear regression for this type of binary classification. * First, you're getting predicted probabilities above 1.0 and below 0.0! That just doesn't make sense. * Second, remember, we want **very high confidence** by the time we pass 0.3 or 0.7, because that's when noise is no longer a factor. We can certainly do much better on this front.  Logistic RegressionLet's see how logistic regression differs: * it's actually an algorithm for classification * the .predict() function behaves quite differently for classification.<jupyter_code># logistic regression
model = LogisticRegression()
model.fit(X, y)
# predict()
model.predict(X)<jupyter_output><empty_output><jupyter_text>As you can see, if you call .predict(), the model will give you the predicted classes directly.However, as we discussed earlier, we almost always prefer to get the **class probabilities**. That's where the **.predict_proba()** function comes in. * For classification problems, we recommend using .predict_proba() instead of .predict(). * It's required for calculating certain metrics and setting custom thresholds, which we'll see in the next module.<jupyter_code># predict_proba
pred = model.predict_proba(X[:10])
                           
pred
# class probabilities for first observation
pred[0]
# positive class probability for first observation
pred[0][1]
# get just the second value (positive) for each prediction
pred = [p[1] for p in pred]

pred
# fit and plot the logistic regression model
model = LogisticRegression()
model.fit(X, y)

# predict probabilities
pred = model.predict_proba(X)

# get just the second value (positive class) for each prediction
pred = [p[1] for p in pred]

# plot the dataset and predictions
plt.scatter(X, y)
plt.plot(X, pred, 'k--')
plt.show()<jupyter_output><empty_output><jupyter_text>Regularized Logistic RegressionLogistic regression has regularized versions that are analogous to those for linear regression. Remember:Regularization is a technique used to prevent overfitting by artificially penalizing model coefficients.It does so by adding a penalty factor to the cost function. * This dampens coefficients or removes features entirely, depending on the type of penalty used. * The "strength" of that penalty factor is **tunable.**As in linear regression, there are two types of penalties. * **L1** penalizes the absolute size of model coefficients. * **L2** penalizes the squared size of model coefficients. Helper function that:    1. Fits any classification model    2. Makes predictions    3. Extracts the positive probabilities    4. Plots them<jupyter_code>def fit_and_plot_classifier(clf):
    # fit model
    clf.fit(X, y)
    
    # predict and take second value (positive class) of each prediction
    pred = clf.predict_proba(X)
    pred = [p[1] for p in pred]
    
    # plot
    plt.scatter(X, y)
    plt.plot(X, pred, 'k--')
    plt.show()
    
    # return fitted model and predictions
    return clf, pred
# use new fit_and_plot_classifier on the logistic regression from toy dataset
clf, pred = fit_and_plot_classifier(LogisticRegression())<jupyter_output><empty_output><jupyter_text>Now, we said earlier that this model looks a bit too conservative. * In fact, that's because the LogisticRegression algorithm in Scikit-Learn is regularized by default! * That's why we weren't getting very high confidence for x>0.7 or x<0.3. So let's see what happens when we play around with that regularization strength.  Regularization strengthAs in Lasso, Ridge, and Elastic-Net, the strength of regularized logistic regression's penalty term is **tunable**.However, for **Logistic Regression**, it's not referred to as **alpha**.  Instead, it's called **C**, which is the inverse of the regularized strength. * Therefore, higher **C** means a weaker penalty, and lower **C** means a stronger penalty. * By default, **C** = 1. In other words, our logistic regression was being regularized by default, which is why it was a bit conservative.<jupyter_code># standard strength
clf, pred = fit_and_plot_classifier(LogisticRegression(C=1))
# stronger penalty (smaller C)
clf, pred = fit_and_plot_classifier(LogisticRegression(C=0.25))
# weaker penalty (bigger C)
clf, pred = fit_and_plot_classifier(LogisticRegression(C=4))
# basically no regularization (very high C)
clf, pred = fit_and_plot_classifier(LogisticRegression(C=10000))<jupyter_output><empty_output><jupyter_text>Key takeaway --> this is a **hyperparameter** that should be tuned, and the modeling process learned in Project 2 is the most efficient and reliable way to do so.  Penalty typeBy default, the Logistic Regression() algorithm uses the $L_2$ penalty. * For linear regression, $L_2$ regularization is called Ridge regression. * For logistic regression, we'll simply call it **$L_2$-regularized logistic regression.**  Likewise...  * For linear regression, $L_2$ regularization is called Lasso regression. * For logistic regression, we'll simply call it **$L_2$-regularized logistic regression.**We can set the **penalty type** like so:<jupyter_code># L1 regularization with weaker penalty
clf, pref = fit_and_plot_classifier(LogisticRegression(penalty='l1', C=4))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09546998867497168, 'fraction_numerical': 0.018346545866364666, 'mean_word_length': 4.999320652173913, 'pattern_counts': {'":': 0, '<': 44, '<?xml version=': 0, '>': 47, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '595563', 'license': 'MIT', 'n_tokens_mistral': 2715, 'n_tokens_neox': 2510, 'n_words': 1080, 'path': 'EmployeeRetention_Classification.ipynb', 'repo_name': 'mattdube/employee_retention', 'size': '8830'}