<jupyter_start><jupyter_text>Example of Logistic regression  Predict student admission based on exams result Data is taken from <NAME>'s CS229 course on Machine Learning at Stanford.  This is the same example [previously shown](https://github.com/Mashimo/datascience/blob/master/01-Regression/LogisticRegression.ipynb) but uses StatsModel package.<jupyter_code>import pandas as pd
data = pd.read_csv("data/ex2data1.txt", header=None, 
                   names=['Exam1', 'Exam2', 'Admitted'])
data<jupyter_output><empty_output><jupyter_text>Historical data from previous students: each student has two exams scores associated and the final admission result (1=yes, 0= no).  Split data into X (training data) and y (target variable)<jupyter_code>X = data.copy() # ou training data
y = X.Admitted.copy() # copy “y” column values out
X.drop(['Admitted'], axis=1, inplace=True) # then, drop y column
# manually add the intercept
X['intercept'] = 1.0  # so we don't need to use sm.add_constant every time<jupyter_output><empty_output><jupyter_text>Create and fit a Logistic model via statsmodels package<jupyter_code>import statsmodels.formula.api as sm
import statsmodels.discrete.discrete_model as sm2<jupyter_output><empty_output><jupyter_text>Since we're doing a logistic regression, we're going to use the statsmodels Logit function.<jupyter_code>X.head()
model = sm2.Logit(y, X)
result = model.fit()
result.summary()<jupyter_output><empty_output><jupyter_text>You get a great overview of the coefficients of the model, how well those coefficients fit, the overall fit quality, and several other statistical measures.  The result object also lets you to isolate and inspect parts of the model output, for example the coefficients are in params field:<jupyter_code>coefficients = result.params
coefficients<jupyter_output><empty_output><jupyter_text>As you see, the model found the same coefficients as in the previous example. The confidence interval gives you an idea for how robust the coefficients of the model are.<jupyter_code>result.conf_int()
import numpy as np
 # odds ratios and 95% CI
confidenceInterval = result.conf_int()
confidenceInterval['OddsRatio'] = coefficients
confidenceInterval.columns = ['2.5%', '97.5%', 'OddsRatio']
np.exp(confidenceInterval)<jupyter_output><empty_output><jupyter_text>Take the exponential of each of the coefficients to generate the odds ratios.  This tells you how a 1 unit increase or decrease in a variable affects the odds of being admitted.  For example, we can expect the odds of being admitted to increase roughly the same for both exams better results.   Plot the decision boundary We can use these beta parameters to plot the decision boundary on the training data.We only need two points to plot a line, so we choose two endpoints: the min and the max among the X training data that we already calculated in the previous example.<jupyter_code>plot_x = np.array([ 95.60854757,  21.63578984])<jupyter_output><empty_output><jupyter_text>The line equation is given by the coefficients:<jupyter_code>line_intercept = -coefficients['intercept'] / coefficients['Exam2']
line_slope = -coefficients['Exam1'] / coefficients['Exam2']
line_intercept
line_slope<jupyter_output><empty_output><jupyter_text>Now we have everything to plot the boundary:<jupyter_code>import matplotlib.pyplot as plt
%matplotlib inline
colours = ['red' if i==0 else 'green' for i in y] # red not admitted, green yes.
# Create a list of values in the best fit line
plot_y = [line_slope * i + line_intercept for i in plot_x]
plot_y
fig,ax = plt.subplots()
ax.scatter(X.Exam1, X.Exam2, c=colours)
ax.plot(plot_x, plot_y)
ax.grid(True)
ax.set_xlabel("Exam 1 score")
ax.set_ylabel("Exam 2 score")
fig.suptitle("Student admission vs. past two exams")<jupyter_output><empty_output><jupyter_text>The blue line is our decision boundary: when your exams score lie below the line then probably (that is the prediction) you will not be admitted to University. If they lie above, probably you will.As you can see, the boundary is not predicting perfectly on the training historical data. It's a model. Not perfect but useful. What we can do is to measure its accuracy.  Accuracy<jupyter_code>def logPredict(modelParams, X):  
    probabilities = modelParams.predict(X)
    return [1 if x >= 0.5 else 0 for x in probabilities]
predictions = logPredict(result, X)
accuracy = np.mean(predictions == y)
print ('accuracy = {0}%'.format(accuracy*100)  )<jupyter_output>accuracy = 89.0%<jupyter_text>Just for fun, let's say that my scores are 40 in the first exam and 78 in the second one:<jupyter_code>myExams = pd.DataFrame({'Exam1': [40.], 'Exam2': [78.], 'intercept': [1.]})
myExams
print ('Your probability = {0}%'.format(result.predict(myExams)[0]*100))<jupyter_output>Your probability = 23.195940705038893%
{'fraction_non_alphanumeric': 0.09126819126819127, 'fraction_numerical': 0.01995841995841996, 'mean_word_length': 5.795197740112994, 'pattern_counts': {'":': 0, '<': 44, '<?xml version=': 0, '>': 45, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '262605', 'license': 'MIT', 'n_tokens_mistral': 1541, 'n_tokens_neox': 1449, 'n_words': 543, 'path': '11_LogisticRegressionSM.ipynb', 'repo_name': 'kafasin/big_data_science', 'size': '4813'}