<jupyter_start><jupyter_text>Logistic RegressionIn this part of the exercise, we built a logistic regression model to predict whether a student gets admitted into a university. Suppose that you are the administrator of a university department and you want to determine each applicant’s chance of admission based on their results on two exams. You have historical data from previous applicants that you can use as a training set for logistic regression. For each training example, you have the applicant’s scores on two exams and the admissions decision. Our task is to build a classification model that estimates an applicant’s probability of admission based the scores from those two exams.The file ex2data1.csv contains a training set.The structure of the dataset described blow:1. First column = First exam score 2. Second column = Second exam score3. Third column = Admit status Our assignment has these sections:1. Visualizing the Data    1. Loading dataset    2. Ploting scatter        1. 2D scatter        2. 3D scatter2. Implementation    1. Sigmoid Function (Warmup)    2. Cost Function and Gradient    3. Learning Parameters Using scipy.optimize.minimize3. Evaluating logistic regression    1. Visualization of Fitted Model    2. Predicting on Test Data    3. Accuracy on Training DataIn each section full description provided.  1. Plotting DatasetBefore starting on any task, it is often useful to understand the data by visualizing it. For this dataset, you can use a scatter plot to visualize the data, since it has only two properties to plot. (Many other problems that you will encounter in real life are multi-dimensional and can’t be plotted on a 2D plot.)  1.A Loading DatasetWe just need to import our data in file to a good structrue to work on it. So best option for us in python is Pandas.<jupyter_code># import library
import pandas as pd
import numpy as np

dataset = pd.read_csv('ex2data1.csv',names = ['Exam #1','Exam #2','Admitted'])
dataset.head()
# splitting to x and y variables for features and target variable
x = dataset.iloc[:,:-1].values
y = dataset.iloc[:,-1].values
print('x[0] ={}, y[0] ={}'.format(x[0],y[0]))
m, n = x.shape
print('#{} Number of training samples, #{} features per sample'.format(m,n))<jupyter_output>x[0] =[ 34.62365962  78.02469282], y[0] =0
#100 Number of training samples, #2 features per sample<jupyter_text>1.B Plotting ScatterIn this step we plot our data in 2D and 3D scatter.  1.B.a 3D Scatter<jupyter_code># imporing libraries
from matplotlib import pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
import random
%matplotlib inline

# visualize our data
fig = plt.figure()
ax = Axes3D(fig)

ax.scatter(x[:,0], x[:,1], y)
plt.show()<jupyter_output><empty_output><jupyter_text>1.B.b 2D Scatter<jupyter_code># Visualising the Training set results
from matplotlib.colors import ListedColormap
X_set, y_set = x[:,[1,2]], y.reshape(m,)
X1, X2 = np.meshgrid(np.arange(start = X_set[:, 0].min() - 1, stop = X_set[:, 0].max() + 1, step = 0.01),
                     np.arange(start = X_set[:, 1].min() - 1, stop = X_set[:, 1].max() + 1, step = 0.01))
plt.xlim(X1.min(), X1.max())
plt.ylim(X2.min(), X2.max())
for i, j in enumerate(np.unique(y_set)):
    plt.scatter(X_set[y_set == j, 0], X_set[y_set == j, 1],
                c = ListedColormap(('red', 'green'))(i), label = j)
plt.title('Admission State (Training set)')
plt.xlabel('Exam #1 Score')
plt.ylabel('Exam #2 Score')
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>2. Implementation  2.A Sigmoid Function (Warm up)Before you start with the actual cost function, recall that the logistic regression hypothesis is defined as: where function g is the sigmoid function. The sigmoid function is defined as: For large positive values of x, the sigmoid should be close to 1, while for large negative values, the sigmoid should be close to 0. Evaluating sigmoid(0) should give you exactly 0.5. For a matrix, your function should perform the sigmoid function on every element. h = hypothesis(x,theta) will compute sigmoid function on θTX and return a number which 0.You can use this library for calculating sigmoid.<jupyter_code>def sigmoid(z):
    return 1/(1+np.exp(-z))
def lr_hypothesis(x,theta):
    return np.dot(x,theta)<jupyter_output><empty_output><jupyter_text>2.B Cost Function and GradientNow you will implement the cost function and gradient for logistic regression and we return the cost and gradient.Recall that the cost function in logistic regression is: And the gradient of the cost is a vector of the same length as θ where the j th element (for j = 0, 1, . . . , n) is defined as follows:Note that while this gradient looks identical to the linear regression gradient, the formula is actually different because linear and logistic regression have different definitions of hθ(x).Implementation Note: We store each example as a row in the the x matrix in python. To take into account the intercept term (θ0), we add an additional first column to x and set it to all ones. This allows us to treat (θ0) as simply another feature.In the following lines, we add another dimension to our data to accommodate the (θ0) intercept term. We also initialize the initial parameters to 0.<jupyter_code># import library
import numpy as np

# appending ones to our x 
z = np.ones(m)
z = z.reshape(m,1)
x = np.append(z,x,axis =1)
print('x.shape = {}, x[0] = {}'.format(x.shape,x[0]))
m,n = x.shape
# define theta as zero
theta = np.zeros(n)
#theta = theta.reshape(n,1)

# reshape (-1,1) because we just have one feature in y column
y = y.reshape(-1,1)<jupyter_output><empty_output><jupyter_text>compute_cost(theta, x, y): computes the cost of using theta as the parameter for logistic regression.<jupyter_code>def compute_cost(theta,x,y):
    theta = theta.reshape(n,1)
    infunc = -y*(np.log(sigmoid(lr_hypothesis(x,theta)))) - (1-y)*(np.log(1 - sigmoid(lr_hypothesis(x,theta))))
    j = np.sum(infunc)/m
    return j<jupyter_output><empty_output><jupyter_text>compute_gradient(theta, x, y): computes the gradient of the cost using theta as the parameter for logistic regression.<jupyter_code>def compute_gradient(theta,x,y):
    theta = theta.reshape(n,1)
    infunc2 = sigmoid(lr_hypothesis(x,theta))-y
    gradient =  np.dot(x.transpose(),infunc2)/m
    gradient = gradient.flatten()
    return gradient<jupyter_output><empty_output><jupyter_text>Now with theta = 0 for n values, we should see that the cost = 0.693 and gradients = [0.100, -12.0092, -11.2628].<jupyter_code>cost_temp = compute_cost(theta,x,y)
gradient_temp = compute_gradient(theta,x,y)
print('if theta = 0  =======>\n cost = {}\n gradient =\n {}'.format(cost_temp,gradient_temp))<jupyter_output>if theta = 0  =======>
 cost = 0.6931471805599453
 gradient =
 [ -0.1        -12.00921659 -11.26284221]<jupyter_text>For validating our fucntions work correctly, we should test it on new theta initializations.Test theta = [-24; 0.2; 0.2] and output should be cost = 0.218 and gradients = [0.043, 2.566, 2.647].<jupyter_code>test_theta = np.array([-24,0.2,0.2])
test_theta = test_theta.reshape(n,1)

cost_temp2 = compute_cost(test_theta,x,y)
gradient_temp2 = compute_gradient(test_theta,x,y)
print('if theta = 0  =======>\n cost = {}\n gradient =\n {}'.format(cost_temp2,gradient_temp2))<jupyter_output>if theta = 0  =======>
 cost = 0.21833019382659796
 gradient =
 [ 0.04290299  2.56623412  2.64679737]<jupyter_text>2.C Learning Parameters Using scipy.optimize.minimizeIn the previous assignment, you found the optimal parameters of a linear regression model by implementing gradient descent. You wrote a cost function and calculated its gradient, then took a gradient descent step accordingly. This time, instead of taking gradient descent steps, you will use an scipy built-in function called minimize.Scipy's minimize is an optimization solver that finds the minimum of an unconstrained1 function. For logistic regression, you want to optimize the cost function J(θ) with parameters θ. Concretely, you are going to use minimize to find the best parameters θ for the logistic regression cost function, given a fixed dataset (of x and y values). You will pass to minimize the following inputs:     The initial values of the parameters we are trying to optimize.     A function that, when given the training set and a particular θ, computes the logistic regression cost with respect to θ for the dataset (x, y) ======> compute_cost     A function that, when given the training set and a particular θ, computes the logistic regression gradient with respect to θ for the dataset (x, y) ======> compute_gradient    1 Constraints in optimization often refer to constraints on the parameters. for example, constraints that bound the possible values θ can take (e.g., θ ≤ 1). Logistic regression does not have such constraints since θ is allowed to take any real value.  For doing this step, we have many complex optimization fucntions. You can visit this page for more information. For optimizing this problem, we use this library scipy.optimize.minimize and TNC solver method. Now some explanation about below code that do all loops and updates and other calculations only in one line.we need two functions, one for cost and the other for gradient that described above. all should have inputs you assgning for X0 and args.All of your functions should return just one variable which for cost, optimizer just minmize its return value but for gradient, because the target variable is theta which you specified by X0, optimizer will update theta to minimize cost.And at last, you define the mathematical model to do this optimization which in our case, it is  Truncated Newton method.<jupyter_code>import scipy.optimize as opt
optimized = opt.minimize(fun = compute_cost, x0 = theta, args = (x, y), method = 'TNC', jac = compute_gradient)<jupyter_output><empty_output><jupyter_text>Expected output for gradients after optimizations is gradients = [- 25.161, 0.206, 0.201].<jupyter_code>theta = optimized.x
theta<jupyter_output><empty_output><jupyter_text>This output, is optimized and best value for our theta, so now we can plot our boundary and start prediction on our test data.so here we update our theta to best one!  3 Evaluating logistic regressionAfter learning the parameters, you can use the model to predict whether a particular student will be admitted.Another way to evaluate the quality of the parameters we have found is to see how well the learned model predicts on our training set.  3.A Visualization of Fitted ModelThe best way is to see model. here model plotted in scatter diagram. For Visualizing of fitted model, we have to predict for every single row in x data and get predicted y and plot new ys.We should get something like this:<jupyter_code># Visualising the Training set results
"""from matplotlib.colors import ListedColormap
X_set, y_set = x[:,[1,2]], y.reshape(m,)
X1, X2 = np.meshgrid(np.arange(start = X_set[:, 0].min() - 1, stop = X_set[:, 0].max() + 1, step = 0.01),
                     np.arange(start = X_set[:, 1].min() - 1, stop = X_set[:, 1].max() + 1, step = 0.01))  
plt.xlim(X1.min(), X1.max())
plt.ylim(X2.min(), X2.max())
for i, j in enumerate(np.unique(y_set)):
    plt.scatter(X_set[y_set == j, 0], X_set[y_set == j, 1],
                c = ListedColormap(('red', 'green'))(i), label = j)
plt.title('K-Nearest Neighbors (Training set)')
plt.xlabel('Age')
plt.ylabel('Estimated Salary')
plt.legend()
plt.show()"""<jupyter_output><empty_output><jupyter_text>3.B Predicting on Test DataFor a student with an Exam 1 score = 45 and an Exam 2 score = 85, you should expect to see an admission probability of 0.776.For this prediction, we need just put this test data as new x to the sigmoid fucntion. For better usage, we implement get_probability(x,theta) function.<jupyter_code>def get_propability(x,theta):
    p = sigmoid(lr_hypothesis(x,theta))
    return p
# we put our test data into numpy array with 1 to intercept theta 0.
test_data = np.array([1, 45, 85])
print('The probability of admission of student with Score #1 = 45 and #2 = 85 ======>> {}'
      .format(get_propability(test_data,theta)))<jupyter_output>The probability of admission of student with Score #1 = 45 and #2 = 85 ======>> 0.7762906220893772<jupyter_text>3.C Accuracy on Training DataIn this step we will implement predictor function. Before this step we can calculate probability of admission of any student with respect ot two exam scores. Now we need to convert this probabilities to the 0 or 1 value because we only have two class. To do this we need a threshold to map porbabilities higher than threshold to 1 and lower to 0. And mathematically approved the best threshold for logistic regression is 0.5. So we have this situation:    1. get_probability(x,theta) >= 0.5 then admission = 1    2. get_probability(x,theta) <= 0.5 then admission = 0<jupyter_code>def predictor(x,theta):
    y_pred = np.zeros(m).reshape(m,)
    for i in range(0,m):
        p = get_propability(x[i],theta)
        if (p >= 0.5):
            y_pred[i] = 1
        else:
            y_pred[i] = 0
    return y_pred
y_pred_train = predictor(x,theta)
pd.DataFrame(y_pred_train).head(5)<jupyter_output><empty_output><jupyter_text>Now we will compare our predicted result to the true one with confusion_matrix of numpy library.<jupyter_code>from sklearn.metrics import confusion_matrix
cm_train = confusion_matrix(y,y_pred_train)

# Function for accuracy
def acc(confusion_matrix):
    t = confusion_matrix[0][0] + confusion_matrix[1][1]
    f = confusion_matrix[0][1] + confusion_matrix[1][0]
    ac = t/(t+f)
    return (t,f,ac)
t_train,f_train,acc_train = acc(cm_train)
print('{} correct, {} wrong ==========> accuracy = {}%'.format(t_train,f_train,acc_train*100))<jupyter_output>89 correct, 11 wrong ==========> accuracy = 89.0%
{'fraction_non_alphanumeric': 0.09683951688724958, 'fraction_numerical': 0.029218196282635424, 'mean_word_length': 4.602917341977309, 'pattern_counts': {'":': 0, '<': 57, '<?xml version=': 0, '>': 70, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '293182', 'license': 'MIT', 'n_tokens_mistral': 4512, 'n_tokens_neox': 4139, 'n_words': 1732, 'path': 'Week 3 - Logistic Regression/Logistic Regression.ipynb', 'repo_name': 'Nikronic/Coursera-Machine-Learning', 'size': '13827'}