<jupyter_start><jupyter_text>Introduction: What is Logistic Regression?Logistic regression is a supervised learning classification algorithm used to predict the probability of a target variable. The nature of target or dependent variable is dichotomous, which means there would be only two possible classes.<jupyter_code>import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_classification
import seaborn as sns<jupyter_output><empty_output><jupyter_text>Behind The Mask The term **logistic** in logistic regression is used because we are applying another function to the weighted sum of input data and parameters of the model and this function is called logit (**sigmoid**) function.Sigmoid function always outputs values between 0 and 1, mapping values to a range and thus can be used to calculate probabilities of input data belonging to a certain class:<jupyter_code>def sigmoid(x):

    % ====================== YOUR CODE HERE ======================================================
    % Instructions: Compute the sigmoid of each value of x (x can be a matrix, vector or scalar).



    % ============================================================================================<jupyter_output><empty_output><jupyter_text>Performance MeasureWeights (represented by theta in our notation) is a vital part of Logistic Regression and other Machine Learning algorithms and we want to find the best values for them. To start we pick random values and we need a way to measure how well the algorithm performs using those random weights. That measure is computed using the **cost function**. The cost function is defined as:The above two functions can be compressed into a single function i.e.<jupyter_code>def compute_cost(X, y, theta):

    # Initialize some useful values
    m = length(y) % number of training examples


    % ====================== YOUR CODE HERE ==========================
    % Instructions: Compute the cost of a particular choice of theta.
    %               
    %
    % ================================================================<jupyter_output><empty_output><jupyter_text>Finding Optimal ParametersThe goal is to minimize the cost by means of increasing or decreasing the weights. This can be done with a function called **Gradient Descent**. Gradient descent is just the **derivative of the cost function with respect to its weights**.Now to minimize our cost function we need to run the gradient descent function on each parameter i.e.Where: α = **learning rate** (usually 0.1)This is implemented as follows :<jupyter_code>def gradient_descent(X, y, theta, learning_rate, iterations):
    
    # Initialize some useful values
    m = len(y)
    cost_history = np.zeros((iterations,1))
    

    % ====================== YOUR CODE HERE ==============================================
    % Instructions: Write a for loop that runs the parameter "iterations" number of times.
        In each iteration do the following -
        > Update theta according to the formula shown above
        > Compute cost for current theta using compute_cost() and enter
          it into the array cost_history at that iteration index.
    %
    %   Return (cost_history, theta)
    %
    %
    %               
    %
    % ====================================================================================<jupyter_output><empty_output><jupyter_text>Predicting ClassesNow let's write the **prediction function**. Since we are dealing with probabilities here, if the resulting value is above 0.50, we round it up to 1, meaning the data sample belongs to the class 1. Consequently, if the probability of a data sample belonging to the class 1 is below 0.50, it simply means that it is part of the other class (class 0).Remember that this is binary classification, so we have only two classes (class 1 and class 0).<jupyter_code>def predict(X, theta):
    
    # Initialize some useful values
    m = size(X, 1); % Number of training examples

    % ====================== YOUR CODE HERE =================================
    % Instructions: Compute the predictions for X using a threshold at 0.5 
        (i.e., if sigmoid(theta.T * x) >= 0.5, predict 1)



    % =======================================================================<jupyter_output><empty_output><jupyter_text>Dataset GenerationAfter writing the code for the necessary functions, let’s create our very own dataset with **make_classification** function from **sklearn.datasets**. We will create **500 sample points** with **two classes** and plot the dataset with the help of seaborn library.<jupyter_code>X, y = make_classification(n_samples=500, n_features=2, n_redundant=0, n_informative=1,
                             n_clusters_per_class=1, random_state=14)

y = y[:,np.newaxis]

sns.set_style('white')
sns.scatterplot(X[:,0],X[:,1],hue=y.reshape(-1));<jupyter_output><empty_output><jupyter_text>Model Testing Finally, here is the algorithm for implementation, that we built for you using the functions you created above.Now, let's run it to calculate the parameters of our model.<jupyter_code>m = len(y)

X = np.hstack((np.ones((m,1)),X))
n = np.size(X,1)
theta = np.zeros((n,1))

iterations = 1500
learning_rate = 0.03

initial_cost = compute_cost(X, y, theta)

print("Initial Cost is: {} \n".format(initial_cost))

(cost_history, params_optimal) = gradient_descent(X, y, params, learning_rate, iterations)

print("Optimal Parameters are: \n", params_optimal, "\n")<jupyter_output><empty_output><jupyter_text>Now we will plot the cost as a function of number of iterations.<jupyter_code>plt.figure()
sns.set_style('white')
plt.plot(range(len(cost_history)), cost_history, 'r')
plt.title("Convergence Graph of Cost Function")
plt.xlabel("Number of Iterations")
plt.ylabel("Cost")
plt.show()<jupyter_output><empty_output><jupyter_text>- Does the cost decrease with every iteration?- Does it converge to a minimum value?  Model AccuracyAfter running the algorithm and getting the optimal parameters, we want to know how successful our model is at predicting the classes of our data.<jupyter_code>y_pred = predict(X, params_optimal)
score = float(sum(y_pred == y))/ float(len(y))

print(score)<jupyter_output><empty_output><jupyter_text>What accuracy score did you achieve?!  Dataset Plotting with Decision BoundaryNow, for the sake of visualization, let’s plot our dataset along with the **decision boundary** of our model.We simply calculate the intercept and slope values using the optimal parameters and plot the boundary that classifies the data into two classes.<jupyter_code>slope = -(params_optimal[1] / params_optimal[2])
intercept = -(params_optimal[0] / params_optimal[2])

sns.set_style('white')
sns.scatterplot(X[:,1],X[:,2],hue=y.reshape(-1));

ax = plt.gca()
ax.autoscale(False)
x_vals = np.array(ax.get_xlim())
y_vals = intercept + (slope * x_vals)
plt.plot(x_vals, y_vals, c="k");<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1577432354634427, 'fraction_numerical': 0.00791594703511802, 'mean_word_length': 5.09561403508772, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 44, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '51253', 'license': 'MIT', 'n_tokens_mistral': 1979, 'n_tokens_neox': 1825, 'n_words': 743, 'path': 'Logistic Regression from scratch/Logistic Regression from scratch.ipynb', 'repo_name': 'abhisngh/Data-Science', 'size': '6948'}