<jupyter_start><jupyter_text>Lab 5: Logistic ClassificationAuthor: <NAME> (이승재)     We use elemental PyTorch to implement linear regression here. However, in most actual applications, abstractions such as nn.Module or nn.Linear are used. You can see those implementations near the end of this notebook.  Reminder: Logistic Regression  Hypothesis $$ H(X) = \frac{1}{1+e^{-W^T X}} $$  Cost $$ cost(W) = -\frac{1}{m} \sum y \log\left(H(x)\right) + (1-y) \left( \log(1-H(x) \right) $$  - If $y \simeq H(x)$, cost is near 0. - If $y \neq H(x)$, cost is high.  Weight Update via Gradient Descent $$ W := W - \alpha \frac{\partial}{\partial W} cost(W) $$  - $\alpha$: Learning rate  Imports<jupyter_code>import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
torch.manual_seed(1)<jupyter_output><empty_output><jupyter_text>Training Data<jupyter_code>x_data = [[1, 2], [2, 3], [3, 1], [4, 3], [5, 3], [6, 2]]
y_data = [[0], [0], [0], [1], [1], [1]]<jupyter_output><empty_output><jupyter_text>Consider the following classification problem: given the number of hours each student spent watching the lecture and working in the code lab, predict whether the student passed or failed a course. For example, the first (index 0) student watched the lecture for 1 hour and spent 2 hours in the lab session ([1, 2]), and ended up failing the course ([0]).<jupyter_code>x_train = torch.FloatTensor(x_data)
y_train = torch.FloatTensor(y_data)<jupyter_output><empty_output><jupyter_text>As always, we need these data to be in `torch.Tensor` format, so we convert them.<jupyter_code>print(x_train.shape)
print(y_train.shape)<jupyter_output>torch.Size([6, 2])
torch.Size([6, 1])<jupyter_text>Computing the Hypothesis $$ H(X) = \frac{1}{1+e^{-W^T X}} $$ PyTorch has a `torch.exp()` function that resembles the exponential function.<jupyter_code>print('e^1 equals: ', torch.exp(torch.FloatTensor([1])))<jupyter_output>e^1 equals:  tensor([2.7183])<jupyter_text>We can use it to compute the hypothesis function conveniently.<jupyter_code>W = torch.zeros((2, 1), requires_grad=True)
b = torch.zeros(1, requires_grad=True)
hypothesis = 1 / (1 + torch.exp(-(x_train.matmul(W) + b)))
print(hypothesis)
print(hypothesis.shape)<jupyter_output>tensor([[0.5000],
        [0.5000],
        [0.5000],
        [0.5000],
        [0.5000],
        [0.5000]], grad_fn=<MulBackward0>)
torch.Size([6, 1])<jupyter_text>Or, we could use `torch.sigmoid()` function! This resembles the sigmoid function:<jupyter_code>print('1/(1+e^{-1}) equals: ', torch.sigmoid(torch.FloatTensor([1])))<jupyter_output>1/(1+e^{-1}) equals:  tensor([0.7311])<jupyter_text>Now, the code for hypothesis function is cleaner.<jupyter_code>hypothesis = torch.sigmoid(x_train.matmul(W) + b)
print(hypothesis)
print(hypothesis.shape)<jupyter_output>tensor([[0.5000],
        [0.5000],
        [0.5000],
        [0.5000],
        [0.5000],
        [0.5000]], grad_fn=<SigmoidBackward>)
torch.Size([6, 1])<jupyter_text>Computing the Cost Function (Low-level) $$ cost(W) = -\frac{1}{m} \sum y \log\left(H(x)\right) + (1-y) \left( \log(1-H(x) \right) $$ We want to measure the difference between `hypothesis` and `y_train`.<jupyter_code>print(hypothesis)
print(y_train)<jupyter_output>tensor([[0.5000],
        [0.5000],
        [0.5000],
        [0.5000],
        [0.5000],
        [0.5000]], grad_fn=<SigmoidBackward>)
tensor([[0.],
        [0.],
        [0.],
        [1.],
        [1.],
        [1.]])<jupyter_text>For one element, the loss can be computed as follows:<jupyter_code>-(y_train[0] * torch.log(hypothesis[0]) + 
  (1 - y_train[0]) * torch.log(1 - hypothesis[0]))<jupyter_output><empty_output><jupyter_text>To compute the losses for the entire batch, we can simply input the entire vector.<jupyter_code>losses = -(y_train * torch.log(hypothesis) + 
           (1 - y_train) * torch.log(1 - hypothesis))
print(losses)<jupyter_output>tensor([[0.6931],
        [0.6931],
        [0.6931],
        [0.6931],
        [0.6931],
        [0.6931]], grad_fn=<NegBackward>)<jupyter_text>Then, we just `.mean()` to take the mean of these individual losses.<jupyter_code>cost = losses.mean()
print(cost)<jupyter_output>tensor(0.6931, grad_fn=<MeanBackward0>)<jupyter_text>Computing the Cost Function with `F.binary_cross_entropy` In reality, binary classification is used so often that PyTorch has a simple function called `F.binary_cross_entropy` implemented to lighten the burden.<jupyter_code>F.binary_cross_entropy(hypothesis, y_train)<jupyter_output><empty_output><jupyter_text>Training with Low-level Binary Cross Entropy Loss<jupyter_code>x_data = [[1, 2], [2, 3], [3, 1], [4, 3], [5, 3], [6, 2]]
y_data = [[0], [0], [0], [1], [1], [1]]
x_train = torch.FloatTensor(x_data)
y_train = torch.FloatTensor(y_data)
W = torch.zeros((2, 1), requires_grad=True)
b = torch.zeros(1, requires_grad=True)
optimizer = optim.SGD([W, b], lr=1)
nb_epochs = 1000

for epoch in range(nb_epochs + 1):
    hypothesis = torch.sigmoid(x_train.matmul(W) + b)
    cost = -(y_train * torch.log(hypothesis) + (1 - y_train) * torch.log(1 - hypothesis)).mean()

    optimizer.zero_grad()
    cost.backward()
    optimizer.step()

    if epoch % 100 == 0:
        print('Epoch {:4d}/{} Cost: {:.6f}'.format(
            epoch, nb_epochs, cost.item()
        ))<jupyter_output>Epoch    0/1000 Cost: 0.693147
Epoch  100/1000 Cost: 0.134722
Epoch  200/1000 Cost: 0.080643
Epoch  300/1000 Cost: 0.057900
Epoch  400/1000 Cost: 0.045300
Epoch  500/1000 Cost: 0.037261
Epoch  600/1000 Cost: 0.031673
Epoch  700/1000 Cost: 0.027556
Epoch  800/1000 Cost: 0.024394
Epoch  900/1000 Cost: 0.021888
Epoch 1000/1000 Cost: 0.019852<jupyter_text>Training with `F.binary_cross_entropy`<jupyter_code>W = torch.zeros((2, 1), requires_grad=True)
b = torch.zeros(1, requires_grad=True)
optimizer = optim.SGD([W, b], lr=1)
nb_epochs = 1000

for epoch in range(nb_epochs + 1):
    hypothesis = torch.sigmoid(x_train.matmul(W) + b)
    cost = F.binary_cross_entropy(hypothesis, y_train)

    optimizer.zero_grad()
    cost.backward()
    optimizer.step()

    if epoch % 100 == 0:
        print('Epoch {:4d}/{} Cost: {:.6f}'.format(
            epoch, nb_epochs, cost.item()
        ))<jupyter_output>Epoch    0/1000 Cost: 0.693147
Epoch  100/1000 Cost: 0.134722
Epoch  200/1000 Cost: 0.080643
Epoch  300/1000 Cost: 0.057900
Epoch  400/1000 Cost: 0.045300
Epoch  500/1000 Cost: 0.037261
Epoch  600/1000 Cost: 0.031673
Epoch  700/1000 Cost: 0.027556
Epoch  800/1000 Cost: 0.024394
Epoch  900/1000 Cost: 0.021888
Epoch 1000/1000 Cost: 0.019852<jupyter_text>Loading Real Data<jupyter_code>import numpy as np
xy = np.loadtxt('data-03-diabetes.csv', delimiter=',', dtype=np.float32)
x_data = xy[:, 0:-1]
y_data = xy[:, [-1]]
x_train = torch.FloatTensor(x_data)
y_train = torch.FloatTensor(y_data)
print(x_train[0:5])
print(y_train[0:5])<jupyter_output>tensor([[-0.2941,  0.4874,  0.1803, -0.2929,  0.0000,  0.0015, -0.5312, -0.0333],
        [-0.8824, -0.1457,  0.0820, -0.4141,  0.0000, -0.2072, -0.7669, -0.6667],
        [-0.0588,  0.8392,  0.0492,  0.0000,  0.0000, -0.3055, -0.4927, -0.6333],
        [-0.8824, -0.1055,  0.0820, -0.5354, -0.7778, -0.1624, -0.9240,  0.0000],
        [ 0.0000,  0.3769, -0.3443, -0.2929, -0.6028,  0.2846,  0.8873, -0.6000]])
tensor([[0.],
        [1.],
        [0.],
        [1.],
        [0.]])<jupyter_text>Training with Real Data using low-level Binary Cross Entropy Loss<jupyter_code>W = torch.zeros((8, 1), requires_grad=True)
b = torch.zeros(1, requires_grad=True)
optimizer = optim.SGD([W, b], lr=1)
nb_epochs = 100

for epoch in range(nb_epochs + 1):
    hypothesis = torch.sigmoid(x_train.matmul(W) + b) # or .mm or @
    cost = -(y_train * torch.log(hypothesis) + (1 - y_train) * torch.log(1 - hypothesis)).mean()
    
    optimizer.zero_grad()
    cost.backward()
    optimizer.step()

    if epoch % 10 == 0:
        print('Epoch {:4d}/{} Cost: {:.6f}'.format(
            epoch, nb_epochs, cost.item()
        ))<jupyter_output>Epoch    0/100 Cost: 0.693148
Epoch   10/100 Cost: 0.572727
Epoch   20/100 Cost: 0.539493
Epoch   30/100 Cost: 0.519708
Epoch   40/100 Cost: 0.507066
Epoch   50/100 Cost: 0.498539
Epoch   60/100 Cost: 0.492549
Epoch   70/100 Cost: 0.488208
Epoch   80/100 Cost: 0.484985
Epoch   90/100 Cost: 0.482543
Epoch  100/100 Cost: 0.480661<jupyter_text>Training with Real Data using `F.binary_cross_entropy`<jupyter_code>W = torch.zeros((8, 1), requires_grad=True)
b = torch.zeros(1, requires_grad=True)
optimizer = optim.SGD([W, b], lr=1)
nb_epochs = 100

for epoch in range(nb_epochs + 1):
    hypothesis = torch.sigmoid(x_train.matmul(W) + b) # or .mm or @
    cost = F.binary_cross_entropy(hypothesis, y_train)

    optimizer.zero_grad()
    cost.backward()
    optimizer.step()

    if epoch % 10 == 0:
        print('Epoch {:4d}/{} Cost: {:.6f}'.format(
            epoch, nb_epochs, cost.item()
        ))<jupyter_output>Epoch    0/100 Cost: 0.693148
Epoch   10/100 Cost: 0.572727
Epoch   20/100 Cost: 0.539493
Epoch   30/100 Cost: 0.519708
Epoch   40/100 Cost: 0.507066
Epoch   50/100 Cost: 0.498539
Epoch   60/100 Cost: 0.492549
Epoch   70/100 Cost: 0.488208
Epoch   80/100 Cost: 0.484985
Epoch   90/100 Cost: 0.482543
Epoch  100/100 Cost: 0.480661<jupyter_text>Checking the Accuracy our our Model After we finish training the model, we want to check how well our model fits the training set.<jupyter_code>hypothesis = torch.sigmoid(x_train.matmul(W) + b)
print(hypothesis[:5])<jupyter_output>tensor([[0.4103],
        [0.9242],
        [0.2300],
        [0.9411],
        [0.1772]], grad_fn=<SliceBackward>)<jupyter_text>We can change **hypothesis** (real number from 0 to 1) to **binary predictions** (either 0 or 1) by comparing them to 0.5.<jupyter_code>prediction = hypothesis >= torch.FloatTensor([0.5])
print(prediction[:5])<jupyter_output>tensor([[False],
        [ True],
        [False],
        [ True],
        [False]])<jupyter_text>Then, we compare it with the correct labels `y_train`.<jupyter_code>print(prediction[:5])
print(y_train[:5])
correct_prediction = prediction.float() == y_train
print(correct_prediction[:5])<jupyter_output>tensor([[True],
        [True],
        [True],
        [True],
        [True]])<jupyter_text>Finally, we can calculate the accuracy by counting the number of correct predictions and dividng by total number of predictions.<jupyter_code>accuracy = correct_prediction.sum().item() / len(correct_prediction)
print('The model has an accuracy of {:2.2f}% for the training set.'.format(accuracy * 100))<jupyter_output>The model has an accuracy of 76.68% for the training set.<jupyter_text>Optional: High-level Implementation with `nn.Module`<jupyter_code>class BinaryClassifier(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(8, 1)
        self.sigmoid = nn.Sigmoid()

    def forward(self, x):
        return self.sigmoid(self.linear(x))
model = BinaryClassifier()
optimizer = optim.SGD(model.parameters(), lr=1)
nb_epochs = 100

for epoch in range(nb_epochs + 1):
    hypothesis = model(x_train)
    cost = F.binary_cross_entropy(hypothesis, y_train)

    optimizer.zero_grad()
    cost.backward()
    optimizer.step()
    
    if epoch % 10 == 0:
        prediction = hypothesis >= torch.FloatTensor([0.5])
        correct_prediction = prediction.float() == y_train
        accuracy = correct_prediction.sum().item() / len(correct_prediction)
        
        print('Epoch {:4d}/{} Cost: {:.6f} Accuracy {:2.2f}%'.format(
            epoch, nb_epochs, cost.item(), accuracy * 100,
        ))<jupyter_output>Epoch    0/100 Cost: 0.704829 Accuracy 45.72%
Epoch   10/100 Cost: 0.572392 Accuracy 67.59%
Epoch   20/100 Cost: 0.539564 Accuracy 73.25%
Epoch   30/100 Cost: 0.520041 Accuracy 75.89%
Epoch   40/100 Cost: 0.507561 Accuracy 76.15%
Epoch   50/100 Cost: 0.499125 Accuracy 76.42%
Epoch   60/100 Cost: 0.493177 Accuracy 77.21%
Epoch   70/100 Cost: 0.488846 Accuracy 76.81%
Epoch   80/100 Cost: 0.485612 Accuracy 76.28%
Epoch   90/100 Cost: 0.483146 Accuracy 76.55%
Epoch  100/100 Cost: 0.481234 Accuracy 76.81%
{'fraction_non_alphanumeric': 0.1540114021317029, 'fraction_numerical': 0.10881599603404114, 'mean_word_length': 3.761605035405193, 'pattern_counts': {'":': 0, '<': 82, '<?xml version=': 0, '>': 84, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '135471', 'license': 'MIT', 'n_tokens_mistral': 5494, 'n_tokens_neox': 4692, 'n_words': 1097, 'path': 'AI 이노베이션 스퀘어 음성지능 과정/20200926/lab-05_logistic_classification.ipynb', 'repo_name': 'donddog/AI_Innovation_Square_Codes', 'size': '12109'}