<jupyter_start><jupyter_text>Information* Class: Special topics: Convolutional Neural Networks* Class id: INT3414 22* Lecturer: <NAME>, Ph.D.* Email: <EMAIL>* Slide & code: https://github.com/chupibk/INT3414_22---  PyTorch packages and tensors- In PyTorch, everything is a tensor!  + from v.0.4.0, there is no distinction between Tensor and Variable anymore- Commonly used packages:   + `torch` provides all the necessary Tensor operators  + `torch.nn` provides neural network modules  + `torch.optim` contains various optimization algorithms---  Importing packages and enable GPU<jupyter_code>#import the package
import torch
import torch.nn as nn
import torch.optim as optim

torch.__version__
# Enable cuda GPU
use_cuda = True
print(torch.cuda.device_count()) 
print(torch.cuda.get_device_name(0)) 
cuda0 = torch.device('cuda:0')<jupyter_output><empty_output><jupyter_text>Define some *tensors*<jupyter_code># data
x = torch.tensor(([2, 9], [1, 5], [3, 6]), dtype=torch.float) # 3x2 tensor
y = torch.tensor(([0], [1], [1]), dtype=torch.float)      # 3x1 tensor
x_test = torch.tensor(([4,8]), dtype=torch.float)             # 1x2 tensor
print(x.size())
print(y.size())
print(x)
#Scale the data to range [0, 1]
x_max, _ = torch.max(x, 0)
x_test_max, _ = torch.max(x_test, 0)


x = torch.div(x, x_max)
x_test = torch.div(x_test, x_test_max)

print(x)<jupyter_output>tensor([[0.6667, 1.0000],
        [0.3333, 0.5556],
        [1.0000, 0.6667]])<jupyter_text>GPU-enabled tensors<jupyter_code># a tensor of 4 observed data, each has 2 features
x = torch.randn(4, 2, device=cuda0, dtype=torch.float) ### NOTE: we assign device=cuda0

print(x)<jupyter_output>tensor([[ 0.2141, -0.0354],
        [ 0.3934,  0.8518],
        [-0.7442,  0.1648],
        [ 0.0674,  1.0748]], device='cuda:0')<jupyter_text>---  A sample neural network for binary classification Using Autograd<jupyter_code>#Let's get some data

import sklearn.datasets as datasets
from sklearn.model_selection import train_test_split

%matplotlib inline

x, y = datasets.make_moons(500, noise=0.2)

# Split into test and training data
x_train, x_test, y_train, y_test = train_test_split(x, y,
    test_size=0.25, random_state=73)

import matplotlib.pyplot as plt

plt.scatter(x[:, 0], x[:, 1], s=40, c=y, cmap="spring")
x.shape


#convert x, y to torch tensor
x_train_t  = torch.FloatTensor(x_train)
y_train_t = torch.FloatTensor(y_train).reshape(-1, 1)

#dimensions
D_in = x.shape[1] # Input dimension, = 2
H = 4 # hidden dimension, okay, let's use only one hidden layer
D_out = 1 # output dimension



#Initialize weights
# w1*x + bias --> h        ##hidden layer 1
# w2*h + bias --> output

# Setting requires_grad=True indicates that we want to compute gradients
# with respect to these tensors during the backward pass
w1 = torch.randn(D_in, H, dtype=torch.float, requires_grad=True) 
w2 = torch.randn(H, D_out, dtype=torch.float, requires_grad=True)


# Train = do feedforward + compute loss + backprop + update

lr = 0.01 #learning rate
loss_func = nn.BCELoss()
for t in range(5000): #epochs
  #forward pass: compute predicted y
  h = x_train_t.mm(w1)
  h_relu = h.clamp(min=0)
  output = h_relu.mm(w2)
  y_pred = torch.sigmoid(output)


  #compute and print loss
  # # loss is a Tensor of shape (1, )
  loss = loss_func( y_pred, y_train_t)

  if t % 100 == 99:
    print(t, loss.item())
  
  # backprop to compute gradients of w1 and w2 with respect to loss
  # for simplicity, let's use Autograd :)
  # This call will compute the
  # gradient of loss with respect to all Tensors with requires_grad=True.
  # After this call w1.grad and w2.grad will be Tensors holding the gradient
  # of the loss with respect to w1 and w2 respectively.
  loss.backward()


  # Manually update weights using gradient descent
  # temporarily set all requies_grad flag to false in order to update weights
  with torch.no_grad():
    w1 -= lr * w1.grad
    w2 -= lr * w2.grad

    # manually zero the gradients after updating weights
    # because PyTorch accumulates gradients on subsequent backward passes
    w1.grad.zero_()
    w2.grad.zero_()<jupyter_output><empty_output><jupyter_text>PyTorch: nn moduleProvide higher-level abstractions over raw computational graphs that are useful for building neural networks.<jupyter_code>import torch
import torch.nn as nn
import numpy as np

import sklearn.datasets as datasets
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt

%matplotlib inline

x, y = datasets.make_moons(500, noise=0.2)

# Split into test and training data
x_train, x_test, y_train, y_test = train_test_split(x, y,
    test_size=0.25, random_state=73)

#convert to tensors
x_train_t  = torch.FloatTensor(x_train)
y_train_t = torch.FloatTensor(y_train).reshape(-1, 1)

import numpy as np

# random input and output data
D_in, H, D_out = 2, 4, 1

# use nn package to define our model as a sequence of layers.
# nn.Sequential is a Module which contains other modules, and applies them in sequences to produce its output.
# Each Linear Module computes output from input using a linear function, and holds internal Tensors for its weight and bias

model = nn.Sequential(
    nn.Linear(D_in, H),
    nn.ReLU(),
    nn.Linear(H, D_out),
    nn.Sigmoid()
)

# nn package also contains defiinitions of popular loss functions
loss_fn = nn.BCELoss()
lr = 0.01 #learning rate
optimizer = torch.optim.Adam(
      model.parameters(),
      lr=lr
)

# log
train_loss = []
train_accuracy = []
for t in range(1000): #num epochs
    # Forward pass: compute predicted y by passing x to the model.
    y_pred = model(x_train_t)

    # Compute and print loss.
    loss = loss_fn(y_pred, y_train_t)
    if t % 100 == 99:
        print(t, loss.item())

    # Clear the buffers of all gradients before backward pass 
    optimizer.zero_grad()

    # Backward pass: compute gradient of the loss with respect to model parameters
    loss.backward()

    # Calling the step function on an Optimizer makes an update to its parameters
    optimizer.step()

    #log the training info
    y_pred_class = np.where(y_pred.detach().numpy()<0.5, 0, 1)
    accuracy = np.sum(y_train.reshape(-1, 1) == y_pred_class) / len(y_train)
    train_accuracy.append(accuracy)  
    train_loss.append(loss.item())

  

#visualize the training log
fig, ax = plt.subplots(2, 1, figsize=(12,8))
ax[0].plot(train_loss)
ax[0].set_ylabel('Loss')
ax[0].set_title('Training Loss')

ax[1].plot(train_accuracy)
ax[1].set_ylabel('Classification Accuracy')
ax[1].set_title('Training Accuracy')

plt.tight_layout()
plt.show()<jupyter_output>99 0.32643601298332214
199 0.2757432162761688
299 0.25963324308395386
399 0.25024038553237915
499 0.24700400233268738
599 0.24559710919857025
699 0.24482783675193787
799 0.24415405094623566
899 0.24353250861167908
999 0.24309110641479492<jupyter_text>More complex model<jupyter_code>import torch
import torch.nn as nn
import torch.nn.functional as F


class DynamicNet(nn.Module):
  def __init__(self, D_in, H, D_out):
    '''
    construct layer instances that we will use in forward pass
    '''
    super(DynamicNet, self).__init__()
    self.input_linear = nn.Linear(D_in, H)
    self.middle_linear = nn.Linear(H, H)
    self.output_linear = nn.Linear(H, D_out)


  def forward(self, x):
    '''
    build the computation graph
    '''
    x = self.input_linear(x)
    x = F.relu(x)
    
    for _ in range(4): #we'll have 4 hidden layers
      x = self.middle_linear(x)
      x = F.relu(x)
    x = self.output_linear(x)
    out = torch.sigmoid(x)
    
    return out



#data
D_in, H, D_out = 2, 4, 1


model = DynamicNet(D_in, H, D_out)

# Construct our loss function and an Optimizer. Training this strange model with
# vanilla stochastic gradient descent is tough, so we use momentum
criterion = torch.nn.BCELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-2)
for t in range(1000):
    # Forward pass: Compute predicted y by passing x to the model
    y_pred = model(x_train_t)

    # Compute and print loss
    loss = criterion(y_pred, y_train_t)
    if t % 100 == 99:
        print(t, loss.item())

    # Zero gradients, perform a backward pass, and update the weights.
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()<jupyter_output>99 0.3758455812931061
199 0.19530275464057922
299 0.13623546063899994
399 0.13012175261974335
499 0.1267622709274292
599 0.12296771258115768
699 0.12078988552093506
799 0.11968906968832016
899 0.11874430626630783
999 0.11808303743600845<jupyter_text>Performance evaluation on test data<jupyter_code># Evaluate the performance on test data
from sklearn.metrics import accuracy_score

# convert to tensor
x_test_t = torch.FloatTensor(x_test)

# do the forward pass
y_pred = model(x_test_t)

#get the predicted classes
y_pred_class = np.where(y_pred.detach().numpy()<0.5, 0, 1)
score = accuracy_score(y_pred_class, y_test)

print(score)

#or do it manually
accuracy = np.sum(y_test.reshape(-1, 1) == y_pred_class) / len(y_test)
print(accuracy)
# Forward pass
y_pred = model(x_train_t)

# Compute loss.
loss = loss_fn(y_pred, y_train_t)

# Clear gradient buffers 
optimizer.zero_grad()

# Backward pass: compute gradients
loss.backward()

# Update parameters
optimizer.step()<jupyter_output><empty_output><jupyter_text>Understand backpropagation(out of scope for INT3414_22)  Example Convolutional Neural Network Load the MNIST dataset<jupyter_code>import torch
import torchvision

train_loader = torch.utils.data.DataLoader(
    torchvision.datasets.MNIST("/files/", train=True, download=True,
                               transform=torchvision.transforms.Compose([
                                  torchvision.transforms.ToTensor(),
                                  torchvision.transforms.Normalize(
                                      (0.1307,), (0.3081,)
                                  )
                               ])),
    batch_size=64, shuffle=True
)

test_loader = torch.utils.data.DataLoader(
    torchvision.datasets.MNIST("/files/", train=False, download=True,
                               transform=torchvision.transforms.Compose([
                                  torchvision.transforms.ToTensor(),
                                  torchvision.transforms.Normalize(
                                      (0.1307,), (0.3081,)
                                  )
                               ])),
    batch_size=64, shuffle=True
)
arr = [ 0, 1, 2, 3]
arr2 = (i for i in arr)
print(type(arr))
type(arr2)

arr
next(arr2)
next(arr2)
list(arr2)
|# let's display some data

examples = enumerate(train_loader)
batch_idx, (example_data, example_targets) = next(examples)
print(example_data.shape)<jupyter_output><empty_output><jupyter_text>|-> a data batch as a tensor of shape: `torch.Size([64, 1, 28, 28])` (i.e., [num_data, num_channel, height, width])This means we have 64 examples of 28x28 pixels in 1 channel (grayscale).<jupyter_code>import matplotlib.pyplot as plt
%matplotlib inline

fig = plt.figure()
for i in range(6):
  plt.subplot(2,3,i+1)
  plt.tight_layout()
  plt.imshow(example_data[i][0], cmap='gray', interpolation='none')
  plt.title("Ground Truth: {}".format(example_targets[i]))
  plt.xticks([])
  plt.yticks([])
fig<jupyter_output><empty_output><jupyter_text>Build a convolutional network<jupyter_code>import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

class MyModel(nn.Module):
  def __init__(self):
    super(MyModel, self).__init__()
    self.conv1 = nn.Conv2d(1, 10, kernel_size=5)   #https://pytorch.org/docs/stable/nn.html?highlight=conv2d#torch.nn.Conv2d
    self.conv2 = nn.Conv2d(10, 20, kernel_size=3)
    self.fc1 = nn.Linear(500, 50) # how to find this 500 as dimesion in?
    self.fc2 = nn.Linear(50, 10)
  
  def forward(self, x):
    x = self.conv1(x)
    x = F.max_pool2d(x, 2)
    x = F.relu(x)
    x = self.conv2(x)
    x = F.max_pool2d(x, 2)
    x = F.relu(x)
    x = torch.flatten(x, 1)
   
    x = self.fc1(x)
    x = F.relu(x)
    x = self.fc2(x)
    x = F.log_softmax(x, dim=1)
    return x


#28x28x1 -conv1-> (28-5+1)x(28-5+1) = 24x24x10 -maxpool--> 12x12x10 -conv2-> (12-3+1)x(12-3+1)x20=10x10x20 -maxpool-> 5x5x20 =500

model = MyModel()

#calculate total parameters .  #--> confirm this by hand :)
# conv1: 5x5x1x10 + 10 + 3x3x10x20 + 20 + 500x50+50 +50x10+10
num_params = sum(p.numel() for p in model.parameters() if p.requires_grad)
print("Total parameters: ", num_params)<jupyter_output>Total parameters:  27640<jupyter_text><jupyter_code>n_epochs = 3
batch_size = 64
learning_rate = 0.01
log_interval = 10

loss = F.nll_loss
optimizer = optim.Adam(model.parameters(),
                       lr=learning_rate,
                     )

train_losses = []
train_counter = []
for epoch in range(1, n_epochs+1):
  for batch_idx, (data, target) in enumerate(train_loader):
    #forward
    output = model(data)
    #loss
    loss = F.nll_loss(output, target)

    #backward
    optimizer.zero_grad()
    loss.backward()
    #update
    optimizer.step()
    if batch_idx % log_interval == 0:
      print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
        epoch, batch_idx * len(data), len(train_loader.dataset),
        100. * batch_idx / len(train_loader), loss.item()))
      train_losses.append(loss.item())
      train_counter.append(
        (batch_idx*64) + ((epoch-1)*len(train_loader.dataset)))
      
#Visualize the training loss
fig = plt.figure()
plt.plot(train_counter, train_losses, color='blue')
plt.xlabel('number of training examples seen')
plt.ylabel('negative log likelihood loss')
fig<jupyter_output><empty_output><jupyter_text>Implementation of AlexNet<jupyter_code>class AlexNet(nn.Module):

    def __init__(self, num_classes=1000):
        super(AlexNet, self).__init__()
        self.features = nn.Sequential(
            nn.Conv2d(3, 64, kernel_size=11, stride=4, padding=2),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),
            nn.Conv2d(64, 192, kernel_size=5, padding=2),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),
            nn.Conv2d(192, 384, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.Conv2d(384, 256, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.Conv2d(256, 256, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),
        )
        self.avgpool = nn.AdaptiveAvgPool2d((6, 6))
        self.classifier = nn.Sequential(
            nn.Dropout(),
            nn.Linear(256 * 6 * 6, 4096),
            nn.ReLU(inplace=True),
            nn.Dropout(),
            nn.Linear(4096, 4096),
            nn.ReLU(inplace=True),
            nn.Linear(4096, num_classes),
        )

    def forward(self, x):
        x = self.features(x)
        x = self.avgpool(x)
        x = torch.flatten(x, 1)
        x = self.classifier(x)
        return x<jupyter_output><empty_output><jupyter_text>HOMEWORK1. Evaluate the model on test data  + Run the model  + Calculate precision, accuracy, F1 score2. Save the model and call the model from saved file3. Create checkpoint during training and Implement training resume from checkpoint  + Hint: During training, save both network state and optimizer state  ```   torch.save(network.state_dict(), '/results/model.pth')   torch.save(optimizer.state_dict(), '/results/optimizer.pth')  ```<jupyter_code><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1171642755590022, 'fraction_numerical': 0.06042393692645728, 'mean_word_length': 3.4302891497280275, 'pattern_counts': {'":': 0, '<': 52, '<?xml version=': 0, '>': 58, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '174210', 'license': 'MIT', 'n_tokens_mistral': 6047, 'n_tokens_neox': 5432, 'n_words': 1391, 'path': 'CNNs_week2+3_foundations_ND.ipynb', 'repo_name': 'kienle371999/mxnet-text-to-image', 'size': '15502'}