<jupyter_start><jupyter_text>1: Basics  1.1: Tensors<jupyter_code># Creating tensors, operating it and particularities

## Empty tensor
x = torch.empty(1) # 1-D empty tensor
x = torch.empty(2, 1) # 2-D empty tensor
# x = torch.empty(n-dimensional tensor) # n-D empty tensor

## Zeros tensor
x = torch.zeros(1) # 1-D zeros tensor

## Ones tensor
x = torch.ones(1) # 1-D ones tensor

## Defining type
x = torch.ones(2,2, dtype=torch.float16)
# print(x.dtype)

## Retrieving tensor size
# print(x.size())

## Creating tensor "from scratch"
x = torch.tensor([[2.5, 5], [1.7, 8]])

## Operations
x = torch.rand(2, 2)
y = torch.rand(2, 2)

### Arithmetic
z = x+y # Element-wise addition
z = torch.add(x,y) # Same result
y.add_(x) # Inplace element-wise addition; functions if "_" before its name make inplace ops

z = x-y # Element-wise subtraction
z = torch.sub(x, y) # Analogous as above
y.sub_(x) # Analogous as above

z = x*y # Element-wise multiplication
z = torch.mul(x, y) # Analogous as above
y.mul_(x) # Analogous as above

z = x / y # Element-wise division
z = torch.div(x, y) # Analogous as above
y.div_(x) # Analogous as above

### Slicing
z = x[:, 1]
z = x[1, 1]

### Reshaping
z = x.view(4)
z = x.view(-1, 2)

### Casting to numpy arrays
z = x.numpy()

### Casting from numpy arrays to pytorch tensors
z = np.array([[1, 3.4], [5, 8.9]])
w = torch.from_numpy(z)

### Working on GPU
# if torch.cuda_is_available():
#     gpu = torch.device("cuda")
#     x = torch.ones(1, 2, device=gpu) # Tensor allocated on GPU
#     y = torch.ones(5) # Tensor allocated on CPU
#     y = y.to(gpu) # Sending tensor instantiated on CPU to GPU

### Observation
#################################################################################
#    When dealing with tensors which you need to compute their gradients        #
#    (e.g in optimization), Pytorch requires defining it explicitly.            #
#################################################################################
# x = torch.ones(4, requires_grad=True) # requires_grad argument is False by default<jupyter_output><empty_output><jupyter_text>---  1.2: Autograd Autograd's package: used to ease the calculation of gradients. Its principle is based on Automatic Differentiation (autodiff)which is a general method for automatically computing the derivative of a value. For more details, I highly recommend checking out<NAME> lecture's slides¹ about the technique and the following medium post² about Pytorch's implementation of it.1: https://www.cs.toronto.edu/~rgrosse/courses/csc321_2018/slides/lec10.pdf 2: https://mustafaghali11.medium.com/how-pytorch-backward-function-works-55669b3b7c62<jupyter_code># First, let's create an input tensor
x = torch.rand(3, requires_grad=True) # Remember that it's necessary to enable requires_grad property

# After creating the input tensor, let's define
# an operation over this tensor.
y = 3*x+2 # Creates the computational graph<jupyter_output><empty_output><jupyter_text>**Computational graph created after operating over the tensor:**<jupyter_code>Image(filename="assets/images/compt_graph.png")
print("X:", x)
print("Y:", y)<jupyter_output>X: tensor([0.1159, 0.7438, 0.5619], requires_grad=True)
Y: tensor([2.3477, 4.2314, 3.6857], grad_fn=<AddBackward0>)<jupyter_text>As seen, tensor X has requires_grad property set as **True** and tensor Y has the last operation over X (addition) as its gradient function. In the next example, Z is a tensor computed multiplying Y by itself and your grad. function is the multiplication op.<jupyter_code>z = y*y
print(z)<jupyter_output>tensor([ 5.5116, 17.9049, 13.5847], grad_fn=<MulBackward0>)<jupyter_text>For the sake of understanding, let's simplify assuming that our output tensor is 1-dimensional.<jupyter_code>z = z.mean()
print(z)<jupyter_output>tensor(12.3338, grad_fn=<MeanBackward0>)<jupyter_text>So, in order to compute Z gradient w.r.t. X, we call the Z backward() function and X grad property.<jupyter_code>z.backward()
print(x.grad, '\nThree gradients of Z w.r.t. X, because X is a 3-D tensor')<jupyter_output>tensor([4.6954, 8.4628, 7.3715]) 
Three gradients of Z w.r.t. X, because X is a 3-D tensor<jupyter_text>On the other hand, if we have a multi-dimensional output tensor, we need to specify a gradient vector as an argument of backward() functiontowards the correct execution. In this sense, it's used a 'ones' tensor as the gradient vector in the Jacobian-vector Product - which isa way for computing partial derivatives, used inside autograd. For more details, please check the references in the head of this section.<jupyter_code>y = 3*x+2
z = y*y
size = list(z.size())
init_grad_vec = torch.ones(size)

z.backward(init_grad_vec)
print(x.grad)<jupyter_output>tensor([18.7815, 33.8514, 29.4859])<jupyter_text>Though Pytorch allocate every gradient of a specific variable, sometimes it's essential to reset our gradients.<jupyter_code>for i in range(3):
    y = 3*x + 2
    z = y*y

    z.backward(init_grad_vec)
    print("Step {} gradient vector: {}".format(i+1, x.grad))

_ = x.grad.zero_()
for i in range(3):
    y = 3*x + 2
    z = y*y

    z.backward(init_grad_vec)
    print("Step {} gradient vector: {}".format(i+1, x.grad))
    
    _ = x.grad.zero_()<jupyter_output>Step 1 gradient vector: tensor([14.0861, 25.3885, 22.1144])
Step 2 gradient vector: tensor([14.0861, 25.3885, 22.1144])
Step 3 gradient vector: tensor([14.0861, 25.3885, 22.1144])<jupyter_text>Sometimes during training we might want to stop tracking variables from the gradient perspective. There are three ways to do so.(As seen below, none of tensors have the property requires_grad in it)<jupyter_code># First way
x.requires_grad_(False)
# print(x)

# Second way
x.detach()
# print(x)

# Third way
with torch.no_grad():
    y = x + 2
#     print(y)<jupyter_output><empty_output><jupyter_text>---  1.3: Backpropagation Backpropagation is an algorithm used to compute gradients of functions w.r.t. parameters that appears previously to the function computation, and them is backpropagated to which one of them. Let's suppose we have a first degree function (w/o linear coefficient, i.e. $y = f(x) = {\theta}x$). Suppose our parameter $\theta = 5$.In our computational graph Pytorch's framework, such operation should be seen as follows:<jupyter_code>Image(filename="assets/images/bprop_ex_compt_graph_1.png")<jupyter_output><empty_output><jupyter_text>Here we have a multiplication operation and the attribution of this result in Y variable (an identity). In the background,the result of the first operation is seen as an intermediate variable - here called Z -, and them it's attributed to Y, completingthe forward pass stage. Given that, to compute the gradient of Y w.r.t. X it's necessary to operate using the chain rule of calculus,in order to compute the gradient of Y w.r.t the intermediate variable Z. --- Briefly, the chain rule assures (not restricted to a 2-step computation. In fact, it's valid, in general, for n-steps): $\dfrac{dy}{dz}*\dfrac{dz}{dx} = \dfrac{dy}{dx}$ --- The complete computation graph with the forward and backward pass is depicted below:<jupyter_code>Image(filename="assets/images/bprop_ex_compt_graph_2.png")<jupyter_output><empty_output><jupyter_text>The Pytorch implementation of the backpropagation is as simple as follows:<jupyter_code># Instantiate data point
x = torch.tensor([4], dtype=torch.float32, requires_grad=True)

# Computational graph
y = 5*x # dy/dx = 5

# Backward pass
y.backward()
dy = x.grad

print(f'dy/dx = {int(dy.item())} (as expected!)')<jupyter_output>dy/dx = 5 (as expected!)<jupyter_text>---  1.4: Gradient Descent Algorithm Toward to wrap all of these concepts and enable the algorithm to learn the paremeters' values, the gradient descent algorithmthis task*. This algorithm can be divided in four steps:* Forward pass* Loss computation* Backward pass* Update parametersThe only which wasn't introduced is the loss computation. In short, since the learning is being supervised, i.e. it's known a priori the correctoutput's value, one way to compute 'how much' the approximation function being learned is 'wrong' is computing the error between itsoutput and the correct value. Here, the loss function used is the well-known Mean Squared Error (MSE)**. \* The purpose here isn't to explain all the details of gradient descent algorithm.\** In the examples below the problem is a regression one. First, let's code it from scratch, using Numpy:<jupyter_code># Set data points
X = np.array([1, 2, 3, 4, 5], dtype=np.float32)
y = np.array([5, 10, 15, 20, 25], dtype=np.float32)

# Randomly initialize parameter
theta = np.random.randn(1)

# Define hyperparameters
lr = 0.01
num_iters = 100

for i in range(1, num_iters+1):
    # Forward pass
    y_hat = theta*X

    # Loss computation
    J = np.mean((y_hat - y)**2) / 2

    # Backward pass
    dJ = np.mean(X*(y_hat - y)) # This is the derivative of the loss function w.r.t. the parameter theta

    # Update parameter
    theta = theta - lr*dJ

    # Print loss function
    if i % 20 == 0:
        space_str = displayMathHelper(num_iters, i)
        display(Math(r'{space}{i}th \ iter. \ Error (J): {J:.3f} \ | \ Parameter \ (\theta): {theta:.3f}'.format(space=space_str, i=i, J=J, theta=theta[0])))<jupyter_output><empty_output><jupyter_text>Now with Pytorch's Autograd (untill gradient computation):<jupyter_code># Set data points
x = torch.tensor([1, 2, 3, 4, 5], dtype=torch.float32)
y = torch.tensor([5, 10, 15, 20, 25], dtype=torch.float32)

# Randomly initialize parameter
theta = torch.rand(1, requires_grad=True)

# Define hyperparameters
lr = 0.01
num_iters = 100

for i in range(1, num_iters+1):
    # Forward pass
    y_hat = theta*x

    # Loss computation
    J = torch.mean((y_hat - y)**2 / 2)

    # Backward pass
    J.backward() # dJ/dtheta

    # Update parameter
    ## Remember that we don't want to track this operation in our computational graph. Given that
    ## we should wrap our update operation inside the torch.no_grad(), as explained before.
    with torch.no_grad():
        theta -= lr*theta.grad

    # Zero theta gradients
    theta.grad.zero_()

    # Print loss function and theta updating
    if i % 20 == 0:
        space_str = displayMathHelper(num_iters, i)
        display(Math(r'{space}{i}th \ iter. \ Error (J): {J:.3f} \ | \ Parameter \ (\theta): {theta:.3f}'.format(space=space_str, i=i, J=J, theta=theta[0])))<jupyter_output><empty_output><jupyter_text>---  1.5: Training Pipeline Previously, we implemented our training in two ways: (i) all from scratch; (ii) from scratch mixed with Pytorch's autograd package. Now we'll write all the training pipeline with Pytorch. First, let's recap how we trained our linear regression model mixingfrom scratch with autograd:<jupyter_code># Set data points
x = torch.tensor([1, 2, 3, 4, 5], dtype=torch.float32)
y = torch.tensor([5, 10, 15, 20, 25], dtype=torch.float32)

# Randomly initialize parameter
theta = torch.rand(1, requires_grad=True)

# Define hyperparameters
lr = 0.01
num_iters = 100

for i in range(1, num_iters+1):
    # Forward pass
    y_hat = theta*x

    # Loss computation
    J = torch.mean((y_hat - y)**2 / 2)

    # Backward pass
    J.backward() # dJ/dtheta

    # Update parameter
    ## Remember that we don't want to track this operation in our computational graph. Given that
    ## we should wrap our update operation inside the torch.no_grad(), as explained before.
    with torch.no_grad():
        theta -= lr*theta.grad

    # Zero theta gradients
    theta.grad.zero_()

    # Print loss function and theta updating
    if i % 20 == 0:
        space_str = displayMathHelper(num_iters, i)
        display(Math(r'{space}{i}th \ iter. \ Error \ (J): {J:.3f} \ | \ Parameter \ (\theta): {theta:.3f}'.format(space=space_str, i=i, J=J, theta=theta[0])))<jupyter_output><empty_output><jupyter_text>As stated above, now we'll implement a linear regression in a fully-Pytorch version. For that, we need to add some steps:1. The model needs to be fed with a matrix whose lines are related to examples and columns to features. In our 1-D example, we need to reshape our covariates and target variables to be a column vector.2. The instance of the model object (in our case, a linear one).3. In order to compute loss and gradients, we need to specify which loss object we want (for a linear regression, good old MSE) and which optimizer. For that example, we use stochastic gradient descent optimizer.Implementation is as follows:<jupyter_code># Set data points
X = torch.tensor([1, 2, 3, 4, 5], dtype=torch.float32)
y = torch.tensor([5, 10, 15, 20, 25], dtype=torch.float32)

## Reshaping as column vectors
X = X.view(X.shape[0], 1)
y = y.view(y.shape[0], 1)

# Design model
n_samples, n_features = X.shape
model = nn.Linear(in_features=n_features, out_features=n_features)

# Define hyperparameters
lr = 0.01
num_iters = 2000

# Construct Loss and Optimizer
loss = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=lr) # Obs.: model.parameters() is a generator object of torch.nn.parameter which has requires_grad set True

final_iter_magnitude = np.floor(np.log10(num_iters)).astype(np.int32)
for i in range(1, num_iters+1):
    # Forward pass
    y_hat = model(X)

    # Loss computation
    J = loss(y_hat, y)

    # Backward pass
    J.backward() # dJ/dtheta

    # Update parameter
    optimizer.step()

    # Zero theta gradients
    optimizer.zero_grad()

    # Print loss function and theta updating
    if i % 400 == 0:
        theta, b = model.parameters()

        space_str = displayMathHelper(num_iters, i)
        display(Math(r'{space}{i}th \ iter. \ Error (J): {J:.3f} \ | \ Parameter \ (\theta): {theta:.3f}'.format(space=space_str, i=i, J=J, theta=theta.item())))<jupyter_output><empty_output><jupyter_text>---  1.6: Linear Regression In this section we'll use the acquired knowledge about how to implement a linear regression fully with Pytorch to a toy dataset. Let's load the dataset:<jupyter_code># Loading data
X_numpy, y_numpy = datasets.make_regression(n_samples=100, n_features=1,
                                            noise=15, random_state=1)

## Transform Numpy vectors to Pytorch tensors
X = torch.from_numpy(X_numpy.astype(np.float32))
y = torch.from_numpy(y_numpy.astype(np.float32))

## Reshaping as column vectors
X = X.view(X.shape[0], 1)
y = y.view(y.shape[0], 1)<jupyter_output><empty_output><jupyter_text>Then, we follow the steps followed before:* Design model (a linear one for that problem)* Construct loss function and optimizer* Forward pass* Backward pass* Update parametersIn order to visualize results, it is plotted data points scatter with the line learned.<jupyter_code># Design model
n_samples, n_features = X.shape
model = nn.Linear(in_features=n_features, out_features=n_features)

# Define hyperparameters
lr = 0.01
num_iters = 1000

# Construct Loss and Optimizer
loss = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=lr)

for i in range(1, num_iters+1):
    # Forward pass
    y_hat = model(X)

    # Loss computation
    J = loss(y_hat, y)

    # Backward pass
    J.backward() # dJ/dtheta

    # Update parameter
    optimizer.step()

    # Zero theta gradients
    optimizer.zero_grad()

    # Print loss function and theta updating
    if i % 200 == 0:
        theta, b = model.parameters()

        space_str = displayMathHelper(num_iters, i)
        display(Math(r'{space}{i}th \ iter. \ Error (J): {J:.3f} \ | \ Parameter \ (\theta): {theta:.3f}'.format(space=space_str, i=i, J=J, theta=theta.item())))
predicted = model(X).detach().numpy() # We don't want that op. on computational graph.

fig, axs = plt.subplots(1, 1, figsize=(8,8))
axs.plot(X_numpy, y_numpy, 'ro', label="Original Data")
axs.plot(X_numpy, predicted, 'b', label="Line Learned")
axs.legend(loc="upper left")
plt.show()<jupyter_output><empty_output><jupyter_text>---  1.7: Logistic Regression Previously, we learned how to implement a linear model in the context of a linear regression problem fully with Pytorch. In this section we'll learn to implement the Logistic Regression classification algorithm.First of all, there isn't an implemented interface on Pytorch as in the case of our linear model. Given that, we'll need to build our own Logistic Regression class. In summary, Logistic Regression is a model used in classification tasks that has two-steps:* First, it's applied a linear transformation of the same kind it's done in linear regression (z = A*x + b).* Second, a non-linear limited function is applied in order to constraint the output value. In our model, we'll use the logistic function (also known as sigmoid function).With this in mind, let's create our Logistic Regression class:<jupyter_code>class LogRegression(nn.Module): # Inherit torch.nn.Module. This enable us to access methods, attributes and other classes that inherit this parent class.

    def __init__(self, n_features, n_classes):
        super().__init__()
        self.Linear = nn.Linear(in_features=n_features, out_features=n_classes)

    def forward(self, x):
        predicted = torch.sigmoid(self.Linear(x))
        return predicted<jupyter_output><empty_output><jupyter_text>After constructing the class, let's generate and load our toy dataset:<jupyter_code># Loading and transforming data. 

## Don't care too much about the data itself. Nevertheless, have in mind that our data is generated 
## with few constraints (e.g., it's a 100 examples size, it has 2 features and 3 classes as possible values 
## to target variable).
X_numpy, y_numpy = datasets.make_classification(n_samples=100, n_features=2, 
                                                n_redundant=0, n_classes=3,
                                                n_clusters_per_class=1, random_state=1)

X = torch.from_numpy(X_numpy.astype(np.float32))
y = torch.from_numpy(y_numpy)
y = y.type(torch.long)<jupyter_output><empty_output><jupyter_text>Training pipeline follows the same steps as before:* Design model* Construct loss function and optimizer* Forward pass* Backward pass* Update parameters<jupyter_code># Design model
n_samples, n_features = X.shape
n_classes = y.unique().shape[0]
model = LogRegression(n_features, n_classes)

# Define hyperparameters
lr=0.01
num_iters=1000

# Construct Loss and Optimizer
loss = nn.CrossEntropyLoss() # Since we're tackling a classification task, loss function needs to fit adequately.
optimizer = torch.optim.Adam(model.parameters(), lr=lr)

for i in range(1, num_iters+1):
    # Forward pass
    y_hat = model(X)

    # Loss computation
    J = loss(y_hat ,y)

    # Backward pass
    J.backward()

    # Update parameters
    optimizer.step()

    # Zero grad
    optimizer.zero_grad()

    # Print loss and weights
    if i % 200 == 0:
        theta, b = model.parameters()
        theta1 = theta[0][0].item()
        theta2 = theta[0][1].item()

        space_str = displayMathHelper(num_iters, i)
        display(
            Math(
                r'{space}{i:4}th \ iter. \ Error \ (J): {J:.3f} \ | \ Parameters: (\theta_1): {theta1:.3f}, \ (\theta_2): {theta2:.3f} '.format(
                    space=space_str, i=i, J=J,
                    theta1=theta1,
                    theta2=theta2
                )
            )
        )<jupyter_output><empty_output><jupyter_text>In order to visualize results, it is plotted model prediction and it's decision boundary (represented by it's non-linear function) beside with original data points.<jupyter_code>h = 0.02

x_min, x_max = X[:, 0].min().item() - 0.5, X[:, 0].max().item() + 0.5
y_min, y_max = X[:, 1].min().item() - 0.5, X[:, 1].max().item() + 0.5

xx, yy = np.meshgrid(np.arange(x_min, x_max, h),
                     np.arange(y_min, y_max, h))

grid_agg = np.c_[xx.ravel(), yy.ravel()]
grid_agg = torch.from_numpy(grid_agg.astype(np.float32))

Z = model(grid_agg).detach()
Z = torch.argmax(Z, 1, keepdim=True)
Z = np.reshape(Z.numpy(), xx.shape)

cm = plt.cm.Pastel1

predicted = model(X).detach()
predicted = torch.argmax(predicted, 1, keepdim=True)
predicted = predicted.view(n_samples)

corr_fst_class_idx = y_numpy==0
corr_snd_class_idx = y_numpy==1
corr_trd_class_idx = y_numpy==2

fst_class_idx = predicted==0
snd_class_idx = predicted==1
trd_class_idx = predicted==2

fig, axs = plt.subplots(1, 2, figsize=(20,8))
axs[0].plot(X_numpy[fst_class_idx, 0], X_numpy[fst_class_idx, 1], 'ro', label='Class 1')
axs[0].plot(X_numpy[snd_class_idx, 0], X_numpy[snd_class_idx, 1], 'bo', label='Class 2')
axs[0].plot(X_numpy[trd_class_idx, 0], X_numpy[trd_class_idx, 1], 'go', label='Class 3')
axs[0].contourf(xx, yy, Z, cmap=cm, alpha=0.7)
axs[0].legend(loc="upper left")
axs[0].set_title("Predicted (w/ Decision Boundary)")

axs[1].plot(X_numpy[corr_fst_class_idx, 0], X_numpy[corr_fst_class_idx, 1], 'ro', label='Class 1')
axs[1].plot(X_numpy[corr_snd_class_idx, 0], X_numpy[corr_snd_class_idx, 1], 'bo', label='Class 2')
axs[1].plot(X_numpy[corr_trd_class_idx, 0], X_numpy[corr_trd_class_idx, 1], 'go', label='Class 3')
axs[1].legend(loc="upper left")
axs[1].set_title("Original Data")

plt.show()<jupyter_output><empty_output><jupyter_text>---  1.8: Dataset and Dataloader Earlier we've managed with hand-crafted and/or toy datasets in our examples. Though, nowadays it's pretty common to deal with large datasets (in the order of millions of examples, hundreds or thousands of features and classes). In this scenario, even with state-of-the-art hardware it's recommended to save memory, paralellize computation, and make efficient use of resources. Given that, it'd be nice to come up with an implementation to ease the process of loading larger datasets. That's exactly what Pytorch's Dataset and DataLoader classes are made for.First, let's create a subclass of Dataset for the Wine Quality dataset (which is in './data/winequality-red.csv' directory, in case you want to take a look). To do so, we need three methods: (i) initialization, (ii) a getter to fetch samples by its index, and (iii) a method to return the number of samples (dataset length):<jupyter_code>class WineDataset(Dataset):

    def __init__(self, file_path):
        dataset = np.loadtxt(file_path, delimiter=",", dtype=np.float32, skiprows=1)
        self.X = torch.from_numpy(dataset[:, 1:])
        self.y = torch.from_numpy(dataset[:, 0]).view(self.X.shape[0], 1) # We want to have a label column vector
        self.n_samples = self.X.shape[0]

    def __getitem__(self, index):
        return self.X[index], self.y[index]

    def __len__(self):
        return self.n_samples
path = './data/winequality-red.csv'
wine_dataset = WineDataset(path)<jupyter_output><empty_output><jupyter_text>Afterward, we can use Pytorch's DataLoader class in order to efficiently load our dataset object aiming to optimize our model's training. In this case, we'll exploit mini-batch technique. [Quick reminder] Here is an explanation of some of the concepts used below:* **Epoch**: Number of times training on the entire dataset* **Iteration or Batch**: Number of batchs/forward and backward passes (optimization is done in each batch instead of the entire data)* **Batch Size**: Number of samples in each batchFor instace, given a dataset of 1600 samples, if the batch size is 4, then the number of batchs is 400 ($\dfrac{1600}{4}$ = 400). The image below illustrates a dataset divided in 4 batchs. On this example, each epoch we are training (forward, backward passing, and updating parameters) on each batch, running through all the dataset in this way, but using just an amount of computing power and memory, since it's only allocated a batch per step of training.<jupyter_code>Image(filename="assets/images/mini-batch.png")
# It's choosen a batch size (i.e., numbers of samples per batch) of 4, and separate this process in 2 (number of workers = 2)
wine_dataloader = DataLoader(dataset=wine_dataset, batch_size=4, shuffle=True, num_workers=2)
n_epochs = 2
n_samples = len(wine_dataset)
n_iters = len(wine_dataloader)

for epoch in range(n_epochs):
    for i, (X, y) in enumerate(wine_dataloader):
        if (i+1) % 50 == 0:
            print(
                r'Epoch {epoch}/{n_epochs} | Batch {i:3}/{n_iters:} | Batch shape {X_shape}'.format(
                    epoch=epoch+1, i=i+1,                                                                      
                    n_epochs=n_epochs, n_iters=n_iters,                                                                                
                    X_shape=X.shape)
            )<jupyter_output>Epoch 1/2 | Batch  50/400 | Batch shape torch.Size([4, 11])
Epoch 1/2 | Batch 100/400 | Batch shape torch.Size([4, 11])
Epoch 1/2 | Batch 150/400 | Batch shape torch.Size([4, 11])
Epoch 1/2 | Batch 200/400 | Batch shape torch.Size([4, 11])
Epoch 1/2 | Batch 250/400 | Batch shape torch.Size([4, 11])
Epoch 1/2 | Batch 300/400 | Batch shape torch.Size([4, 11])
Epoch 1/2 | Batch 350/400 | Batch shape torch.Size([4, 11])
Epoch 1/2 | Batch 400/400 | Batch shape torch.Size([3, 11])
Epoch 2/2 | Batch  50/400 | Batch shape torch.Size([4, 11])
Epoch 2/2 | Batch 100/400 | Batch shape torch.Size([4, 11])
Epoch 2/2 | Batch 150/400 | Batch shape torch.Size([4, 11])
Epoch 2/2 | Batch 200/400 | Batch shape torch.Size([4, 11])
Epoch 2/2 | Batch 250/400 | Batch shape torch.Size([4, 11])
Epoch 2/2 | Batch 300/400 | Batch shape torch.Size([4, 11])
Epoch 2/2 | Batch 350/400 | Batch shape torch.Size([4, 11])
Epoch 2/2 | Batch 400/400 | Batch shape torch.Size([3, 11])<jupyter_text>---  1.9: Dataset Transforms Transforms are built-in transformations applied to datasets, wheter its datatype is a tensor, a ndarray or a PIL image.A few examples of transformations are:    Images    -------    Pad, CenterCrop, GrayScale, RandomAffine, Resize, Scale.    Tensors    -------    LinearTransformation, Normalize, RandomErasing    Conversions    -----------    ToPILImage: tensor -> ndarray    ToTensor:   ndarray -> tensor    Generic    -------    Lambda function    Custom    ------    Write your own class    Compose    -------    Design a list of transformations:        composed = torchvision.transforms.Compose(            [Rescale(256),            RandomCrop(128)]        )For more details, a complete list of all built-in transform functions is available at: https://pytorch.org/vision/stable/transforms.html Let's use our own custom class dataset implemented before in order to make two transformations:1. Transform from ndarray to tensor2. Exponentiate our data<jupyter_code># Custom WineDataset class
class WineDataset(Dataset):

    def __init__(self, file_path, transform):
        dataset = np.loadtxt(file_path, delimiter=",", dtype=np.float32, skiprows=1)
        self.X = dataset[:, 1:]
        self.y = dataset[:, [0]]
        self.n_samples = self.X.shape[0]
        self.transform = transform

    def __getitem__(self, index):
        sample = self.X[index], self.y[index]

        if self.transform:
            sample = self.transform(sample)

        return sample

    def __len__(self):
        return self.n_samples
# Transform classes

class ToTensor:
    ''' 
    Transform ndarray data into tensor
    '''

    def __call__(self, sample):
        inputs, labels = sample
        sample = torch.from_numpy(inputs), torch.from_numpy(labels) 

        return sample

class PowTransform:
    ''' 
     Raise inputs to a power 'n'
    '''

    def __init__(self, n):
        self.power = n

    def __call__(self, sample):
        inputs, labels = sample
        sample = inputs.pow(self.power), labels

        return sample
path = './data/winequality-red.csv'
composed = torchvision.transforms.Compose([ToTensor(), PowTransform(2)])
# Without transformation
wine_dataset = WineDataset(file_path=path, transform=None)

first_data = wine_dataset[0]
features, labels = first_data

print(type(features), type(labels))
print(features)
# With transformation
wine_dataset = WineDataset(file_path=path, transform=composed)

first_data = wine_dataset[0]
features, labels = first_data

print(type(features), type(labels))
print(features)<jupyter_output><class 'torch.Tensor'> <class 'torch.Tensor'>
tensor([5.4760e+01, 4.9000e-01, 0.0000e+00, 3.6100e+00, 5.7760e-03, 1.2100e+02,
        1.1560e+03, 9.9560e-01, 1.2320e+01, 3.1360e-01, 8.8360e+01])<jupyter_text>---  1.10: Softmax & Cross-Entropy Softmax and Cross-Entropy are two frequently used functions in the field and, in particular, in neural network models. Let's dive into the math and intuition behind these functions:  1. Softmax Softmax formula is represented below:$$S(Y_i) = \dfrac{e^{Y_i}}{\sum_{j} e^{Y_j}}$$What this function does is normalize all $Y_i$ values to keep them between $[0,1]$. In other words, it transforms real values to probability values, making sure that it respects Kolmogorov axioms (Probability Theory axioms). In our case, we use it in order to pick up the class with the greatest probability in our model's prediction. As an example, take a look in the image below. In a 3-classes classification problem, for that particular fiction observation, our model would choose the first one (with almost $80\%$ probability that this is observation's class).<jupyter_code>Image(filename="assets/images/softmax_ex.png")<jupyter_output><empty_output><jupyter_text>As we've done before, let's implement Softmax function in two ways: (i) from scratch; (ii) with Pytorch.<jupyter_code># Softmax function from scratch
def softmax(logits):
    return np.exp(logits)/np.sum(np.exp(logits))

logits = np.array([4, 0.1, 2.6])
y_hat = softmax(logits)

print('Probabilities:', np.round(y_hat, 2), '(as expected!)')
# Softmax function with Pytorch
logits = torch.tensor([4, 0.1, 2.6])
y_hat = torch.softmax(logits, dim=0)

print('Probabilities:', np.round(y_hat.numpy(), 2), '(as expected!)')<jupyter_output>Probabilities: [0.79 0.02 0.19] (as expected!)<jupyter_text>2. Cross-Entropy As stated above, Cross-Entropy is another famous function used in the field. In this case, it's used as the loss function which we want to optimize in classification problems. Its formula is represented below:$$H(\hat{Y}, Y) = -\dfrac{1}{N}\sum_{i}^{N}Y_{i}log(\hat{Y_{i}})$$Cross-Entropy is a measure of the difference between two probability distributions of two random variables (or functions). Intuitively, if the probability distribution of our model's prediction is similar to that of the real label, them cross-entropy will have a small value. Otherwise, i.e. the two probability distributions are different, cross-entropy is high. Examples are shown below:1)${Y = [1, 0, 0] \\ \hat{Y_{1}} = [0.79, 0.02, 0.19]} H_{1}(\hat{Y_{1}}, Y) = 0.24$2)${Y = [1, 0, 0] \\ \hat{Y_{2}} = [0.19, 0.02, 0.79]} H_{2}(\hat{Y_{2}}, Y) = 1.66$**Obs. (1):** To compute cross-entropy, labels must be one-hot encoded (just like it is in the example) and predictions must be probabilities (i.e., it must be transformed by softmax).**Obs. (2):** Pytorch's Cross-Entropy interface (_torch.nn.CrossEntropyLoss()_) ease things a little bit. Our label vector doesn't need to be one-hot encoded, so label values are just the class value. Also, it internally applies softmax to the prediction vector. Thus we need to pass the logits vector. This observation is **paramount** because when we start coding our neural networks in the future we'll need to remind that Pytorch's Cross-Entropy Loss already pass results in a Softmax layer, therefore we don't need to add this layer again. Nonetheless, in the case the intended task is a binary classification, Pytorch's Binary Cross-Entropy Loss (_torch.nn.BCELoss()_) will require a Sigmoid function layer in the end of the neural network. Please keep that in mind!<jupyter_code># Cross-Entropy function from scratch
def cross_entropy(y, y_hat):
    N = 1
    return -np.sum(y*np.log(y_hat))/N

y = np.array([1, 0, 0])
y_hat_1 = np.array([0.79, 0.02, 0.19])
y_hat_2 = np.array([0.19, 0.02, 0.79])

CE_1 = cross_entropy(y, y_hat_1)
CE_2 = cross_entropy(y, y_hat_2)

display(Math(r'H_1 = {ce_1:.2f} \ | \ H_2 = {ce_2:.2f} \ (as \ expected!)'.format(ce_1=CE_1, ce_2=CE_2)))
# Cross-Entropy function with Pytorch
y = torch.tensor([0]) # Only one label in a 3-classes classification problem.
y_hat_1 = torch.tensor([[4, 0.1, 2.6]])
y_hat_2 = torch.tensor([[2.6, 0.1, 4]])

loss = nn.CrossEntropyLoss()
CE_1 = loss(y_hat_1, y)
CE_2 = loss(y_hat_2, y)

display(Math(r'H_1 = {ce_1:.2f} \ | \ H_2 = {ce_2:.2f} \ (as \ expected!)'.format(ce_1=CE_1, ce_2=CE_2)))<jupyter_output><empty_output><jupyter_text>---  1.11: Activation Functions Activation functions are important features of neural networks because they allow them to **model non-linear relations** present in the data. An analogy often used is that of the all-or-none principle that control our nerve cells' response given a stimulus and the intuition behind activation functions on artificial neural networks (yet, this comparison is quite questionable).Next, we're going to see some of the activation functions implemented in Pytorch and their idiosyncrasies:  1. Step Function This is the most simple activation function. It's described as follows:$$f(x) = \left\{\begin{array}{ll}    1,  & x \geq a \\    0,  & x < a\\\end{array}\right.$$That means whenever the neuron value is greater than a value ($a$, in this case), then it activates.**Obs.:** This function is hardly ever used.Below, it's a graphic representation of that function:<jupyter_code>X = np.linspace(-2, 4, 1000, dtype=np.float64)
X = torch.from_numpy(X)
inputs = X
values = torch.tensor([1], dtype=torch.float64)

a = 1
step = torch.heaviside(inputs-a, values) # Pytorch's step function
step = step.numpy()

fig, axs = plt.subplots(1, 1, figsize=(8,8))
axs.plot(inputs, step, 'b-', label="Step")
axs.legend(loc="upper left")
axs.set_ylim([-2, 3])
plt.grid()
plt.show()<jupyter_output><empty_output><jupyter_text>2. Logistic Logistic function, also kwon as sigmoid, is an activation function that outputs a probability value and is usually used in the last layer of a binary classification problem. It's formula is represented below:$$f(x) = \dfrac{1}{1+e^{-x}}$$Graphically, logistic function is as follows:<jupyter_code>X = np.linspace(-15, 15, 1000, dtype=np.float64)
X = torch.from_numpy(X)

logistic = nn.Sigmoid() # Pytorch's logistic function (also known as just sigmoid)
logits = logistic(X)
logits = logits.numpy()

fig, axs = plt.subplots(1, 1, figsize=(8,8))
axs.plot(X, logits, 'b-', label="Logistic")
axs.legend(loc="upper left")
axs.set_xlim([-16, 16])
plt.grid()
plt.show()<jupyter_output><empty_output><jupyter_text>3. Tanh Tanh is another function from the sigmoid family. Nonetheless, instead of having its output value between $[0,1]$, it's scaled and shifted in order to be between $[-1,1]$. The formula and plot is represented below:$$f(x) = \dfrac{2}{1 + e^{-2x}} - 1$$<jupyter_code>X = np.linspace(-15, 15, 1000, dtype=np.float64)
X = torch.from_numpy(X)

sigmoid_tanh = nn.Tanh() # Pytorch's tanh function
tanh = sigmoid_tanh(X)
tanh = tanh.numpy()

fig, axs = plt.subplots(1, 1, figsize=(8,8))
axs.plot(X, tanh, 'b-', label="Tanh")
axs.legend(loc="upper left")
axs.set_xlim([-16, 16])
plt.grid()
plt.show()<jupyter_output><empty_output><jupyter_text>4. ReLU Probably the most popular activation function (and the best candidate if you're in doubt which one to use), ReLU outputs $0$ in the case input is **negative**, and is the idendity ($f(x) = x$) otherwise. ReLU formula is below:$$f(x) = max(0, x)$$Next, it's presented ReLU graphically:<jupyter_code>X = np.linspace(-15, 15, 1000, dtype=np.float64)
X = torch.from_numpy(X)

relu_func = nn.ReLU() # Pytorch's ReLU function
relu = relu_func(X)
relu = relu.numpy()

fig, axs = plt.subplots(1, 1, figsize=(8,8))
axs.plot(X, relu, 'b-', label="ReLU")
axs.legend(loc="upper left")
axs.set_xlim([-16, 16])
plt.grid()
plt.show()<jupyter_output><empty_output><jupyter_text>5. Leaky ReLU Leaky ReLU is a variation of ReLU function, but it's not $0$ for negative input values. What this function seeks to improve is what is called vanishing gradient problem, which is the result of a null gradient value, due to null activation. To addres this problem, it's used a small term ($\alpha$) (in the order of $0.01$) for negative input values, and so the gradient does not vanish. Leaky ReLU formula and its plot is as follows:$$f(x) = \left\{\begin{array}{ll}    x,  & x \geq 0 \\    {\alpha}x,  & x < 0\\\end{array}\right.$$<jupyter_code>X = np.linspace(-15, 15, 1000, dtype=np.float64)
X = torch.from_numpy(X)

leaky_relu_func = nn.LeakyReLU(negative_slope=0.02) # Pytorch's LeakyReLU function
leaky_relu = leaky_relu_func(X)
leaky_relu = leaky_relu.numpy()

fig, axs = plt.subplots(1, 1, figsize=(8,8))
axs.plot(X, leaky_relu, 'b-', label="Leaky ReLU")
axs.legend(loc="upper left")
axs.set_xlim([-16, 16])
plt.grid()
plt.show()<jupyter_output><empty_output><jupyter_text>6. Softmax Softmax formula is represented below:$$S(Y_i) = \dfrac{e^{Y_i}}{\sum_{j} e^{Y_j}}$$What this function does is normalize all $Y_i$ values to keep them between $[0,1]$. In other words, it transforms real values to probability values, making sure that it respects Kolmogorov axioms (Probability Theory axioms). In our case, we use it in order to pick up the class with the greatest probability in our model's prediction. As an example, take a look in the image below. In a 3-classes classification problem, for that particular fiction observation, our model would choose the first one (with almost $80\%$ probability that this is observation's class).<jupyter_code>Image(filename="assets/images/softmax_ex.png")
logits = torch.tensor([4, 0.1, 2.6])
y_hat = torch.softmax(logits, dim=0) # Pytorch's softmax function

print('Probabilities:', np.round(y_hat.numpy(), 2), '(as expected!)')<jupyter_output>Probabilities: [0.79 0.02 0.19] (as expected!)<jupyter_text>---  1.12: Feedforward Neural Networks We've seen, so far, how to construct and train our models with Pytorch, what are activation and loss functions (and some supported), how to load and transform datasets.In this sense, let's build a Multilayer Perceptron (MLP), and train it on MNIST dataset*.A MLP is a Feedforward Neural Network which has basically three entities:1. Input layer2. Hidden layer3. Output layerFrom one layer to another, two operations are done: (i) a linear combination between activations from previous layer and currently layer's parameters ($z = X*W + b$); (ii) neurons activation, via an activation function ($\sigma(z) = f(z)$, for f as an activation function). In our example, we'll create a MLP in which it'll be possible to choose the deepth and size of layers.*MNIST is a dataset of handwritten digits (0-9) images. Pytorch already has an API in order to download MNIST dataset, which we'll see later. For more details, please check out the dataset website: http://yann.lecun.com/exdb/mnist/.<jupyter_code>class NeuralNet(nn.Module):

    def __init__(self, in_units, hid_units, out_units):
        super().__init__()
        self.num_in_units = in_units

        self.h1 = nn.Linear(in_features=in_units, out_features=hid_units)
        self.h2 = nn.Linear(in_features=hid_units, out_features=hid_units)
        self.output_layer = nn.Linear(in_features=hid_units, out_features=out_units)

    def forward(self, x):
        x = x.view(-1, self.num_in_units)
        out = nn.ReLU()(self.h1(x))
        out = nn.ReLU()(self.h2(out))
        prediction = self.output_layer(out)

        return prediction
# Define hyperparameters
lr=0.001
num_epochs=5
batch_size = 32<jupyter_output><empty_output><jupyter_text>MNIST dataset contains $60.000$ training examples, and $10.000$ test examples. Here, we're loading it with $torchvision.datasets$ Pytorch's API, and passing it through a transformation from PIL images (original data format) to Pytorch Tensors.<jupyter_code># Loading and transforming data
composed = torchvision.transforms.Compose([torchvision.transforms.ToTensor()])
mnist_train_dataset = torchvision.datasets.MNIST(root='./data', train=True, download=True, transform=composed)
train_dataloader = DataLoader(dataset=mnist_train_dataset, batch_size=batch_size, shuffle=True)

mnist_test_dataset = torchvision.datasets.MNIST(root='./data', train=False, download=True, transform=composed)
test_dataloader = DataLoader(dataset=mnist_test_dataset, batch_size=batch_size, shuffle=True)<jupyter_output><empty_output><jupyter_text>A few examples are shown below in order to understand the data a little bit:<jupyter_code>examples = enumerate(train_dataloader)
batch_idx, (example_features, example_targets) = next(examples)

fig = plt.figure()

for i in range(6):
    plt.subplot(2, 3, i+1)
    plt.tight_layout()
    plt.imshow(example_features[i][0], cmap='gray')
    plt.title(r'Ground Truth: {label}'.format(label=example_targets[i]))
    plt.xticks([])
    plt.yticks([])
fig.show()<jupyter_output><empty_output><jupyter_text>Let's train our model and check train and test accuracy, in order to see our classifier performance:<jupyter_code># Setting device
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

# Design model
n_samples = len(mnist_train_dataset)
n_test_samples = len(mnist_test_dataset)
n_features = (
    mnist_train_dataset.train_data.shape[1]*mnist_train_dataset.train_data.shape[2]
)
n_classes = mnist_train_dataset.targets.unique().shape[0]
model = NeuralNet(n_features, 512, n_classes).to(device)

# Construct Loss and Optimizer
loss = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=lr)

train_losses = []
# test_losses = []

for epoch in range(num_epochs):
    train_acc = 0
    test_acc = 0
    for batch_idx, (X_train, y_train) in enumerate(train_dataloader):
        X_train = X_train.to(device)
        y_train = y_train.to(device)

        # Forward pass
        y_hat = model(X_train)

        pred = torch.max(y_hat.data, 1)[1]
        batch_corr = (pred==y_train).sum()
        train_acc += batch_corr

        # Loss computation
        J = loss(y_hat, y_train)

        # Backward pass
        J.backward()

        # Update parameters
        optimizer.step()

        # Zero grad
        optimizer.zero_grad()

        # Print train loss
        if batch_idx % 20 == 0:
            print('Train Epoch {}/{} [{:>5}/{} ({:>2.0f}%)] | Loss: {}'.format(
                epoch+1, num_epochs, batch_idx * batch_size, n_samples, 
                100*batch_idx / len(train_dataloader), J))
            train_losses.append(J)

    with torch.no_grad():
        for batch_idx, (X_test, y_test) in enumerate(test_dataloader):
            X_test = X_test.to(device)
            y_test = y_test.to(device)

            # Forward pass
            y_val = model(X_test)

            pred = torch.max(y_val.data, 1)[1]
            batch_corr = (pred==y_test).sum()
            test_acc += batch_corr

    print('\nTrain Acc.: [{:>5}/{} ({:>2.0f}%)]'.format(train_acc, n_samples, 100 * train_acc / n_samples))
    print('Test Acc.:  [{:>5}/{} ({:>2.0f}%)]\n'.format(test_acc, n_test_samples, 100 * test_acc / n_test_samples))<jupyter_output>/home/cdeberaldini/miniconda3/envs/torch_env/lib/python3.8/site-packages/torchvision/datasets/mnist.py:62: UserWarning: train_data has been renamed data
  warnings.warn("train_data has been renamed data")<jupyter_text>That's great! Our model had an accuracy greater than $95\%$ on test set, and we were capable of implementing a flexible MLP and training it in an elegant and easy way. ---  1.13: Convolutional Neural Networks (CNNs) Convolutional Neural Networks (shortened as CNNs) are a special family of neural networks. Its singularity comes from the namesake operation applied in it - convolution. In short, a CNN is a neural network composed of some specific building blocks represented in layers. Those layers are as described:**1. Convolution Layers:** This is the basic feature of CNNs. In this context, convolution is an operation of linearly combining the inputs from each of these layers with their respectives filters/kernels. Generally, conv. layers are followed by activations ones, and these two layers combined are often called as Convolution Layer. In this tutorial, we're using the latter definition for conv. layers (Convolution + Activation). Convolution operation is represented on [1] gif.**2. Pooling Layers:** Mainly, conv. layers are also followed by Pooling layers. Pooling is used to reduce dimensionality and keep the most important informations in a feature map. A Max Pooling operation is represented on [2] gif.**3. Fully-Connected Layer:** After stacking a large (and deep) number of conv. and pooling layers, we stack a few fully-connected layers on top of this stack. Given the features learned from conv. and pooling layers, the fully-connected layer will classify our input object in our task's classes.Figure [3] shows a complete Conv. Network architecture with those foundational blocks.With this brief review, let's build a Conv. Network with two conv. layers, two fully-connected layers, and with arbitrary convolutional hyperparameters. Also, it'll be trained on CIFAR-10*.*CIFAR-10 is a well-known dataset with 60.000 images of 10 (balanced) classes. Dataset is splitted in 50.000 training images and 10.000 test images. As with MNIST, Pytorch already has an API in order to download CIFAR-10 dataset, which we'll see later. For more details, please check out the dataset website: https://www.cs.toronto.edu/~kriz/cifar.html**OBS.:** This tutorial's intention isn't to go further into details about CNNs, but to be able to construct a CNN model with Pytorch. Please, check out this link for additional informations: https://cezannec.github.io/Convolutional_Neural_Networks/<jupyter_code>Image(filename="assets/images/no_padding_no_strides.gif")<jupyter_output><empty_output><jupyter_text>[1]: Convolution operation. In this example, a $4x4$ input matrix (lighter blue) is convoluted by a $3x3$ filter/kernel (darker blue, passing through the input) resulting in a $2x2$ convolution output.Source: <NAME>, <NAME>. "A guide to convolution arithmetic for deep learning". https://github.com/vdumoulin/conv_arithmetic. The convolution layer output's size is computed using the following equation:$$n_{OUT}^{H} = \dfrac{n_{IN}^{H} + 2p - f_{H}}{s} + 1$$$$n_{OUT}^{W} = \dfrac{n_{IN}^{W} + 2p - f_{W}}{s} + 1$$Where $n_{OUT}^{*}$ is the corresponding output size, $p$ and $s$ stands, respectively, for the padding and stride, and $f_{*}$ is the filter size w.r.t. dimension ($* \in \{H, W\}$, in 2D conv.). In the above example, we have $n_{IN}^{H} = n_{IN}^{W} = 4, p = 0, s = 1$, and $f_{H} = f_{W} = 3$ (i.e., $3x3$ kernel). Using the presented equations we have $n_{OUT}^{H} = n_{OUT}^{W} = 2$.<jupyter_code>Image(filename="assets/images/max_pool.gif")<jupyter_output><empty_output><jupyter_text>[2]: Max Pooling operation. In this example, a $4x4$ input matrix is pooled by a $2x2$ filter/kernel (red, passing through the input) resulting in a $2x2$ pooling output, due to a stride of 2 ($s = 2$).Source: _Quora_. "What is pooling in a deep architecture?". Accessed 15 November 2021, https://qr.ae/pGlY4X.<jupyter_code>Image(filename="assets/images/conv_architecture.png", height=750, width=705)<jupyter_output><empty_output><jupyter_text>[3]: CNN general architecture. Source: _MathWorks_. "What is a Convolutional Neural Network? 3 things you need to know". Accessed 14 November 2021, https://www.mathworks.com/discovery/convolutional-neural-network-matlab.html.<jupyter_code>class ConvNet(nn.Module):

    def __init__(self, hparams):
        super().__init__()

        n_C_prev, n_H_prev, n_W_prev = 3, 32, 32
        n_C, f, stride, pad = [hparams[key] for key in list(hparams.keys())[:4]]
        n_H, n_W = int((n_H_prev + 2*pad - f)/stride + 1), int((n_W_prev + 2*pad - f)/stride + 1)

        pool_mode, f_pool, stride_pool = [hparams[key] for key in list(hparams.keys())[4:7]]
        n_H, n_W = int((n_H - f_pool)/stride_pool + 1), int((n_W - f_pool)/stride_pool + 1)

        self.conv1 = nn.Conv2d(in_channels=n_C_prev, out_channels=n_C, kernel_size=(f, f), stride=stride, padding=pad)
        if pool_mode == 'mean':
            self.pool1 = nn.MaxPool2d(kernel_size=f_pool, stride=stride_pool)
        elif pool_mode == 'avg':
            self.pool1 = nn.AvgPool2d(kernel_size=f_pool, stride=stride_pool)

        n_C_prev, n_H_prev, n_W_prev = n_C, n_H, n_W
        n_C, f, stride, pad = [hparams[key] for key in list(hparams.keys())[7:11]]
        n_H, n_W = int((n_H_prev + 2*pad - f)/stride + 1), int((n_W_prev + 2*pad - f)/stride + 1)

        pool_mode, f_pool, stride_pool = [hparams[key] for key in list(hparams.keys())[11:]]
        n_H, n_W = int((n_H - f_pool)/stride_pool + 1), int((n_W - f_pool)/stride_pool + 1)

        self.conv2 = nn.Conv2d(in_channels=n_C_prev, out_channels=n_C, kernel_size=(f, f), stride=stride, padding=pad)
        if pool_mode == 'mean':
            self.pool2 = nn.MaxPool2d(kernel_size=f_pool, stride=stride_pool)
        elif pool_mode == 'avg':
            self.pool2 = nn.AvgPool2d(kernel_size=f_pool, stride=stride_pool)

        self.fc1 = nn.Linear(in_features=n_C*n_H*n_W, out_features=256)
        self.fc2 = nn.Linear(in_features=256, out_features=128)
        self.output = nn.Linear(in_features=128, out_features=10)

    def forward(self, X):
        out = nn.ReLU()(self.conv1(X))
        out = self.pool1(out)

        out = nn.ReLU()(self.conv2(out))
        out = self.pool2(out)

        out = nn.Flatten()(out)
        out = nn.ReLU()(self.fc1(out))
        out = nn.ReLU()(self.fc2(out))
        output = self.output(out)

        return output
# Define hyperparameters
lr=0.001
num_epochs= 10
batch_size = 32
composed = torchvision.transforms.Compose([torchvision.transforms.ToTensor(), torchvision.transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])
train_dataset = torchvision.datasets.CIFAR10(root='./data', train=True, transform=composed, download=True)
train_dataloader = DataLoader(dataset=train_dataset, batch_size=batch_size, shuffle=True)

test_dataset = torchvision.datasets.CIFAR10(root='./data', train=False, transform=composed, download=True)
test_dataloader = DataLoader(dataset=test_dataset, batch_size=batch_size, shuffle=False)
examples = enumerate(train_dataloader)
batch_idx, (example_features, example_targets) = next(examples)
classes_dict = {
    0: 'Airplane',
    1: 'Automobile',
    2: 'Bird',
    3: 'Cat',
    4: 'Deer',
    5: 'Dog',
    6: 'Frog',
    7: 'Horse',
    8: 'Ship',
    9: 'Truck'
}

fig = plt.figure()

for i in range(4):
    plt.subplot(2, 2, i+1)
    plt.tight_layout()
    plt.imshow(example_features[i].permute((1, 2, 0))*0.5 + 0.5)
    plt.title(r'{label}'.format(label=classes_dict[example_targets[i].item()]))
    plt.xticks([])
    plt.yticks([])
fig.show()
# CNN hyperparameters
hparams = {
    'num_f_1': 32,
    'f_1': 3,
    'stride_1': 1,
    'pad_1': 0,
    'pool_1': 'mean',
    'f_pool_1': 2,
    'stride_pool_1': 1,
    'num_f_2': 32,
    'f_2': 3,
    'stride_2': 1,
    'pad_2': 0,
    'pool_2': 'mean',
    'f_pool_2': 2,
    'stride_pool_2': 1
}
# Setting device
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

# Design model
n_samples = len(train_dataset)
n_test_samples = len(test_dataset)
model = ConvNet(hparams).to(device)

# Construct Loss and Optimizer
loss = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=lr)

train_losses = []
# test_losses = []

for epoch in range(num_epochs):
    train_acc = 0
    test_acc = 0
    for batch_idx, (X_train, y_train) in enumerate(train_dataloader):
        X_train = X_train.to(device)
        y_train = y_train.to(device)

        # Forward pass
        y_hat = model(X_train)

        pred = torch.max(y_hat.data, 1)[1]
        batch_corr = (pred==y_train).sum()
        train_acc += batch_corr

        # Loss computation
        J = loss(y_hat, y_train)
        train_losses.append(J)

        # Backward pass
        J.backward()

        # Update parameters
        optimizer.step()

        # Zero grad
        optimizer.zero_grad()

        # Print train loss
        if batch_idx % int(n_samples/(100*batch_size)) == 0:
            print('Train Epoch {}/{} [{:>5}/{} ({:>2.0f}%)] | Loss: {}'.format(
                epoch+1, num_epochs, batch_idx * batch_size, n_samples, 
                100*batch_idx / len(train_dataloader), J))
            # train_losses.append(J)

    with torch.no_grad():
        for batch_idx, (X_test, y_test) in enumerate(test_dataloader):
            X_test = X_test.to(device)
            y_test = y_test.to(device)

            # Forward pass
            y_val = model(X_test)

            pred = torch.max(y_val.data, 1)[1]
            batch_corr = (pred==y_test).sum()
            test_acc += batch_corr

    print('\nTrain Acc.: [{:>5}/{} ({:>2.0f}%)]'.format(train_acc, n_samples, 100 * train_acc / n_samples))
    print('Test Acc.:  [{:>5}/{} ({:>2.0f}%)]\n'.format(test_acc, n_test_samples, 100 * test_acc / n_test_samples))<jupyter_output>Train Epoch 1/10 [    0/50000 ( 0%)] | Loss: 2.3300554752349854
Train Epoch 1/10 [  480/50000 ( 1%)] | Loss: 2.0199551582336426
Train Epoch 1/10 [  960/50000 ( 2%)] | Loss: 2.0171780586242676
Train Epoch 1/10 [ 1440/50000 ( 3%)] | Loss: 1.910845398902893
Train Epoch 1/10 [ 1920/50000 ( 4%)] | Loss: 1.9986704587936401
Train Epoch 1/10 [ 2400/50000 ( 5%)] | Loss: 1.9018077850341797
Train Epoch 1/10 [ 2880/50000 ( 6%)] | Loss: 1.8397245407104492
Train Epoch 1/10 [ 3360/50000 ( 7%)] | Loss: 1.7842048406600952
Train Epoch 1/10 [ 3840/50000 ( 8%)] | Loss: 1.9856284856796265
Train Epoch 1/10 [ 4320/50000 ( 9%)] | Loss: 1.653296709060669
Train Epoch 1/10 [ 4800/50000 (10%)] | Loss: 1.586301326751709
Train Epoch 1/10 [ 5280/50000 (11%)] | Loss: 1.3201878070831299
Train Epoch 1/10 [ 5760/50000 (12%)] | Loss: 1.6494166851043701
Train Epoch 1/10 [ 6240/50000 (12%)] | Loss: 1.6111509799957275
Train Epoch 1/10 [ 6720/50000 (13%)] | Loss: 1.285415530204773
Train Epoch 1/10 [ 7200/50000 (14%)] | Loss:[...]<jupyter_text>---  1.14: Transfer Learning Transfer Learning is one of the greatest problem in ML field. Briefly, what Transfer Learning seeks to do is reuse the knowledge acquired by a model trained in a specific task (say, image classification) and apply it in a different scenario (e.g., a different task or same task but different domains). Since many DL models are just too big, it's not efficient to train them all over again. Instead, given that they've learned common concepts, such as edge detection in different angles, objects' characteristics, and so on (giving an example in the context of computer vision), let's reuse this knowledge. To do so, we need to load a choosen model and its parameters trained in the first scenario, and replace its final layers. Training this new top layers for our specific task will then give us remarkable results, due to the pre-trained model, and will save us time. This short description is better illustrated in [1].<jupyter_code>Image(filename="assets/images/transfer_learning.jpg", height=750, width=705)<jupyter_output><empty_output><jupyter_text>[1]: Transfer Learning framework. One deep neural network architecture trained to one task is reused to another. Only a few layers (final ones) are retrained (fine-tuned, in the field jargon).Source: Do, Synho et al. “Basics of Deep Learning: A Radiologist's Guide to Understanding Published Radiology Articles on Deep Learning.” Korean journal of radiology vol. 21,1 (2020): 33-41. doi:10.3348/kjr.2019.0312 Pytorch's _torchvision_ module has some important pre-trained models (e.g., VGG family, ResNet, MobileNet, Inception). To use it, it's just required to instantiate a model object and load its parameters. For this tutorial, we're gonna use ResNet to train a chess' pieces classifier. We'll use a Kaggle dataset called "Chessman image dataset", from <NAME> - for more informations, please check Kaggle's dataset page https://www.kaggle.com/niteshfre/chessman-image-dataset.<jupyter_code># Loading pre-trained model. In this case, ResNet50.
tf_model = torchvision.models.resnet50(pretrained=True)
class ChessDataset(Dataset):
    def __init__(self, transforms=None):
        labels = ['Bishop', 'King', 'Knight', 'Pawn', 'Queen', 'Rook']
        file_paths = ['./data/chess/' + label + '/' for label in labels]

        X, y = np.empty((1, 400, 400, 3), dtype=np.uint8), np.array([], dtype=np.int32)
        for i, file_path in enumerate(file_paths):
            imgs = np.array(os.listdir(file_path), dtype=np.str_)
            for img in imgs:
                if not (bool(re.match(r'.*\.ipynb_checkpoints', img)) or bool(re.match(r'.*\.gif', img))):
                    img_path = file_paths[i] + img
                    np_img = np.array(PIL.Image.open(img_path).resize((400, 400)))
                    np_img = np.reshape(
                        np_img,  ((1,) + np_img.shape))
                    X = np.append(X, np_img, axis=0)
                    y = np.append(y, i)
        X = np.delete(X, 0, 0)

        self.X = X
        self.y = y
        self.n_samples = X.shape[0]
        self.transform = composed

    def __getitem__(self, index):
        sample = self.X[index]
        
        if self.transform:
            sample = self.transform(sample)

        return sample, self.y[index]

    def __len__(self):
        return self.n_samples
# Define hyperparameters
lr = 1e-3
batch_size = 8
num_epochs = 10
composed = torchvision.transforms.Compose([torchvision.transforms.ToTensor(),
                                           torchvision.transforms.Normalize(
                                               mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
                                          ])

chess_dataset = ChessDataset(transforms=composed)
chess_dataloader = DataLoader(chess_dataset, batch_size=batch_size, shuffle=True)
examples = enumerate(chess_dataloader)
batch_idx, (example_features, example_targets) = next(examples)
classes_dict = {
    0: 'Bishop',
    1: 'King',
    2: 'Knight',
    3: 'Pawn',
    4: 'Queen',
    5: 'Rook'
}

mean_tensor = torch.tensor([[[0.229]], [[0.224]], [[0.225]]])
std_tensor = torch.tensor([[[0.485]], [[0.456]], [[0.406]]])

fig = plt.figure()

for i in range(4):
    plt.subplot(2, 2, i+1)
    plt.tight_layout()
    plt.imshow(
        (example_features[i]*mean_tensor + std_tensor).permute((1, 2, 0))
    )
    plt.title(r'{label}'.format(label=classes_dict[example_targets[i].item()]))
    plt.xticks([])
    plt.yticks([])
fig.show()
# Setting device
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

# Design model
n_samples = len(chess_dataset)
n_classes = len(classes_dict)

num_fc_feat = tf_model.fc.in_features
## To train just the last layers of the pre-trained model
## we'll freeze (do not train) the first layers, and replace
## the last ones accordingly to our task. In this case,
## a FC layer with #output_neurons = #classes of our problem.
for param in tf_model.parameters():
    param.requires_grad = False # Freezing layers
tf_model.fc = nn.Linear(num_fc_feat, n_classes) # Replacing original FC to ours
tf_model = tf_model.to(device)

# Construct Loss and Optimizer
loss = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(tf_model.parameters(), lr=lr)

# train_losses = []

for epoch in range(num_epochs):
    train_acc = 0
    for batch_idx, (X_train, y_train) in enumerate(chess_dataloader):
        X_train = X_train.to(device)
        y_train = y_train.to(device)

        # Forward pass
        y_hat = tf_model(X_train)

        pred = torch.max(y_hat.data, 1)[1]
        batch_corr = (pred==y_train).sum()
        train_acc += batch_corr

        # Loss computation
        J = loss(y_hat, y_train)

        # Backward pass
        J.backward()

        # Update parameters
        optimizer.step()

        # Zero grad
        optimizer.zero_grad()

        # Print train loss
        if batch_idx % int(n_samples/(65*batch_size)) == 0:
            print('Train Epoch {}/{} [{:>5}/{} ({:>2.0f}%)] | Loss: {}'.format(
                epoch+1, num_epochs, batch_idx * batch_size, n_samples, 
                100*batch_idx / len(chess_dataloader), J))
            # train_losses.append(J)

    print('\nTrain Acc.: [{:>5}/{} ({:>2.0f}%)]\n'.format(train_acc, n_samples, 100 * train_acc / n_samples))<jupyter_output>Train Epoch 1/10 [    0/555 ( 0%)] | Loss: 1.8011231422424316
Train Epoch 1/10 [    8/555 ( 1%)] | Loss: 2.214264154434204
Train Epoch 1/10 [   16/555 ( 3%)] | Loss: 2.233550786972046
Train Epoch 1/10 [   24/555 ( 4%)] | Loss: 2.045016050338745
Train Epoch 1/10 [   32/555 ( 6%)] | Loss: 1.9106197357177734
Train Epoch 1/10 [   40/555 ( 7%)] | Loss: 1.8819496631622314
Train Epoch 1/10 [   48/555 ( 9%)] | Loss: 2.0445985794067383
Train Epoch 1/10 [   56/555 (10%)] | Loss: 1.9184272289276123
Train Epoch 1/10 [   64/555 (11%)] | Loss: 1.907508134841919
Train Epoch 1/10 [   72/555 (13%)] | Loss: 2.1166234016418457
Train Epoch 1/10 [   80/555 (14%)] | Loss: 1.9764569997787476
Train Epoch 1/10 [   88/555 (16%)] | Loss: 2.3132290840148926
Train Epoch 1/10 [   96/555 (17%)] | Loss: 1.4106663465499878
Train Epoch 1/10 [  104/555 (19%)] | Loss: 2.2681310176849365
Train Epoch 1/10 [  112/555 (20%)] | Loss: 1.8622990846633911
Train Epoch 1/10 [  120/555 (21%)] | Loss: 1.803499698638916
Train Epoch 1[...]<jupyter_text>Even though our dataset is small, it's still interesting that fine-tuning a big model trained on a different dataset and task could lead us to a good result in the training data! ---  1.15: TensorBoard TensorBoard [1] is a visualization toolkit implemented by TensorFlow's team. Although it's tool from a differente framework, it's possible to use it with Pytorch. Some of the TensorBoard's main features are:- Tracking and visualizing metrics such as loss and accuracy;- Visualizing the model graph (ops and layers);- Viewing histograms of weights, biases, or other tensors as they change over time;- Projecting embeddings to a lower dimensional space;- Displaying images, text, and audio data;- Profiling TensorFlow programs;So, let's install tensorboard in our env:```(venv)$ conda install -c conda-forge tensorboard```It's possible to run TensorBoard in two ways: (i) starting TensorBoard server to run in localhost; (ii) use TensorBoard Notebook extension. To run TensorBoard in localhost, write in command line (specifying a logdir):```(venv)$ tensorboard --logdir=dir_name```To run it directly in Notebook, load the extension and run TensorBoard in a code cell as follows:```%load_ext tensorboard``````%tensorboard --logdir dir_name```We're gonna use it in our Feedforward NNs tutorial (section 1.12) for three visualizations:1. Display MNIST dataset;2. Visualization of model graph (computational graph);3. Tracking training loss and accuracy;[1]: TensorBoard Web Page: https://www.tensorflow.org/tensorboard<jupyter_code>%load_ext tensorboard
%tensorboard --logdir runs
tb_writer = SummaryWriter('./runs')
class NeuralNet(nn.Module):

    def __init__(self, in_units, hid_units, out_units):
        super().__init__()
        self.num_in_units = in_units

        self.h1 = nn.Linear(in_features=in_units, out_features=hid_units)
        self.relu = nn.ReLU()
        self.h2 = nn.Linear(in_features=hid_units, out_features=hid_units)
        self.output_layer = nn.Linear(in_features=hid_units, out_features=out_units)

    def forward(self, x):
        x = x.view(-1, self.num_in_units)

        out = self.relu(self.h1(x))
        out = self.relu(self.h2(out))
        prediction = self.output_layer(out)

        return prediction
# Define hyperparameters
lr=0.001
num_epochs=1
batch_size = 32
# Loading and transforming data
composed = torchvision.transforms.Compose([torchvision.transforms.ToTensor()])
mnist_train_dataset = torchvision.datasets.MNIST(root='./data', train=True, download=True, transform=composed)
train_dataloader = DataLoader(dataset=mnist_train_dataset, batch_size=batch_size, shuffle=True)

mnist_test_dataset = torchvision.datasets.MNIST(root='./data', train=False, download=True, transform=composed)
test_dataloader = DataLoader(dataset=mnist_test_dataset, batch_size=batch_size, shuffle=True)
examples = enumerate(train_dataloader)
batch_idx, (example_features, example_targets) = next(examples)
img_grid = torchvision.utils.make_grid(example_features)
# Display MNIST images. After running this cell
# rerun %tensorboard --logdir runs cell above.
tb_writer.add_image('MNIST_images', img_grid)
# Setting device
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

# Design model
n_samples = len(mnist_train_dataset)
n_test_samples = len(mnist_test_dataset)
n_features = (
    mnist_train_dataset.train_data.shape[1]*mnist_train_dataset.train_data.shape[2]
)
n_classes = mnist_train_dataset.targets.unique().shape[0]
model = NeuralNet(n_features, 512, n_classes).to(device)

# Construct Loss and Optimizer
loss = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=lr)
# Construct a visualization of the computational graph
# (i.e., our model). Again, rerun %tensorboard --logidr runs cell 
# above.
tb_writer.add_graph(model, example_features.to(device))
n_batchs = len(train_dataloader)
train_loss = 0.0
train_corr = 0

for epoch in range(num_epochs):
    train_acc = 0
    test_acc = 0
    for batch_idx, (X_train, y_train) in enumerate(train_dataloader):
        X_train = X_train.to(device)
        y_train = y_train.to(device)

        # Forward pass
        y_hat = model(X_train)

        pred = torch.max(y_hat.data, 1)[1]
        batch_corr = (pred==y_train).sum()
        train_acc += batch_corr

        # Loss computation
        J = loss(y_hat, y_train)

        train_corr += batch_corr.item()
        train_loss += J.item()

        # Backward pass
        J.backward()

        # Update parameters
        optimizer.step()

        # Zero grad
        optimizer.zero_grad()

        # Print train loss
        if (batch_idx + 1) % 100 == 0:
            print('Train Epoch {}/{} [{:>5}/{} ({:>2.0f}%)] | Loss: {}'.format(
                epoch+1, num_epochs, batch_idx * batch_size, n_samples, 
                100*batch_idx / len(train_dataloader), J))

            # Adding train loss and accuracy to track them. Likewise,
            # rerun %tensorboard --logidr runs cell above to see the result.
            tb_writer.add_scalar('Training Loss', train_loss / (100*batch_size), epoch*n_batchs + batch_idx)
            tb_writer.add_scalar('Accuracy', train_corr / (100*batch_size), epoch*n_batchs + batch_idx)

            train_loss = 0.0
            train_corr = 0

    with torch.no_grad():
        for batch_idx, (X_test, y_test) in enumerate(test_dataloader):
            X_test = X_test.to(device)
            y_test = y_test.to(device)

            # Forward pass
            y_val = model(X_test)

            pred = torch.max(y_val.data, 1)[1]
            batch_corr = (pred==y_test).sum()
            test_acc += batch_corr

    print('\nTrain Acc.: [{:>5}/{} ({:>2.0f}%)]'.format(train_acc, n_samples, 100 * train_acc / n_samples))
    print('Test Acc.:  [{:>5}/{} ({:>2.0f}%)]\n'.format(test_acc, n_test_samples, 100 * test_acc / n_test_samples))<jupyter_output>Train Epoch 1/1 [ 3168/60000 ( 5%)] | Loss: 0.477774441242218
Train Epoch 1/1 [ 6368/60000 (11%)] | Loss: 0.13809886574745178
Train Epoch 1/1 [ 9568/60000 (16%)] | Loss: 0.2757168412208557
Train Epoch 1/1 [12768/60000 (21%)] | Loss: 0.12989328801631927
Train Epoch 1/1 [15968/60000 (27%)] | Loss: 0.3065938353538513
Train Epoch 1/1 [19168/60000 (32%)] | Loss: 0.07973434776067734
Train Epoch 1/1 [22368/60000 (37%)] | Loss: 0.31801822781562805
Train Epoch 1/1 [25568/60000 (43%)] | Loss: 0.07133185118436813
Train Epoch 1/1 [28768/60000 (48%)] | Loss: 0.2804717421531677
Train Epoch 1/1 [31968/60000 (53%)] | Loss: 0.49961113929748535
Train Epoch 1/1 [35168/60000 (59%)] | Loss: 0.1850898414850235
Train Epoch 1/1 [38368/60000 (64%)] | Loss: 0.14962603151798248
Train Epoch 1/1 [41568/60000 (69%)] | Loss: 0.023404883220791817
Train Epoch 1/1 [44768/60000 (75%)] | Loss: 0.041594672948122025
Train Epoch 1/1 [47968/60000 (80%)] | Loss: 0.44139742851257324
Train Epoch 1/1 [51168/60000 (85%)] | Loss: [...]<jupyter_text>---  1.16: Save and Load Model After training a model, it's important to save it in order to use it later. Therefore, let's see how we can save a trained model and load it.<jupyter_code>class ModelNN(nn.Module):
    def __init__(self, in_units):
        super().__init__()
        self.linear = nn.Linear(in_units, 1)

    def forward(self, x):
        return nn.Sigmoid(self.linear(x))
# First way: save the complete model (compt. graph and parameters)

## Save
model = ModelNN(in_units=5)
torch.save(model, "model_1.pth")

## Load
loaded_model = torch.load("model_1.pth")
loaded_model.eval()

print('Model: {} \nModel\'s Parameters: {} \n\nLoaded Model: {} \nLoaded Model\'s Parameters: {}'.format(
    model, model.state_dict(), 
    loaded_model, loaded_model.state_dict()
))
# Second way: save only the model's parameters

## Save
model = ModelNN(in_units=5)
torch.save(model.state_dict(), "model_2.pth")

## Load
loaded_model = ModelNN(in_units=5)
loaded_model.load_state_dict(torch.load("model_2.pth"))
loaded_model.eval()

print('Model: {} \nModel\'s Parameters: {} \n\nLoaded Model: {} \nLoaded Model\'s Parameters: {}'.format(
    model, model.state_dict(), 
    loaded_model, loaded_model.state_dict()
))
# Saving checkpoints (not only for model params, but also for optimizer state)

## Save
model = ModelNN(in_units=5)
optimizer = torch.optim.SGD(model.parameters(), lr=0.001)

checkpoint = {
    "epoch": 90,
    "model_state": model.state_dict(),
    "optim_state": optimizer.state_dict()
}

torch.save(checkpoint, "checkpoint_1.pth")

## Load
loaded_checkpoint = torch.load("checkpoint_1.pth")

loaded_model = ModelNN(in_units=5)
loaded_optimizer = torch.optim.SGD(loaded_model.parameters(), lr=0)

loaded_model.load_state_dict(loaded_checkpoint["model_state"])
loaded_optimizer.load_state_dict(loaded_checkpoint["optim_state"])

print('Model: {} \nModel\'s Parameters: {} \n\nLoaded Model: {} \nLoaded Model\'s Parameters: {}\n'.format(
    model, model.state_dict(), 
    loaded_model, loaded_model.state_dict()
))

print('Optim: {} \nOptim\'s Parameters: {} \n\nLoaded Optim: {} \nLoaded Optim\'s Parameters: {}'.format(
    optimizer, optimizer.state_dict(), 
    loaded_optimizer, loaded_optimizer.state_dict()
))<jupyter_output>Model: ModelNN(
  (linear): Linear(in_features=5, out_features=1, bias=True)
) 
Model's Parameters: OrderedDict([('linear.weight', tensor([[ 0.0509, -0.2935,  0.0417,  0.3653,  0.2452]])), ('linear.bias', tensor([-0.2429]))]) 

Loaded Model: ModelNN(
  (linear): Linear(in_features=5, out_features=1, bias=True)
) 
Loaded Model's Parameters: OrderedDict([('linear.weight', tensor([[ 0.0509, -0.2935,  0.0417,  0.3653,  0.2452]])), ('linear.bias', tensor([-0.2429]))])

Optim: SGD (
Parameter Group 0
    dampening: 0
    lr: 0.001
    momentum: 0
    nesterov: False
    weight_decay: 0
) 
Optim's Parameters: {'state': {}, 'param_groups': [{'lr': 0.001, 'momentum': 0, 'dampening': 0, 'weight_decay': 0, 'nesterov': False, 'params': [0, 1]}]} 

Loaded Optim: SGD (
Parameter Group 0
    dampening: 0
    lr: 0.001
    momentum: 0
    nesterov: False
    weight_decay: 0
) 
Loaded Optim's Parameters: {'state': {}, 'param_groups': [{'lr': 0.001, 'momentum': 0, 'dampening': 0, 'weight_decay': 0, 'nes[...]
{'fraction_non_alphanumeric': 0.12735961100476464, 'fraction_numerical': 0.044097304698961844, 'mean_word_length': 3.916197279846133, 'pattern_counts': {'":': 3, '<': 180, '<?xml version=': 0, '>': 202, 'https://': 10, 'lorem ipsum': 0, 'www.': 5, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 24, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '46', 'dir': 'jupyter-structured-clean-dedup', 'id': '269347', 'license': 'MIT', 'n_tokens_mistral': 26303, 'n_tokens_neox': 23733, 'n_words': 7227, 'path': 'pytorch-tutorial.ipynb', 'repo_name': 'caiodeberaldini/pytorch-tutorial', 'size': '71606'}