<jupyter_start><jupyter_text>--- IntroductionThis assignment will cover the basics of PyTorch. It is largely adapted (with permission) from https://web.eecs.umich.edu/~justincj/teaching/eecs498/FA2019/assignment1.html. PyTorch will be used throughout the semester for homework assignments and your final projects, so it is important to become familiar with it.This notebook will walk you through many of the important features of PyTorch that you will need to use throughout the semester. In some cells you will see code blocks that look like this:```python TODO: Create a variable x with value 536.                                    Replace "pass" statement with your codepass END OF YOUR CODE```You should replace the `pass` statement with your own code and leave the blocks intact, like this:```python TODO: Create a variable x with value 536.                                    Replace "pass" statement with your codex = 536 END OF YOUR CODE```When completing the notebook, please adhere to the following rules:- Do not write or modify any code outside of code blocks.- Do not add or delete any cells from the notebook. You may add new cells to perform scratch work, but delete them before submitting.- Run all cells before submitting. You will only get credit for code that has been run.- Submit your notebook as `netid.ipynb`, where `netid` is your actual netid.  PyTorch [PyTorch](https://pytorch.org/) is an open source machine learning framework. At its core, PyTorch provides a few key features:- A multidimensional **Tensor** object, similar to [NumPy](https://numpy.org/) but with GPU acceleration.- An optimized **autograd** engine for automatically computing derivatives.- A clean, modular API for building and deploying **deep learning models**.We will use PyTorch for all programming assignments throughout the semester. This notebook will focus on the **Tensor API**.You can find more information about PyTorch by following one of the [official tutorials](https://pytorch.org/tutorials/) or by reading the [documentation](https://pytorch.org/docs/1.7.0/). To use PyTorch, we first need to import the `torch` package.We also check the version; the assignments in this course will use PyTorch version 1.7.0, since this is the default version in Google Colab.<jupyter_code>import torch
print(torch.__version__)<jupyter_output>1.7.0+cu101<jupyter_text>Tensor Basics  Creating and accessing tensors A PyTorch **tensor** is a multidimensional grid of values, all of the same type, and is indexed by a tuple of integers.The number of dimensions is the **rank** of the tensor; the **shape** of a tensor is a tuple of integers giving the size of the tensor along each dimension.We can initialize a PyTorch tensor from nested Python lists:<jupyter_code># Create a rank 1 tensor from a Python list
a = torch.tensor([1, 2, 3])
print('Here is a:')
print(a)
print('type of a:', type(a))
print('rank of a:', a.dim())
print('shape of a:', a.shape)<jupyter_output>Here is a:
tensor([1, 2, 3])
type of a: <class 'torch.Tensor'>
rank of a: 1
shape of a: torch.Size([3])<jupyter_text>We can access or mutate elements of a PyTorch tensor using square brackets.Accessing an element from a PyTorch tensor returns a PyTorch scalar; we can convert this to a Python scalar using the `.item()` method:<jupyter_code># Access elements using square brackets
print('a[0]:', a[0])
print('type(a[0]):', type(a[0]))
print('type(a[0].item()):', type(a[0].item()))
# Mutate elements using square brackets
a[1] = 10
print('a after mutating:')
print(a)<jupyter_output>a after mutating:
tensor([ 1, 10,  3])<jupyter_text>The example above shows a one-dimensional tensor; we can similarly create tensors with two or more dimensions:<jupyter_code># Create a two-dimensional tensor
b = torch.tensor([[1, 2, 3], [4, 5, 6]])
print('Here is b:')
print(b)
print('rank of b:', b.dim())
print('shape of b:', b.shape)
# Access elements from a multidimensional tensor
print('b[0, 1]:', b[0, 1])
print('b[1, 2]:', b[1, 2])
# Mutate elements of a multidimensional tensor
b[1, 1] = 100
print('b after mutating:')
print(b)<jupyter_output>b after mutating:
tensor([[  1,   2,   3],
        [  4, 100,   6]])<jupyter_text>Now it's your turn:1. Construct a tensor `c` of shape `(3, 2)` filled with zeros by initializing from nested Python lists.2. Then set element `(0, 1)` to 10, and element `(1, 0)` to 100.<jupyter_code>c = None
################################################################################
# TODO: Construct a tensor c filled with all zeros, initializing from nested   #
# Python lists.                                                                #
################################################################################
# Replace "pass" statement with your code
start = [[0, 0], [0, 0], [0, 0]]
c = torch.Tensor(start)
print('Here is c:')
print(c)
# END OF YOUR CODE
assert torch.is_tensor(c), 'c is a tensor'
assert c.shape == (3, 2), 'c has correct shape'
assert (c == 0).all(), 'c is filled with all zeros'
################################################################################
# TODO: Set element (0, 1) of c to 10, and element (1, 0) of c to 100.         #
################################################################################
# Replace "pass" statement with your code
c[0, 1] = 10
c[1, 0] = 100
print('c after mutating:')
print(c)
# END OF YOUR CODE
assert torch.is_tensor(c), 'c is a tensor'
assert c.shape == (3, 2), 'c has correct shape'
assert c[0, 1] == 10, 'c[0, 1] correct'
assert c[1, 0] == 100, 'c[1, 0] correct'<jupyter_output><empty_output><jupyter_text>Tensor constructors PyTorch provides many convenience methods for constructing tensors; this avoids the need to use Python lists. For example:- [`torch.zeros()`](https://pytorch.org/docs/1.7.0/generated/torch.zeros.htmltorch.zeros): Creates a tensor of all zeros.- [`torch.ones()`](https://pytorch.org/docs/1.7.0/generated/torch.ones.htmltorch.ones): Creates a tensor of all ones.- [`torch.rand()`](https://pytorch.org/docs/1.7.0/generated/torch.rand.htmltorch.rand): Creates a tensor with uniform random numbers.You can find a full list of tensor creation operations [here](https://pytorch.org/docs/1.7.0/torch.htmlcreation-ops) and [here](https://pytorch.org/docs/1.7.0/torch.htmltorch.torch.default_generator).<jupyter_code># Create a tensor of all zeros
a = torch.zeros(2, 3)
print('tensor of zeros:')
print(a)

# Create a tensor of all ones
b = torch.ones(1, 2)
print('\ntensor of ones:')
print(b)

# Create a 3x3 identity matrix
c = torch.eye(3)
print('\nidentity matrix:')
print(c)

# Tensor of random values
d = torch.rand(4, 5)
print('\nrandom tensor:')
print(d)<jupyter_output>tensor of zeros:
tensor([[0., 0., 0.],
        [0., 0., 0.]])

tensor of ones:
tensor([[1., 1.]])

identity matrix:
tensor([[1., 0., 0.],
        [0., 1., 0.],
        [0., 0., 1.]])

random tensor:
tensor([[0.0586, 0.5054, 0.1277, 0.7184, 0.3947],
        [0.4013, 0.9105, 0.9345, 0.0187, 0.6280],
        [0.4440, 0.2133, 0.5801, 0.2999, 0.0200],
        [0.7531, 0.2776, 0.4876, 0.8555, 0.6873]])<jupyter_text>Data types PyTorch provides [a large set of numeric data types](https://pytorch.org/docs/1.7.0/tensor_attributes.htmltorch-dtype) that you can use to construct tensors, of which the most commonly used are:- `torch.float32`: Standard floating-point type; used to store learnable parameters, network activations, etc. Nearly all arithmetic is done using this type.- `torch.int64`: Typically used to store indices.PyTorch tries to guess a data type when you create a tensor; functions that construct tensors typically have a `dtype` argument that you can use to explicitly specify a data type.Each tensor has a `dtype` attribute that you can use to check its data type:<jupyter_code># Let torch choose the data type
x0 = torch.tensor([1, 2])   # List of integers
x1 = torch.tensor([1., 2.]) # List of floats
x2 = torch.tensor([1., 2])  # Mixed list
print('dtype when torch chooses for us:')
print('List of integers:', x0.dtype)
print('List of floats:', x1.dtype)
print('Mixed list:', x2.dtype)

# Force a particular data type
y0 = torch.tensor([1, 2], dtype=torch.float32)  # 32-bit float
y1 = torch.tensor([1, 2], dtype=torch.int32)    # 32-bit (signed) integer
y2 = torch.tensor([1, 2], dtype=torch.int64)    # 64-bit (signed) integer
print('\ndtype when we force a data type:')
print('32-bit float:', y0.dtype)
print('32-bit integer:', y1.dtype)
print('64-bit integer:', y2.dtype)

# Other creation ops also take a dtype argument
z0 = torch.ones(1, 2)  # Let torch choose for us
z1 = torch.ones(1, 2, dtype=torch.int16) # 16-bit (signed) integer
z2 = torch.ones(1, 2, dtype=torch.uint8) # 8-bit (unsigned) integer
print('\ntorch.ones with different dtypes:')
print('default dtype:', z0.dtype)
print('16-bit integer:', z1.dtype)
print('8-bit unsigned integer:', z2.dtype)<jupyter_output>dtype when torch chooses for us:
List of integers: torch.int64
List of floats: torch.float32
Mixed list: torch.float32

dtype when we force a data type:
32-bit float: torch.float32
32-bit integer: torch.int32
64-bit integer: torch.int64

torch.ones with different dtypes:
default dtype: torch.float32
16-bit integer: torch.int16
8-bit unsigned integer: torch.uint8<jupyter_text>We can **cast** a tensor to another data type using the [`.to()`](https://pytorch.org/docs/1.7.0/tensors.htmltorch.Tensor.to) method; there are also convenience methods like [`.float()`](https://pytorch.org/docs/1.7.0/tensors.htmltorch.Tensor.float) and [`.long()`](https://pytorch.org/docs/1.7.0/tensors.htmltorch.Tensor.long) that cast to particular data types:<jupyter_code>x0 = torch.eye(3, dtype=torch.int64)
x1 = x0.float()  # Cast to 32-bit float
x2 = x0.double() # Cast to 64-bit float
x3 = x0.to(torch.float32) # Alternate way to cast to 32-bit float
x4 = x0.to(torch.float64) # Alternate way to cast to 64-bit float
print('x0:', x0.dtype)
print('x1:', x1.dtype)
print('x2:', x2.dtype)
print('x3:', x3.dtype)
print('x4:', x4.dtype)<jupyter_output>x0: torch.int64
x1: torch.float32
x2: torch.float64
x3: torch.float32
x4: torch.float64<jupyter_text>PyTorch provides several ways to create a tensor with the same data type as another tensor:- PyTorch provides tensor constructors such as [`torch.zeros_like()`](https://pytorch.org/docs/1.7.0/generated/torch.zeros_like.htmltorch.zeros_like) that create new tensors with the **same shape and type** as a given tensor.- Tensor objects have instance methods such as [`.new_zeros()`](https://pytorch.org/docs/1.7.0/tensors.htmltorch.Tensor.new_ones) that create tensors the **same type but possibly different shapes**.- The tensor instance method [`.to()`](https://pytorch.org/docs/1.7.0/tensors.htmltorch.Tensor.to) can take a tensor as an argument, in which case it casts to the data type of the argument.<jupyter_code>x0 = torch.eye(3, dtype=torch.float64)  # Shape (3, 3), dtype torch.float64
x1 = torch.zeros_like(x0)               # Shape (3, 3), dtype torch.float64
x2 = x0.new_zeros(4, 5)                 # Shape (4, 5), dtype torch.float64
x3 = torch.ones(6, 7).to(x0)            # Shape (6, 7), dtype torch.float64
print('x0 shape is %r, dtype is %r' % (x0.shape, x0.dtype))
print('x1 shape is %r, dtype is %r' % (x1.shape, x1.dtype))
print('x2 shape is %r, dtype is %r' % (x2.shape, x2.dtype))
print('x3 shape is %r, dtype is %r' % (x3.shape, x3.dtype))<jupyter_output>x0 shape is torch.Size([3, 3]), dtype is torch.float64
x1 shape is torch.Size([3, 3]), dtype is torch.float64
x2 shape is torch.Size([4, 5]), dtype is torch.float64
x3 shape is torch.Size([6, 7]), dtype is torch.float64<jupyter_text>Your turn: Use a tensor creation operation to create a tensor of shape (2, 3, 4) filled entirely with the integer 7.Hint: [`torch.full()`](https://pytorch.org/docs/1.7.0/generated/torch.full.htmltorch.full)<jupyter_code>e = None
################################################################################
# TODO: Create a tensor of shape (2, 3, 4) filled entirely with the integer 7, #
# stored in e. Use 64-bit integers.                                            #
################################################################################
# Replace "pass" statement with your code
e = torch.full((2, 3, 4), 7, dtype = torch.int64)
print('Here is e:')
print(e)
# END OF YOUR CODE
assert torch.is_tensor(e), 'e is a tensor'
assert e.shape == (2, 3, 4), 'e has correct shape'
assert (e == 7).all(), 'e is filled with sevens'<jupyter_output><empty_output><jupyter_text>Your turn: Create a 64-bit floating-point tensor of shape (6,) (six-element vector) filled with evenly-spaced values between 10 (inclusive) and 20 (inclusive).Hint: [`torch.linspace()`](https://pytorch.org/docs/1.7.0/generated/torch.linspace.htmltorch.linspace)<jupyter_code>x = None
##############################################################################
# TODO: Make x contain a six-element vector of 64-bit floating-bit values,   #
# evenly spaced between 10 (inclusive) and 20 (inclusive).                   #
##############################################################################
# Replace "pass" statement with your code
x = torch.linspace(10, 20, steps = 6, dtype = torch.float64)
print('Here is x:')
print(x)
# END OF YOUR CODE
assert torch.is_tensor(x), 'x is a tensor'
assert x.shape == (6,), 'x has correct shape'
assert x.dtype == torch.float64, 'x has correct dtype'<jupyter_output><empty_output><jupyter_text>Tensor Indexing We have already seen how to get and set individual elements of PyTorch tensors. PyTorch also provides many other ways of indexing into tensors. Getting comfortable with these different options will make it easy to modify different parts of tensors.  Slice indexing Similar to Python lists and NumPy arrays, PyTorch tensors can be **sliced** using the syntax `start:stop` or `start:stop:step`. The `stop` index is always non-inclusive: it is the first element not to be included in the slice.Start and stop indices can be negative, in which case they count backward from the end of the tensor:<jupyter_code>a = torch.tensor([0, 11, 22, 33, 44, 55, 66])
print(0, a)        # (0) Original tensor
print(1, a[2:5])   # (1) Elements between index 2 and 5
print(2, a[2:])    # (2) Elements after index 2
print(3, a[:5])    # (3) Elements before index 5
print(4, a[:])     # (4) All elements
print(5, a[1:5:2]) # (5) Every second element between indices 1 and 5
print(6, a[:-1])   # (6) All but the last element
print(7, a[-4::2]) # (7) Every second element, starting from the fourth-last<jupyter_output>0 tensor([ 0, 11, 22, 33, 44, 55, 66])
1 tensor([22, 33, 44])
2 tensor([22, 33, 44, 55, 66])
3 tensor([ 0, 11, 22, 33, 44])
4 tensor([ 0, 11, 22, 33, 44, 55, 66])
5 tensor([11, 33])
6 tensor([ 0, 11, 22, 33, 44, 55])
7 tensor([33, 55])<jupyter_text>For multidimensional tensors, you can provide a slice or integer for each dimension of the tensor:<jupyter_code># Create the following rank 2 tensor with shape (3, 4)
# [[ 1  2  3  4]
#  [ 5  6  7  8]
#  [ 9 10 11 12]]
a = torch.tensor([[1,2,3,4], [5,6,7,8], [9,10,11,12]])
print('Original tensor:')
print(a)
print('shape:', a.shape)

# Get row 1, and all columns. 
print('\nSingle row:')
print(a[1, :])
print(a[1])  # Gives the same result; we can omit : for trailing dimensions
print('shape:', a[1].shape)

print('\nSingle column:')
print(a[:, 1])
print('shape:', a[:, 1].shape)

# Get the first two rows and the last three columns
print('\nFirst two rows, last two columns:')
print(a[:2, -3:])
print('shape:', a[:2, -3:].shape)

# Get every other row, and columns at index 1 and 2
print('\nEvery other row, middle columns:')
print(a[::2, 1:3])
print('shape:', a[::2, 1:3].shape)<jupyter_output>Original tensor:
tensor([[ 1,  2,  3,  4],
        [ 5,  6,  7,  8],
        [ 9, 10, 11, 12]])
shape: torch.Size([3, 4])

Single row:
tensor([5, 6, 7, 8])
tensor([5, 6, 7, 8])
shape: torch.Size([4])

Single column:
tensor([ 2,  6, 10])
shape: torch.Size([3])

First two rows, last two columns:
tensor([[2, 3, 4],
        [6, 7, 8]])
shape: torch.Size([2, 3])

Every other row, middle columns:
tensor([[ 2,  3],
        [10, 11]])
shape: torch.Size([2, 2])<jupyter_text>There are two common ways to access a single row or column of a tensor: using an integer will reduce the rank by one, and using a length-one slice will keep the rank unchanged:<jupyter_code># Create the following rank 2 tensor with shape (3, 4)
a = torch.tensor([[1,2,3,4], [5,6,7,8], [9,10,11,12]])
print('Original tensor')
print(a)

row_r1 = a[1, :]    # Rank 1 view of the second row of a  
row_r2 = a[1:2, :]  # Rank 2 view of the second row of a
print('\nTwo ways of accessing a single row:')
print(row_r1, row_r1.shape)
print(row_r2, row_r2.shape)

# We can make the same distinction when accessing columns
col_r1 = a[:, 1]
col_r2 = a[:, 1:2]
print('\nTwo ways of accessing a single column:')
print(col_r1, col_r1.shape)
print(col_r2, col_r2.shape)<jupyter_output>Original tensor
tensor([[ 1,  2,  3,  4],
        [ 5,  6,  7,  8],
        [ 9, 10, 11, 12]])

Two ways of accessing a single row:
tensor([5, 6, 7, 8]) torch.Size([4])
tensor([[5, 6, 7, 8]]) torch.Size([1, 4])

Two ways of accessing a single column:
tensor([ 2,  6, 10]) torch.Size([3])
tensor([[ 2],
        [ 6],
        [10]]) torch.Size([3, 1])<jupyter_text>Slicing a tensor returns a **view** into the same data, so modifying it will also modify the original tensor. To avoid this, you can use the [`.clone()`](https://pytorch.org/docs/1.7.0/generated/torch.clone.htmltorch.clone) method to make a copy of the tensor:<jupyter_code># Create a tensor, a slice, and a clone of a slice
a = torch.tensor([[1, 2, 3, 4], [5, 6, 7, 8]])
b = a[0, 1:]
c = a[0, 1:].clone()
print('Before mutating:')
print(a)
print(b)
print(c)

a[0, 1] = 20  # a[0, 1] and b[0] point to the same element
b[1] = 30     # b[1] and a[0, 2] point to the same element
c[2] = 40     # c is a clone, so it has its own data
print('\nAfter mutating:')
print(a)
print(b)
print(c)

print(a.storage().data_ptr() == c.storage().data_ptr())<jupyter_output>Before mutating:
tensor([[1, 2, 3, 4],
        [5, 6, 7, 8]])
tensor([2, 3, 4])
tensor([2, 3, 4])

After mutating:
tensor([[ 1, 20, 30,  4],
        [ 5,  6,  7,  8]])
tensor([20, 30,  4])
tensor([ 2,  3, 40])
False<jupyter_text>Your turn: Practice indexing tensors with slices.<jupyter_code># We will use this helper function to check your results
def check(orig, actual, expected):
  expected = torch.tensor(expected)
  same_elements = (actual == expected).all()
  same_storage = (orig.storage().data_ptr() == actual.storage().data_ptr())
  return same_elements and same_storage
# Create the following rank 2 tensor of shape (3, 5)
# [[ 1  2  3  4  5]
#  [ 6  7  8  9 10]
#  [11 12 13 14 15]]
a = torch.tensor([[1, 2, 3, 4, 5], [6, 7, 8, 9, 10], [11, 12, 13, 14, 15]])

b, c, d, e = None, None, None, None
##############################################################################
# TODO: Extract the last row of a, and store it in b; it should have rank 1. #
##############################################################################
# Replace "pass" statement with your code
b = a[-1, :]
print(b)
# END OF YOUR CODE
assert check(a, b, [11, 12, 13, 14, 15]), 'b correct'
##############################################################################
# TODO: Extract the third col of a, and store it in c; it should have rank 2 #
##############################################################################
# Replace "pass" statement with your code
c = a[:, 2:3]
print(c)
# END OF YOUR CODE
assert check(a, c, [[3], [8], [13]]), 'c correct'
##############################################################################
# TODO: Use slicing to extract the first two rows and first three columns    #
# from a; store the result into d.                                           #
##############################################################################
# Replace "pass" statement with your code
d = a[:2, :3]
print(d)
# END OF YOUR CODE
assert check(a, d, [[1, 2, 3], [6, 7, 8]]), 'd correct'
##############################################################################
# TODO: Use slicing to extract a subtensor of a consisting of rows with      #
# index 0 and 2 and columns with index 1 and 4; store the result into e.     #
#                                                                            #
# HINT: Get every other row and every 3rd column starting with index 1.      #
# If you get the right solution, but check() returns False, this means       #
# your solution is copying values (instead of pointing to the original       #
# values), which we do not want here.                                        #
##############################################################################
# Replace "pass" statement with your code
e = a[::2, 1::3]
print(e)
# END OF YOUR CODE
assert check(a, e, [[2, 5], [12, 15]]), 'e correct'<jupyter_output><empty_output><jupyter_text>So far we have used slicing to **access** subtensors; we can also use slicing to **modify** subtensors by writing assignment expressions where the left-hand side is a slice expression, and the right-hand side is a constant or a tensor of the correct shape:<jupyter_code>f = torch.zeros(2, 4, dtype=torch.int64)
f[:, :2] = 1
f[:, 2:] = torch.tensor([[2, 3], [4, 5]])
print(f)<jupyter_output>tensor([[1, 1, 2, 3],
        [1, 1, 4, 5]])<jupyter_text>Your turn: Use slicing to modify a tensor.<jupyter_code>x = torch.zeros(4, 6, dtype=torch.int64)
def modify(x):
  ##############################################################################
  # TODO: Use slicing to modify the tensor x so it has the following contents: #
  #     [[1, 0, 2, 2, 2, 2],                                                   #
  #      [0, 1, 2, 2, 2, 2],                                                   #
  #      [3, 4, 3, 4, 5, 5],                                                   #
  #      [3, 4, 3, 4, 5, 5]]                                                   #
  # This can be achieved using 5 slicing assignment operations (next section   #
  # may be helpful), but you will still get full credit if you use more than 5.#
  ##############################################################################
  # Replace "pass" statement with your code
  x[:2, :2] = torch.eye(2)
  x[:2, 2:] = 2
  x[2:, 0:3:2] = 3
  x[2:, 1:4:2] = 4
  x[2:, -2:] = 5
  # END OF YOUR CODE
  return x
expected = [
    [1, 0, 2, 2, 2, 2],
    [0, 1, 2, 2, 2, 2],
    [3, 4, 3, 4, 5, 5],
    [3, 4, 3, 4, 5, 5],
]
y = modify(x)
assert check(x, y, expected), 'slicing correct'<jupyter_output><empty_output><jupyter_text>Integer array indexing When you index into a PyTorch tensor using slicing, the resulting tensor view will always be a subarray of the original tensor. This is powerful, but can be restrictive.We can also use **index arrays** to index tensors; this lets us construct new tensors with a lot more flexibility than using slices.As an example, we can use index arrays to reorder the rows or columns of a tensor:<jupyter_code># Create the following rank 2 tensor with shape (3, 4)
# [[ 1  2  3  4]
#  [ 5  6  7  8]
#  [ 9 10 11 12]]
a = torch.tensor([[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12]])
print('Original tensor:')
print(a)

# Create a new tensor of shape (5, 4) by reordering rows from a:
# - First two rows are the same as the first row of a
# - Third row is the same as the last row of a
# - Fourth and fifth rows are the same as the second row of a
idx = [0, 0, 2, 1, 1]  # index arrays can be Python lists of integers
print('\nReordered rows:')
print(a[idx])

# Create a new tensor of shape (3, 4) by reversing the columns from a
idx = torch.tensor([3, 2, 1, 0])  # Index arrays can be int64 torch tensors
print('\nReordered columns:')
print(a[:, idx])<jupyter_output>Original tensor:
tensor([[ 1,  2,  3,  4],
        [ 5,  6,  7,  8],
        [ 9, 10, 11, 12]])

Reordered rows:
tensor([[ 1,  2,  3,  4],
        [ 1,  2,  3,  4],
        [ 9, 10, 11, 12],
        [ 5,  6,  7,  8],
        [ 5,  6,  7,  8]])

Reordered columns:
tensor([[ 4,  3,  2,  1],
        [ 8,  7,  6,  5],
        [12, 11, 10,  9]])<jupyter_text>More generally, given index arrays `idx0` and `idx1` with `N` elements each, `a[idx0, idx1]` is equivalent to:```torch.tensor([  a[idx0[0], idx1[0]],  a[idx0[1], idx1[1]],  ...,  a[idx0[N - 1], idx1[N - 1]],])```(A similar pattern extends to tensors with more than two dimensions)We can for example use this to get or set the diagonal of a tensor:<jupyter_code>a = torch.tensor([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
print('Original tensor:')
print(a)

idx = [0, 1, 2]
print('\nGet the diagonal:')
print(a[idx, idx])

# Modify the diagonal
a[idx, idx] = torch.tensor([11, 22, 33])
print('\nAfter setting the diagonal:')
print(a)<jupyter_output>Original tensor:
tensor([[1, 2, 3],
        [4, 5, 6],
        [7, 8, 9]])

Get the diagonal:
tensor([1, 5, 9])

After setting the diagonal:
tensor([[11,  2,  3],
        [ 4, 22,  6],
        [ 7,  8, 33]])<jupyter_text>One useful trick with integer array indexing is selecting or mutating one element from each row or column of a matrix:<jupyter_code># Create a new tensor from which we will select elements
a = torch.tensor([[1, 2, 3], [4, 5, 6], [7, 8, 9], [10, 11, 12]])
print('Original tensor:')
print(a)

# Take one element from each row of a:
# from row 0, take element 1;
# from row 1, take element 2;
# from row 2, take element 1;
# from row 3, take element 0
idx0 = torch.arange(a.shape[0])  # Quick way to build [0, 1, 2, 3]
idx1 = torch.tensor([1, 2, 1, 0])
print('\nSelect one element from each row:')
print(a[idx0, idx1])

# Now set each of those elements to zero
a[idx0, idx1] = 0
print('\nAfter modifying one element from each row:')
print(a)<jupyter_output>Original tensor:
tensor([[ 1,  2,  3],
        [ 4,  5,  6],
        [ 7,  8,  9],
        [10, 11, 12]])

Select one element from each row:
tensor([ 2,  6,  8, 10])

After modifying one element from each row:
tensor([[ 1,  0,  3],
        [ 4,  5,  0],
        [ 7,  0,  9],
        [ 0, 11, 12]])<jupyter_text>Your turn: Practice with integer array indexing.<jupyter_code># Build a tensor of shape (4, 3):
# [[ 1,  2,  3],
#  [ 4,  5,  6],
#  [ 7,  8,  9],
#  [10, 11, 12]]
a = torch.tensor([[1, 2, 3], [4, 5, 6], [7, 8, 9], [10, 11, 12]])
print('Here is a:')
print(a)

b, c, d = None, None, None
##############################################################################
# TODO: Use integer array indexing to create a tensor of shape (4, 4) where: #
# - The first two columns are the same as the first column of a.             #
# - The next column is the same as the third column of a.                    #
# - The last column is the same as the second column of a.                   #
# Store the resulting tensor in b.                                           #
##############################################################################
# Replace "pass" statement with your code
b = a[:, [0, 0, 2, 1]]
# END OF YOUR CODE
print('Here is b:')
print(b)
##############################################################################
# TODO: Use integer array indexing to create a new tensor which is the same  #
# as a, but has its rows reversed. Store the result in c.                    #
##############################################################################
# Replace "pass" statement with your code
c = a[[3, 2, 1, 0]]
# END OF YOUR CODE
print('Here is c:')
print(c)
##############################################################################
# TODO: Use integer array indexing to create a new tensor by selecting one   #
# element from each column of a:                                             #
# - From the first column, take the second element.                          #
# - From the second column, take the first element.                          #
# - From the third column, take the fourth element.                          #
# Store the result in d.                                                     #
##############################################################################
# Replace "pass" statement with your code
d = a[[1, 0, 3], [0, 1, 2]]
# END OF YOUR CODE
print('Here is d:')
print(d)<jupyter_output>Here is d:
tensor([ 4,  2, 12])<jupyter_text>Boolean mask indexing Boolean mask indexing lets you pick out arbitrary elements of a tensor according to a boolean mask. Frequently this type of indexing is used to select or modify the elements of a tensor that satisfy some condition:<jupyter_code>a = torch.tensor([[1, 2], [3, 4], [5, 6]])
print('Original tensor:')
print(a)

# Find the elements of a that are bigger than 3. The mask has the same shape as
# a, where each element of the mask tells whether the corresponding element of a
# is greater than 3.
mask = (a > 3)
print('\nMask tensor:')
print(mask)

# We can use the mask to construct a rank-1 tensor containing the elements of a
# that are selected by the mask
print('\nSelecting elements with the mask:')
print(a[mask])

# We can also use boolean masks to modify tensors; for example this sets all
# elements <= 3 to zero:
a[a <= 3] = 0
print('\nAfter modifying with a mask:')
print(a)<jupyter_output>Original tensor:
tensor([[1, 2],
        [3, 4],
        [5, 6]])

Mask tensor:
tensor([[False, False],
        [False,  True],
        [ True,  True]])

Selecting elements with the mask:
tensor([4, 5, 6])

After modifying with a mask:
tensor([[0, 0],
        [0, 4],
        [5, 6]])<jupyter_text>Your turn: Practice with boolean masks by implementing the following function.<jupyter_code>def num_negative(x):
  """
  Return the number of negative values in the tensor x.

  Inputs:
  - x: A tensor of any shape.

  Returns:
  - num_neg: Number of negative values in x.
  """
  num_neg = 0
  ##############################################################################
  # TODO: Use boolean masks to count the number of negative elements in x.     #
  ##############################################################################
  # Replace "pass" statement with your code
  num_neg = list((x[x < 0]).shape)[0]
  # END OF YOUR CODE
  return num_neg
# Make a few test cases
torch.manual_seed(536)
x0 = torch.tensor([[-1, -1, 0], [0, 1, 2], [3, 4, 5]])
x1 = torch.tensor([0, 1, 2, 3])
x2 = torch.randn(100, 100)
assert num_negative(x0) == 2
assert num_negative(x1) == 0
assert num_negative(x2) == 4955<jupyter_output><empty_output><jupyter_text>Now implement a function that creates a tensor of **one-hot vectors** from a list of Python integers.A one-hot vector for an integer $n$ is a vector that has a one in its $n$th slot, and zeros in all other slots. One-hot vectors are commonly used to represent categorical variables in machine learning models.For example, given a list of integers `[1, 4, 3, 2]`, your function should produce the tensor:```[[0 1 0 0 0], [0 0 0 0 1], [0 0 0 1 0], [0 0 1 0 0]]```Here the first row corresponds to the first element of the list: it has a one at index 1, and zeros at all other indices. The second row corresponds to the second element of the list: it has a one at index 4, and zeros at all other indices. The other rows follow the same pattern.<jupyter_code>def make_one_hot(x):
  """
  Construct a LongTensor of one-hot-vectors from a list of Python non-negative
  integers.

  Inputs:
  - x: A list of N ints (non-negative).

  Returns:
  - y: A tensor of shape (N, C) where C = 1 + max(x) is one more than the max
       value in x. The n-th row of y is a one-hot-vector representation of x[n].
       In other words, if x[n] = c then y[n, c] = 1; all other elements of y are
       zeros. The ones and zeros in y should be 64-bit integers.
  """
  y = None
  ##############################################################################
  # TODO: Complete the implementation of this function. You are NOT allowed to #
  # use PyTorch's built-in one_hot functions.                                  #
  ##############################################################################
  # Replace "pass" statement with your code
  y = torch.zeros(len(x), max(x) + 1)
  for i, j in enumerate(x):
    y[i, j] = 1
  # END OF YOUR CODE
  return y
def check_one_hot(x, y):
  C = y.shape[1]
  for i, n in enumerate(x):
    if n >= C: return False
    for j in range(C):
      expected = 1 if j == n else 0
      if y[i, j].item() != expected: return False
  return True

x0 = [1, 4, 3, 2]
y0 = make_one_hot(x0)
print('Here is y0:')
print(y0)
assert check_one_hot(x0, y0), 'y0 correct'<jupyter_output>Here is y0:
tensor([[0., 1., 0., 0., 0.],
        [0., 0., 0., 0., 1.],
        [0., 0., 0., 1., 0.],
        [0., 0., 1., 0., 0.]])<jupyter_text>Reshape Operations  View PyTorch provides many ways to manipulate the shape of tensors. The simplest example is [`.view()`](https://pytorch.org/docs/1.7.0/tensors.htmltorch.Tensor.view): This returns a new tensor with the same number of elements as its input, but with a different shape.We can use `.view()` to flatten matrices into vectors, and to convert rank-1 vectors into rank-2 row or column matrices:<jupyter_code>x0 = torch.tensor([[1, 2, 3, 4], [5, 6, 7, 8]])
print('Original tensor:')
print(x0)
print('shape:', x0.shape)

# Flatten x0 into a rank 1 vector of shape (8,)
x1 = x0.view(8)
print('\nFlattened tensor:')
print(x1)
print('shape:', x1.shape)

# Convert x1 to a rank 2 "row vector" of shape (1, 8)
x2 = x1.view(1, 8)
print('\nRow vector:')
print(x2)
print('shape:', x2.shape)

# Convert x1 to a rank 2 "column vector" of shape (8, 1)
x3 = x1.view(8, 1)
print('\nColumn vector:')
print(x3)
print('shape:', x3.shape)

# Convert x1 to a rank 3 tensor of shape (2, 2, 2):
x4 = x1.view(2, 2, 2)
print('\nRank 3 tensor:')
print(x4)
print('shape:', x4.shape)<jupyter_output>Original tensor:
tensor([[1, 2, 3, 4],
        [5, 6, 7, 8]])
shape: torch.Size([2, 4])

Flattened tensor:
tensor([1, 2, 3, 4, 5, 6, 7, 8])
shape: torch.Size([8])

Row vector:
tensor([[1, 2, 3, 4, 5, 6, 7, 8]])
shape: torch.Size([1, 8])

Column vector:
tensor([[1],
        [2],
        [3],
        [4],
        [5],
        [6],
        [7],
        [8]])
shape: torch.Size([8, 1])

Rank 3 tensor:
tensor([[[1, 2],
         [3, 4]],

        [[5, 6],
         [7, 8]]])
shape: torch.Size([2, 2, 2])<jupyter_text>As a convenience, calls to `.view()` may include a single -1 argument; this puts enough elements on that dimension so that the output has the same shape as the input. This makes it easy to write some reshape operations in a way that is agnostic to the shape of the tensor:<jupyter_code># We can reuse these functions for tensors of different shapes
def flatten(x):
  return x.view(-1)

def make_row_vec(x):
  return x.view(1, -1)

x0 = torch.tensor([[1, 2, 3], [4, 5, 6]])
x0_flat = flatten(x0)
x0_row = make_row_vec(x0)
print('x0:')
print(x0)
print('x0_flat:')
print(x0_flat)
print('x0_row:')
print(x0_row)

x1 = torch.tensor([[1, 2], [3, 4]])
x1_flat = flatten(x1)
x1_row = make_row_vec(x1)
print('\nx1:')
print(x1)
print('x1_flat:')
print(x1_flat)
print('x1_row:')
print(x1_row)<jupyter_output>x0:
tensor([[1, 2, 3],
        [4, 5, 6]])
x0_flat:
tensor([1, 2, 3, 4, 5, 6])
x0_row:
tensor([[1, 2, 3, 4, 5, 6]])

x1:
tensor([[1, 2],
        [3, 4]])
x1_flat:
tensor([1, 2, 3, 4])
x1_row:
tensor([[1, 2, 3, 4]])<jupyter_text>As its name implies, a tensor returned by `.view()` shares the same data as the input, so changes to one will affect the other:<jupyter_code>x = torch.tensor([[1, 2, 3], [4, 5, 6]])
x_flat = x.view(-1)
print('x before modifying:')
print(x)
print('x_flat before modifying:')
print(x_flat)

x[0, 0] = 10   # x[0, 0] and x_flat[0] point to the same data
x_flat[1] = 20 # x_flat[1] and x[0, 1] point to the same data

print('\nx after modifying:')
print(x)
print('x_flat after modifying:')
print(x_flat)<jupyter_output>x before modifying:
tensor([[1, 2, 3],
        [4, 5, 6]])
x_flat before modifying:
tensor([1, 2, 3, 4, 5, 6])

x after modifying:
tensor([[10, 20,  3],
        [ 4,  5,  6]])
x_flat after modifying:
tensor([10, 20,  3,  4,  5,  6])<jupyter_text>Swapping axes Another common reshape operation you might want to perform is transposing a matrix. You might be surprised if you try to transpose a matrix with `.view()`: The `view()` function takes elements in row-major order, so **you cannot transpose matrices with `.view()`**.In general, you should only use `.view()` to add new dimensions to a tensor, or to collapse adjacent dimensions of a tensor.For other types of reshape operations, you usually need to use a function that can swap axes of a tensor. The simplest such function is `.t()`, specificially for transposing matrices. It is available both as a [function in the `torch` module](https://pytorch.org/docs/1.7.0/generated/torch.t.htmltorch.t), and as a [tensor instance method](https://pytorch.org/docs/1.7.0/tensors.htmltorch.Tensor.t):<jupyter_code>x = torch.tensor([[1, 2, 3], [4, 5, 6]])
print('Original matrix:')
print(x)
print('\nTransposing with .view() DOES NOT WORK!')
print(x.view(3, 2))
print('\nTransposed matrix:')
print(torch.t(x))
print(x.t())<jupyter_output>Original matrix:
tensor([[1, 2, 3],
        [4, 5, 6]])

Transposing with .view() DOES NOT WORK!
tensor([[1, 2],
        [3, 4],
        [5, 6]])

Transposed matrix:
tensor([[1, 4],
        [2, 5],
        [3, 6]])
tensor([[1, 4],
        [2, 5],
        [3, 6]])<jupyter_text>For tensors with more than two dimensions, we can use the function [`torch.transpose()`](https://pytorch.org/docs/1.7.0/generated/torch.transpose.htmltorch.transpose) to swap arbitrary dimensions, or the [`.permute()`](https://pytorch.org/docs/1.7.0/tensors.htmltorch.Tensor.permute) method to arbitrarily permute dimensions:<jupyter_code># Create a tensor of shape (2, 3, 4)
x0 = torch.tensor([
     [[1,  2,  3,  4],
      [5,  6,  7,  8],
      [9, 10, 11, 12]],
     [[13, 14, 15, 16],
      [17, 18, 19, 20],
      [21, 22, 23, 24]]])
print('Original tensor:')
print(x0)
print('shape:', x0.shape)

# Swap axes 1 and 2; shape is (2, 4, 3)
x1 = x0.transpose(1, 2)
print('\nSwap axes 1 and 2:')
print(x1)
print(x1.shape)

# Permute axes; the argument (1, 2, 0) means:
# - Make the old dimension 1 appear at dimension 0;
# - Make the old dimension 2 appear at dimension 1;
# - Make the old dimension 0 appear at dimension 2
# This results in a tensor of shape (3, 4, 2)
x2 = x0.permute(1, 2, 0)
print('\nPermute axes')
print(x2)
print('shape:', x2.shape)<jupyter_output>Original tensor:
tensor([[[ 1,  2,  3,  4],
         [ 5,  6,  7,  8],
         [ 9, 10, 11, 12]],

        [[13, 14, 15, 16],
         [17, 18, 19, 20],
         [21, 22, 23, 24]]])
shape: torch.Size([2, 3, 4])

Swap axes 1 and 2:
tensor([[[ 1,  5,  9],
         [ 2,  6, 10],
         [ 3,  7, 11],
         [ 4,  8, 12]],

        [[13, 17, 21],
         [14, 18, 22],
         [15, 19, 23],
         [16, 20, 24]]])
torch.Size([2, 4, 3])

Permute axes
tensor([[[ 1, 13],
         [ 2, 14],
         [ 3, 15],
         [ 4, 16]],

        [[ 5, 17],
         [ 6, 18],
         [ 7, 19],
         [ 8, 20]],

        [[ 9, 21],
         [10, 22],
         [11, 23],
         [12, 24]]])
shape: torch.Size([3, 4, 2])<jupyter_text>Contiguous tensors Some combinations of reshape operations will fail with cryptic errors. The exact reasons for this have to do with the way that tensors and views of tensors are implemented, and are beyond the scope of this assignment. However if you're curious, [this blog post by <NAME>](http://blog.ezyang.com/2019/05/pytorch-internals/) gives a clear explanation of the problem.What you need to know is that you can typically overcome these sorts of errors by either calling [`.contiguous()`](https://pytorch.org/docs/1.7.0/tensors.htmltorch.Tensor.contiguous) before `.view()`, or using [`.reshape()`](https://pytorch.org/docs/1.7.0/tensors.htmltorch.Tensor.reshape) instead of `.view()`:<jupyter_code>x0 = torch.randn(2, 3, 4)

try:
  # This sequence of reshape operations will crash
  x1 = x0.transpose(1, 2).view(8, 3)
except RuntimeError as e:
  print(type(e), e)
  
# We can solve the problem using either .contiguous() or .reshape()
x1 = x0.transpose(1, 2).contiguous().view(8, 3)
x2 = x0.transpose(1, 2).reshape(8, 3)
print('x1 shape:', x1.shape)
print('x2 shape:', x2.shape)<jupyter_output><class 'RuntimeError'> view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead.
x1 shape: torch.Size([8, 3])
x2 shape: torch.Size([8, 3])<jupyter_text>Your turn: Given the 1-dimensional input tensor `x0` containing the numbers 0 through 23 in order, apply a sequence of reshape operations to `x0` to create the following tensor:```x1 = tensor([[ 0,  1,  2,  3, 12, 13, 14, 15],             [ 4,  5,  6,  7, 16, 17, 18, 19],             [ 8,  9, 10, 11, 20, 21, 22, 23]])```Hint: You will need to create an intermediate tensor of rank 3.<jupyter_code>x0 = torch.arange(24)
print('Here is x0:')
print(x0)

x1 = None
##############################################################################
# TODO: Use reshape operations to create x1 from x0.                         #
##############################################################################
# Replace "pass" statement with your code
x1 = x0.reshape(3, 8)
# END OF YOUR CODE
print('Here is x1:')
print(x1)<jupyter_output>Here is x1:
tensor([[ 0,  1,  2,  3,  4,  5,  6,  7],
        [ 8,  9, 10, 11, 12, 13, 14, 15],
        [16, 17, 18, 19, 20, 21, 22, 23]])<jupyter_text>Tensor Operations  Elementwise operations Basic mathematical functions operate elementwise on tensors, and are available as operator overloads, as functions in the `torch` module, and as instance methods on tensor objects; all produce the same results:<jupyter_code>x = torch.tensor([[1, 2, 3, 4]], dtype=torch.float32)
y = torch.tensor([[5, 6, 7, 8]], dtype=torch.float32)

# Elementwise sum; all give the same result
print('Elementwise sum:')
print(x + y)
print(torch.add(x, y))
print(x.add(y))

# Elementwise difference
print('\nElementwise difference:')
print(x - y)
print(torch.sub(x, y))
print(x.sub(y))

# Elementwise product
print('\nElementwise product:')
print(x * y)
print(torch.mul(x, y))
print(x.mul(y))

# Elementwise division
print('\nElementwise division')
print(x / y)
print(torch.div(x, y))
print(x.div(y))

# Elementwise power
print('\nElementwise power')
print(x ** y)
print(torch.pow(x, y))
print(x.pow(y))<jupyter_output>Elementwise sum:
tensor([[ 6.,  8., 10., 12.]])
tensor([[ 6.,  8., 10., 12.]])
tensor([[ 6.,  8., 10., 12.]])

Elementwise difference:
tensor([[-4., -4., -4., -4.]])
tensor([[-4., -4., -4., -4.]])
tensor([[-4., -4., -4., -4.]])

Elementwise product:
tensor([[ 5., 12., 21., 32.]])
tensor([[ 5., 12., 21., 32.]])
tensor([[ 5., 12., 21., 32.]])

Elementwise division
tensor([[0.2000, 0.3333, 0.4286, 0.5000]])
tensor([[0.2000, 0.3333, 0.4286, 0.5000]])
tensor([[0.2000, 0.3333, 0.4286, 0.5000]])

Elementwise power
tensor([[1.0000e+00, 6.4000e+01, 2.1870e+03, 6.5536e+04]])
tensor([[1.0000e+00, 6.4000e+01, 2.1870e+03, 6.5536e+04]])
tensor([[1.0000e+00, 6.4000e+01, 2.1870e+03, 6.5536e+04]])<jupyter_text>PyTorch also provides many standard mathematical functions; these are available both as functions in the `torch` module and as instance methods on tensors.You can find a full list of all available mathematical functions in the [documentation](https://pytorch.org/docs/1.7.0/torch.htmlpointwise-ops); many functions in the `torch` module have corresponding instance methods [on tensor objects](https://pytorch.org/docs/1.7.0/tensors.html):<jupyter_code>x = torch.tensor([[1, 2, 3, 4]], dtype=torch.float32)

print('Square root:')
print(torch.sqrt(x))
print(x.sqrt())

print('\nTrig functions:')
print(torch.sin(x))
print(x.sin())
print(torch.cos(x))
print(x.cos())<jupyter_output>Square root:
tensor([[1.0000, 1.4142, 1.7321, 2.0000]])
tensor([[1.0000, 1.4142, 1.7321, 2.0000]])

Trig functions:
tensor([[ 0.8415,  0.9093,  0.1411, -0.7568]])
tensor([[ 0.8415,  0.9093,  0.1411, -0.7568]])
tensor([[ 0.5403, -0.4161, -0.9900, -0.6536]])
tensor([[ 0.5403, -0.4161, -0.9900, -0.6536]])<jupyter_text>Reduction operations So far we've seen basic arithmetic operations on tensors that operate elementwise. We may sometimes want to perform operations that aggregate over part or all of a tensor, such as a summation; these are called **reduction** operations.Like the elementwise operations above, most reduction operations are available both as functions in the `torch` module and as instance methods on tensor objects.The simplest reduction operation is summation. We can use the [`.sum()`](https://pytorch.org/docs/1.7.0/generated/torch.sum.htmltorch.sum) function to reduce a tensor. We can also specify one or several dimensions along which to reduce using the `dim` argument:<jupyter_code>x = torch.tensor([[1, 2, 3], 
                  [4, 5, 6]], dtype=torch.float32)
print('Original tensor:')
print(x)

print('\nSum over entire tensor:')
print(torch.sum(x))
print(x.sum())

# We can sum over each row:
print('\nSum of each row:')
print(torch.sum(x, dim=0))
print(x.sum(dim=0))

# Sum over each column:
print('\nSum of each column:')
print(torch.sum(x, dim=1))
print(x.sum(dim=1))<jupyter_output>Original tensor:
tensor([[1., 2., 3.],
        [4., 5., 6.]])

Sum over entire tensor:
tensor(21.)
tensor(21.)

Sum of each row:
tensor([5., 7., 9.])
tensor([5., 7., 9.])

Sum of each column:
tensor([ 6., 15.])
tensor([ 6., 15.])<jupyter_text>Other useful reduction operations include [`.mean()`](https://pytorch.org/docs/1.7.0/generated/torch.mean.htmltorch.mean), [`.min()`](https://pytorch.org/docs/1.7.0/generated/torch.min.htmltorch.min), and [`.max()`](https://pytorch.org/docs/1.7.0/generated/torch.max.htmltorch.max). You can find a full list of all available reduction operations in the [documentation](https://pytorch.org/docs/1.7.0/torch.htmlreduction-ops).Some reduction operations return more than one value; for example `.min()` returns both the minimum value over the specified dimension, as well as the index where the minimum value occurs:<jupyter_code>x = torch.tensor([[2, 4, 3, 5], [3, 3, 5, 2]], dtype=torch.float32)
print('Original tensor:')
print(x, x.shape)

# Finding the overall minimum only returns a single value
print('\nOverall minimum:', x.min())

# Compute the minimum along each column; we get both the value and location:
# The minimum of the first column is 2, and it appears at index 0;
# the minimum of the second column is 3 and it appears at index 1; etc
col_min_vals, col_min_idxs = x.min(dim=0)
print('\nMinimum along each column:')
print('values:', col_min_vals)
print('idxs:', col_min_idxs)

# Compute the minimum along each row; we get both the value and the minimum
row_min_vals, row_min_idxs = x.min(dim=1)
print('\nMinimum along each row:')
print('values:', row_min_vals)
print('idxs:', row_min_idxs)<jupyter_output>Original tensor:
tensor([[2., 4., 3., 5.],
        [3., 3., 5., 2.]]) torch.Size([2, 4])

Overall minimum: tensor(2.)

Minimum along each column:
values: tensor([2., 3., 3., 2.])
idxs: tensor([0, 1, 0, 1])

Minimum along each row:
values: tensor([2., 2.])
idxs: tensor([0, 3])<jupyter_text>By default, reduction operations *reduce* the rank of tensors: the dimension over which you perform the reduction will be removed from the shape of the output. If you pass `keepdim=True` to a reduction operation, the specified dimension will not be removed; the output tensor will instead have a shape of 1 in that dimension.When you are working with multidimensional tensors, thinking about rows and columns can become confusing; instead it's more useful to think about the shape that will result from each operation. For example:<jupyter_code># Create a tensor of shape (128, 10, 3, 64, 64)
x = torch.randn(128, 10, 3, 64, 64)
print(x.shape)

# Take the mean over dimension 1; shape is now (128, 3, 64, 64)
x = x.mean(dim=1)
print(x.shape)

# Take the sum over dimension 2; shape is now (128, 3, 64)
x = x.sum(dim=2)
print(x.shape)

# Take the mean over dimension 1, but keep the dimension from being eliminated
# by passing keepdim=True; shape is now (128, 1, 64)
x = x.mean(dim=1, keepdim=True)
print(x.shape)<jupyter_output>torch.Size([128, 10, 3, 64, 64])
torch.Size([128, 3, 64, 64])
torch.Size([128, 3, 64])
torch.Size([128, 1, 64])<jupyter_text>Your turn: Use reduction and indexing operations to implement a function that sets the minimum value along each row of a tensor to zero.Hint: [`torch.argmin()`](https://pytorch.org/docs/1.7.0/generated/torch.argmin.htmltorch.argmin)<jupyter_code>def zero_row_min(x):
  """
  Return a copy of x, where the minimum value along each row has been set to 0.

  For example, if x is:
  x = torch.tensor([
        [10, 20, 30],
        [ 2,  5,  1],
      ])

  Then y = zero_row_min(x) should be:
  torch.tensor([
    [0, 20, 30],
    [2,  5,  0],
  ])

  Inputs:
  - x: Tensor of rank 2 with shape (N, M).

  Returns:
  - y: Tensor of rank 2 that is a copy of x, except the minimum value along each
       row is replaced with 0.
  """
  y = x.clone()
  ##############################################################################
  # TODO: Complete the implementation of this function.                        #
  ##############################################################################
  # Replace "pass" statement with your code
  row_min_idx = x.argmin(dim = 1)
  for i in range(len(row_min_idx)):
    y[i, row_min_idx[i]] = 0
  # END OF YOUR CODE
  return y
x0 = torch.tensor([[10, 20, 30], [2, 5, 1]])
print('Here is x0:')
print(x0)
y0 = zero_row_min(x0)
print('Here is y0:')
print(y0)<jupyter_output>Here is x0:
tensor([[10, 20, 30],
        [ 2,  5,  1]])
Here is y0:
tensor([[ 0, 20, 30],
        [ 2,  5,  0]])<jupyter_text>Matrix operations Note that `*` is elementwise multiplication, not matrix multiplication. PyTorch provides a number of linear algebra functions that compute different types of vector and matrix products. The most commonly used are:- [`torch.dot()`](https://pytorch.org/docs/1.7.0/generated/torch.dot.htmltorch.dot): Computes inner product of vectors.- [`torch.mm()`](https://pytorch.org/docs/1.7.0/generated/torch.mm.htmltorch.mm): Computes matrix-matrix products.- [`torch.mv()`](https://pytorch.org/docs/1.7.0/generated/torch.mv.htmltorch.mv): Computes matrix-vector products.- [`torch.addmm()`](https://pytorch.org/docs/1.7.0/generated/torch.addmm.htmltorch.addmm) / [`torch.addmv()`](https://pytorch.org/docs/1.7.0/generated/torch.addmv.htmltorch.addmv): Computes matrix-matrix and matrix-vector multiplications plus a bias.- [`torch.bmm()`](https://pytorch.org/docs/1.7.0/generated/torch.bmm.htmltorch.bmm) / [`torch.baddmm()`](https://pytorch.org/docs/1.7.0/generated/torch.baddmm.htmltorch.baddmm): Batched versions of `torch.mm()` and `torch.addmm()`, respectively.- [`torch.matmul()`](https://pytorch.org/docs/1.7.0/generated/torch.matmul.htmltorch.matmul): General matrix product that performs different operations depending on the rank of the inputs.You can find a full list of the available linear algebra operators in the [documentation](https://pytorch.org/docs/1.7.0/torch.htmlblas-and-lapack-operations).Here is an example of using `torch.dot()` to compute inner products. Like the other mathematical operators we've seen, most linear algebra operators are available both as functions in the `torch` module and as instance methods of tensors:<jupyter_code>v = torch.tensor([9, 10], dtype=torch.float32)
w = torch.tensor([11, 12], dtype=torch.float32)

# Inner product of vectors
print('Dot products:')
print(torch.dot(v, w))
print(v.dot(w))

# .dot() only works for vectors -- it will give an error for tensors of rank > 1
x = torch.tensor([[1, 2], [3, 4]], dtype=torch.float32)
y = torch.tensor([[5, 6], [7, 8]], dtype=torch.float32)
try:
  print(x.dot(y))
except RuntimeError as e:
  print(e)
  
# Instead we use .mm() for matrix-matrix products:
print('\nMatrix-matrix product:')
print(torch.mm(x, y))
print(x.mm(y))<jupyter_output>Dot products:
tensor(219.)
tensor(219.)
1D tensors expected, but got 2D and 2D tensors

Matrix-matrix product:
tensor([[19., 22.],
        [43., 50.]])
tensor([[19., 22.],
        [43., 50.]])<jupyter_text>With all the different linear algebra operators that PyTorch provides, there is usually more than one way to compute something. For example, to compute matrix-vector products we can use `torch.mv()`; we can reshape the vector to have rank 2 and use `torch.mm()`; or we can use `torch.matmul()`. All give the same results, but the outputs might have different ranks:<jupyter_code>print('Here is x (rank 2):')
print(x)
print('\nHere is v (rank 1):')
print(v)

# Matrix-vector multiply with torch.mv() produces a rank-1 output
print('\nMatrix-vector product with torch.mv() (rank 1 output):')
print(torch.mv(x, v))
print(x.mv(v))

# We can reshape the vector to have rank 2 and use torch.mm() to perform
# matrix-vector products, but the result will have rank 2
print('\nMatrix-vector product with torch.mm() (rank 2 output):')
print(torch.mm(x, v.view(2, 1)))
print(x.mm(v.view(2, 1)))

print('\nMatrix-vector product with torch.matmul() (rank 1 output):')
print(torch.matmul(x, v))
print(x.matmul(v))<jupyter_output>Here is x (rank 2):
tensor([[1., 2.],
        [3., 4.]])

Here is v (rank 1):
tensor([ 9., 10.])

Matrix-vector product with torch.mv() (rank 1 output):
tensor([29., 67.])
tensor([29., 67.])

Matrix-vector product with torch.mm() (rank 2 output):
tensor([[29.],
        [67.]])
tensor([[29.],
        [67.]])

Matrix-vector product with torch.matmul() (rank 1 output):
tensor([29., 67.])
tensor([29., 67.])<jupyter_text>Your turn: Use [`torch.bmm()`](https://pytorch.org/docs/1.1.0/torch.htmltorch.bmm) to perform a batched matrix multiply.<jupyter_code>B, N, M, P = 3, 2, 5, 4
x = torch.rand(B, N, M)  # Random tensor of shape (B, N, M)
y = torch.rand(B, M, P)  # Random tensor of shape (B, M, P)

# We can use a for loop to (inefficiently) compute a batch of matrix multiply
# operations
z1 = torch.empty(B, N, P)  # Empty tensor of shape (B, N, P)
for i in range(B):
  z1[i] = x[i].mm(y[i])
print('Here is the result of batched matrix multiply with a loop:')
print(z1)
z2 = None
##############################################################################
# TODO: Use torch.bmm() to compute a batched matrix multiply between x and   #
# y; store the result in z2.                                                 #
##############################################################################
# Replace "pass" statement with your code
z2 = torch.bmm(x, y)
# END OF YOUR CODE
print('Here is the result of batched matrix multiply with torch.bmm():')
print(z2)

# The two may not return exactly the same result; different linear algebra
# routines often return slightly different results due to the fact that
# floating-point math is non-exact and non-associative.
diff = (z1 - z2).abs().max().item()
print('\nDifference:', diff)
assert diff < 1e-6, 'Difference within threshold'<jupyter_output>Here is the result of batched matrix multiply with torch.bmm():
tensor([[[1.1163, 1.1587, 1.9690, 1.8232],
         [1.4139, 1.6047, 2.7143, 2.7186]],

        [[0.3707, 1.3418, 0.9090, 1.5271],
         [0.5270, 1.6415, 0.8949, 1.6092]],

        [[0.8863, 0.8614, 1.6273, 0.5403],
         [1.0417, 0.4843, 0.6703, 0.9262]]])

Difference: 2.384185791015625e-07<jupyter_text>Broadcasting Broadcasting is a powerful mechanism that allows PyTorch to work with arrays of different shapes when performing arithmetic operations. Frequently we have a smaller tensor and a larger tensor, and we want to use the smaller tensor multiple times to perform some operation on the larger tensor.For example, suppose that we want to add a constant vector to each row of a tensor. We could do it like this:<jupyter_code># We will add the vector v to each row of the matrix x,
# storing the result in the matrix y
x = torch.tensor([[1, 2, 3], [4, 5, 6], [7, 8, 9], [10, 11, 12]])
v = torch.tensor([1, 0, 1])
y = torch.zeros_like(x)   # Create an empty matrix with the same shape as x

# Add the vector v to each row of the matrix x with an explicit loop
for i in range(4):
    y[i, :] = x[i, :] + v

print(y)<jupyter_output>tensor([[ 2,  2,  4],
        [ 5,  5,  7],
        [ 8,  8, 10],
        [11, 11, 13]])<jupyter_text>This works; however when the tensor `x` is very large, computing an explicit loop in Python could be slow. Note that adding the vector `v` to each row of the tensor `x` is equivalent to forming a tensor `vv` by stacking multiple copies of `v` vertically, then performing elementwise summation of `x` and `vv`. We could implement this approach like this:<jupyter_code>vv = v.repeat((4, 1))  # Stack 4 copies of v on top of each other
print(vv)              # Prints "[[1 0 1]
                       #          [1 0 1]
                       #          [1 0 1]
                       #          [1 0 1]]"
y = x + vv  # Add x and vv elementwise
print(y)<jupyter_output>tensor([[ 2,  2,  4],
        [ 5,  5,  7],
        [ 8,  8, 10],
        [11, 11, 13]])<jupyter_text>PyTorch broadcasting allows us to perform this computation without actually creating multiple copies of `v`. Consider this version, using broadcasting:<jupyter_code># We will add the vector v to each row of the matrix x,
# storing the result in the matrix y
x = torch.tensor([[1, 2, 3], [4, 5, 6], [7, 8, 9], [10, 11, 12]])
v = torch.tensor([1, 0, 1])
y = x + v  # Add v to each row of x using broadcasting
print(y)<jupyter_output>tensor([[ 2,  2,  4],
        [ 5,  5,  7],
        [ 8,  8, 10],
        [11, 11, 13]])<jupyter_text>Due to broadcasting, the line `y = x + v` works even though `x` has shape (4, 3) and `v` has shape (3,); it works as if `v` actually had shape (4, 3), where each row was a copy of `v`, and the sum was performed elementwise.Broadcasting two tensors together follows these rules:1.   If the tensors do not have the same rank, prepend the shape of the lower rank tensor with 1's until both shapes have the same length.2.   The two tensors are said to be *compatible* in a dimension if they have the same size in the dimension, or if one of the tensors has size 1 in that dimension.3.   The tensors can be broadcast together if they are compatible in all dimensions.4.   After broadcasting, each tensor behaves as if it had shape equal to the elementwise maximum of shapes of the two input tensors.5.   In any dimension where one tensor had size 1 and the other tensor had size greater than 1, the first tensor behaves as if it were copied along that dimension.If this explanation does not make sense, try reading the explanation from the [documentation](https://pytorch.org/docs/1.7.0/notes/broadcasting.html).Not all functions support broadcasting. For example, [`torch.mm()`](https://pytorch.org/docs/1.7.0/generated/torch.mm.htmltorch.mm) does not support broadcasting, but [`torch.matmul()`](https://pytorch.org/docs/1.7.0/generated/torch.matmul.htmltorch.matmul) does.Broadcasting can let us easily implement many different operations. For example we can compute an outer product of vectors:<jupyter_code># Compute outer product of vectors
v = torch.tensor([1, 2, 3])  # v has shape (3,)
w = torch.tensor([4, 5])     # w has shape (2,)
# To compute an outer product, we first reshape v to be a column
# vector of shape (3, 1); we can then broadcast it against w to yield
# an output of shape (3, 2), which is the outer product of v and w:
print(v.view(3, 1) * w)<jupyter_output>tensor([[ 4,  5],
        [ 8, 10],
        [12, 15]])<jupyter_text>We can add a vector to each row of a matrix:<jupyter_code>x = torch.tensor([[1, 2, 3], [4, 5, 6]])  # x has shape (2, 3)
v = torch.tensor([1, 2, 3])               # v has shape (3,)
print('Here is the matrix:')
print(x)
print('\nHere is the vector:')
print(v)

# x has shape (2, 3) and v has shape (3,) so they broadcast to (2, 3),
# giving the following matrix:
print('\nAdd the vector to each row of the matrix:')
print(x + v)<jupyter_output>Here is the matrix:
tensor([[1, 2, 3],
        [4, 5, 6]])

Here is the vector:
tensor([1, 2, 3])

Add the vector to each row of the matrix:
tensor([[2, 4, 6],
        [5, 7, 9]])<jupyter_text>We can add a vector to each column of a matrix:<jupyter_code>x = torch.tensor([[1, 2, 3], [4, 5, 6]])  # x has shape (2, 3)
w = torch.tensor([4, 5])                  # w has shape (2,)
print('Here is the matrix:')
print(x)
print('\nHere is the vector:')
print(w)

# x has shape (2, 3) and w has shape (2,). We reshape w to (2, 1);
# then when we add the two the result broadcasts to (2, 3):
print('\nAdd the vector to each column of the matrix:')
print(x + w.view(-1, 1))

# Another solution is the following:
# 1. Transpose x so it has shape (3, 2)
# 2. Since w has shape (2,), adding will broadcast to (3, 2)
# 3. Transpose the result, resulting in a shape (2, 3)
print((x.t() + w).t())<jupyter_output>Here is the matrix:
tensor([[1, 2, 3],
        [4, 5, 6]])

Here is the vector:
tensor([4, 5])

Add the vector to each column of the matrix:
tensor([[ 5,  6,  7],
        [ 9, 10, 11]])
tensor([[ 5,  6,  7],
        [ 9, 10, 11]])<jupyter_text>Multiply a tensor by a set of constants:<jupyter_code>x = torch.tensor([[1, 2, 3], [4, 5, 6]])  # x has shape (2, 3)
c = torch.tensor([1, 10, 11, 100])        # c has shape (4)
print('Here is the matrix:')
print(x)
print('\nHere is the vector:')
print(c)

# We do the following:
# 1. Reshape c from (4,) to (4, 1, 1)
# 2. x has shape (2, 3). Since they have different ranks, when we multiply the
#    two, x behaves as if its shape were (1, 2, 3)
# 3. The result of the broadcast multiplication between tensors of shape
#    (4, 1, 1) and (1, 2, 3) has shape (4, 2, 3)
# 4. The result y has shape (4, 2, 3), and y[i] (shape (2, 3)) is equal to
#    c[i] * x
y = c.view(-1, 1, 1) * x
print('\nMultiply x by a set of constants:')
print(y)<jupyter_output>Here is the matrix:
tensor([[1, 2, 3],
        [4, 5, 6]])

Here is the vector:
tensor([  1,  10,  11, 100])

Multiply x by a set of constants:
tensor([[[  1,   2,   3],
         [  4,   5,   6]],

        [[ 10,  20,  30],
         [ 40,  50,  60]],

        [[ 11,  22,  33],
         [ 44,  55,  66]],

        [[100, 200, 300],
         [400, 500, 600]]])<jupyter_text>Your turn: Write a function that normalizes the columns of a matrix. It should compute the mean and standard deviation of each column, then subtract the mean and divide by the standard deviation for each element in the column.Example:```x = [[ 0,  30,  600],     [ 1,  10,  200],     [-1,  20,  400]]```- The first column has mean 0 and std 1.- The second column has mean 20 and std 10.- The third column has mean 400 and std 200.After normalizing the columns, the result should be:```y = [[ 0,  1,  1],     [ 1, -1, -1],     [-1,  0,  0]]```<jupyter_code>def normalize_columns(x):
  """
  Normalize the columns of a matrix by subtracting the mean and dividing by the
  (unbiased) standard deviation.

  Inputs:
  - x: Tensor of shape (N, M).

  Returns:
  - y: Tensor of shape (N, M) which is a copy of x with normalized columns.
  """
  y = x.clone()
  ##############################################################################
  # TODO: Complete the implementation of this function. Do not modify x.       #
  # Your implementation should not use any loops; instead you should use       #
  # reduction and broadcasting operations.                                     #
  ##############################################################################
  # Replace "pass" statement with your code
  m = x.mean(dim = 0)
  s = x.std(dim = 0)
  y = (y - m)/s
  # END OF YOUR CODE
  return y
x0 = torch.tensor([[0., 30., 600.], [1., 10., 200.], [-1., 20., 400.]])
y0 = normalize_columns(x0)
print('Here is x0:')
print(x0)
print('Here is y0:')
print(y0)<jupyter_output>Here is x0:
tensor([[  0.,  30., 600.],
        [  1.,  10., 200.],
        [ -1.,  20., 400.]])
Here is y0:
tensor([[ 0.,  1.,  1.],
        [ 1., -1., -1.],
        [-1.,  0.,  0.]])<jupyter_text>Running on GPU One of the most important features of PyTorch is that it can use graphics processing units (GPUs) to accelerate its tensor operations.You can enable GPUs in Colab via Runtime -> Change Runtime Type -> Hardware Accelerator -> GPU.This may cause the Colab runtime to restart, so we will re-import `torch` in the next cell.<jupyter_code>import torch

if torch.cuda.is_available:
  print('PyTorch can use GPUs!')
else:
  print('PyTorch cannot use GPUs.')<jupyter_output>PyTorch can use GPUs!<jupyter_text>We have already seen that PyTorch tensors have a [`dtype`](https://pytorch.org/docs/1.7.0/tensor_attributes.htmltorch-dtype) attribute specifying their data type. All PyTorch tensors also have a [`device`](https://pytorch.org/docs/1.7.0/tensor_attributes.htmltorch-device) attribute that specifies the device where the tensor is stored -- either CPU or CUDA (for NVIDIA GPUs). A tensor on a CUDA device will automatically use that device to accelerate all of its operations.Just as with data types, we can use the [`.to()`](https://pytorch.org/docs/1.7.0/tensors.htmltorch.Tensor.to) method to change the device of a tensor. We can also use the convenience methods [`.cuda()`](https://pytorch.org/docs/1.7.0/tensors.html?highlight=cudatorch.Tensor.cuda) and [`.cpu()`](https://pytorch.org/docs/1.7.0/tensors.html?highlight=cputorch.Tensor.cpu) methods to move tensors between CPU and GPU:<jupyter_code># Construct a tensor on the CPU
x0 = torch.tensor([[1, 2], [3, 4]], dtype=torch.float32)
print('x0 device:', x0.device)

# Move it to the GPU using .to()
x1 = x0.to('cuda')
print('x1 device:', x1.device)

# Move it to the GPU using .cuda()
x2 = x0.cuda()
print('x2 device:', x2.device)

# Move it back to the CPU using .to()
x3 = x1.to('cpu')
print('x3 device:', x3.device)

# Move it back to the CPU using .cpu()
x4 = x2.cpu()
print('x4 device:', x4.device)

# We can construct tensors directly on the GPU as well
y = torch.tensor([[1, 2, 3], [4, 5, 6]], dtype=torch.float64, device='cuda')
print('y device / dtype:', y.device, '/', y.dtype)

# Calling x.to(y) where y is a tensor will return a copy of x with the same
# device and dtype as y
x5 = x0.to(y)
print('x5 device / dtype:', x5.device, '/', x5.dtype)<jupyter_output>x0 device: cpu
x1 device: cuda:0
x2 device: cuda:0
x3 device: cpu
x4 device: cpu
y device / dtype: cuda:0 / torch.float64
x5 device / dtype: cuda:0 / torch.float64<jupyter_text>Performing large tensor operations on a GPU can be **a lot faster** than running the equivalent operation on CPU.Here we compare the speed of adding two tensors of shape (10000, 10000) on CPU and GPU.Note that GPU code may run asynchronously with CPU code, so when timing the speed of operations on the GPU it is important to use [`torch.cuda.synchronize()`](https://pytorch.org/docs/1.7.0/cuda.html?highlight=synchronizetorch.cuda.synchronize) to synchronize the CPU and GPU:<jupyter_code>import time

a_cpu = torch.randn(10000, 10000, dtype=torch.float32)
b_cpu = torch.randn(10000, 10000, dtype=torch.float32)

a_gpu = a_cpu.cuda()
b_gpu = b_cpu.cuda()
torch.cuda.synchronize()

t0 = time.time()
c_cpu = a_cpu + b_cpu
t1 = time.time()
c_gpu = a_gpu + b_gpu
torch.cuda.synchronize()
t2 = time.time()

# Check that they computed the same thing
diff = (c_gpu.cpu() - c_cpu).abs().max().item()
print('Max difference between c_gpu and c_cpu:', diff)

cpu_time = 1000.0 * (t1 - t0)
gpu_time = 1000.0 * (t2 - t1)
print('CPU time: %.2f ms' % cpu_time)
print('GPU time: %.2f ms' % gpu_time)
print('GPU speedup: %.2f x' % (cpu_time / gpu_time))<jupyter_output>Max difference between c_gpu and c_cpu: 0.0
CPU time: 256.21 ms
GPU time: 9.64 ms
GPU speedup: 26.59 x<jupyter_text>Your turn: Use the GPU to accelerate the following matrix multiplication operation. You should see 10~20x speedup by using the GPU. If your first run seems slow, run again.<jupyter_code>import time

x = torch.rand(512, 4096)
w = torch.rand(4096, 4096)

t0 = time.time()
y0 = x.mm(w)
t1 = time.time()

x_gpu = None
w_gpu = None
t2 = 0
y1 = None
t3 = 0
##############################################################################
# TODO: Write a bit of code that:                                            #
# - Moves x and w to the GPU (store them in x_gpu and w_gpu, respectively).  #
# - Performs matrix multiplication of x_gpu and w_gpu.                       #
# - Moves the result back to the CPU (store the result in y1).               #
#                                                                            #
# Properly set the start time t2 and end time t3 so that you only time the   #
# matrix multiplication on the GPU (i.e., excluding the time for moving the  #
# tensors between CPU and GPU). Use torch.cuda.synchronize() when            #
# appropriate.                                                               #
##############################################################################
# Replace "pass" statement with your code
x_gpu = x.cuda()
w_gpu = w.cuda()
torch.cuda.synchronize()

t2 = time.time()
y1_gpu = x_gpu.mm(w_gpu)
torch.cuda.synchronize()
t3 = time.time()

y1 = y1_gpu.cpu()
# END OF YOUR CODE
diff = (y0 - y1).abs().max().item()
print('Max difference between y0 and y1:', diff)
assert diff < 5e-3, 'Difference within tolerance'

cpu_time = 1000.0 * (t1 - t0)
gpu_time = 1000.0 * (t3 - t2)
print('CPU time: %.2f ms' % cpu_time)
print('GPU time: %.2f ms' % gpu_time)
print('GPU speedup: %.2f x' % (cpu_time / gpu_time))<jupyter_output>Max difference between y0 and y1: 0.000732421875
CPU time: 283.71 ms
GPU time: 8.72 ms
GPU speedup: 32.54 x<jupyter_text>More Practice Using what we've learned, let's implement a few common functions that you'll see in this course.<jupyter_code># We will use this helper function to check your results
def max_diff(actual, expected):
  return (actual - expected).abs().max().item()<jupyter_output><empty_output><jupyter_text>Sigmoid First, we'll implement the sigmoid function. It is defined as:$$ \text{sigmoid}(x) = \frac{1}{1 + e^{-x}}. $$This function outputs a value between 0 and 1, and is often used in binary classification. Implement an elementwise sigmoid function below.Given an input tensor `x`, your function should return a tensor of the same shape where the sigmoid function is applied to each element of `x`.You are NOT allowed to use PyTorch's built-in sigmoid functions.<jupyter_code>def sigmoid(x):
  """
  Elementwise sigmoid.

  Inputs:
  - x: Tensor of floats of any shape.

  Returns:
  - y: Tensor of the same shape as x where the sigmoid function is applied
       to each element of x.
  """
  y = None
  ##############################################################################
  # TODO: Complete the implementation of this function. Do not modify x.       #
  # Your implementation should not use any loops.                              #
  ##############################################################################
  # Replace "pass" statement with your code
  y = 1/(1 + torch.exp(-x))
  # END OF YOUR CODE
  return y
assert max_diff(sigmoid(torch.tensor([1.0])), torch.sigmoid(torch.tensor([1.0]))) < 1e-4
assert max_diff(sigmoid(torch.tensor([1.0, 2.0, 3.0])), torch.sigmoid(torch.tensor([1.0, 2.0, 3.0]))) < 1e-4<jupyter_output><empty_output><jupyter_text>MSE Next, let's implement the mean squared error (MSE) function. The MSE is defined as:$$ \text{MSE}(x, y) = \frac{1}{n} \sum_{i=0}^{n-1}(x_i - y_i)^2, $$where $x$ and $y$ are tensors of the same shape and $n$ is the total number of elements in each tensor.You are NOT allowed to use PyTorch's built-in MSE functions.<jupyter_code>def mse(x, y):
  """
  Mean squared error between the inputs x and y.

  Inputs:
  - x, y: Tensors of floats of the same shape.

  Returns:
  - mse: The scalar mean squared error between x and y.
  """
  mse = None
  ##############################################################################
  # TODO: Complete the implementation of this function. Do not modify x or y.  #
  # Your implementation should not use any loops.                              #
  ##############################################################################
  # Replace "pass" statement with your code
  mse = ((x - y)**2).sum()/torch.numel(x)
  # END OF YOUR CODE
  return mse
mse_x1 = torch.tensor([1.0, 2.0, 3.0])
mse_y1 = torch.tensor([4.0, 5.0, 6.0])
assert mse(mse_x1, mse_y1) == 9.0<jupyter_output><empty_output><jupyter_text>Softmax Lastly we'll implement the softmax function. The softmax function is often used to normalize a vector to a probability distribution since the outputs sum to 1 and each element of the output is between 0 and 1. For a given vector $x$, it is defined as:$$ \text{softmax}(x_{i}) = \frac{\exp(x_i)}{\sum_j \exp(x_j)}. $$Note that for each element, the denominator sums over all elements in the vector and is the same for each element.You are NOT allowed to use PyTorch's built-in softmax functions.<jupyter_code>def softmax(x):
  """
  Applies the softmax function to the input vector x.

  Inputs:
  - x: A vector of floats.

  Returns:
  - y: A vector of floats of the same size as x.
  """
  y = None
  ##############################################################################
  # TODO: Complete the implementation of this function. Do not modify x.       #
  # Your implementation should not use any loops.                              #
  ##############################################################################
  # Replace "pass" statement with your code
  y = x.exp()/x.exp().sum()
  # END OF YOUR CODE
  return y
softmax_x1 = torch.tensor([1.0, 1.0, 1.0, 1.0])
assert max_diff(softmax(softmax_x1), torch.tensor([0.25, 0.25, 0.25, 0.25])) == 0

softmax_x2 = torch.tensor([1.0, 2.0, 3.0])
assert max_diff(softmax(softmax_x2), torch.nn.functional.softmax(softmax_x2, dim=0)) < 1e-4<jupyter_output><empty_output><jupyter_text>Softmax - numerical stability Because of the exponentials in the above softmax definition, the intermediate terms $\exp(x_i)$ can be very large and thus numerically unstable. Therefore, it is important to use a normalization trick that solves this problem. Specifically, if we multiply the numerator and the denominator of the softmax expression by a positive constant $C$, we get:$$ \text{softmax}(x_{i}) = \frac{\exp(x_i)}{\sum_j \exp(x_j)} = \frac{C\exp(x_i)}{C\sum_j \exp(x_j)} = \frac{\exp(x_i + \log(C))}{\sum_j \exp(x_j + \log(C))}. $$We can choose $\log(C)$ arbitrarily. A common choice is to set $ \log(C) = -\max_k(x_k)$. This effectively shifts all the elements in $x$ to be less than or equal to 0, leading to numerically stable computations. The resulting formula is:$$ \text{softmax}(x_{i}) = \frac{\exp(x_i -\max_k(x_k))}{\sum_j \exp(x_j -\max_k(x_k))}. $$Note that $ \log(C) = -\max_k(x_k) $ only needs to be calculated once for the entire vector.<jupyter_code>def softmax_stable(x):
  """
  Applies the softmax function to the input vector x.
  Uses the normalization trick for numerical stable computation.

  Inputs:
  - x: A vector of floats.

  Returns:
  - y: A vector of floats of the same size as x.
  """
  y = None
  ##############################################################################
  # TODO: Complete the implementation of this function. Do not modify x.       #
  # Your implementation should not use any loops.                              #
  ##############################################################################
  # Replace "pass" statement with your code
  cx = x - x.max()
  y = cx.exp()/cx.exp().sum()
  # END OF YOUR CODE
  return y
# Previous implementation may lead to NaNs for larger inputs:
print(softmax(torch.tensor([100.0, 100.0, 100.0])))

# The numerically stable version should solve this problem:
print(softmax_stable(torch.tensor([100.0, 100.0, 100.0])))
softmax_x1 = torch.tensor([1.0, 1.0, 1.0, 1.0])
assert max_diff(softmax_stable(softmax_x1), torch.tensor([0.25, 0.25, 0.25, 0.25])) == 0

softmax_x2 = torch.tensor([1.0, 2.0, 3.0])
assert max_diff(softmax_stable(softmax_x2), torch.nn.functional.softmax(softmax_x2, dim=0)) < 1e-4

softmax_x3 = torch.tensor([100.0, 200.0, 300.0])
assert max_diff(softmax_stable(softmax_x3), torch.nn.functional.softmax(softmax_x3, dim=0)) < 1e-4<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.18313972523802663, 'fraction_numerical': 0.04967455505061749, 'mean_word_length': 3.7201582493663845, 'pattern_counts': {'":': 0, '<': 199, '<?xml version=': 0, '>': 195, 'https://': 54, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 55, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '58', 'dir': 'jupyter-structured-clean-dedup', 'id': '268579', 'license': 'Apache-2.0', 'n_tokens_mistral': 27827, 'n_tokens_neox': 25442, 'n_words': 9497, 'path': 'MyScratch/OtherSources/ys688.ipynb', 'repo_name': 'samjz06/d2l-pytorch', 'size': '76362'}