<jupyter_start><jupyter_text>PyTorch Playground In this code tutorial we will learn the PyTorch's basic tensor manipulation. The code in this tutorial is borrowd from the great <NAME>. Follow him here: https://github.com/leriomaggio  What is PyTorch-  A library for Tensor manipulation;-  a deep learning framework that provides maximum flexibility and speed  PyTorch BasicsOne of the **striking feature** of PyTorch is its *natural* integration with **NumPy** which aided a lot the adoption among researchers.Indeed, learning `PyTorch APIs` is by far easier than learning the API of other frameworks (e.g. `TensorFlow`)[$^1$](fn1)`NumPy` integration + **Dynamic Graph** computation makes `torch` a great tool for researcher and practitioners.Actually there are people who spent so much time and effort to learn `TF 1.x` API so that they are very reluctuant to even updated to `TF 2.x`<jupyter_code>import torch
import numpy as np

np.random.seed(42)

torch.manual_seed(7)<jupyter_output><empty_output><jupyter_text>Tensors**Tensors** are the **main** data structure supported by PyTorch (*everything is built around `torch.Tensor`*) * Scalar is a single number.* Vector is an array of numbers.* Matrix is a 2-D array of numbers.* Tensors are N-D arrays of numbers.  Creating Tensors You can create tensors by specifying the shape as arguments.  Here is a tensor with `6` rows and `4` columns<jupyter_code>t = torch.Tensor(6, 4)
t.shape  # shape of tensor
t.type()  # dtype of tensor
t[2:4, :]  #slicing<jupyter_output><empty_output><jupyter_text>It's common in prototyping to create a tensor with random numbers of a specific shape.<jupyter_code>x = torch.rand(2, 3)
x.shape<jupyter_output><empty_output><jupyter_text>You can also initialize tensors of ones or zeros.<jupyter_code>zeros = torch.zeros(4, 4)
ones = torch.ones(4, 4)<jupyter_output><empty_output><jupyter_text>Tensors can be initialized and then filled in place. Note: operations that **end in an underscore** (`_`) are in place operations by convention<jupyter_code>x = torch.Tensor(3,4).fill_(5)
print(x.type())
print(x.shape)
print(x)<jupyter_output>torch.FloatTensor
torch.Size([3, 4])
tensor([[5., 5., 5., 5.],
        [5., 5., 5., 5.],
        [5., 5., 5., 5.]])<jupyter_text>`torch.Tensor` from `list` and `np.ndarray` Tensors can be initialized from a list of lists (for 2D tensor)<jupyter_code>x = torch.Tensor([[1, 2,],  
                  [3, 4,],
                  [5, 6,]
                 ])
print(x.type())
x<jupyter_output>torch.FloatTensor<jupyter_text>... or `np.array`<jupyter_code>npy = np.random.rand(4, 4)
torch.from_numpy(npy)<jupyter_output><empty_output><jupyter_text>Tensor Types The `FloatTensor` is the **default** tensor type.However, we can control the type of **tensors** by explicit casting, or specialised constructors.<jupyter_code>x = torch.FloatTensor([[1, 2, 3],  
                       [4, 5, 6]])
# converts to long
x = x.long()
x.type()
# Specify `dtype` in the constructor
x = torch.tensor([[1, 2, 3], 
                  [4, 5, 6]], dtype=torch.int64)
x.type()
x.float().type()
x.type()<jupyter_output><empty_output><jupyter_text>**What happened??** Method like `.float()` or `.long()` return a new copy of the Tensor.  ExcerciseTry to create a long tensor from a `random` NumPy array and then cast it as `float`<jupyter_code># Your code here<jupyter_output><empty_output><jupyter_text>Element-wise Operations<jupyter_code>x = torch.rand(4, 5)
x
x + x
x * 2<jupyter_output><empty_output><jupyter_text>The convention of `_` indicating in-place operations continues:<jupyter_code>x = torch.arange(12).reshape(3, 4)
print(x)
print(x.add_(x))<jupyter_output>tensor([[ 0,  1,  2,  3],
        [ 4,  5,  6,  7],
        [ 8,  9, 10, 11]])
tensor([[ 0,  2,  4,  6],
        [ 8, 10, 12, 14],
        [16, 18, 20, 22]])<jupyter_text>`arange` and `reshape`<jupyter_code>x = torch.arange(6)
x
x = x.view(2, 3)
x<jupyter_output><empty_output><jupyter_text>**Q**: What is the equavalent of `torch.view` in `numpy`?<jupyter_code>x = torch.arange(6).view(2, 3)

x
torch.sum(x, dim=0)  # over axis = 0
torch.sum(x, dim=1)  # over axis = 1 (column-wise)
# np.swapaxes

torch.transpose(x, 0, 1)<jupyter_output><empty_output><jupyter_text>Long Tensors are used for indexing operations and mirror the `int64` numpy type<jupyter_code>x = torch.LongTensor([[1, 2, 3],  
                      [4, 5, 6],
                      [7, 8, 9]])
print(x.dtype)
print(x.numpy().dtype)<jupyter_output>torch.int64
int64<jupyter_text>You can convert a FloatTensor to a LongTensor<jupyter_code>x = torch.FloatTensor([[1, 2, 3],  
                       [4, 5, 6],
                       [7, 8, 9]])
print(x.dtype)
print(x.numpy().dtype)

x = x.long()
print(x.dtype)
print(x.numpy().dtype)<jupyter_output>torch.float32
float32
torch.int64
int64<jupyter_text>Special Tensor initializations We can create a vector of incremental numbers<jupyter_code>x = torch.arange(0, 10)
print(x)<jupyter_output>tensor([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])<jupyter_text>Sometimes it's useful to have an integer-based arange for indexing<jupyter_code>x = torch.arange(0, 10).long()
print(x)<jupyter_output>tensor([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])<jupyter_text>OperationsUsing the tensors to do linear algebra is a foundation of modern Deep Learning practices **Reshaping** allows you to move the numbers in a tensor around.  In PyTorch, reshaping is called `view`:(*from the documentation: *)> Returns a new tensor with the same data as the self tensor but of a different shape.>  The returned tensor **shares the same** data and must have the same number of elements, but may have a different size. For a tensor to be viewed, the new view size must be compatible with its original size and stride We can use view to add size-1 dimensions, which can be useful for combining with other tensors.  **This is called broadcasting.**<jupyter_code>x = torch.arange(0, 20)

print(x.view(1, 20))
print(x.view(2, 10))
print(x.view(4, 5))
print(x.view(5, 4))
print(x.view(10, 2))
print(x.view(20, 1))
x = torch.arange(12).view(3, 4)
y = torch.arange(4).view(1, 4)
z = torch.arange(3).view(3, 1)

print(x)
print(y)
print(z)
print(x + y)
print(x + z)<jupyter_output>tensor([[ 0,  1,  2,  3],
        [ 4,  5,  6,  7],
        [ 8,  9, 10, 11]])
tensor([[0, 1, 2, 3]])
tensor([[0],
        [1],
        [2]])
tensor([[ 0,  2,  4,  6],
        [ 4,  6,  8, 10],
        [ 8, 10, 12, 14]])
tensor([[ 0,  1,  2,  3],
        [ 5,  6,  7,  8],
        [10, 11, 12, 13]])<jupyter_text>Unsqueeze and squeeze will add and remove 1-dimensions.<jupyter_code>x = torch.arange(12).view(3, 4)
print(x.shape)

x = x.unsqueeze(dim=1)
print(x.shape)

x = x.squeeze()
print(x.shape)<jupyter_output>torch.Size([3, 4])
torch.Size([3, 1, 4])
torch.Size([3, 4])<jupyter_text>Swapping AxesAnother very common operation to perform requires `swapping` axes when analysing images, due to different internal representation between `numpy` and `torch`.For instance, for a 3D tensor representing a simple `RGB` (three channels) image:`torch` $\mapsto$ (`channel`, `row`, `col`);`numpy` $\mapsto$ (`row`, `col`, `channel`). Transposing allows you to switch the dimensions to be on different axis. So we can make it so all the rows are columsn and vice versa.<jupyter_code>x = torch.arange(2352).view(28, 28, 3)
print("x: \n", x.shape) 
print("tranpose: \n", x.transpose(2, 0).shape)
print("x - again: \n", x.shape)<jupyter_output>x: 
 torch.Size([28, 28, 3])
tranpose: 
 torch.Size([3, 28, 28])
x - again: 
 torch.Size([28, 28, 3])<jupyter_text>Understanding Dimensions in `Tensor` A `3D` tensor would represent a `batch` of sequences, where each sequence item has a feature vector.  It is common to switch the batch and sequence dimensions so that we can more easily index the sequence in a sequence model.Note: `transpose` will only let you swap `2` axes.  `permute` allows for multiple axes:<jupyter_code>batch_size = 3
seq_size = 28
feature_size = 28

x = torch.arange(batch_size * seq_size * feature_size).view(batch_size, seq_size, feature_size)<jupyter_output><empty_output><jupyter_text>Permute is a more general version of tranpose:<jupyter_code>print("x.permute(1, 0, 2).shape: \n", x.permute(1, 0, 2).shape)
x = torch.arange(0, 12).view(3,4).float()
print(x)

x2 = torch.ones(4, 2)
x2[:, 1] += 1
print(x2)

print(x.mm(x2))<jupyter_output>tensor([[ 0.,  1.,  2.,  3.],
        [ 4.,  5.,  6.,  7.],
        [ 8.,  9., 10., 11.]])
tensor([[1., 2.],
        [1., 2.],
        [1., 2.],
        [1., 2.]])
tensor([[ 6., 12.],
        [22., 44.],
        [38., 76.]])<jupyter_text>See the [PyTorch Math Operations Documentation](https://pytorch.org/docs/stable/torch.htmlmath-operations) for more!  Computing GradientsCentral to Tensors is the *free* computation of **gradients** - **when required**!To specify that a `Tensor` requires `gradient`, the option `requires_grad` has to be provided.<jupyter_code>x = torch.tensor([[2.0, 3.0]], requires_grad=True)
z = 3 * x
print(z)<jupyter_output>tensor([[6., 9.]], grad_fn=<MulBackward0>)<jupyter_text>In the **next** example, we are going to slightly complicate a bit more the (**graph**) of operations..1. We create a tensor and multiply it by `3`.  2. We create a scalar output using `sum()`.  A Scalar output is needed as the the loss variable. Then,3. We call `backward` on the loss means it computes its rate of change with res`pect to the inputs.  Since the scalar was created with `sum`, each position in `z` and `x` are independent with respect to the loss scalar. The rate of change of `x` with respect to the output is just the constant `3` that we multiplied `x` by.<jupyter_code>x = torch.tensor([[2.0, 3.0]], requires_grad=True)
print("x: \n", x)
print("---")
z = 3 * x
print("z = 3*x: \n", z)
print("---")

loss = z.sum()
print("loss = z.sum(): \n", loss)
print("---")

loss.backward()

print("after loss.backward(), x.grad: \n", x.grad)<jupyter_output>x: 
 tensor([[2., 3.]], requires_grad=True)
---
z = 3*x: 
 tensor([[6., 9.]], grad_fn=<MulBackward0>)
---
loss = z.sum(): 
 tensor(15., grad_fn=<SumBackward0>)
---
after loss.backward(), x.grad: 
 tensor([[3., 3.]])<jupyter_text>CUDA Tensors PyTorch's operations can seamlessly be used on the GPU or on the CPU.  There are a couple basic operations for interacting in this way.<jupyter_code>print(torch.cuda.is_available())  # should be True if executed on Colab
x = torch.rand(3,3)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(device)
x = torch.rand(3, 3).to(device)
print(x.device)
cpu_device = torch.device("cpu")
# this will break if X is on GPU!
y = torch.rand(3, 3)
x + y
y = y.to(device)
x = x.to(device)
x + y
if torch.cuda.is_available(): # only is GPU is available
    a = torch.rand(3,3).to(device='cuda:0') #  CUDA Tensor
    print(a)
    
    b = torch.rand(3,3).cuda()
    print(b)

    print(a + b)

    a = a.cpu() # Error expected
    print(a + b)<jupyter_output>tensor([[0.6286, 0.7653, 0.1132],
        [0.8559, 0.6721, 0.6267],
        [0.5691, 0.7437, 0.9592]], device='cuda:0')
tensor([[0.3887, 0.2214, 0.3742],
        [0.1953, 0.7405, 0.2529],
        [0.2332, 0.9314, 0.9575]], device='cuda:0')
tensor([[1.0173, 0.9867, 0.4874],
        [1.0512, 1.4126, 0.8795],
        [0.8022, 1.6751, 1.9168]], device='cuda:0')<jupyter_text>Exercise 1Create a 2D tensor and then add a dimension of size 1 inserted at the 0th axis.<jupyter_code>a = torch.rand(3,3)
a = a.unsqueeze(0)
print(a)
print(a.shape)<jupyter_output>tensor([[[0.4703, 0.1049, 0.5137],
         [0.2674, 0.4990, 0.7447],
         [0.7213, 0.4414, 0.5550]]])
torch.Size([1, 3, 3])
{'fraction_non_alphanumeric': 0.15417061203408797, 'fraction_numerical': 0.050443315830248775, 'mean_word_length': 4.040347071583514, 'pattern_counts': {'":': 0, '<': 94, '<?xml version=': 0, '>': 96, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '26', 'dir': 'jupyter-structured-clean-dedup', 'id': '229520', 'license': 'Apache-2.0', 'n_tokens_mistral': 4651, 'n_tokens_neox': 4273, 'n_words': 1313, 'path': 'tutorial_pytorch/1_torch_playground.ipynb', 'repo_name': 'franchg/intro_deep_learning', 'size': '11625'}