<jupyter_start><jupyter_text>PyTorch tutorialsTensors are fundamental data structures for deep learning.Indices required | Computer Science | Mathematics--- | --- | ---0 | number | scalar1 | array | vector2 | 2D array | matrixany dimension greater than 2 called n-dimension tensor / nd-tensor. 3D array called, 3D-tensor, etc<jupyter_code>import torch
import numpy as np<jupyter_output><empty_output><jupyter_text>Accessing tensors with indices<jupyter_code>a = [1,2,3,4]
a[2]
dd = [[1,2,3],
     [4,5,6],
     [7,8,9]]
dd[0][1]<jupyter_output><empty_output><jupyter_text>Rank, Axes and ShapeBuild up concept of tensors.1. Rank of a tensor indicates  of dimensions present within a tensor. In short it tells how many indices are required to access the tensor.2. Rank =  of axes in the tensor 3. Shape of the tensor is determined by the shape of each axis. (rows, columns, channels/depth)<jupyter_code>a = [[1,2,3],
    [4,5,6],
    [7,8,9]]
a[0]
a[1]
a[2]
a[0][0]
a[1][0]
a[2][0]
dd = [[[111, 112, 113], [121, 122, 123]],
     [[211, 212, 213], [221, 222, 223]],
     [[311, 312, 313], [321, 322, 323]]]
dd
t = torch.tensor(dd)
t
t.shape  # size and shape of tensor is the same
type(t)
t[0][0][2]
t[1][1][0]
t[2][0][1]<jupyter_output><empty_output><jupyter_text>Tensor ReshapingChanges the shape but not the underlying data.<jupyter_code>a = torch.tensor(a)
a
a.shape
a.reshape(1,9)
a.reshape(1,9).shape<jupyter_output><empty_output><jupyter_text>CNN TensorsShape of CNN tensor typically has length of 4. i.e we have rank 4 tensor with 4 axis. last axis is where data values are locateda = [a0,a1,a2,a3]image data usually represented as:[ of data samples,  channels, height, width], hence a 4D-tensor.1. First axis: total samples / batch size of samples.2. Second axis: color channels or depth of image3. Third axis: height / rows in an image4. Fourth axis: width, columns in an imageAs image passes through the network, its shape changes. That means, the number of channels will increase of decrease based on number of filters present in the layer, while the height and width will change too, based on the convolution operations being performed in each layer.When the convolution operation is performed, the output of each convolutional layer are modified images. The number of these images corresponds directly to the number of filters present in these layers. These are called **Feature Maps**. These feature maps are created due to convolution between the input color channels and the convolution filters.  PyTorch TensorsApart from Rank, Axes, and Shape which are the fundamental attributes for all the tensors for all the deep learning frameworks, the attributes which are specific to PyTorch tensors are datatype, device and layout.<jupyter_code>t = torch.Tensor()
t  # we can create an empty tensor
print(t.dtype)
print(t.device)
print(t.layout)<jupyter_output>torch.float32
cpu
torch.strided<jupyter_text>PyTorch attributes are:1. DataType: specifies type of data contained in a tensor. Each data has CPU and GPU datatype. Tensor operations between tensors must happen with tensors of the same datatype.2. Device: indicates where the tensor computations will be performed. Whether CPU or GPU. 3. Layout: strided (tells us way in which data is laid out in memory). It is the default layout and not usually changed.Tensors contain data of a uniform type. Tensor computations depends on the type and device.<jupyter_code>t1 = torch.tensor([1,2,3])
t2 = torch.tensor([1.,2.,3.,4.])
t1.dtype
t2.dtype
t1 + t2   # will fail due to non compatible datatypes
t1.device
t2 = t1.cuda()
t2.device
t2
t1
t1 + t2   # if tensor there on GPU and CPU then can't be added since there on different device<jupyter_output><empty_output><jupyter_text>Creating Tensors from data4 primary ways of creating data objects in PyTorch1. torch.Tensor(data)  - odd ball, since it is class Constructor.below three may look similar in output but work differently internally2. torch.tensor(data)3. torch.as_tensor(data)4. torch.from_numpy(data)<jupyter_code>data = np.array([1,2,3])
type(data)
torch.Tensor(data)    # creates float value array<jupyter_output><empty_output><jupyter_text>lower case tensor is factory function which builds tensor objects, while capital case Tensor is Tensor constructor calling<jupyter_code>torch.tensor(data) 
torch.as_tensor(data)
torch.from_numpy(data)
torch.eye(2)
torch.zeros(2,2)
torch.ones(2,2)
torch.rand(2,2)<jupyter_output><empty_output><jupyter_text>Creating PyTorch Tensors for Deep LearningPyTorch tensors are instances of torch.Tensor class. As discussed above, the torch.Tensor is a Constructor which helps us return object of the Tensor class. while the other 3 methods are factory methods. Factory methods are the notation of object oriented programming, in which they take in parameters and returns the instance of the class.<jupyter_code>data = np.array([1,2,3])
type(data)
t1 = torch.Tensor(data)  # constructor method
t2 = torch.tensor(data)   # factory method
t3 = torch.as_tensor(data)  # factory method
t4 = torch.from_numpy(data)  # factory method
print(t1)
print(t2)
print(t3)
print(t4)
print(t1.dtype)
print(t2.dtype)
print(t3.dtype)
print(t4.dtype)
torch.get_default_dtype()<jupyter_output><empty_output><jupyter_text>As can be seen above, the default datatype for the tensor is float32 which is assigned irrespective of the type of the data which is passed to it. the above line proves that the global default datatype of data for the data in PyTorch is **float32**.Factory functions on the other hand choose dtype based on the inputs which are provided. example shown below. Below 2 lines show automatic type inference based on the data provided to it.<jupyter_code>torch.tensor(np.array([1,2,3]))
torch.tensor(np.array([1.,2.,3.]))<jupyter_output><empty_output><jupyter_text>data types can be explicitly set as shown below.<jupyter_code>torch.tensor(np.array([1,2,3]), dtype=torch.float64)<jupyter_output><empty_output><jupyter_text>constructor however does not have type casting functionality. there is no type inference. it can be shown below<jupyter_code>t = np.array([1,2,3], dtype=np.float64)
t.dtype
t
t5 = torch.Tensor(t)
t5.dtype
t5 = torch.Tensor(t, dtype=torch.float64)  # unable to typecast since a Constructor<jupyter_output><empty_output><jupyter_text>Memory sharing vs copying<jupyter_code>data = np.array([1,2,3])
data
t1 = torch.Tensor(data)  # constructor method
t2 = torch.tensor(data)   # factory method
t3 = torch.as_tensor(data)  # factory method
t4 = torch.from_numpy(data)  # factory method
data[0] = 0
data[1] = 0
data[2] = 0
# contain the original data values as given in the array
print(t1)  
print(t2)
print(t3)
print(t4)<jupyter_output>tensor([0, 0, 0])
tensor([0, 0, 0])<jupyter_text>From above, t1 and t2 show the same values as before. t3 and t4 contains data after the change, i.e it mirrors the data. This has to do with how memory is allocated by the tensors.t1 and t2 create additional copies of input data in memory, i.e they copy the data into memory, while t3 and t4, share the data in memory with the numpy array. This has to do with sharing of memory for performance.i.etorch.as_tensor() and torch.from_numpy() share data between memory and the tensor, whiletorch.tensor() and torch.Tensor() copy data from memory to tensor. Best option usually is to use torch.tensor() function for converting numpy data to tensor objects. Second option, if we would like to have performance boost, is to use torch.as_tensor()Things to remember:1. torch.as_tensor() does not work with built in Python data structures.2. since numpy.ndarray allocate memory on the CPU, the as_tensor() must copy data from CPU to GPU.3. as_tensor() performance improvement will be greater if there are lots of back and forth operations between numpy.ndarray and tensor objects.  Flatten, Reshape and SqueezeTensor operations usually fall into 4 higher level operations:1. Reshaping operations2. Element-wise operations3. Reduction operations4. Access operationsReshaping are one of the most important type of tensor operations<jupyter_code>t = torch.tensor([[1,1,1,1],
                 [2,2,2,2],
                 [3,3,3,3]], dtype=torch.float32)<jupyter_output><empty_output><jupyter_text>For the above tensor t, elements in the first axis are vectors, while those in the second axis are numbers.Axis 1 = rows / height of image / x Axis 2 = columns / width of image / y Axis 3 = depth /  of channels in an image / z<jupyter_code># rank 2 tensor with 2 axes with shape (3,4), 2 ways to access it as below
t.shape
t.size()
len(t.shape)   # gives rank of the tensor
torch.tensor(t.shape).prod()   # total 12 elements in this tensor
t.numel() # returns number of elements in the tensor.<jupyter_output><empty_output><jupyter_text>Reshaping a Tensor<jupyter_code>t1 = t.reshape(1,12)
print(t1.shape)
print(len(t1))
t2 = t.reshape(2,6)
print(t2.shape)
print(len(t2))
t3 = t.reshape(4,3)
print(t3.shape)
print(len(t3))
t4 = t.reshape(12,1)
print(t4.shape)
print(len(t4))<jupyter_output>torch.Size([12, 1])
12<jupyter_text>Reshape operations do not affect rank of a tensor if reshape has 2 factors of the total number of elements. In the above example, the tensor had 12 elements in total. Therefore, 2 element factors of 12 are:1. (1,12) = 1 x 122. (2,6) = 2 x 63. (3,4) = 3 x 44. (4,3) = 4 x 35. (6,2) = 6 x 26. (12,1) = 12 x 1These are the possible reshape combinations for a tensor having total elements 12 in it with 2 factor elements. However, we can have 3 factors as well, in which case the rank of the tensor will change: One such example is: 2 x 2 x 3 = (2,2,3) = 12 elements in total. There can be more combinations as well.<jupyter_code>t5 = t.reshape(2,2,3)
t5
t5.shape
len(t5.shape)<jupyter_output><empty_output><jupyter_text>Squeezing a TensorSqueezing drops the axis with a length of 1. While unsqueezing adds an extra axis to the tensor. Below, we can see that initially the tensor had a shape of (1,12); on squeezing it drops the axis with length 1. But on Unsqueezing it adds an axis along the dimension we specify, in this case for dimension 0.<jupyter_code>print(t.reshape(1,12))
print(t.reshape(1,12).shape)
print(t.reshape(1,12).squeeze())
print(t.reshape(1,12).squeeze().shape)
print(t.reshape(1,12).squeeze().unsqueeze(dim=0))
print(t.reshape(1,12).squeeze().unsqueeze(dim=0).shape)<jupyter_output>tensor([[1., 1., 1., 1., 2., 2., 2., 2., 3., 3., 3., 3.]])
torch.Size([1, 12])<jupyter_text>Flattening a TensorSqueezing helps us expand or shrink a tensor. A common usecase for squeezing is best described with a Flatten function. Flattening a tensor essentially means creating a 1D array from all the elements from the tensor. All the other axis except one with length 1 is kept. It helps us reduce the rank of the tensor. Flattening helps in converting a 2D convolutional layer into a 1D fully connected layer at the end stage of the CNN pipeline. For example a 28 x 28 image is flattened into a (784,) array before passing it to the fully connected layer.<jupyter_code>def flatten(t):
    t = t.reshape(1,-1)
    t = t.squeeze()
    return t
flatten(t)
t.reshape(1,12)<jupyter_output><empty_output><jupyter_text>From above, we can observe that, reshape retains the additional axis, while flattening (essentially squeezing removes it). Notice that in reshape, we used (1,-1). This tell PyTorch that the first dimension of tensor will be of length 1 while the second dimension should be inferred on its own, based on the number of elements present in the tensor. -1 indicates self inference of the dimension value. Here we get a (1,12) tensor. If we want automatic inference of rows and just have 1 column, we do the opposite. It can be seen below<jupyter_code>t.reshape(-1,1)<jupyter_output><empty_output><jupyter_text>However, if you want to achieve flattening without squeezing, we can exploit this functionality of -1 indexing. Flatten function above gives us shape as length of the tensor, which is 12. We can do it in 1 more way as below.<jupyter_code>flatten(t)
flatten(t).shape
t.reshape(-1,)
t.reshape(-1,).shape<jupyter_output><empty_output><jupyter_text>Concatenating TensorsConcatenating tensors is possible only when width or height of the tensor matches, not to forget depth too, if that is involved. For instance, two 2x2 tensors can concatenate to be a (4,2) or a (2,4) tensor, since the height and width of both the tensors are one and the same. However, if we have a (2,4) and a (2x6) tensor, we can only concatenate along the columns (axis=1). Concatenating along an axis simply means increasing the length of that axis from its previous length. Here concatenation will increase num of columns. If we try to concatenate the rows, it will throw an error as below due to mismatching column numbers.<jupyter_code>a = torch.rand(2,4)
a
b = torch.rand(2,6)
b
c = torch.cat((a,b), dim=1)
c
c.shape
c = torch.cat((a,b), dim=0)
c
a = torch.rand(2,2)
a
b = torch.rand(2,2)
b
# since both are 2x2 concatenating in either dimension will yield a new tensor.
c = torch.cat((a,b), dim=0)
d = torch.cat((a,b), dim=1) 
print(c)
print(d)
print(c.shape)
print(d.shape)<jupyter_output>torch.Size([4, 2])
torch.Size([2, 4])<jupyter_text>CNN Tensor Flatten operation ExplainedIn CNN we need to flatten the convolutional layers. But, when we do so, we don't do on a single image or a tensor as explained above, but on a batch of images. Images can be 3 channel if color or 1 channel if grayscale. When we flatten a batch of images in a convolutional layer, we essentially want to have a 1D tensor for each image, but would like to have number of samples in each batch as well Consider an example. Say we have a batch of 3 images. Each image is grayscale and is of shape (4,4). The resulting batch tensor shape will be (3,4,4). If we have a single channel then it will be (3,1,4,4) or if we have color images then (3,3,4,4). In either case, our batch tensor will be of form, (batch size, num_channels, height, width) format. On flattening a CNN tensor, we essentially want (batch_size,  of elements in flattened grayscale / color images) format.<jupyter_code>t1 = torch.tensor([
    [1,1,1,1],
    [1,1,1,1],
    [1,1,1,1],
    [1,1,1,1]
])

t2 = torch.tensor([
    [2,2,2,2],
    [2,2,2,2],
    [2,2,2,2],
    [2,2,2,2]
])

t3 = torch.tensor([
    [3,3,3,3],
    [3,3,3,3],
    [3,3,3,3],
    [3,3,3,3]
])<jupyter_output><empty_output><jupyter_text>We need to combine above tensors into a batch. We stack them vertically.<jupyter_code>t = torch.stack((t1,t2,t3), axis=0)
t
t.shape<jupyter_output><empty_output><jupyter_text>**NOTE: Numpy arrays or images are in (height, width, channels) format while PyTorch represents images in (channels, height, width) format.**Adding an additional axis as shown below does not change the total element count in the tensor.<jupyter_code>t = t.reshape(3,1,4,4)
t<jupyter_output><empty_output><jupyter_text>Let's index the above tensor. We do 4 things.1. We first access one image2. We then access the channel of that image3. We then access a row of the image in that channel4. We lastly access a pixel value in the row.<jupyter_code>t[0]   # first image
t[0][0]  # channel 1 consisting of an image
t[0][0][0]   # first row
t[0][0][0][0]  # first pixel value<jupyter_output><empty_output><jupyter_text>There are different ways to flatten an image tensor. Few are explained below:<jupyter_code>t.reshape(1,-1)[0]
t.reshape(-1)
t.view(t.numel())
t.flatten()   # provided by PyTorch<jupyter_output><empty_output><jupyter_text>We need not flatten all the images into a single long vector without retaining the batch shape. Instead we need to retain the batch size but just flatten individual images. We essentially want (batch_size,  of elements in flattened grayscale / color images) format as discussed earlier.<jupyter_code>t.flatten(start_dim=1)  # built in flatten method specifying from which dimension to flatten the tensor
t.flatten(start_dim=1).shape<jupyter_output><empty_output><jupyter_text>We can observe that, we want to retain the shape 0, i.e the batch_size but flatten everything beyond that (channels, height and width) into a single 1D tensor. For our example, we get 3 images flattened into a 16 element 1D-tensor.<jupyter_code>t.reshape(t.shape[0],-1)    # same operation performed using reshape operation<jupyter_output><empty_output><jupyter_text>Broadcasting and Element-wise operationsOperations that operate on tensor elements with same index locations.<jupyter_code>t1 = torch.tensor([
    [1,2],
    [3,4]
], dtype=torch.float32)

t2 = torch.tensor([
    [9,8],
    [7,6]
], dtype=torch.float32)
t1
# elements in the same indices, i.e corresponding elements in both t1 and t2
print(t1[0][0])
print(t2[0][0])<jupyter_output>tensor(1.)
tensor(9.)<jupyter_text>Two tensors must have same shape in order to perform elementwise operations. Having same shape essentially means having same length of elements in either of the x, y or z axis; or any other axes in that case<jupyter_code># element wise addition
t1 + t2<jupyter_output><empty_output><jupyter_text>Scalar values are rank 0 tensor since they have no shape. As a result, we can add them without the above criteria being satisfied of 2 tensors being of same rank. Any rank greater than 0 needs to be same for elementwise operation to succeed; scalar operations are special case like below<jupyter_code>t1 + 2
t1.add(2)
t1 - 2
t1.sub(2)
t2 * 2
t2.mul(2)
t2 / 2
t2.div(2)<jupyter_output><empty_output><jupyter_text>One might wonder that the scalar value of rank 0 is operated iteratively on all tensor elements of other tensor so as to succeed in achieving above results. Although, may sound true, it uses concept of **Broadcasting**. Broadcasting is a way in which tensors of different shapes interact to have element wise operations. Essentially, a scalar value is broadcasted to a rank / shape as same as the other tensor and then the element wise operation is performed internally.<jupyter_code>np.broadcast_to(2, t1.shape)
t1 + 2
t1 + torch.tensor(np.broadcast_to(2, t1.shape), dtype=torch.float32)   # result same as above<jupyter_output><empty_output><jupyter_text>Lets take another example. Here, t2 will be broadcasted to match higher rank tensor t1 before any further operations<jupyter_code>t1 = torch.tensor([
    [1,1],
    [1,1]
], dtype=torch.float32)

t2 = torch.tensor([2,4], dtype=torch.float32)
t1.shape
t2.shape
np.broadcast_to(t2.numpy(), t1.shape)
t1 + t2<jupyter_output><empty_output><jupyter_text>Comparison operationThis too happens element wise and outputs a True/False vector based on element wise comparsion<jupyter_code>t = torch.tensor([
    [0,5,7],
    [6,0,7],
    [0,8,0]
], dtype=torch.float32)
t.eq(0)   # elements equal to 0
t.ge(0)  # elements greater than or equal to 0
t.gt(0)  # elements greater than 0
t.lt(0)  # elements less than 0
t.le(7) # elements less than and equal to 0<jupyter_output><empty_output><jupyter_text>Above operation is essentially, as below<jupyter_code>t <= torch.tensor(np.broadcast_to(7, t.shape), dtype=torch.float32)<jupyter_output><empty_output><jupyter_text>Other element wise operations also have similar functionality. This includes built-in functions which operate on each element<jupyter_code>t.abs()
t.sqrt()
t.neg()
t.neg().abs()<jupyter_output><empty_output><jupyter_text>ArgMax and Reduction OperationsReduction operation on a tensor is one which reduces number of elements present in the tensor<jupyter_code>t = torch.tensor([
    [0,1,0],
    [2,0,2],
    [0,3,0]
], dtype=torch.float32)<jupyter_output><empty_output><jupyter_text>First reduction operation: **Summation**<jupyter_code>t.sum()   # reduced tensor to rank 0 scalar value
t.numel()   # original tensor has 9 elements
t.sum().numel()   # finally only 1 element left
t.sum().numel() < t.numel()<jupyter_output><empty_output><jupyter_text>Some other reduction operations include:<jupyter_code>t.mean()
t.prod()
t.std()<jupyter_output><empty_output><jupyter_text>Does reduction operations always lead to a scalar value?  - **NO**If the axis along which the reduction is to be performed is mentioned, the reduction operation will yield a vector or higher dimensional tensor<jupyter_code>t = torch.tensor([
    [1,1,1,1],
    [2,2,2,2],
    [3,3,3,3]
], dtype=torch.float32)
t.shape
t.sum(dim=0)
t.sum(dim=1)<jupyter_output><empty_output><jupyter_text>What just happened? Elementwise additions along along dim=0 means what? Simply put, it says add all the rows togther individually. What does that mean? Let's see all the rows<jupyter_code>t[0]
t[1]
t[2]
t[0] + t[1] + t[2]<jupyter_output><empty_output><jupyter_text>The result above is same as above, of summing with dim=0. This means the dimensions we specify are the ones which get reduced to a lower number essentially to 1. If we do, dim=1, we essentially mean to say we want to reduce all the columns to a single column, i.e add all the columns up. ArgMaxArgMax is a mathematical concept which returns the index value of the maximum element in the array. Given an array of many numbers, which number is a maximum and what is it's corresponding index? We get the index value as an answer<jupyter_code>t = torch.tensor([
    [1,0,0,2],
    [0,3,3,0],
    [4,0,0,5]
], dtype=torch.float32)
t.max()
t.argmax()
t.flatten()<jupyter_output><empty_output><jupyter_text>As can be seen above, the maximum value is 5 in the above tensor but the argmax returned the index 11. This is because, if no axis is specified, the argmax returns maximum value of the tensor provided it is flattened.<jupyter_code>t.max(dim=0)   # returns maximum elements as well as indices<jupyter_output><empty_output><jupyter_text>Well, it may seem confusing but lets see what just happened. Above code asks, what is the maximum value along the row for all elements having compared elementwise. Essentially this means, we have 3 rows:1. row1 = 1,0,0,22. row2 = 0,3,3,03. row3 = 4,0,0,5Along the row we see, 0 < 1 < 4, for first element in all the rows. The maximum is 4. It belongs to row 3. Next we compare 0,3,0; i.e the second elements of all the rows and we get 0 < 3, i.e row 1 has maximum element. Likewise, we have (4,3,3,5) as maximum elements having compared all the rows elementwise. The corresponding result is the row number which has maximum value element (2,1,1,2<jupyter_code>t.argmax(dim=0)   # just returns indices
t.max(dim=1)
t.argmax(dim=1)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09627481597144769, 'fraction_numerical': 0.03399509257193843, 'mean_word_length': 5.006430868167203, 'pattern_counts': {'":': 0, '<': 180, '<?xml version=': 0, '>': 175, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '45', 'dir': 'jupyter-structured-clean-dedup', 'id': '588638', 'license': 'MIT', 'n_tokens_mistral': 7490, 'n_tokens_neox': 7166, 'n_words': 2854, 'path': 'pytorch/Deep Lizard PyTorch.ipynb', 'repo_name': 'nikunjlad/Coding-Playground', 'size': '22415'}