<jupyter_start><jupyter_text>Tensor OperationsThis section covers:* Indexing and slicing* Reshaping tensors (tensor views)* Shifting tensors between CPU and GPU* Tensor arithmetic and basic operations* Dot products* Matrix multiplication* Element-wise multiplication* Broadcasting Perform standard imports<jupyter_code>import torch
import numpy as np<jupyter_output><empty_output><jupyter_text>Indexing and slicingExtracting specific values from a tensor works just the same as with NumPy arrays![IS.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAwQAAAF/CAIAAAB445meAAAJMmlDQ1BkZWZhdWx0X3JnYi5pY2MAAEiJlZVnUJNZF8fv8zzphUASQodQQ5EqJYCUEFoo0quoQOidUEVsiLgCK4qINEWQRQEXXJUia0UUC4uCAhZ0gywCyrpxFVFBWXDfGZ33HT+8/5l7z2/+c+bec8/5cAEgiINlwct7YlK6wNvJjhkYFMwE3yiMn5bC8fR0A9/VuxEArcR7ut/P+a4IEZFp/OW4uLxy+SmCdACg7GXWzEpPWeGjy0wPj//CZ1dYsFzgMt9Y4eh/eexLzr8s+pLj681dfhUKABwp+hsO/4b/c++KVDiC9NioyGymT3JUelaYIJKZttIJHpfL9BQkR8UmRH5T8P+V/B2lR2anr0RucsomQWx0TDrzfw41MjA0BF9n8cbrS48hRv9/z2dFX73kegDYcwAg+7564ZUAdO4CQPrRV09tua+UfAA67vAzBJn/eqiVDQ0IgALoQAYoAlWgCXSBETADlsAWOAAX4AF8QRDYAPggBiQCAcgCuWAHKABFYB84CKpALWgATaAVnAad4Dy4Aq6D2+AuGAaPgRBMgpdABN6BBQiCsBA<KEY>)<jupyter_code>x = torch.arange(20).reshape(5,4)
print(x)
print(x[0][1])
print(x[2][1])
# Grabbing the right hand column values
x[1:4,1:] # x[start:end:step, start:end:step]<jupyter_output><empty_output><jupyter_text>Reshape tensorsview() and reshape() do essentially the same thing by returning a reshaped tensor without changing the original tensor in place.There's a good discussion of the differences here.Also learn about squeeze() and unsqueeze()<jupyter_code>x = torch.arange(10)
print(x)
x.view(2,5)
x.view(5,2)
# x is unchanged
x<jupyter_output><empty_output><jupyter_text>Views reflect the most current data<jupyter_code>z = x.view(2,5).clone()
x[0]=1
print(z)<jupyter_output>tensor([[0, 1, 2, 3, 4],
        [5, 6, 7, 8, 9]])<jupyter_text>Views can infer the correct sizeBy passing in -1 PyTorch will infer the correct value from the given tensor, when you are aware of fixing all dimensions except one while reshaping.<jupyter_code>y = torch.Tensor([[1,2,3,4,5,6,7,8,9,10], [11,12,13,14,15,16,17,18,19,20]])
print(y.shape)
y = y.view(2,2,-1)
print(y)
print(y.shape)
y.view(4,-1)
y.view(-1,5)<jupyter_output><empty_output><jupyter_text>Adopt another tensor's shape with .view_as()view_as(input) only works with tensors that have the same number of elements.<jupyter_code>x1 = torch.rand(5,4)
x2 = torch.rand(2,10)
print(x1)
print()
print(x2)
x2 = x2.view_as(x1)
x2 #= x2.reshape(x1.shape)
# unchanged<jupyter_output><empty_output><jupyter_text>squeeze() and unsqueeze()<jupyter_code>y = torch.Tensor([[1,2,3,4,5,6,7,8,9,10], [11,12,13,14,15,16,17,18,19,20]])
print("Original size: ", y.shape)
y_row = torch.unsqueeze(y,0) # unsqueeze about row as axis=0
print("Unsqueezed size:", y_row.shape) 
print(y_row)
y_col = torch.unsqueeze(y,1) # unsqueeze about col as axis=1
print(y_col.shape)
print(y_col)
y_axis2 = torch.unsqueeze(y,2) # unsqueeze about axis=2
print(y_axis2.shape)
print(y_axis2)
y = torch.Tensor([4])
print("Original size: ", y.shape)
print(y)
y_row = torch.unsqueeze(y,0) #added another dimension across row
print(y_row.shape)
print(y_row)
y_col= torch.unsqueeze(y,1) #added another dimension across column
print(y_col.shape)
print(y_col)
y_restored = torch.squeeze(y_row,0) #added another dimension across row
print(y_restored.shape)
print(y_restored)
y = torch.zeros(2, 1, 2, 1, 2)
print(y.shape)
y_sq = torch.squeeze(y)
print(y_sq)
print(y_sq.shape)
print(torch.squeeze(y,0).shape) # won't change as axis=0 is not equal to 1
print(torch.squeeze(y,1).shape)<jupyter_output>torch.Size([2, 1, 2, 1, 2])
tensor([[[0., 0.],
         [0., 0.]],

        [[0., 0.],
         [0., 0.]]])
torch.Size([2, 2, 2])
torch.Size([2, 1, 2, 1, 2])
torch.Size([2, 2, 1, 2])<jupyter_text>Shifting tensors between CPU and GPU<jupyter_code># Tensor loading into GPU, back to CPU, detach(), etc.
# Check device (first) = cuda, if i want to perform a non-gpu task then first i have to bring the tensors from gpu memory to cpu one.
# if gradient is attached to tensor in cpu or gpu, so call detach (if cpu) and detach().cpu() (if gpu)
t = torch.Tensor([1,2,3])
t.type()
# To check where a particular tensor is located
t.device
dev = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
print(dev)
# For now, let's assume dev = 'gpu', you can shift tensors or models from gpu to cpu or vice versa using tensor_name.to(dev) / model.to(dev)
t.to(dev)
t.type()
# You can also use tensor_name.cpu()
t.cpu()<jupyter_output><empty_output><jupyter_text>Tensor ArithmeticAdding tensors can be performed a few different ways depending on the desired result.As a simple expression:<jupyter_code>a = torch.tensor([1,2,3], dtype=torch.float32)
b = torch.tensor([4,5,6], dtype=torch.float32)
print(a + b)<jupyter_output>tensor([5., 7., 9.])<jupyter_text>As arguments passed into a torch operation:<jupyter_code>print(torch.add(a, b))
print(a)<jupyter_output>tensor([5., 7., 9.])
tensor([1., 2., 3.])<jupyter_text>Changing a tensor in-place<jupyter_code>a.add_(b)  # equivalent to a=torch.add(a,b)
print(a)<jupyter_output>tensor([5., 7., 9.])<jupyter_text>NOTE: Any operation that changes a tensor in-place is post-fixed with an underscore _.    In the above example: a.add_(b) changed a.  Basic Tensor OperationsArithmeticOPERATIONFUNCTIONDESCRIPTIONa + ba.add(b)element wise additiona - ba.sub(b)subtractiona * ba.mul(b)multiplicationa / ba.div(b)divisiona % ba.fmod(b)modulo (remainder after division)aba.pow(b)power&nbsp; Monomial OperationsOPERATIONFUNCTIONDESCRIPTION|a|torch.abs(a)absolute value1/atorch.reciprocal(a)reciprocal$\sqrt{a}$torch.sqrt(a)square rootlog(a)torch.log(a)natural logeatorch.exp(a)exponential12.34  ==>  12.torch.trunc(a)truncated integer12.34  ==>  0.34torch.frac(a)fractional component Summary StatisticsOPERATIONFUNCTIONDESCRIPTION$\sum a$torch.sum(a)sum$\bar a$torch.mean(a)meanamaxtorch.max(a)maximumamintorch.min(a)minimumtorch.max(a,b) returns a tensor of size acontaining the element wise max between a and b NOTE: Most arithmetic operations require float values. Those that do work with integers return integer tensors.For example, torch.div(a,b) performs floor division (truncates the decimal) for integer types, and classic division for floats.  Dot productsA dot product is the sum of the products of the corresponding entries of two 1D tensors. If the tensors are both vectors, the dot product is given as:$\begin{bmatrix} a & b & c \end{bmatrix} \;\cdot\; \begin{bmatrix} d & e & f \end{bmatrix} = ad + be + cf$If the tensors include a column vector, then the dot product is the sum of the result of the multiplied matrices. For example:$\begin{bmatrix} a & b & c \end{bmatrix} \;\cdot\; \begin{bmatrix} d \\ e \\ f \end{bmatrix} = ad + be + cf$Dot products can be expressed as torch.dot(a,b) or `a.dot(b)` or `b.dot(a)`<jupyter_code>a = torch.tensor([1,2,3], dtype=torch.float)
b = torch.tensor([4,5,6], dtype=torch.float)
print(a.mul(b)) # for reference
print()
print(a.dot(b))<jupyter_output>tensor([ 4., 10., 18.])

tensor(32.)<jupyter_text>NOTE: There's a slight difference between torch.dot() and numpy.dot(). While torch.dot() only accepts 1D arguments and returns a dot product, numpy.dot() also accepts 2D arguments and performs matrix multiplication. We show matrix multiplication below.  Matrix multiplication2D Matrix multiplication is possible when the number of columns in tensor A matches the number of rows in tensor B. In this case, the product of tensor A with size $(x,y)$ and tensor B with size $(y,z)$ results in a tensor of size $(x,z)$$\begin{bmatrix} a & b & c \\d & e & f \end{bmatrix} \;\times\; \begin{bmatrix} m & n \\ p & q \\ r & s \end{bmatrix} = \begin{bmatrix} (am+bp+cr) & (an+bq+cs) \\(dm+ep+fr) & (dn+eq+fs) \end{bmatrix}$Matrix multiplication can be computed using torch.mm(a,b) or `a.mm(b)` or `a @ b`<jupyter_code>a = torch.tensor([[0,2,4],[1,3,5]], dtype=torch.float)
b = torch.tensor([[6,7],[8,9],[10,11]], dtype=torch.float)

print('a: ',a.size())
print('b: ',b.size())
print('a x b: ',torch.mm(a,b).size())
print(torch.mm(a,b))
print(a.mm(b))
print(a @ b)<jupyter_output>tensor([[56., 62.],
        [80., 89.]])<jupyter_text>Element-Wise Multiplication<jupyter_code>#TODO: Show usage of mul() operator. A.mul(B) or torch.mul(A,B)
a = torch.tensor([[1,2,4],[1,3,5]])
b = torch.tensor([[6,8,9],[2,10,11]])
a.mul(b) # Use mul_ to store the output matrix in 'a' itself
b.mul(a)
torch.mul(a,b)
a 
# unchanged<jupyter_output><empty_output><jupyter_text>BroadcastingTwo tensors are “broadcastable” if the following rules hold:* Each tensor has at least one dimension.* When iterating over the dimension sizes, starting at the trailing dimension (last dimension in the list), the dimension sizes must either be equal, one of them is 1, or one of them does not exist.<jupyter_code>t1 = torch.randn(3, 4)
t2 = torch.randn(1, 4)

print(t1,t2,sep='\n\n')
# Notice how the two tensors dont have the same dimensions but we can still perform arithmetic operations. Why?
t1 + t2
t1 - t2
t1 = torch.randn(3, 1, 4, 1)
t2 = torch.randn(3, 4, 1)

print(t1,t2,sep='\n\n')
(t1 + t2).size()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13985108449336356, 'fraction_numerical': 0.03971080176972051, 'mean_word_length': 5.968421052631579, 'pattern_counts': {'":': 0, '<': 55, '<?xml version=': 0, '>': 57, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '391049', 'license': 'MIT', 'n_tokens_mistral': 3818, 'n_tokens_neox': 3605, 'n_words': 900, 'path': 'Tensor_Operations.ipynb', 'repo_name': 'Aatmaj-Zephyr/PyTorch-101', 'size': '52551'}