<jupyter_start><jupyter_text>Code Snippets for Natural Language Programming   0. Contents- 1 One-hot representation with scikit-lear  1. Creating one-hot representation with scikit-learn<jupyter_code>from sklearn.feature_extraction.text import CountVectorizer

corpus = [
    "This is the first document.",
    "This document is the second document.",
    "And this is the third one.",
    "Is this the first document?"
]

vectorizer = CountVectorizer()
X = vectorizer.fit_transform(corpus)
vectorizer.get_feature_names_out()

print(X.toarray())<jupyter_output>[[0 1 1 1 0 0 1 0 1]
 [0 2 0 1 0 1 1 0 1]
 [1 0 0 1 1 0 1 1 1]
 [0 1 1 1 0 0 1 0 1]]<jupyter_text>1. Tensors  1.0. General<jupyter_code>def describe(x):
    """A function used to give information """
    print("Type: {}".format(x.type()))
    print("Shape/size: {}".format(x.shape))
    print("Values: \n {}".format(x))
import torch
# describe(torch.Tensor(2, 3))
import torch 
# describe(torch.rand(2,3)) # create a tensor from uniform random distribution
# describe(torch.randn(2,3)) # create a tensor from random normal distribution<jupyter_output><empty_output><jupyter_text>We can also create tensors all filled with the same scalar. For creating tensor of zeros or ones, we have built-in functions, and for filling it with specific values, we can use the fill_() method. Any PyTorch method with an underscore refers to an in-place operation; that is, it modifies the content in place without creating a new object.<jupyter_code># Creating a filled tensor 

import torch 
# describe(torch.zeros(2,3))
x = torch.ones(2,3)
# describe(x)
x.fill_(5)
# describe(x)
# Creating and initializing a tensor from lists

x = torch.tensor([[1,2,3],
                  [4,5,6]])
# We can also create from Numpy but pay attention that in that case the type will
# be torch.DoubleTensor instead of the torch.FloatTensor

import numpy as np
npy = np.random.rand(2, 3)
describe(torch.from_numpy(npy))<jupyter_output>Type: torch.DoubleTensor
Shape/size: torch.Size([2, 3])
Values: 
 tensor([[0.9334, 0.9989, 0.0594],
        [0.4688, 0.8958, 0.8736]], dtype=torch.float64)<jupyter_text>You can convert a tensor to a different type(float, long, double, etc. ) by specifying it at initialization or later using one of the typecasting methods. There are two ways to specify the initialization type: either by directly calling the constructor of a specific tensor type, such as FloatTensor or LongTensor, or using a special method, torch.tensor().<jupyter_code>x = torch.FloatTensor([[1, 2, 3], [4, 5, 6]])
x = x.long()
x = torch.tensor([[1, 2, 3],[4, 5, 6]], dtype = torch.int64)
x = x.float()<jupyter_output><empty_output><jupyter_text>We use the shape property and size() method of a tensor object to access the measurements of its dimensions. The two ways of accessing these measurements are mostly synonymous. Inspecting the shape of the tensor is an indispansable tool in debugging PyTorch code.  1.1 Tensor Operations After you have created your tensors, you can operate on them like you would do with traditional programming language types, like +,-,*,/. Instead of the operators, you can also use functions like .add().<jupyter_code>x = torch.randn(2, 3)
describe(x)
describe(torch.add(x,x))
describe(x+x)
x = torch.arange(6)
describe(x)
x = x.view(2,3)
describe(x)
describe(torch.sum(x, dim = 0))
describe(torch.transpose(x,0,1))<jupyter_output>Type: torch.LongTensor
Shape/size: torch.Size([3, 2])
Values: 
 tensor([[0, 3],
        [1, 4],
        [2, 5]])<jupyter_text>1.2 Indexing, slicing, and joining<jupyter_code>x = torch.arange(6).view(2,3)
describe(x[:1,:2])
describe(x[0,1])
indices = torch.LongTensor([0,2])
describe(torch.index_select(x, dim=1,index = indices))
indices = torch.LongTensor([0,0,1])
describe(torch.index_select(x, dim=0,index = indices))
row_indices = torch.arange(2).long()
col_indices = torch.LongTensor([0,1])
describe(x[row_indices, col_indices])<jupyter_output>Type: torch.LongTensor
Shape/size: torch.Size([2])
Values: 
 tensor([0, 4])<jupyter_text>Notice that the indices are a LongTensor; this is a requirement for indexing using PyTorch functions.<jupyter_code>x = torch.arange(6).view(2,3)
torch.cat([x,x], dim = 0)
torch.cat([x,x], dim = 1)
torch.stack([x,x])<jupyter_output><empty_output><jupyter_text>Python also implements highly efficient linear algebra operations on tensors, such as multiplication, inverse, and trace.<jupyter_code>x1 = torch.arange(6).view(2,3)
describe(x1)
x2 = torch.ones(3,2).long()
x2[:,1] += 1
describe(x2)
torch.mm(x1,x2)<jupyter_output><empty_output><jupyter_text>1.3 Tensors and Computational Graphs PyTorch tensor class encapsulates the data (the tensor itself) and a range of operations, such as algebraic operations, indexing, and reshaping operations. However, when the requires_grad Boolean flag is set to True on a tensor, bookkeeping operations are enabled that can track the gradient at the tensor as well as the gradient function, both of which are needed to facilitate the fradient-based learning discussed in "The Supervised Learning Paradigm".<jupyter_code>import torch 
x = torch.ones(2,3, requires_grad=True)
print(x.grad is None)
y = (x + 2) * (x + 5) + 3
print(y.grad is None)
z = y.mean()
z.backward()<jupyter_output><empty_output><jupyter_text>When you create a tensor with requires_grad = True, you are requiring PyTorch to manage bookkeeping information that computes gradients. First, PyTorch will keep track fo the values of the forward pass.Then, at the end of the computations, a single scalar is used to compute a backward pass. The backward pass is initiated using the backward() method on a tensor resulting from the evaluation of a loss function. The backward pass computes a gradient value for a tensor object that participated in the forward pass.  In general, the gradient is a value that represents the slope of a function output with respect to the function input. IN the computational graph setting, gradients exist for each parameter in the model and can be thought of as the parameter's contribution to the error signal.In PyTorch, you can access the gradients for the nodes in the computational graph by using the .grad member variable. Optimisers use the .grad variable to update the values of the parameters.   1.4 CUDA Tensors  So far, we have been allocating our tensors on the CPU memory. When doing linear algebra operations, it might make sense to utilize a GPU, if you have one. To use a GPU you need first to allocate the tensor on the GPU's memory. Access to the GPUs is via a specialized API called CUDA. The CUDA API was created by NVIDIA and is limited to use on only NVIDIA GPUs. PyTorch offers CUDA tensor objects that are indistinguishable in use from the regular CPU-bound tensors except for the way they are allocated internally.  PyTOrch makes it very easy to create these CUDA tensors, transfering the tensor from the CPU to the GPU while maintating its underlying type. THe preffered method in PyTorch is to be device agnostic and write code that works wheter it's on GPU or the CPU. You can check whether a GPU is available by using torch.cuda.is_available() and retrieve the device name with torch.decice().Then, all future tensors are instantiated and moved to the target device by using the .to(device) method.<jupyter_code>import torch 
print(torch.cuda.is_available())
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(device)
x = torch.rand(3,3).to(device)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09587945059341245, 'fraction_numerical': 0.023869849313241764, 'mean_word_length': 5.028938906752412, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 37, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '476478', 'license': 'MIT', 'n_tokens_mistral': 2396, 'n_tokens_neox': 2265, 'n_words': 939, 'path': 'code/snippets.ipynb', 'repo_name': 'GeorgePaps/NLP', 'size': '7499'}