<jupyter_start><jupyter_text>One Hot Vector Representation of Words__<NAME> (DTU/2K16/MC/13)____Natural Language Processing - Dr. <NAME>__In this notebook we se how to extract one hot vectors from a given corpus and in this case our corpus will be a resume. Our first step is importing all required packages. 1. Import Packages<jupyter_code>from collections import Counter
import pprint
import random

import pandas
import numpy as np
import nltk
nltk.download('stopwords')
from nltk.corpus import stopwords<jupyter_output>[nltk_data] Downloading package stopwords to
[nltk_data]     C:\Users\anish\AppData\Roaming\nltk_data...
[nltk_data]   Package stopwords is already up-to-date!<jupyter_text>2. Importing the CorpusIn this case our corpus will be the resume which we will divide into 6 parts and each part will represent a document.<jupyter_code>resume_file = open('../assets/resume.txt', 'r')
resume = resume_file.read().lower()
resume_file.close()
print(resume)<jupyter_output><NAME>
software developer + clean code enthusiast

phone : 8287428181
email : <EMAIL>
home : sandesh vihar, pitampura, new delhi - 110034
date of birth : 7th april 1998
languages : english, hindi, french

work experience
what after college (4 months)
delhi, india
creating content to teach core java and python with data structures and algorithms and giving online classes to students.
giving python classes workshops to students all around india and teaching core data structures and the python api
with emphasis on data structures, algorithms and problem solving. see a sample python batch here:
https://github.com/anishlearnstocode/python-workshop-6

also teaching java to students in batches of 10 days, where the full java api and data types are covered along with many
important algorithms are aso taught. see a sample java batch here: https://github.com/anishlearnstocode/java-wac-batch-32

summer research fellow at university of auckland (2 months)
auckland, new zealand
w[...]<jupyter_text>3. Tokenizing the ResumeWe will now tokenize the Resume and obtain words/tokens from our corpus. We will further divide up these tokens into 6 parts to obtain our 6 documents.<jupyter_code># creating utility function to tokenize the document
def tokenize(document: str, stopwords_en=stopwords.words('english'), tokenizer=nltk.RegexpTokenizer(r'\w+')) -> list:
    document = document.lower()
    return [token for token in tokenizer.tokenize(document) if token not in stopwords_en and token.isalpha()]
# getting the tokens
tokens = tokenize(resume)
# printing first 25 tokens
print(tokens[: 25])<jupyter_output>['anish', 'sachdeva', 'software', 'developer', 'clean', 'code', 'enthusiast', 'phone', 'email', 'outlook', 'com', 'home', 'sandesh', 'vihar', 'pitampura', 'new', 'delhi', 'date', 'birth', 'april', 'languages', 'english', 'hindi', 'french', 'work']<jupyter_text>4. Dividing Corpus into 6 documentsWe now divide the corpus into 6 documents, basically we equally divide the tokens list into 6 different fragments.<jupyter_code># dividing corpus into 6 documents
k = len(tokens) // 6
documents = []
for i in range(5):
    documents.append(tokens[i * k: (i + 1) * k])
documents.append(tokens[5 * k:])

# printing the second document
pprint.pp(documents[1])<jupyter_output>['java',
 'api',
 'data',
 'types',
 'covered',
 'along',
 'many',
 'important',
 'algorithms',
 'aso',
 'taught',
 'see',
 'sample',
 'java',
 'batch',
 'https',
 'github',
 'com',
 'anishlearnstocode',
 'java',
 'wac',
 'batch',
 'summer',
 'research',
 'fellow',
 'university',
 'auckland',
 'months',
 'auckland',
 'new',
 'zealand',
 'worked',
 'geometry',
 'mobius',
 'transformations',
 'differential',
 'geometry',
 'dr',
 'pedram',
 'hekmati',
 'department',
 'mathematics',
 'university',
 'auckland',
 'worked',
 'various',
 'topics',
 'mathematics',
 'abelian',
 'group',
 'theory',
 'measure',
 'theory',
 'graph',
 'theory',
 'differential',
 'geometry',
 'attended',
 'lectures',
 'conferences',
 'notable',
 'speakers',
 'throughout',
 'academia',
 'industry',
 'met',
 'ed',
 'witten',
 'currently',
 'forefront',
 'applied',
 'mathematics',
 'physics',
 'mathematical',
 'topology',
 'higher',
 'dimensional',
 'spaces',
 'met']<jupyter_text>5. Calculating 5 Most Common Tokens from Each DocumentWe now calculate the 5 Most common Tokens from each document and it is not necessary that we obtain $5 \times 6 = 30$ tokens in the end as some tokens might be repeated.<jupyter_code># calculating most common 5 tokens from each document
most_common = set()
for document in documents:
    frequencies = Counter(document)
    for word, frequency in frequencies.most_common(5):
        most_common.add(word)

# print number of most common tokens
print('Number of Most Common Tokens:', len(most_common), end='\n')

# print the most common tokens
pprint.pprint(most_common)<jupyter_output>Number of Most Common Tokens: 27
{'algorithms',
 'also',
 'applications',
 'auckland',
 'cern',
 'college',
 'com',
 'computer',
 'data',
 'geometry',
 'group',
 'guitar',
 'java',
 'london',
 'many',
 'mathematics',
 'participated',
 'plectrum',
 'python',
 'requests',
 'research',
 'structures',
 'students',
 'theory',
 'trinity',
 'university',
 'worked'}<jupyter_text>6. Creating the One Hot Vectors for Each Word in the `most_common` SetWe will iterate over all the words in `most_common` set and for each word we will se whether it is present in a particular document or not. If it is then we mark it as $1$ in the corresponding row otherwise as $0$.<jupyter_code>vectors = {}
for word in most_common:
    vector = np.zeros((6), dtype=int)
    for index, document in enumerate(documents):
        vector[index] = word in document
    vectors[word] = vector
# let us see the vector output for a sample word, you can aso modify the word below to see one-hot vector representation
word = 'students'
print(word + ':', vectors[word].T)
# we can see the vector representations for all the words in most_common set of words
for word in most_common:
    print(word + ':', vectors[word].T)<jupyter_output>computer: [0 0 0 0 1 0]
python: [1 0 0 0 1 0]
also: [1 0 1 1 0 0]
college: [1 0 0 0 0 1]
mathematics: [0 1 0 0 1 1]
applications: [0 0 1 0 0 0]
structures: [1 0 0 0 1 1]
research: [0 1 1 0 0 1]
group: [0 1 1 0 0 0]
java: [1 1 1 1 1 1]
worked: [0 1 1 1 0 0]
guitar: [0 0 0 0 0 1]
students: [1 0 0 1 0 0]
trinity: [0 0 0 0 0 1]
theory: [0 1 1 0 0 0]
plectrum: [0 0 0 0 0 1]
auckland: [0 1 0 0 0 1]
data: [1 1 0 1 1 1]
london: [0 0 0 0 0 1]
many: [0 1 0 1 0 0]
geometry: [0 1 0 0 0 0]
participated: [0 0 0 1 0 0]
com: [1 1 0 0 0 1]
cern: [0 0 1 1 0 1]
requests: [0 0 0 1 0 0]
university: [0 1 0 0 1 1]
algorithms: [1 1 0 0 1 1]<jupyter_text>7. Making the Table of Words & Vector RepresentationsWe now create a $6 \times 27$ table representing the words and their repective vectors.<jupyter_code>table = pandas.DataFrame(data=vectors)
print(table.iloc[:, 0:7])
print(table.iloc[:, 7:16])
print(table.iloc[:, 16:22])
print(table.iloc[:, 22:])<jupyter_output>com  cern  requests  university  algorithms
0    1     0         0           0           1
1    1     0         0           1           1
2    0     1         0           0           0
3    0     1         1           0           0
4    0     0         0           1           1
5    1     1         0           1           1
{'fraction_non_alphanumeric': 0.11837624301866231, 'fraction_numerical': 0.03827816373791037, 'mean_word_length': 3.9877717391304346, 'pattern_counts': {'":': 0, '<': 26, '<?xml version=': 0, '>': 27, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '108734', 'license': 'MIT', 'n_tokens_mistral': 2642, 'n_tokens_neox': 2380, 'n_words': 953, 'path': 'notebooks/one-hot-vector.ipynb', 'repo_name': 'anishLearnsToCode/bow-representation', 'size': '7372'}