<jupyter_start><jupyter_text>NLP1Improve on document term occurence matrix/one hot encoded vectors for word/document representationswith embeddings. The advantage of embeddings over matrix representations followed by dimensionality reductionis embeddings encode word similarity directly without the dimensionsionality reduction step. If you have the following sentences:Mathematician can runMathematician likes coffeeMathematician majored in physicsPhysicst can runPhysicst likes coffeePhysicst majored in physicsWe can give each word a vector, a vector for mathematician and one for physicist and we can see the wordssurrouding these 2 words are similar. We can use a dot product and cos between the 2 vectors to represent similarity. This concept of word vectorsdoesn't scale so we use word embeddings which is a mini DL vector.<jupyter_code>import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

torch.manual_seed(42)

vocab = set()
with open('howard.txt','r') as fh:
    for x in fh:
        #print(x)
        vocab = vocab | set(x.split())

print (len(vocab))
index_to_word= []
word_to_index = []
#create index to word and word to index mappings
for index,word in enumerate(vocab):
    index_to_word.append(word)
    word_to_index.append(index)<jupyter_output>2823
{'fraction_non_alphanumeric': 0.05267175572519084, 'fraction_numerical': 0.006870229007633588, 'mean_word_length': 5.213270142180095, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 4, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '204743', 'license': 'Apache-2.0', 'n_tokens_mistral': 379, 'n_tokens_neox': 355, 'n_words': 158, 'path': 'twitter_nlp/ethan/Untitled1.ipynb', 'repo_name': 'dougc333/DeepLearning', 'size': '1310'}