<jupyter_start><jupyter_text>How To Use The Word Embedder<jupyter_code>import nltk
from nltk.tokenize import word_tokenize
nltk.download('punkt')<jupyter_output>[nltk_data] Downloading package punkt to
[nltk_data]     C:\Users\shreyanshchordia\AppData\Roaming\nltk_data...
[nltk_data]   Package punkt is already up-to-date!<jupyter_text>Setting the dataset<jupyter_code>sentences = [
    'Hey how are you?',
    'I am great.',
    "What's the issue in it?",
    'We survived the  day!',
    'I am tired.',
    'I need to rest.',
    'We need to resolve our differences',
    'How far are you going to go with this, huh?'
]<jupyter_output><empty_output><jupyter_text>Tokenizing the sentences<jupyter_code>tokenized_sentences = [word_tokenize(sentence) for sentence in sentences]
for sentence in tokenized_sentences:
    print(sentence)<jupyter_output>['Hey', 'how', 'are', 'you', '?']
['I', 'am', 'great', '.']
['What', "'s", 'the', 'issue', 'in', 'it', '?']
['We', 'survived', 'the', 'day', '!']
['I', 'am', 'tired', '.']
['I', 'need', 'to', 'rest', '.']
['We', 'need', 'to', 'resolve', 'our', 'differences']
['How', 'far', 'are', 'you', 'going', 'to', 'go', 'with', 'this', ',', 'huh', '?']<jupyter_text>Word Embedder<jupyter_code>from WordEmbedder import Embedder, token_seq_to_num_seq, num_seq_to_token_seq, generate_vocabulary<jupyter_output><empty_output><jupyter_text>As simple as it could be!<jupyter_code>emb = Embedder(dimensions=50)
embedder = emb.get_embedder()
# embedder returns embeddings of known words 
# for unknown (out of vocabulary) words it returns array of zeros
print(embedder['beautiful', 'ffnceo'])<jupyter_output>[[ 0.54623    1.2042    -1.1288    -0.1325     0.95529    0.040524
  -0.47863   -0.3397    -0.28056    0.71761   -0.53691   -0.0045698
   0.73217    0.12101    0.28093   -0.088097   0.59733    0.55264
   0.056646  -0.50247   -0.63204    1.1439    -0.31053    0.1263
   1.3155    -0.52444   -1.5041     1.158      0.68795   -0.85051
   2.3236    -0.41789    0.44519   -0.019216   0.28969    0.53258
  -0.023008   0.58958   -0.72397   -0.85216   -0.17761    0.14432
   0.40658   -0.52003    0.09081    0.082961  -0.021975  -1.6214
   0.34579   -0.010919 ]
 [ 0.         0.         0.         0.         0.         0.
   0.         0.         0.         0.         0.         0.
   0.         0.         0.         0.         0.         0.
   0.         0.         0.         0.         0.         0.
   0.         0.         0.         0.         0.         0.
   0.         0.         0.         0.         0.         0.
   0.         0.         0.         0.         0.         0.
   0.         0.  [...]<jupyter_text>Generating vocabulary for the dataset<jupyter_code>vocab, vocab_dict = generate_vocabulary(tokenized_sentences)
print(f'Vocabulary:\n{vocab}\n\nVocabulary Dictionary that maps words to the count of their occurances:\n{vocab_dict}')<jupyter_output>Vocabulary:
['?', 'I', '.', 'to', 'are', 'you', 'am', 'the', 'We', 'need', 'Hey', 'how', 'great', 'What', "'s", 'issue', 'in', 'it', 'survived', 'day', '!', 'tired', 'rest', 'resolve', 'our', 'differences', 'How', 'far', 'going', 'go', 'with', 'this', ',', 'huh']

Vocabulary Dictionary that maps words to the count of their occurances:
{'?': 3, 'I': 3, '.': 3, 'to': 3, 'are': 2, 'you': 2, 'am': 2, 'the': 2, 'We': 2, 'need': 2, 'Hey': 1, 'how': 1, 'great': 1, 'What': 1, "'s": 1, 'issue': 1, 'in': 1, 'it': 1, 'survived': 1, 'day': 1, '!': 1, 'tired': 1, 'rest': 1, 'resolve': 1, 'our': 1, 'differences': 1, 'How': 1, 'far': 1, 'going': 1, 'go': 1, 'with': 1, 'this': 1, ',': 1, 'huh': 1}<jupyter_text>Generating embedding matrix for the vocabulary Embedding Matrix is required when you use the Embedding Layer from tensorflow.keras.layers on Pre-Trained word embeddings.The Layer demands for a numpy matrix that contains embeddings of all the words in the chosen vocabulary<jupyter_code>embedding_matrix, id2token, token2id = emb.get_embedding_matrix(vocab, num_words=10)
print(f"Embeddings of the first word of vocabulary:\n{embedding_matrix[0]}\n\nID-TO-TOKEN Mapper:\n{id2token}\n\nTOKEN-TO-ID Mapper:\n{token2id}")<jupyter_output>Embeddings of the first word of vocabulary:
[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0.]

ID-TO-TOKEN Mapper:
{0: '<oov>', 1: '?', 2: 'i', 3: '.', 4: 'to', 5: 'are', 6: 'you', 7: 'am', 8: 'the', 9: 'we'}

TOKEN-TO-ID Mapper:
{'<oov>': 0, '?': 1, 'i': 2, '.': 3, 'to': 4, 'are': 5, 'you': 6, 'am': 7, 'the': 8, 'we': 9}<jupyter_text>Converting Tokenized sentences to Number sequences<jupyter_code>num_sequences = token_seq_to_num_seq(tokenized_sentences, token2id, oov_token='<oov>')
# 0 for words that are not in the vocab of 10 words
for sequence in num_sequences: print(sequence)<jupyter_output>[0 0 5 6 1]
[2 7 0 3]
[0 0 8 0 0 0 1]
[9 0 8 0 0]
[2 7 0 3]
[2 0 4 0 3]
[9 0 4 0 0 0]
[0 0 5 6 0 4 0 0 0 0 0 1]<jupyter_text>Retrieving Tokenized sentences back from the Number sequences<jupyter_code>retrieved_tokenized_sequences = num_seq_to_token_seq(num_sequences, id2token)
# we lose words ('<oov>') on retrieval because num_words < total number of words in the vocabulary
for sequence in retrieved_tokenized_sequences: print(sequence)<jupyter_output>['<oov>', '<oov>', 'are', 'you', '?']
['i', 'am', '<oov>', '.']
['<oov>', '<oov>', 'the', '<oov>', '<oov>', '<oov>', '?']
['we', '<oov>', 'the', '<oov>', '<oov>']
['i', 'am', '<oov>', '.']
['i', '<oov>', 'to', '<oov>', '.']
['we', '<oov>', 'to', '<oov>', '<oov>', '<oov>']
['<oov>', '<oov>', 'are', 'you', '<oov>', 'to', '<oov>', '<oov>', '<oov>', '<oov>', '<oov>', '?']<jupyter_text>Extra features of the Embedder Class You can use object of the Embedder class to find:   1. Similarity score between two words   2. Analogy Score that determines, how good an analogy is.   3. K most similar words   4. K most apt words to satisfy the analogy<jupyter_code>emb2 = Embedder(dimensions=50)
print('bad -> ',emb2.most_similar_to('bad',k=3),'\n\nbaby -> ',emb2.most_similar_to('baby',k=3))
print('Analogy of\n\nking -> queen    man -> ',emb2.get_top_k_by_analogy('king','queen','man',k=2))
print('Similarity score:\n\nbad -> worse    ',emb2.cosine_similarity('bad','worse'), '\n\nhello -> why    ', emb2.cosine_similarity('hello','why'))
print('Analogy Score:\n\ngood->best   bad->worst = ',emb2.cosine_sim_analogy('good','best','bad','worst'),'\n\ngood->woman  yes->hello = ',emb2.cosine_sim_analogy('good','woman','yes','hello'))<jupyter_output>Analogy Score:

good->best   bad->worst =  0.75127333 

good->woman  yes->hello =  0.48011282
{'fraction_non_alphanumeric': 0.19909570459683495, 'fraction_numerical': 0.08183873398643557, 'mean_word_length': 3.2511210762331837, 'pattern_counts': {'":': 1, '<': 64, '<?xml version=': 0, '>': 77, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '360360', 'license': 'MIT', 'n_tokens_mistral': 2986, 'n_tokens_neox': 2664, 'n_words': 733, 'path': 'how_to_use_WordEmbedder.ipynb', 'repo_name': 'shreyanshchordia/word-embedder', 'size': '6635'}