<jupyter_start><jupyter_text>ELMO( Embeddings from Language Models) Elmo is used for building char-level embedding unlike Glove/Word2vec/BOW which used for word embeddings.Computes contextualized word representations using character-based word representations and bidirectional LSTMs, as described in the paper "Deep contextualized word representations"1. Captures contextual meaning of word ,having diffrent embedding for different words2. Handle out of Vocabulary words 3. Capture Morphological words embeddingsInstead of using a fixed embedding for each word, ELMo looks at the entire sentence before assigning each word in it an embedding. It uses a bi-directional LSTM trained on a specific task to be able to create those embeddings. ELMo provided a significant step towards pre-training in the context of NLP.1. For In depth knowledge of ELMO Architecture refer :https://www.mihaileric.com/posts/deep-contextualized-word-representations-elmo/2. Elmo Architecture papaer refer : https://arxiv.org/pdf/1508.06615.pdf3. For Highway Netwrk refer : https://towardsdatascience.com/review-highway-networks-gating-function-to-highway-image-classification-5a33833797b5<jupyter_code>import tensorflow as tf
import tensorflow_hub as hub<jupyter_output><empty_output><jupyter_text>Elmo Embedding<jupyter_code>elmo =hub.Module("https://tfhub.dev/google/elmo/2", trainable=False) ### Load Elmo Model

text1="She sat on the river bank across from a series of wide, large steps leading up a hill to the park where the Arch stood, framed against a black sky."
text2="How could a man with four million in the bank be in financial danger?"

embeddings = elmo(
[text1, text22],
signature="default",
as_dict=True)["elmo"]
with tf.Session() as session:
    session.run([tf.global_variables_initializer(), tf.tables_initializer()])
    embeddings = session.run(embeddings)<jupyter_output>INFO:tensorflow:Saver not created because there are no variables in the graph to restore<jupyter_text>Word Embedding<jupyter_code>word_embeddings = elmo(
[text1, text2],
signature="default",
as_dict=True)["word_emb"]
with tf.Session() as session:
    session.run([tf.global_variables_initializer(), tf.tables_initializer()])
    word_embeddings = session.run(word_embeddings)
word_embeddings.shape<jupyter_output><empty_output><jupyter_text>LSTM Layer1 Embeding<jupyter_code>lstm1_embeddings = elmo(
[text1, text2],
signature="default",
as_dict=True)["lstm_outputs1"]
with tf.Session() as session:
    session.run([tf.global_variables_initializer(), tf.tables_initializer()])
    lstm1_embeddings = session.run(lstm1_embeddings)
lstm1_embeddings.shape<jupyter_output><empty_output><jupyter_text>InputsThe module defines two signatures: default, and tokens.With the default signature, the module takes untokenized sentences as input. The input tensor is a string tensor with shape [batch_size]. The module tokenizes each string by splitting on spaces.With the tokens signature, the module takes tokenized sentences as input. The input tensor is a string tensor with shape [batch_size, max_length] and an int32 tensor with shape [batch_size] corresponding to the sentence length. The length input is necessary to exclude padding in the case of sentences with varying length. The output dictionary contains: 1. word_emb: the character-based word representations with shape [batch_size, max_length, 512]. 2. lstm_outputs1: the first LSTM hidden state with shape [batch_size, max_length, 1024]. 3. lstm_outputs2: the second LSTM hidden state with shape [batch_size, max_length, 1024]. 4. elmo: the weighted sum of the 3 layers, where the weights are trainable. This tensor has shape [batch_size, max_length, 1024] 5. default: a fixed mean-pooling of all contextualized word representations with shape [batch_size, 1024].<jupyter_code>embeddings.shape
embeddings[0][5]  ##Bank in text1
embeddings[1][9]  ##Bank in text2<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0789272030651341, 'fraction_numerical': 0.01966794380587484, 'mean_word_length': 6.26530612244898, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 20, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '271591', 'license': 'Apache-2.0', 'n_tokens_mistral': 1176, 'n_tokens_neox': 1112, 'n_words': 410, 'path': 'Elmo.ipynb', 'repo_name': 'Swty13/NLP-Books', 'size': '3915'}