<jupyter_start><jupyter_text>Tutorial - Easy Embeddings> Using EasyWord, Stacked, and Document Embeddings in the AdaptNLP framework  Finding Available Models with HubsWe can search for available models to utilize with Embeddings with the `HFModelHub` and `FlairModelHub`. We'll see an example below:<jupyter_code>from adaptnlp import (
    EasyWordEmbeddings, 
    EasyStackedEmbeddings, 
    EasyDocumentEmbeddings, 
    HFModelHub, 
    FlairModelHub, 
    DetailLevel
)
hub = HFModelHub()
models = hub.search_model_by_name('gpt2'); models<jupyter_output><empty_output><jupyter_text>For this tutorial we'll use the `gpt2` base model:<jupyter_code>model = models[-1]; model<jupyter_output><empty_output><jupyter_text>Producing Embeddings using `EasyWordEmbeddings` First we'll use some basic example text:<jupyter_code>example_text = "This is Albert.  My last name is Einstein.  I like physics and atoms."<jupyter_output><empty_output><jupyter_text>And then instantiate our embeddings tagger:<jupyter_code>embeddings = EasyWordEmbeddings()<jupyter_output><empty_output><jupyter_text>Now let's run our `gpt2` model we grabbed earlier to generate some `EmbeddingResult` objects:<jupyter_code>res = embeddings.embed_text(example_text, model_name_or_path=model)<jupyter_output><empty_output><jupyter_text>The result of this is a variety of filtered results for your disposal. The default level of information (`DetailLevel.Low`) will return an ordered dictionary with the keys of:- `inputs`, an array of your original sentence- `sentence_embeddings`, any sentence_embeddings you may have (if applicable) as an ordered dictionary of (sentence, embeddings)- `token_embeddings`, a similar `OrderedDict` to the `sentence_embeddings`, where the key `0` will be the embeddings of the first word, `1` is the second, and so forth:<jupyter_code>res['inputs']<jupyter_output><empty_output><jupyter_text>To grab our sentence or token embeddings, simply look it up by its key:> Note: Only `StackedEmbeddings` will have sentence embeddings<jupyter_code>res['token_embeddings'][0].shape<jupyter_output><empty_output><jupyter_text>Using different models is extremely easy to do. Let's try using BERT embeddings with the `bert-base-cased` model instead.Rather than passing in a `HFModelResult` or `FlairModelResult`, we can also just pass in the raw string name of the model as well:<jupyter_code>res = embeddings.embed_text(example_text, model_name_or_path='bert-base-cased')<jupyter_output><empty_output><jupyter_text>Just like in the last example, we can look at the embeddings in the same way:<jupyter_code>res['token_embeddings'][0].shape<jupyter_output><empty_output><jupyter_text>We can also convert our output to an easy to use dictionary, which can have a bit more information. First let's not filter our results by passing in `detail_level = None`:<jupyter_code>res = embeddings.embed_text(example_text, 
                            model_name_or_path='bert-base-cased',
                           detail_level=None)
res<jupyter_output><empty_output><jupyter_text>We can see see that result is now an `EmbeddingResult`, which has all the information we key'd with as available attributes:<jupyter_code>res.inputs<jupyter_output><empty_output><jupyter_text>If we want to filter the object ourselves and convert it to a dictionary, we can use the `to_dict()` function:<jupyter_code>o = res.to_dict()
print(o['inputs'], o['token_embeddings'][0].shape)<jupyter_output>['This is Albert.  My last name is Einstein.  I like physics and atoms.'] torch.Size([768])<jupyter_text>You can specify the level of detail wanted by passing in "low", "medium", or "high" to the `to_dict` method, or use the convience `DetailLevel` class:<jupyter_code>res_dict = res.to_dict(DetailLevel.Medium)
print(o['inputs'], o['token_embeddings'][0].shape)<jupyter_output>['This is Albert.  My last name is Einstein.  I like physics and atoms.'] torch.Size([768])<jupyter_text>Each level returns more data from the outputs:- Available at all levels:  - `original_sentence`: The original sentence  - `tokenized_sentence`: The tokenized sentence  - `sentence_embeddings`: Embeddings from the actual sentence (if available)  - `token_embeddings`: Concatenated embeddings from all the tokens passed- `DetailLevel.Low` (or 'low'):  - Returns information available at all levels- `DetailLevel.Medium` (or 'medium'):  - Everything from `DetailLevel.Low`  - For each token a dictionary of the embeddings and word index is added- `DetailLevel.High` (or 'high'):  - Everything from `DetailLevel.Medium`  - This will also include the original Flair `Sentence` result from the model Let's look at a final example with roBERTa embeddings:<jupyter_code>res = embeddings.embed_text(example_text, model_name_or_path="roberta-base")<jupyter_output><empty_output><jupyter_text>And our generated embeddings:<jupyter_code>#hide_input
print(f'Original text: {res["inputs"]}')
print(f'Model: roberta-base')
print(f'Embedding: {res["token_embeddings"][0].shape}')<jupyter_output>Original text: ['This is Albert.  My last name is Einstein.  I like physics and atoms.']
Model: roberta-base
Embedding: torch.Size([768])<jupyter_text>Producing Stacked Embeddings with `EasyStackedEmbeddings` `EasyStackedEmbeddings` allows you to use a variable number of language models to produce our embeddings shown above. For our example we'll combine the `bert-base-cased` and `distilbert-base-cased` models.First we'll instantiate our `EasyStackedEmbeddings`:<jupyter_code>embeddings = EasyStackedEmbeddings("bert-base-cased", "distilbert-base-cased")<jupyter_output>May need a couple moments to instantiate...<jupyter_text>And then generate our stacked word embeddings through our `embed_text` function:<jupyter_code>res = embeddings.embed_text(example_text)<jupyter_output><empty_output><jupyter_text>We can see our results below:<jupyter_code>#hide_input
print(f'Original text: {res["inputs"]}')
print(f'Model: roberta-base')
print(f'Embedding: {res["token_embeddings"][0]}')<jupyter_output>Original text: ['This is Albert.  My last name is Einstein.  I like physics and atoms.']
Model: roberta-base
Embedding: tensor([-0.6795, -0.2041,  1.0153,  ...,  0.2426, -0.2324,  0.3107])<jupyter_text>Document Embeddings with `EasyDocumentEmbeddings`Similar to the `EasyStackedEmbeddings`, `EasyDocumentEmbeddings` allows you to pool the embeddings from multiple models together with `embed_pool` and `embed_rnn`.We'll use our `bert-base-cased` and `distilbert-base-cased` models again:<jupyter_code>embeddings = EasyDocumentEmbeddings("bert-base-cased", "distilbert-base-cased")<jupyter_output>May need a couple moments to instantiate...<jupyter_text>This time we will use the `embed_pool` method to generate `DocumentPoolEmbeddings`. These do an average over all the word embeddings in a sentence:<jupyter_code>res = embeddings.embed_pool(example_text)<jupyter_output><empty_output><jupyter_text>As a result rather than having embeddings by token, we have embeddings *by document*<jupyter_code>res['inputs']
res['token_embeddings'][0]
#hide_input
print(f'Original text: {res["inputs"]}')
print(f'Model: roberta-base')
print(f'Embedding: {res["token_embeddings"][0].shape}')<jupyter_output>Original text: ['This is Albert.  My last name is Einstein.  I like physics and atoms.']
Model: roberta-base
Embedding: torch.Size([1536])<jupyter_text>We can also generate `DocumentRNNEmbeddings` as well. Document RNN Embeddings run an RNN over all the words in the sentence and use the final state of the RNN as the embedding.First we'll call `embed_rnn`:<jupyter_code>sentences = embeddings.embed_rnn(example_text)<jupyter_output><empty_output><jupyter_text>And then look at our generated embeddings:<jupyter_code>#hide_input
print(f'Original text: {res["inputs"]}')
print(f'Model: roberta-base')
print(f'Embedding: {res["token_embeddings"][0].shape}')<jupyter_output>Original text: ['This is Albert.  My last name is Einstein.  I like physics and atoms.']
Model: roberta-base
Embedding: torch.Size([1536])
{'fraction_non_alphanumeric': 0.10225713929417633, 'fraction_numerical': 0.007731637361266991, 'mean_word_length': 6.801556420233463, 'pattern_counts': {'":': 0, '<': 85, '<?xml version=': 0, '>': 87, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '506007', 'license': 'Apache-2.0', 'n_tokens_mistral': 2617, 'n_tokens_neox': 2483, 'n_words': 756, 'path': 'nbs/04a_tutorial.embeddings.ipynb', 'repo_name': 'mfredriksz/adaptnlp', 'size': '8019'}