<jupyter_start><jupyter_text>textgenrnn 1.3 Encoding Textby [<NAME>](http://minimaxir.com)*Max's open-source projects are supported by his [Patreon](https://www.patreon.com/minimaxir). If you found this project helpful, any monetary contributions to the Patreon are appreciated and will be put to good creative use.*  Introtextgenrnn can also be used to generate sentence vectors much more powerful than traditional word vectors.**IMPORTANT NOTE**: The sentence vectors only account for the first `max_length - 1` tokens. (in the pretrained model, that is the first **39 characters**). If you want more robust sentence vectors, train a new model with a very high `max_length` and/or use word-level training.<jupyter_code>from textgenrnn import textgenrnn

textgen = textgenrnn()<jupyter_output>Using TensorFlow backend.<jupyter_text>The function `encode_text_vectors` takes the Attention layer output of the model and can use PCA and TSNE to compress it into a more reasonable size.The size of the Attention layer is `dim_embeddings + (rnn_size * rnn_layers)`. In the case of the included pretrained model, the size is `100 + (128 * 2) = 356`.By default, `encode_text_vectors` uses PCA to project and calibrate this high-dimensional output to the number of provided texts, or 50D, whichever is lower.<jupyter_code>texts = ['Never gonna give you up, never gonna let you down',
            'Never gonna run around and desert you',
            'Never gonna make you cry, never gonna say goodbye',
            'Never gonna tell a lie and hurt you']

word_vector = textgen.encode_text_vectors(texts)

print(word_vector)
print(word_vector.shape)<jupyter_output>[[ 4.2585015e+00 -3.2080102e+00 -8.6409906e-03  7.3456152e-07]
 [-3.3668094e+00 -7.7063727e-01 -7.0967728e-01  7.3456215e-07]
 [ 2.1060679e+00  4.5051994e+00 -3.9030526e-02  7.3456152e-07]
 [-2.9977567e+00 -5.2655154e-01  7.5734943e-01  7.3456158e-07]]
(4, 4)<jupyter_text>Additionally, you can pass `tsne_dims` to further project the texts into 2D or 3D; great for data visualization. (NB: t-SNE is a random-seeded algorithm; for consistent output, set `tsne_seed` to make the output deterministic)<jupyter_code>word_vector = textgen.encode_text_vectors(texts, tsne_dims=2, tsne_seed=123)

print(str(word_vector))
print(word_vector.shape)<jupyter_output>[[115.05732  149.46983 ]
 [-35.552177   7.491257]
 [110.7458     3.17162 ]
 [-31.240635 153.78947 ]]
(4, 2)<jupyter_text>If you want to encode a single text, you'll have to set `pca_dims=None`.<jupyter_code>word_vector = textgen.encode_text_vectors("What is love?", pca_dims=None)

print(str(word_vector)[0:50])
print(word_vector.shape)<jupyter_output>[[ 4.78423387e-01 -6.65371776e-01 -1.54521123e-01 
(1, 356)<jupyter_text>You can also have the model return the `pca` object, which can then be used to learn more about the projection, and/or used in an encoding pipeline to transform any arbitrary text.<jupyter_code>word_vector, pca = textgen.encode_text_vectors(texts, return_pca=True)

print(pca)
pca.explained_variance_ratio_<jupyter_output><empty_output><jupyter_text>In this case, 56.9% of the variance is explained by the 1st component, and 98.5% of the variance is explained by the first 2 components.<jupyter_code>def transform_text(text, textgen, pca):
    text = textgen.encode_text_vectors(text, pca_dims=None)
    text = pca.transform(text)
    return text

single_encoded_text = transform_text("Never gonna give", textgen, pca)

print(single_encoded_text)<jupyter_output>[[ 0.7431461  -0.30268374  0.30652896 -0.27054822]]<jupyter_text>Sentence Vector Similarity For example you could calculate pairwise similarity...<jupyter_code>from sklearn.metrics.pairwise import cosine_similarity

word_vectors = textgen.encode_text_vectors(texts)
similarity = cosine_similarity(single_encoded_text, word_vectors)

print(similarity)<jupyter_output>[[ 0.8607758  -0.78297377  0.04231021 -0.65008384]]<jupyter_text>...or use textgenrnn's native similarity metrics!<jupyter_code>textgen.similarity("Never gonna give", texts)<jupyter_output><empty_output><jupyter_text>By default similarity is calculated using the PCA-transformed values, but you can calculate similarity on the raw values as well if needed.<jupyter_code>textgen.similarity("Never gonna give", texts, use_pca=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0962294702752718, 'fraction_numerical': 0.0844321073328707, 'mean_word_length': 5.591463414634147, 'pattern_counts': {'":': 0, '<': 32, '<?xml version=': 0, '>': 32, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '282298', 'license': 'MIT', 'n_tokens_mistral': 1696, 'n_tokens_neox': 1446, 'n_words': 440, 'path': 'docs/textgenrnn-encode-text.ipynb', 'repo_name': 'ZacharyMcGee/textgenrnn', 'size': '4326'}