<jupyter_start><jupyter_text>Doc2Vec This tutorial is available as an IPython notebook at [Malaya/example/doc2vec](https://github.com/huseinzol05/Malaya/tree/master/example/doc2vec).     This module trained on both standard and local (included social media) language structures, so it is save to use for both.<jupyter_code>%%time
import malaya
string1 = 'Pemuda mogok lapar desak kerajaan prihatin isu iklim'
string2 = 'Perbincangan isu pembalakan perlu babit kerajaan negeri'
string3 = 'kerajaan perlu kisah isu iklim, pemuda mogok lapar'
string4 = 'Kerajaan dicadang tubuh jawatankuasa khas tangani isu alam sekitar'
news1 = 'Tun Dr <NAME> mengakui pembubaran Parlimen bagi membolehkan pilihan raya diadakan tidak sesuai dilaksanakan pada masa ini berikutan isu COVID-19'
tweet1 = 'DrM sembang pilihan raya tak boleh buat sebab COVID 19'<jupyter_output><empty_output><jupyter_text>Doc2Vec using Word Vector```pythondef doc2vec_wordvector(wordvector):    """    Doc2vec interface for text similarity using Word Vector.    Parameters    ----------    wordvector : object        malaya.wordvector.WordVector object.        should have `get_vector_by_name` method.    Returns    -------    result: malaya.similarity.Doc2VecSimilarity    """```  Using InterfaceI will use `malaya.wordvector.load(model = 'news')`, pretty accurate related to local issues.<jupyter_code>%%time

vocab_news, embedded_news = malaya.wordvector.load(model = 'news')
w2v = malaya.wordvector.WordVector(embedded_news, vocab_news)
doc2vec = malaya.similarity.doc2vec_wordvector(w2v)<jupyter_output>CPU times: user 178 ms, sys: 118 ms, total: 296 ms
Wall time: 301 ms<jupyter_text>predict batch of strings with probability```pythondef predict_proba(    self,    left_strings: List[str],    right_strings: List[str],    aggregation: Callable = np.mean,    similarity: str = 'cosine',    soft: bool = False,):    """    calculate similarity for two different batch of texts.    Parameters    ----------    left_strings : list of str    right_strings : list of str    aggregation : Callable, optional (default=numpy.mean)    similarity : str, optional (default='mean')        similarity supported. Allowed values:        * ``'cosine'`` - cosine similarity.        * ``'euclidean'`` - euclidean similarity.        * ``'manhattan'`` - manhattan similarity.    soft: bool, optional (default=False)        word not inside word vector will replace with nearest word if True, else, will skip.    Returns    -------    result: List[float]    """```<jupyter_code>%%time

doc2vec.predict_proba([string1], [string2])
%%time

doc2vec.predict_proba([string1, string2], [string3, string4])
%%time

doc2vec.predict_proba([string1, string2], [string3, tweet1])<jupyter_output>CPU times: user 1.68 ms, sys: 381 µs, total: 2.06 ms
Wall time: 1.75 ms<jupyter_text>visualize heatmap```pythondef heatmap(    self,    strings: List[str],    aggregation: Callable = np.mean,    similarity: str = 'cosine',    soft: bool = False,    visualize: bool = True,    annotate: bool = True,    figsize: Tuple[int, int] = (7, 7),):    """    plot a heatmap based on output from bert similarity.    Parameters    ----------    strings : list of str        list of strings    aggregation : Callable, optional (default=numpy.mean)    similarity : str, optional (default='mean')        similarity supported. Allowed values:        * ``'cosine'`` - cosine similarity.        * ``'euclidean'`` - euclidean similarity.        * ``'manhattan'`` - manhattan similarity.    soft: bool, optional (default=True)        word not inside word vector will replace with nearest word if True, else, will skip.    visualize : bool        if True, it will render plt.show, else return data.    figsize : tuple, (default=(7, 7))        figure size for plot.    Returns    -------    result: list        list of results.    """```<jupyter_code>doc2vec.heatmap([string1, string2, string3, string4])<jupyter_output><empty_output><jupyter_text>Different similarity function will return different percentage.  Doc2Vec using Vectorizer ModelWe can use any Vectorizer models provided by Malaya to use encoder similarity interface, example, BERT, XLNET. Again, these encoder models not trained to do similarity classification, it just encode the strings into vector representation.```pythondef doc2vec_vectorizer(vectorizer):    """    Doc2vec interface for text similarity using Vectorizer model.    Parameters    ----------    vectorizer : object        vectorizer interface object, BERT, XLNET.        should have `vectorize` method.    Returns    -------    result: malaya.similarity.VectorizerSimilarity    """```  using ALXLNET<jupyter_code>alxlnet = malaya.transformer.load(model = 'alxlnet')
doc2vec_vectorizer = malaya.similarity.doc2vec_vectorizer(alxlnet)<jupyter_output>INFO:tensorflow:memory input None
INFO:tensorflow:Use float type <dtype: 'float32'>
WARNING:tensorflow:From /Users/huseinzolkepli/Documents/Malaya/malaya/transformers/alxlnet/modeling.py:810: dropout (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.dropout instead.
WARNING:tensorflow:From /Users/huseinzolkepli/Documents/tf-1.15/env/lib/python3.7/site-packages/tensorflow_core/python/layers/core.py:271: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `layer.__call__` method instead.
WARNING:tensorflow:From /Users/huseinzolkepli/Documents/Malaya/malaya/transformers/alxlnet/modeling.py:110: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.Dense instead.
INFO:tensorflow:Restoring parameters from /Users/huseinz[...]<jupyter_text>predict for 2 strings with probability```pythondef predict_proba(    self,    left_strings: List[str],    right_strings: List[str],    similarity: str = 'cosine',):    """    calculate similarity for two different batch of texts.    Parameters    ----------    left_strings : list of str    right_strings : list of str    similarity : str, optional (default='mean')        similarity supported. Allowed values:        * ``'cosine'`` - cosine similarity.        * ``'euclidean'`` - euclidean similarity.        * ``'manhattan'`` - manhattan similarity.    Returns    -------    result: List[float]    """```<jupyter_code>%%time

doc2vec_vectorizer.predict_proba([string1], [string2])
%%time

doc2vec_vectorizer.predict_proba([string1, string2], [string3, string4])<jupyter_output>CPU times: user 504 ms, sys: 118 ms, total: 621 ms
Wall time: 139 ms<jupyter_text>visualize heatmap```pythondef heatmap(    self,    strings: List[str],    similarity: str = 'cosine',    visualize: bool = True,    annotate: bool = True,    figsize: Tuple[int, int] = (7, 7),):    """    plot a heatmap based on output from bert similarity.    Parameters    ----------    strings : list of str        list of strings.    similarity : str, optional (default='mean')        similarity supported. Allowed values:        * ``'cosine'`` - cosine similarity.        * ``'euclidean'`` - euclidean similarity.        * ``'manhattan'`` - manhattan similarity.    visualize : bool        if True, it will render plt.show, else return data.    figsize : tuple, (default=(7, 7))        figure size for plot.    Returns    -------    result: list        list of results    """```<jupyter_code>doc2vec_vectorizer.heatmap([string1, string2, string3, string4])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11456541870617806, 'fraction_numerical': 0.01587511575605239, 'mean_word_length': 3.9250814332247557, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 27, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '446711', 'license': 'MIT', 'n_tokens_mistral': 2366, 'n_tokens_neox': 2232, 'n_words': 684, 'path': 'example/doc2vec/load-doc2vec-similarity.ipynb', 'repo_name': 'AetherPrior/malaya', 'size': '7569'}