<jupyter_start><jupyter_text>Sentence Transformers: Multilingual Sentence Embeddings using BERT / RoBERTa / XLM-RoBERTa & Co. with PyTorchhttps://github.com/UKPLab/sentence-transformersBERT / RoBERTa / XLM-RoBERTa produces out-of-the-box rather bad sentence embeddings. This repository fine-tunes BERT / RoBERTa / DistilBERT / ALBERT / XLNet with __a siamese or triplet network structure__ to produce semantically meaningful sentence embeddings that can be used in unsupervised scenarios: Semantic textual similarity via cosine-similarity, clustering, semantic search.We provide an increasing number of state-of-the-art pretrained models that can be used to derive sentence embeddings. See Pretrained Models. Details of the implemented approaches can be found in our publication: Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (EMNLP 2019).You can use this code to easily train your own sentence embeddings, that are tuned for your specific task. We provide various dataset readers and you can tune sentence embeddings with different loss function, depending on the structure of your dataset. For further details, see Train your own Sentence Embeddings.  install<jupyter_code>!pip install -U sentence-transformers
!pip install tokenizers==0.8.1<jupyter_output>Collecting tokenizers==0.8.1
  Using cached https://files.pythonhosted.org/packages/2b/3e/7cf9b5daa88371c96d9b63d31917e30ba93b1d89421aef79c00e806bc54d/tokenizers-0.8.1-cp37-cp37m-macosx_10_11_x86_64.whl
[31mERROR: transformers 3.0.2 has requirement tokenizers==0.8.1.rc1, but you'll have tokenizers 0.8.1 which is incompatible.[0m
Installing collected packages: tokenizers
  Found existing installation: tokenizers 0.8.1rc1
    Uninstalling tokenizers-0.8.1rc1:
      Successfully uninstalled tokenizers-0.8.1rc1
Successfully installed tokenizers-0.8.1<jupyter_text>论文Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks<jupyter_code>from IPython.display import IFrame
IFrame('https://arxiv.org/pdf/1908.10084', width=1200, height=550)<jupyter_output><empty_output><jupyter_text>AbstractBERT (Devlin et al., 2018) and RoBERTa (Liuet  al.,  2019)  has  set  a  new  state-of-the-art performance on sentence-pair regression tasks like  semantic  textual  similarity  (STS).  How-ever,  it  requires  that  both  sentences  are  fed into the network, which causes a massive com-putational  overhead:   Finding  the  most  sim-ilar  pair  in  a  collection  of  10,000  sentences requires about 50 million inference computa-tions (~65 hours) with BERT. The construction of BERT makes it unsuitable for semantic sim-ilarity search as well as for unsupervised tasks like clustering.In this publication, we present Sentence-BERT(SBERT),  a  modification  of  the  pretrained BERT network that use siamese and triplet net-work structures to derive semantically mean-ingful sentence embeddings that can be com-pared using cosine-similarity. This reduces the effort for finding the most similar pair from 65hours with BERT / RoBERTa to about 5 sec-onds with SBERT, while maintaining the ac-curacy from BERT. We evaluate SBERT and SRoBERTa on com-mon  STS  tasks  and  transfer  learning  tasks,where   it   outperforms   other   state-of-the-art sentence embeddings methods.  model structure  Getting Started-Sentences Embedding with a Pretrained Model<jupyter_code>from sentence_transformers import SentenceTransformer
model = SentenceTransformer('bert-base-nli-mean-tokens')
# Then provide some sentences to the model.

sentences = ['This framework generates embeddings for each input sentence',
    'Sentences are passed as a list of string.', 
    'The quick brown fox jumps over the lazy dog.']
sentence_embeddings = model.encode(sentences)
# And that's it already. We now have a list of numpy arrays with the embeddings.
for sentence, embedding in zip(sentences, sentence_embeddings):
    print("Sentence:", sentence)
    
    print("Embedding:", embedding.size)
    print("")<jupyter_output>Sentence: This framework generates embeddings for each input sentence
Embedding: 768

Sentence: Sentences are passed as a list of string.
Embedding: 768

Sentence: The quick brown fox jumps over the lazy dog.
Embedding: 768
{'fraction_non_alphanumeric': 0.06239580852583949, 'fraction_numerical': 0.032626815908549656, 'mean_word_length': 4.974395448079658, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '617094', 'license': 'Apache-2.0', 'n_tokens_mistral': 1335, 'n_tokens_neox': 1202, 'n_words': 463, 'path': 'sentence-transformers.ipynb', 'repo_name': 'mayi140611/learning', 'size': '4199'}