<jupyter_start><jupyter_text>SpaCy 3 - New Toys- categories: [nlp] The new version of SpaCy in pre-release hase several new features i'm eagerly awaiting.- Transformer Model- Trainable Sentence Splitter- Improved Interface for Training- Groups of Overlapable Spans<jupyter_code>import json
import spacy

from spacy.tokens import Doc
from spacy.training import Example

spacy.__version__<jupyter_output><empty_output><jupyter_text>ModelsThere is now a transformers based model for greater accuracy<jupyter_code>model_acc = spacy.load("en_core_web_trf")
model_acc.components<jupyter_output><empty_output><jupyter_text>As well as an improved version of the model we all know and love<jupyter_code>model_eff = spacy.load("en_core_web_sm")
model_eff.components<jupyter_output><empty_output><jupyter_text>Example Class I used to dread getting everything formatted into Gold objects previously... The new Example makes a uniform and simple way of formatting data for training<jupyter_code>predicted = Doc(model_eff.vocab, words=["Apply", "some", "sun", "screen"])

token_ref = ["Apply", "some", "sun", "screen"]
tags_ref = ["VERB", "DET", "NOUN", "NOUN"]
sent_refs = [1, 0, 0, 0]

example = Example.from_dict(predicted, {"words": token_ref, "tags": tags_ref, "sent_starts": sent_refs})
example.to_dict()<jupyter_output><empty_output><jupyter_text>A Statistical Sentence SplitterThis has been one of my dreams for quite a while. I mentioned it a few times on the SpaCy forums too. Very glad to see it make it's way into the library now.<jupyter_code>senter = model_eff.get_pipe("senter")
optimizer = model_eff.initialize()

examples = [example]
losses = senter.update(examples, sgd=optimizer)
losses<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09936083672283556, 'fraction_numerical': 0.002905287623474724, 'mean_word_length': 7.0093023255813955, 'pattern_counts': {'":': 3, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '568099', 'license': 'Apache-2.0', 'n_tokens_mistral': 581, 'n_tokens_neox': 551, 'n_words': 171, 'path': '_notebooks/2021-01-22-spacy-version-3.ipynb', 'repo_name': 'kslader8/kslader8-thoughts', 'size': '1721'}