<jupyter_start><jupyter_text>Download data, if needed, for stanza and e2edutch<jupyter_code>stanza.download('nl')
e2edutch.download.download_data()<jupyter_output><empty_output><jupyter_text>Here's the piece of text we want to analyze:<jupyter_code>txt = 'Dit is een stukje tekst. Deze tekst dient als voorbeeld.'<jupyter_output><empty_output><jupyter_text>First run stanza<jupyter_code>nlp = stanza.Pipeline('nl', processors='tokenize,lemma,pos,depparse')
stanza_doc = nlp(txt)
stanza_doc<jupyter_output><empty_output><jupyter_text>To use the output of stanza for e2edutch, we only need the output of the tokenization:<jupyter_code>e2e_doc = {'doc_key': 'example',
          'sentences': [[token['text'] for token in sentence]
                       for sentence in stanza_doc.to_dict()]}
predictor = Predictor()
e2e_doc['predicted_clusters'] = predictor.predict(e2e_doc)
e2e_doc['predicted_clusters']<jupyter_output><empty_output><jupyter_text>Let's print out the tokens together with the clusters, in conll-2012 format<jupyter_code>with io.StringIO() as sout:
    e2edutch.conll.output_conll(sout, {'example': e2e_doc['sentences']}, 
                                      {'example': e2e_doc['predicted_clusters']})
    conll_e2e = sout.getvalue()
print(conll_e2e)<jupyter_output>#begin document (example);

example	0	Dit	(0)
example	1	is	-
example	2	een	-
example	3	stukje	-
example	4	tekst	-
example	5	.	-

example	0	Deze	(1
example	1	tekst	1)
example	2	dient	-
example	3	als	-
example	4	voorbeeld	(1)
example	5	.	-

#end document<jupyter_text>Combine the stanza tokens with the clusters:<jupyter_code>clusters_brackets = e2edutch.conll.clusters_to_brackets(e2e_doc['sentences'], e2e_doc['predicted_clusters'])
clusters_brackets
conll_list_stanza = stanza.utils.conll.CoNLL.convert_dict(stanza_doc.to_dict())
for sent_stanza, sent_e2e in zip(conll_list_stanza, clusters_brackets):
    for tok_stanza, tok_e2e in zip(sent_stanza, sent_e2e):
        print('\t'.join(tok_stanza)+'\t'+tok_e2e)
    print('\n')<jupyter_output>1	Dit	dit	PRON	VNW|aanw|pron|stan|vol|3o|ev	Person=3|PronType=Dem	4	nsubj	_	start_char=0|end_char=3	(0)
2	is	zijn	AUX	WW|pv|tgw|ev	Number=Sing|Tense=Pres|VerbForm=Fin	4	cop	_	start_char=4|end_char=6	-
3	een	een	DET	LID|onbep|stan|agr	Definite=Ind	4	det	_	start_char=7|end_char=10	-
4	stukje	stuk	NOUN	N|soort|ev|dim|onz|stan	Gender=Neut|Number=Sing	0	root	_	start_char=11|end_char=17	-
5	tekst	tekst	NOUN	N|soort|ev|basis|zijd|stan	Gender=Com|Number=Sing	4	nmod	_	start_char=18|end_char=23	-
6	.	.	PUNCT	LET	_	4	punct	_	start_char=23|end_char=24	-


1	Deze	deze	DET	VNW|aanw|det|stan|prenom|met-e|rest	_	2	det	_	start_char=25|end_char=29	(1
2	tekst	tekst	NOUN	N|soort|ev|basis|zijd|stan	Gender=Com|Number=Sing	3	nsubj	_	start_char=30|end_char=35	1)
3	dient	dienen	VERB	WW|pv|tgw|met-t	Number=Sing|Tense=Pres|VerbForm=Fin	0	root	_	start_char=36|end_char=41	-
4	als	als	ADP	VZ|init	_	5	mark	_	start_char=42|end_char=45	-
5	voorbeeld	voorbeeld	NOUN	N|soort|ev|basis|onz|stan	Gender=Neut|Number=Sing	3	xc[...]
{'fraction_non_alphanumeric': 0.11514351699109204, 'fraction_numerical': 0.033322335862751566, 'mean_word_length': 5.890909090909091, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 23, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '456790', 'license': 'Apache-2.0', 'n_tokens_mistral': 1381, 'n_tokens_neox': 1338, 'n_words': 250, 'path': 'notebooks/Run_stanza_and_e2eDutch.ipynb', 'repo_name': 'Filter-Bubble/nlp-demo', 'size': '3031'}