<jupyter_start><jupyter_text>Models intersection strategies (ensemble)We use two models: **AllenNLP** and **Huggingface**. In multiple tests AllenNLP turned out much better - it has better precision and recall (on [Google GAP dataset](https://github.com/google-research-datasets/gap-coreference)), and finds much more clusters at all. That's why we decide to take AllenNLP answers as a ground truth.   However AllenNLP also makes mistakes - it has about 8% of false positives which we would like to minimize. That's why we propose **several intersections of AllenNLP and Huggingface outputs (an ensemble)** to modify the results and gain more confidence about the final clusters.  We propose four intersection strategies:- **strict** - we take only those clusters that are identical both in AllenNLP and Huggingface (intersection of clusters)- **partial** - we take all of the spans that are identical both in AllenNLP and Huggingface (intersection of spans/mentions)- **fuzzy** - we take all of the spans that are the same but also we find spans that overlap (are relating to the same entity but are composed of different number of tokens) and choose the shorter one  Example**Text**  *In 1311 it was settled on Peter and Lucy for life with remainder to <NAME> and his wife Beatrice. Peter had died by 1329 but Lucy lived until 1337 and she was succeeded by <NAME> who died in 1343. William's son, Sir <NAME> inherited and maintained ownership jointly with his wife Felice until he died in 1370.*  **AllenNLP clusters**   <NAME> --> <NAME>; his; <NAME>verard who died in 1343; William's   Peter --> Peter; Peter   Lucy --> Lucy; Lucy; she   William's son, Sir <NAME> --> William's son, Sir <NAME>; his; he**Huggingface clusters**  <NAME> --> <NAME>; his; William Everard; Sir Edmund Everard; his  Peter --> Peter; Peter; he  Lucy --> Lucy; Lucy; she  his wife Beatrice --> his wife Beatrice; his wife Felice Strategies1. **Strict**      Lucy --> Lucy; Lucy; she2. **Partial**      William Everard --> William Everard; his      Peter --> Peter; Peter      Lucy --> Lucy; Lucy; she3. **Fuzzy**      William Everard --> William Everard; his; William Everard      Peter --> Peter; Peter      Lucy --> Lucy; Lucy; she      Sir Edmund Everard --> Sir Edmund Everard; his<jupyter_code>text = "<NAME> (born January 8, 1999) is an American basketball player. He currently plays for the Auburn Tigers in the Southeastern Conference. Wiley attended Spain Park High School in Hoover, Alabama, where he averaged 27.1 points, 12.7 rebounds and 2.9 blocked shots as a junior in 2015-16, before moving to Florida, where he went to Calusa Preparatory School in Miami, Florida, while playing basketball at The Conrad Academy in Orlando."

clusters = predictor.predict(text)['clusters']
doc = nlp(text)
print("~~~ AllenNLP clusters ~~~")
print_clusters(doc, clusters)
print("\n~~~ Huggingface clusters ~~~")
for cluster in doc._.coref_clusters:
    print(cluster)
strict = StrictIntersectionStrategy(predictor, nlp)
partial = PartialIntersectionStrategy(predictor, nlp)
fuzzy = FuzzyIntersectionStrategy(predictor, nlp)
for intersection_strategy in [strict, partial, fuzzy]:
    print(f'\n~~~ {intersection_strategy.__class__.__name__} clusters ~~~')
    print_clusters(doc, intersection_strategy.clusters(text))<jupyter_output>~~~ StrictIntersectionStrategy clusters ~~~
Florida - [Florida; Florida]

~~~ PartialIntersectionStrategy clusters ~~~
Wiley - [He; Wiley; he; he]
Florida - [Florida; Florida]

~~~ FuzzyIntersectionStrategy clusters ~~~
<NAME> - [<NAME>; He; Wiley; he; he]
Florida - [Florida; Florida]
{'fraction_non_alphanumeric': 0.10013908205841446, 'fraction_numerical': 0.013073713490959666, 'mean_word_length': 4.56656346749226, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 33, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '390282', 'license': 'Apache-2.0', 'n_tokens_mistral': 1039, 'n_tokens_neox': 947, 'n_words': 453, 'path': 'intersection_strategies.ipynb', 'repo_name': 'NeuroSYS-pl/coreference-resolution', 'size': '3721'}