<jupyter_start><jupyter_text>Setup1. In `cd NEMO`1. Run YAP API server `./yap api` (if you specify a port, change it in `config.py`)1. `pip install -r requirements.txt`1. install bclm https://github.com/OnlpLab/bclm1. Run NEMO API server `uvicorn api_main:app --reload --port 8090`1. Have a look at the swagger OpenAPI documentation by opening http://localhost:8090/docs in your browser<jupyter_code>import requests
requests.get('http://localhost:8090/').json()
texts = [
    'מלך השערים במונדיאל 2006 היה מירוסלב קלוזה.',
    "עשרות אנשים מגיעים מתאילנד לישראל.\nתופעה זו התבררה אתמול בוועדת העבודה והרווחה של הכנסת."
]<jupyter_output><empty_output><jupyter_text>Run NER Model<jupyter_code>%%time 
payload = { 'sentences': texts[1],
            'model_name': 'token-multi', # / 'token-single' / 'morph' 
            #'tokenized': True,
          }

res = requests.get('http://localhost:8090/run_ner_model', params=payload).json()
list(zip(res[0]['tokenized_text'], res[0]['ncrf_preds'])), list(zip(res[1]['tokenized_text'], res[1]['ncrf_preds']))<jupyter_output>CPU times: user 6.83 ms, sys: 3.34 ms, total: 10.2 ms
Wall time: 107 ms<jupyter_text>multi_to_single (get token-single predictions using a token-multi model)<jupyter_code>%%time
payload = { 'sentences': texts[1],
            #'model_name': 'token-multi',
            #'tokenized': True,
          }

res = requests.get('http://localhost:8090/multi_to_single', params=payload).json()
res<jupyter_output>CPU times: user 4.83 ms, sys: 2.32 ms, total: 7.15 ms
Wall time: 82 ms<jupyter_text>MD + NER  multi_align_hybrid<jupyter_code>%%time
payload = { 'sentences': texts[1],
            #'model_name': 'token-multi',
            #'tokenized': True,
          }

res = requests.get('http://localhost:8090/multi_align_hybrid', params=payload).json()
res
print(res[0]['md_lattice'])<jupyter_output>0	1	עשרות	עשר	CDT	CDT	gen=F|num=P	1
1	2	אנשים	איש	NN	NN	gen=M|num=P	2
2	3	מגיעים	הגיע	BN	BN	gen=M|num=P|per=A	3
3	4	מ	מ	PREPOSITION	PREPOSITION	_	4
4	5	תאילנד	תאילנד	NNP	NNP	gen=F|num=S	4
5	6	ל	ל	PREPOSITION	PREPOSITION	_	5
6	7	ישראל	ישראל	NNP	NNP	gen=F|num=S	5
7	8	.	_	yyDOT	yyDOT	_	6<jupyter_text>morph_yap<jupyter_code>%%time
payload = { 'sentences': texts[1],
            #'model_name': 'morph',
            #'tokenized': True,
          }

res = requests.get('http://localhost:8090/morph_yap', params=payload).json()
res<jupyter_output>CPU times: user 9.87 ms, sys: 494 µs, total: 10.4 ms
Wall time: 243 ms<jupyter_text>morph_hybridThe following models got the best results in our experiments.<jupyter_code>%%time
payload = { 'sentences': texts[1],
            #'multi_model_name': 'token-multi',
            #'morph_model_name': 'morph',
            #'tokenized': False,
            # 'align_tokens': False,
          }

res = requests.get('http://localhost:8090/morph_hybrid', params=payload).json()
res<jupyter_output>CPU times: user 6.27 ms, sys: 2.09 ms, total: 8.36 ms
Wall time: 371 ms<jupyter_text>morph_hybrid_align_tokens<jupyter_code>%%time
payload = { 'sentences': texts[1],
            #'tokenized': False,
            #'align_tokens': False,
          }

res = requests.get('http://localhost:8090/morph_hybrid_align_tokens', params=payload).json()
res<jupyter_output>CPU times: user 7.09 ms, sys: 3.78 ms, total: 10.9 ms
Wall time: 357 ms<jupyter_text>Working with sequence labels`iobes` can be used to parse the predictions (`pip install iobes`)<jupyter_code>import iobes
import pandas as pd
payload = { 'sentences': '\n'.join(texts),}

res = requests.get('http://localhost:8090/morph_hybrid_align_tokens', params=payload).json()
def get_ents(docs, label_key, text_key):
    ents = []
    for i, doc in enumerate(docs):
        labels = doc[label_key]
        morphs = doc[text_key]
        spans = iobes.parse_spans_iobes(labels)
        for span in spans:
            text = ' '.join(morphs[span.start:span.end])
            ents.append({ 'sent_id': i+1, 
                          'text': text, 
                          'cat': span.type})
    return pd.DataFrame(ents)
ents_hyb = get_ents(res, 'morph_ncrf_preds', 'morph_forms')
ents_hyb
ents_hyb = get_ents(res, 'multi_ncrf_preds_align_morph', 'morph_forms')
ents_hyb
ents_hyb = get_ents(res, 'multi_ncrf_preds_align_single', 'tokenized_text')
ents_hyb<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13964277429830665, 'fraction_numerical': 0.03363488749710044, 'mean_word_length': 3.872316384180791, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 27, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '427063', 'license': 'Apache-2.0', 'n_tokens_mistral': 1816, 'n_tokens_neox': 1699, 'n_words': 367, 'path': 'api/api_usage.ipynb', 'repo_name': 'cjer/NEMO', 'size': '4311'}