<jupyter_start><jupyter_text>Example NotebookThe purpose of this notebook is to demonstrate how to work with NQ data in Elasticsearch and assumes that `prepare_data.py` has been executed per the repo's README.md.  Connect to Local ElasticSearch Instance<jupyter_code># assumes ElasticSearch is already running
es = connect_es(host='localhost', port=9200)
es.ping()
# ensure our demo_index is available
'demo_index' in es.indices.get_alias('*').keys()
es.count(index='demo_index')['count']
len(qa_records)
es.indices.get_alias('*')<jupyter_output><empty_output><jupyter_text>Load NQ Question/Answer Records<jupyter_code># load data
qa_records = load_pkl_file('../data/eval_data/qa_records_fullsys.pkl')
qa_records[0]<jupyter_output><empty_output><jupyter_text>Ask a Question to ElasticSearch<jupyter_code>question = 'Who is the mother in how i met your mother?'

res = run_question_query(es_obj=es,
                           index_name='demo_index',
                           question_text=question,
                           n_results=5)
res<jupyter_output><empty_output><jupyter_text>Apply BERT Reader to Get Answer<jupyter_code>reader = DocumentReader("deepset/bert-base-cased-squad2")
# lets check the first article returned

context = res['hits']['hits'][0]['_source']['document_text_clean']
reader.tokenize(question, context)

print(f'Question: {question} \n')
print(f"Answer: {reader.get_answer()}")<jupyter_output>WARNING:transformers.tokenization_utils:Token indices sequence length is longer than the specified maximum sequence length for this model (11 > 512). Running this sequence through the model will result in indexing errors
{'fraction_non_alphanumeric': 0.10401459854014598, 'fraction_numerical': 0.007907542579075427, 'mean_word_length': 5.606425702811245, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '60611', 'license': 'MIT', 'n_tokens_mistral': 515, 'n_tokens_neox': 506, 'n_words': 129, 'path': 'notebooks/Example_Notebook.ipynb', 'repo_name': 'areed1242/qa-evaluation-framework', 'size': '1644'}