<jupyter_start><jupyter_text>Task: Question Answering for NTT Healthcare<jupyter_code>from haystack import Finder
from haystack.indexing.cleaning import clean_wiki_text
from haystack.indexing.utils import convert_files_to_dicts, fetch_archive_from_http
from haystack.reader.farm import FARMReader
from haystack.reader.transformers import TransformersReader
from haystack.utils import print_answers
# Connect to Elasticsearch

from haystack.database.elasticsearch import ElasticsearchDocumentStore
document_store = ElasticsearchDocumentStore(host="localhost", username="", password="", index="document")<jupyter_output>09/13/2020 18:06:52 - INFO - elasticsearch -   PUT http://localhost:9200/document [status:200 request:0.152s]
09/13/2020 18:06:52 - INFO - elasticsearch -   PUT http://localhost:9200/label [status:200 request:0.153s]<jupyter_text>Cleaning documents and inserting in elasticsearch<jupyter_code>import os
doc_dir = "../data"
print(os.getcwd())

# Convert files to dicts
# You can optionally supply a cleaning function that is applied to each doc (e.g. to remove footers)
# It must take a str as input, and return a str.
dicts = convert_files_to_dicts(dir_path=doc_dir, clean_func=clean_wiki_text, split_paragraphs=True)

# We now have a list of dictionaries that we can write to our document store.
# If your texts come from a different source (e.g. a DB), you can of course skip convert_files_to_dicts() and create the dictionaries yourself.
# The default format here is: {"name": "<some-document-name>, "text": "<the-actual-text>"}
# (Optionally: you can also add more key-value-pairs here, that will be indexed as fields in Elasticsearch and
# can be accessed later for filtering or shown in the responses of the Finder)

# Let's have a look at the first 3 entries:
print(dicts[:3])

# Now, let's write the dicts containing documents to our DB.
document_store.write_documents(dicts)
print("==============FINISHED TO INTRODUCE DOCUMENTS=================")<jupyter_output>/home/teo/haystack/tutorials
[{'text': "Most eukaryotic genes are interrupted by intronic sequences that must be removed from pre-messenger RNAs by the splicing machinery. These introns are typically spliced out in a sequential order, resulting in the production of a linear mRNA. However, the splicing machinery can also ''backsplice'' and join a splice donor to an upstream splice acceptor (e.g., join the end of exon 2 to the beginning of exon 2), thereby generating a circular RNA with covalently linked ends ( Figure 1 ). Thousands of genes can generate circular RNAs that accumulate in the cytoplasm, but most rarely do so because backsplicing is far less efficient (1%) than canonical splicing (reviewed in Wilusz, 2018) . Nevertheless, some circular RNAs accumulate to high levels and sequester microRNAs or RNA binding proteins or, alternatively, serve as templates for translation. Most other individual circular RNAs are expressed at exceedingly low levels, so it has remained unclear what[...]<jupyter_text>Initialize Retriever<jupyter_code>from haystack.retriever.sparse import ElasticsearchRetriever
retriever = ElasticsearchRetriever(document_store=document_store)
# Alternative: An in-memory TfidfRetriever based on Pandas dataframes for building quick-prototypes with SQLite document store.

# from haystack.retriever.sparse import TfidfRetriever
# retriever = TfidfRetriever(document_store=document_store)<jupyter_output><empty_output><jupyter_text>Initialize model to search question in elasticsearch passages<jupyter_code># Load a  local model or any of the QA models on
# Hugging Face's model hub (https://huggingface.co/models)

reader = FARMReader(model_name_or_path="deepset/roberta-base-squad2", use_gpu=False)
print("=========teminado FARMReader")
# Alternative:
# reader = TransformersReader(model="distilbert-base-uncased-distilled-squad", tokenizer="distilbert-base-uncased", use_gpu=-1)<jupyter_output><empty_output><jupyter_text>Take the passages and find the best answer/passage<jupyter_code>finder = Finder(reader, retriever)<jupyter_output><empty_output><jupyter_text>Question and Answer<jupyter_code># You can configure how many candidates the reader and retriever shall return
# The higher top_k_retriever, the better (but also the slower) your answers. 
prediction = finder.get_answers(question="does microsoft help to stop the covid??", top_k_retriever=10, top_k_reader=5)
print_answers(prediction, details="minimal")<jupyter_output>[   {   'answer': 'we do not suggest this for small virtual meetings as it '
                  'stagnates collaboration',
        'context': 'ntended interruptions. Importantly, we do not suggest this '
                   'for small virtual meetings as it stagnates collaboration. '
                   'Third, encourage participants to '},
    {   'answer': 'More data will likely reduce the confidence intervals for '
                  'the frequencies of each incubation day (Figure 1) , giving '
                  'a clearer picture of the actual frequency distribution of '
                  'all incubation periods',
        'context': ' More data will likely reduce the confidence intervals for '
                   'the frequencies of each incubation day (Figure 1) , giving '
                   'a clearer picture of the actual frequency distribution of '
                   'all incubation periods'},
    {   'answer': 'timely city lockdown prevented a subsequent outbreak burst',
        'c[...]
{'fraction_non_alphanumeric': 0.08234217749313816, 'fraction_numerical': 0.012259835315645014, 'mean_word_length': 4.877419354838709, 'pattern_counts': {'":': 2, '<': 24, '<?xml version=': 0, '>': 24, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '319112', 'license': 'Apache-2.0', 'n_tokens_mistral': 1610, 'n_tokens_neox': 1473, 'n_words': 566, 'path': 'demo.ipynb', 'repo_name': 'tcapilla/nttextractiveqa', 'size': '5465'}