<jupyter_start><jupyter_text>Elasticsearch With HaystackWe will be communicating with our Elasticsearch document store via Haystack. It should be already installed if the `requirements.txt` was used. Otherwise, we need to install Haystack using pip:On Windows:```pip install farm-haystack -f https://download.pytorch.org/whl/torch_stable.html```Anything else:```pip install farm-haystack```We will start by indexing the SQuAD dev data. So let's load that into our notebook first.<jupyter_code>import json

with open('../../data/squad/dev.json', 'r') as f:
    squad = json.load(f)<jupyter_output><empty_output><jupyter_text>Next, we initialize a connection between Haystack and our local Elasticsearch instance like so:<jupyter_code>from haystack.document_store.elasticsearch import ElasticsearchDocumentStore

document_store = ElasticsearchDocumentStore(
    host='localhost',
    index='squad_docs',
)
# document_store = ElasticsearchDocumentStore()<jupyter_output>03/14/2022 08:22:33 - INFO - faiss -   Loading faiss.
03/14/2022 08:22:37 - INFO - elasticsearch -   HEAD http://localhost:9200/squad_docs [status:200 request:0.125s]
03/14/2022 08:22:37 - INFO - elasticsearch -   GET http://localhost:9200/squad_docs [status:200 request:0.005s]
03/14/2022 08:22:37 - INFO - elasticsearch -   PUT http://localhost:9200/squad_docs/_mapping [status:200 request:0.021s]
03/14/2022 08:22:37 - INFO - elasticsearch -   HEAD http://localhost:9200/label [status:200 request:0.004s]<jupyter_text>Great, we've established our connection, now let's try querying our Elasticsearch instance. We will do this through the `requests` library.<jupyter_code>import requests<jupyter_output><empty_output><jupyter_text>Let's check our cluster *health* (eg the general status of our Elasticsearch instance). We do this by sending a **GET** request to the `_cluster/health` endpoint.<jupyter_code>res = requests.get('http://localhost:9200/_cluster/health')

res.json()<jupyter_output><empty_output><jupyter_text>Okay we can see that the cluster is definitely running. The cluster status is *yellow*, ideally we want to aim for *green* but the reason we see yellow here is because not all replica shards have been allocated to nodes. The details of this don't really matter, but it essentially just means that we don't have a full set of backup (*replica*) data shards - which is only a problem if our *primary* data sources get corrupted/lost. That is beyond the scope of what we are doing here however. Adding DataRight now our Elasticsearch instance contains a single, empty index called *'squad_docs'*. We need to populate this with our `squad` data. We populate our index through the `document_store.write_documents()` method, where our *\* must be a list of dictionaries in the format:```json{    'text': '',    'meta': {        'other': ''    }}```We **must** include the `'text'` key. The *text* must contain the text from each sample, which in our case is a *context* string. The `'meta'` data is optional, but is usually used to contain anything else that might be relevant, so for example we might want to include the *group* that the context came from (eg 'Beyonce', or 'Matter').<jupyter_code>squad_docs = []

for sample in squad:
    squad_docs.append({
        'text': sample['context']
    })<jupyter_output><empty_output><jupyter_text>Then we add our data to the index like this:<jupyter_code>document_store.write_documents(squad_docs)<jupyter_output>03/08/2022 08:58:30 - INFO - elasticsearch -   POST http://localhost:9200/_bulk?refresh=wait_for [status:200 request:0.775s]
03/08/2022 08:58:31 - INFO - elasticsearch -   POST http://localhost:9200/_bulk?refresh=wait_for [status:200 request:0.879s]
03/08/2022 08:58:32 - INFO - elasticsearch -   POST http://localhost:9200/_bulk?refresh=wait_for [status:200 request:1.056s]
03/08/2022 08:58:33 - INFO - elasticsearch -   POST http://localhost:9200/_bulk?refresh=wait_for [status:200 request:1.026s]
03/08/2022 08:58:34 - INFO - elasticsearch -   POST http://localhost:9200/_bulk?refresh=wait_for [status:200 request:1.024s]
03/08/2022 08:58:35 - INFO - elasticsearch -   POST http://localhost:9200/_bulk?refresh=wait_for [status:200 request:1.261s]
03/08/2022 08:58:36 - INFO - elasticsearch -   POST http://localhost:9200/_bulk?refresh=wait_for [status:200 request:1.016s]
03/08/2022 08:58:37 - INFO - elasticsearch -   POST http://localhost:9200/_bulk?refresh=wait_for [status:200 request:1.013s]
[...]<jupyter_text>When we're retrieving data from Elasticsearch we will be retrieving documents using either the TF-IDF, or BM25 algorithms.**TF-IDF** is a common *relevance* scoring algorithm, the built is calculated using:* **TF**, the volume of words in the query (question) that appear in the document.* **IDF**, the inverse of the fraction of documents that contain the same word (eg common words like *'the'* don't score well, whereas *'Beyonce'* would).We integrate TD-IDF using:<jupyter_code>document_store
from haystack.retriever.sparse import TfidfRetriever

retriever = TfidfRetriever(document_store)<jupyter_output>03/14/2022 08:23:39 - INFO - elasticsearch -   POST http://localhost:9200/squad_docs/_search?scroll=1d&size=10000 [status:200 request:0.063s]
03/14/2022 08:23:39 - INFO - elasticsearch -   POST http://localhost:9200/_search/scroll [status:200 request:0.007s]
03/14/2022 08:23:39 - INFO - elasticsearch -   DELETE http://localhost:9200/_search/scroll [status:200 request:0.003s]
03/14/2022 08:23:39 - INFO - haystack.retriever.sparse -   Found 1204 candidate paragraphs from 1204 docs in DB<jupyter_text>We can see here that when building our retriever, it identified a total of *16209* 'candidate paragraphs'. These are all of the contexts from our `squad` data:<jupyter_code>len(squad)<jupyter_output><empty_output><jupyter_text>For now, we can return data from Elasticsearch, using the **TF-IDF** algorithm, with the `retrieve` method.<jupyter_code>query = "Physics is a very abstract subject"

retriever.retrieve(query)<jupyter_output><empty_output><jupyter_text>This query returns a huge number of duplicates. The reason we have these is because our data contained duplicates of the same context because each context could be tied to several different questions. So now, we need to restart by first deleting everything inside our *squad_docs* index. Then re-indexing our deduplicated data.We can delete every document in our index by sending a **POST** request to the `/_delete_by_query` endpoint:<jupyter_code>res = requests.post(
    'http://localhost:9200/squad_docs/_delete_by_query',
    json={
        'query': {
            'match_all': {}
        }
    }
)

res.json()<jupyter_output><empty_output><jupyter_text>Our response shows `'deleted': 16209`, which means all *16209* documents have been deleted from our *squad_docs* index. We can confirm this by calling the `/_count` endpoint too:<jupyter_code>res = requests.get('http://localhost:9200/squad_docs/_count')

res.json()<jupyter_output><empty_output><jupyter_text>Now that we've cleared the index, it's time to remove duplicates from our SQuAD contexts and re-index them.<jupyter_code># create list of contexts (we cannot do this using current dictionary format)
contexts = [sample['context'] for sample in squad]

# convert to set to remove duplicates, then back to list
contexts = list(set(contexts))

# convert back to dictionary format we need
squad_docs = [{'text': sample} for sample in contexts]<jupyter_output><empty_output><jupyter_text>Finally, we can re-index our Elasticsearch as we did before.<jupyter_code>document_store.write_documents(squad_docs)<jupyter_output>03/08/2022 09:06:03 - INFO - elasticsearch -   POST http://localhost:9200/_bulk?refresh=wait_for [status:200 request:0.234s]
03/08/2022 09:06:04 - INFO - elasticsearch -   POST http://localhost:9200/_bulk?refresh=wait_for [status:200 request:1.069s]
03/08/2022 09:06:05 - INFO - elasticsearch -   POST http://localhost:9200/_bulk?refresh=wait_for [status:200 request:1.072s]<jupyter_text>Because we have changed the contents of our index, we initialize our retriever once more.<jupyter_code>retriever = TfidfRetriever(document_store)<jupyter_output>03/14/2022 08:24:41 - INFO - elasticsearch -   POST http://localhost:9200/squad_docs/_search?scroll=1d&size=10000 [status:200 request:0.021s]
03/14/2022 08:24:41 - INFO - elasticsearch -   POST http://localhost:9200/_search/scroll [status:200 request:0.004s]
03/14/2022 08:24:41 - INFO - elasticsearch -   DELETE http://localhost:9200/_search/scroll [status:200 request:0.002s]
03/14/2022 08:24:41 - INFO - haystack.retriever.sparse -   Found 1204 candidate paragraphs from 1204 docs in DB<jupyter_text>And this time we see that our retriever found *1204* documents (much less than the *16209* we found before). Now it's time to query our data again!<jupyter_code>retriever.retrieve(query)<jupyter_output><empty_output><jupyter_text>Now we're returning a set of relevant documents, without duplicates.Finally, let's return back to the other *sparse retriever* that we can use with Elasticsearch. We already used **TF-IDF**, by switching `TfidfRetriever` for `ElasticsearchRetriever` we can switch to the **BM25** algorithm, which is an *improved* version of **TF-IDF** and is recommended by Haystack.So, let's initialize that and make another query with it.<jupyter_code># import BM25 retriever
from haystack.retriever.sparse import ElasticsearchRetriever

# intialize
retriever = ElasticsearchRetriever(document_store)

# and query
retriever.retrieve(query)<jupyter_output>03/14/2022 08:25:11 - INFO - elasticsearch -   POST http://localhost:9200/squad_docs/_search [status:200 request:0.037s]<jupyter_text>Okay great, this is a pretty big notebook but it covers everything we need to know to get started with Haystack + Elastic (and a little more). * * * Playground===========Below are some chunks of text where I fool around with the Haystack Library. Since the BM25 algo is better, we'll use this one.Let's start by ordering the results by its score<jupyter_code># import BM25 retriever
from haystack.retriever.sparse import ElasticsearchRetriever

# intialize
retriever = ElasticsearchRetriever(document_store)

# and query
query = "Physics is a very abstract subject"
results = retriever.retrieve(query)


for result in results:
    print(result.text[:128], result.score)
results[0].to_dict()
question_2 = 'What was the Bill of Rights constituted in 1689?'
results = retriever.retrieve(question_2)

answers = [result.to_dict() for result in results]
answers<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1052731231657673, 'fraction_numerical': 0.06399697055760674, 'mean_word_length': 5.333333333333333, 'pattern_counts': {'":': 0, '<': 63, '<?xml version=': 0, '>': 63, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '304393', 'license': 'MIT', 'n_tokens_mistral': 3743, 'n_tokens_neox': 3277, 'n_words': 1054, 'path': 'course/question_answering/09_elastic_in_haystack.ipynb', 'repo_name': 'augustodn/NPL_with_Transformers', 'size': '10563'}