<jupyter_start><jupyter_text>Long-Form Question Answering[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/deepset-ai/haystack/blob/master/tutorials/Tutorial12_LFQA.ipynb)  Prepare environment Colab: Enable the GPU runtimeMake sure you enable the GPU runtime to experience decent speed in this tutorial.  **Runtime -> Change Runtime type -> Hardware accelerator -> GPU**<jupyter_code># Make sure you have a GPU running
!nvidia-smi
# Install the latest master of Haystack
!pip install git+https://github.com/deepset-ai/haystack.git

# If you run this notebook on Google Colab, you might need to
# restart the runtime after installing haystack.
from haystack.utils import convert_files_to_dicts, fetch_archive_from_http, clean_wiki_text
from haystack.nodes import Seq2SeqGenerator<jupyter_output><empty_output><jupyter_text>Document StoreFAISS is a library for efficient similarity search on a cluster of dense vectors.The `FAISSDocumentStore` uses a SQL(SQLite in-memory be default) database under-the-hoodto store the document text and other meta data. The vector embeddings of the text areindexed on a FAISS Index that later is queried for searching answers.The default flavour of FAISSDocumentStore is "Flat" but can also be set to "HNSW" forfaster search at the expense of some accuracy. Just set the faiss_index_factor_str argument in the constructor.For more info on which suits your use case: https://github.com/facebookresearch/faiss/wiki/Guidelines-to-choose-an-index<jupyter_code># from haystack.document_stores import FAISSDocumentStore
# document_store = FAISSDocumentStore(vector_dim=128, faiss_index_factory_str="Flat")
# Download and install Ealstic Search not required for windows
# ! wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.9.2-linux-x86_64.tar.gz -q
# ! tar -xzf elasticsearch-7.9.2-linux-x86_64.tar.gz
# ! chown -R daemon:daemon elasticsearch-7.9.2

import os
from subprocess import Popen, PIPE, STDOUT
es_server = Popen(['elasticsearch-7.9.2/bin/elasticsearch'],
                   stdout=PIPE, stderr=STDOUT,
                   preexec_fn=lambda: os.setuid(1)  # as daemon
                  )
# wait until ES has started
! sleep 30

from haystack.utils import launch_es
launch_es()
from haystack.document_stores import ElasticsearchDocumentStore
document_store = ElasticsearchDocumentStore(host="localhost", username="", password="",
                                            index="document", return_embedding=True)<jupyter_output><empty_output><jupyter_text>Cleaning & indexing documentsSimilarly to the previous tutorials, we download, convert and index some Game of Thrones articles to our DocumentStore<jupyter_code># Let's first get some files that we want to use
doc_dir = "data/article_txt_got"
s3_url = "https://s3.eu-central-1.amazonaws.com/deepset.ai-farm-qa/datasets/documents/wiki_gameofthrones_txt.zip"
fetch_archive_from_http(url=s3_url, output_dir=doc_dir)

# Convert files to dicts
dicts = convert_files_to_dicts(dir_path=doc_dir, clean_func=clean_wiki_text, split_paragraphs=True)
    
# Now, let's write the dicts containing documents to our DB.
document_store.write_documents(dicts)<jupyter_output><empty_output><jupyter_text>Initalize Retriever and Reader/Generator Retriever**Here:** We use a `RetribertRetriever` and we invoke `update_embeddings` to index the embeddings of documents in the `FAISSDocumentStore`<jupyter_code># from haystack.nodes import EmbeddingRetriever

# retriever = EmbeddingRetriever(document_store=document_store,
#                                embedding_model="yjernite/retribert-base-uncased",
#                                model_format="retribert")

# document_store.update_embeddings(retriever)

from haystack.nodes import DensePassageRetriever
retriever = DensePassageRetriever(document_store=document_store,
                                  query_embedding_model="facebook/dpr-question_encoder-single-nq-base",
                                  passage_embedding_model="facebook/dpr-ctx_encoder-single-nq-base",
                                  max_seq_len_query=64,
                                  max_seq_len_passage=256,
                                  batch_size=16,
                                  use_gpu=True,
                                  embed_title=True,
                                  use_fast_tokenizers=True)
# document_store.update_embeddings(retriever)
# document_store.save(index_path='FAISS_GoT')
# document_store.load(index_path='FAISS_GoT')<jupyter_output><empty_output><jupyter_text>Before we blindly use the `RetribertRetriever` let's empirically test it to make sure a simple search indeed finds the relevant documents.<jupyter_code>from haystack.utils import print_documents
from haystack.pipelines import DocumentSearchPipeline

p_retrieval = DocumentSearchPipeline(retriever)
res = p_retrieval.run(
    query="Tell me something about <NAME>?",
    params={"Retriever": {"top_k": 10}}
)
print_documents(res, max_text_len=512)<jupyter_output>Query: Tell me something about <NAME>?

{   'content': '\n'
               '=== Background ===\n'
               'Arya is the third child and younger daughter of Eddard and '
               '<NAME> and is nine years old at the beginning of the '
               'book series.  She has five siblings: an older brother Robb, an '
               'older sister Sansa, two younger brothers Bran and Rickon, and '
               'an older illegitimate half-brother, <NAME>.',
    'name': '43_Arya_Stark.txt'}

{   'content': '\n'
               "===''A Feast for Crows''===\n"
               "After Lysa's death, Sansa becomes mistress of the Eyrie and "
               "still pretends to be Baelish's illegitimate daughter, Alayne "
               'Stone. Baelish successfully pacifies the lords of the Vale, '
               "who suspected Baelish's hand in Lysa's death. Afterwards, "
               'Baelish reveals to Sansa his plans to eventually marry her to '
               'the heir t[...]<jupyter_text>Reader/GeneratorSimilar to previous Tutorials we now initalize our reader/generator.Here we use a `Seq2SeqGenerator` with the *yjernite/bart_eli5* model (see: https://huggingface.co/yjernite/bart_eli5)<jupyter_code>generator = Seq2SeqGenerator(model_name_or_path="yjernite/bart_eli5")<jupyter_output><empty_output><jupyter_text>PipelineWith a Haystack `Pipeline` you can stick together your building blocks to a search pipeline.Under the hood, `Pipelines` are Directed Acyclic Graphs (DAGs) that you can easily customize for your own use cases.To speed things up, Haystack also comes with a few predefined Pipelines. One of them is the `GenerativeQAPipeline` that combines a retriever and a reader/generator to answer our questions.You can learn more about `Pipelines` in the [docs](https://haystack.deepset.ai/docs/latest/pipelinesmd).<jupyter_code>from haystack.pipelines import GenerativeQAPipeline
pipe = GenerativeQAPipeline(generator, retriever)<jupyter_output><empty_output><jupyter_text>Voilà! Ask a question!<jupyter_code>print_documents(pipe.run(
    query="What is Arya's sister's name?",
    params={"Retriever": {"top_k": 1}}
))
print_documents(pipe.run(query="Who was the luckiest character in the show?", params={"Retriever": {"top_k": 1}}))<jupyter_output>Query: Who was the luckiest character in the show?

{   'content': '\n'
               '==Production==\n'
               "As with the previous episode, the show's opening title "
               'sequence is modified to depict the characters in their '
               'role-playing garb, while the soundtrack has been altered to '
               "include the penis-themed chorus singing to the ''Game of "
               "Thrones'' opening theme introduced in the previous episode. "
               'Series co-creators <NAME> and <NAME> said that they '
               'experimented with different styles of opening sequences before '
               'settling on the penis-themed chorus version; a Japanese '
               'Princess Kenny opening sequence was one of the original ideas.',
    'name': '101_Titties_and_Dragons.txt'}
{'fraction_non_alphanumeric': 0.08468975645575817, 'fraction_numerical': 0.006975890343899156, 'mean_word_length': 3.627406568516421, 'pattern_counts': {'":': 6, '<': 37, '<?xml version=': 0, '>': 40, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 13, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '562275', 'license': 'Apache-2.0', 'n_tokens_mistral': 2483, 'n_tokens_neox': 2356, 'n_words': 690, 'path': 'ESDS_GoT_LFQA_via_Haystack.ipynb', 'repo_name': 'superchargez/haystack', 'size': '8197'}