<jupyter_start><jupyter_text>Imports<jupyter_code>%load_ext autoreload
%autoreload 2
from pathlib import Path
import pandas as pd
from tqdm import tqdm
import json
import os
from subprocess import Popen, PIPE, STDOUT
import matplotlib.pyplot as plt

from haystack.preprocessor.cleaning import clean_wiki_text
from haystack.preprocessor.utils import convert_files_to_dicts, fetch_archive_from_http
from haystack.reader.farm import FARMReader
from haystack.reader.transformers import TransformersReader
from haystack.utils import print_answers
from haystack.document_store.elasticsearch import ElasticsearchDocumentStore
from haystack.document_store.faiss import FAISSDocumentStore
from haystack.retriever.sparse import ElasticsearchRetriever
from haystack.pipeline import ExtractiveQAPipeline
from haystack.generator.transformers import RAGenerator
from haystack.retriever.dense import DensePassageRetriever<jupyter_output>03/09/2021 09:39:05 - INFO - faiss -   Loading faiss with AVX2 support.
03/09/2021 09:39:05 - INFO - faiss -   Loading faiss.<jupyter_text>Load data The subjQA dataset covers 6 categories, with a trian/dev/test split per category:<jupyter_code>data = Path('./data/subjqa')
!ls {data}
!ls {data/'electronics/splits'}<jupyter_output>dev.csv  test.csv  train.csv<jupyter_text>Let's start by picking a single category like `electronics` and loading each CSV in a DataFrame:<jupyter_code>category = "electronics"
dfs = {p.stem: pd.read_csv(p) for p in (data/'electronics/splits/').glob("*.csv")}

for k,v in dfs.items(): display(v.head(1))<jupyter_output><empty_output><jupyter_text>From the [paper](https://arxiv.org/pdf/2004.14283.pdf), here are the breakdown of examples per category / split:| Domain | Train | Dev | Test | Total || :---: | :---: | :---: | :---: | :---: || TripAdvisor  | 1165 | 230 | 512 | 1686 || Restaurants | 1400 | 267 | 266 | 1683 || Movies | 1369 | 261 | 291 | 1677 || Books | 1314 | 256 | 345 | 1668 || Electronics | 1295 | 255 | 358 | 1659 || Grocery | 1124 | 218 | 591 | 1725 |Let's try to verify them by counting the number of unique question IDs per split:<jupyter_code>for k,v in dfs.items(): print(k, v["q_review_id"].nunique())<jupyter_output>train 1295
test 358
dev 255<jupyter_text>Looks good.<jupyter_code>dfs["train"].shape
# peek at single example with valid answer
row = dfs["train"].query("human_ans_spans != 'ANSWERNOTFOUND'").iloc[0]
row
print(f"Question: {row['question']}", "\n")
print(f"Answer: {row['human_ans_spans']}", "\n")
print(f"Review: {row['review']}")
# check spans match indices
start_idx, end_idx = eval(row["human_ans_indices"])
row['review'][start_idx:end_idx]<jupyter_output><empty_output><jupyter_text>I noticed that some spans don't match the indices, let's take a look:<jupyter_code>train_df = dfs["train"].copy()
# convert span indices to tuples
train_df["human_ans_indices"] = train_df["human_ans_indices"].apply(lambda x : eval(x))
def extract_answer_spans(row):
    start_idx, end_idx = row["human_ans_indices"]
    return row["review"][start_idx:end_idx]
train_df["answer_spans"] = train_df.apply(extract_answer_spans, axis=1)
train_df.head(1)
mask = train_df["human_ans_spans"].str.len() != train_df["answer_spans"].str.len()
train_df.loc[mask].head()[["human_ans_spans", "answer_spans"]]<jupyter_output><empty_output><jupyter_text>So it seems that some sort of tokenization has been introduced for some examples in the training set. How about the dev and test ones?<jupyter_code>for k,v in dfs.items():
    # convert to tuples
    v["human_ans_indices"] = v["human_ans_indices"].apply(lambda x : eval(x))
    v["answer_spans"] = v.apply(extract_answer_spans, axis=1)
    mask = v["human_ans_spans"].str.len() != v["answer_spans"].str.len()
    print(f"{k} fraction of labelling errors: {100 * len(v.loc[mask]) / len(v):.2f}%", "\n")<jupyter_output>train fraction of labelling errors: 7.76% 

test fraction of labelling errors: 7.52% 

dev fraction of labelling errors: 7.28%<jupyter_text>Assume we can use the indices as answer spans!  Convert to SQuAD format Need this format```json{    "data": [        {            "title": "Beyoncé",            "paragraphs": [                {                    "qas": [                        {                            "question": "When did Beyonce start becoming popular?",                            "id": "56be85543aeaaa14008c9063",                            "answers": [                                {                                    "text": "in the late 1990s",                                    "answer_start": 269                                }                            ],                            "is_impossible": false                        }                        ...                    ],                    "context": "Beyoncé ..."                },                ...            ]        }    ]}```<jupyter_code># peek at single row
row = train_df.query("human_ans_spans != 'ANSWERNOTFOUND'").iloc[0]
row<jupyter_output><empty_output><jupyter_text>So we could have the following field mappings:* title -> item_id* question -> question* id -> q_review_id* answers -> answer_spans (corrects for problems in human_ans_spans)* answer_start -> first element of human_ans_indices* is_impossible -> if human_ans_spans == ANSWERNOTFOUND* context -> review Groupby `item_id` and build data structure from that? First build up paragraphs from row:<jupyter_code>pars = [
    {"qas": 
     [
         {"question": row["question"], 
          "id": row["q_review_id"], 
          "answers": [
              {"text": row["answer_spans"], 
               "answer_start": row["human_ans_indices"][0]}
          ], 
          "is_impossible": True if row["answer_spans"] == "ANSWERNOTFOUND" else False}],
     "context": row["review"]
    }]
pars<jupyter_output><empty_output><jupyter_text>Test on a subset of two items:<jupyter_code>sample_df = train_df.query("item_id == 'B006ZS5ATM' | item_id == 'B0074BW614'")
sample_df.head(1)
sample_df.shape
sample_df.groupby('item_id')['q_review_id'].count()
def create_paragraphs(group):
    pars = []
    
    for idx, row in group.iterrows():
        qas = {"qas": [
            {"question": row["question"],
             "id": row["q_review_id"], 
             "answers": [
                 {"text": row["answer_spans"], 
                  "answer_start": row["human_ans_indices"][0]}
                 ],
             "is_impossible": True if row["answer_spans"] == "ANSWERNOTFOUND" else False}
        ],
               "context": row["review"]
               }
        pars.append(qas)
        
    return pars
groups = sample_df.groupby('item_id').apply(create_paragraphs).to_frame(name="paragraphs").reset_index().rename(columns={"item_id":'title'})
groups
len(groups.iloc[1]["paragraphs"])
groups.to_dict(orient='records')[0]
def convert_to_squad(dfs, output_path: Path):
    for split, df in dfs.items():
        groups = df.groupby('item_id').apply(create_paragraphs).to_frame(name="paragraphs").reset_index().rename(columns={"item_id":'title'})
        squad_data = {}
        squad_data["data"] = groups.to_dict(orient='records')

        with open(output_path/f"{category}-{split}.json", 'w', encoding='utf-8') as f:
            json.dump(squad_data, f)
squad_path = Path("data/subjqa/squad_format")
convert_to_squad(dfs, squad_path)<jupyter_output><empty_output><jupyter_text>Evaluate reader Paper uses BERT-base, so let's try RoBERTa-base which should be a little better:<jupyter_code>reader = FARMReader(model_name_or_path="deepset/roberta-base-squad2", use_gpu=True, context_window_size=500)
reader_eval_results = reader.eval_on_file(squad_path, f"{category}-test.json", device='cuda')
print("Reader Top-N-Accuracy:", reader_eval_results["top_n_accuracy"])
print("Reader Exact Match:", reader_eval_results["EM"])
print("Reader F1-Score:", reader_eval_results["f1"])<jupyter_output>Reader Top-N-Accuracy: 0.7047353760445683
Reader Exact Match: 0.2395543175487465
Reader F1-Score: 0.30428376775021326<jupyter_text>In the paper they get an F1-score of around 23-24 which is compatible with the 6-7 point difference one sees for BERT vs RoBERTa on SQuAD.  Fine-tune<jupyter_code>reader.train(data_dir=squad_path,
             train_filename=f"{category}-train.json", 
             dev_filename=f"{category}-dev.json", 
             use_gpu=True, n_epochs=1, save_dir="models/haystack/subjqa/roberta-base-squad2-subjqa",
             batch_size=16,
             evaluate_every=100)
finetuned_reader =  FARMReader(model_name_or_path="models/haystack/subjqa/roberta-base-squad2-subjqa", 
                               use_gpu=True, context_window_size=500)
finetuned_reader_eval_results = finetuned_reader.eval_on_file(squad_path, f"{category}-test.json", device='cuda')
print("Reader Top-N-Accuracy:", finetuned_reader_eval_results["top_n_accuracy"])
print("Reader Exact Match:", finetuned_reader_eval_results["EM"])
print("Reader F1-Score:", finetuned_reader_eval_results["f1"])<jupyter_output>Preprocessing Dataset data/subjqa/squad_format/electronics-test.json: 100%|██████████| 718/718 [00:01<00:00, 408.99 Dicts/s]
Evaluating: 100%|██████████| 35/35 [00:19<00:00,  1.76it/s]<jupyter_text>Again, the paper shows values slightly below 0.6 F1-score so we're consistent there.<jupyter_code>pd.DataFrame.from_dict(results)
results = {'RoBERTa-baseline':reader_eval_results, 'RoBERTa-finetuned': finetuned_reader_eval_results}
results_df = pd.DataFrame.from_dict(results)
results_df
results_df.plot(kind='bar')
plt.show()<jupyter_output><empty_output><jupyter_text>Retrieval playground  Boot ES<jupyter_code>! wget -nc https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.9.2-linux-x86_64.tar.gz -q
! tar -xzf elasticsearch-7.9.2-linux-x86_64.tar.gz
! chown -R daemon:daemon elasticsearch-7.9.2

es_server = Popen(['elasticsearch-7.9.2/bin/elasticsearch'],
                   stdout=PIPE, stderr=STDOUT,
                   preexec_fn=lambda: os.setuid(1)  # as daemon
                  )
# wait until ES has started
! sleep 30
document_store = ElasticsearchDocumentStore(host="localhost", username="", password="", index="document")
document_store.delete_all_documents()<jupyter_output>03/07/2021 16:26:49 - INFO - elasticsearch -   POST http://localhost:9200/document/_delete_by_query [status:200 request:4.074s]<jupyter_text>Index docs<jupyter_code>row
train_df["answer_spans"].apply(lambda x : True if x != "ANSWERNOTFOUND" else False)
for k,v in dfs.items():
    v["is_answerable"] = v["answer_spans"].apply(lambda x : 1 if x != "ANSWERNOTFOUND" else 0)
dfs["train"].iloc[0]
docs = [{"text": row["review"], "meta":{"item_id": row["item_id"], "is_answerable": row["is_answerable"]}} for _, row in dfs["test"].iterrows()]
docs[0]
document_store.write_documents(docs)<jupyter_output>03/07/2021 16:26:57 - INFO - elasticsearch -   POST http://localhost:9200/_bulk?refresh=wait_for [status:200 request:1.158s]
03/07/2021 16:26:58 - INFO - elasticsearch -   POST http://localhost:9200/_bulk?refresh=wait_for [status:200 request:1.220s]<jupyter_text>Retriever<jupyter_code>retriever = ElasticsearchRetriever(document_store=document_store)<jupyter_output><empty_output><jupyter_text>Reader<jupyter_code>reader = FARMReader(model_name_or_path="deepset/roberta-base-squad2", use_gpu=True, context_window_size=500)
pipe = ExtractiveQAPipeline(reader, retriever)
dfs["test"].query("is_answerable == 1").iloc[0]
query = "is the keyboard good for writing??"
# Logitech keyboard: https://www.amazon.com/dp/B005DKZTMG
asin = "B005DKZTMG"
number_of_answers_to_fetch = 3

prediction = pipe.run(query=query, filters={"item_id": [asin]}, top_k_retriever=10, top_k_reader=number_of_answers_to_fetch)
print(f"Question: {prediction['query']}")
print("\n")
for i in range(number_of_answers_to_fetch):
    print(f"#{i+1}")
    print(f"Answer: {prediction['answers'][i]['answer']}")
    print(f"ASIN: {prediction['answers'][i]['meta']['item_id']}")
    print(f"Is answerable?: {prediction['answers'][i]['meta']['is_answerable']}")
    print(f"Context: {prediction['answers'][i]['context']}")
    print('\n\n')<jupyter_output>03/07/2021 16:33:51 - INFO - elasticsearch -   POST http://localhost:9200/document/_search [status:200 request:0.009s]
Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 14.51 Batches/s]
Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 11.52 Batches/s]
Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 20.95 Batches/s]
Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 29.75 Batches/s]
Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 45.54 Batches/s]
Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 46.99 Batches/s]
Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 46.58 Batches/s]
Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 46.66 Batches/s]
Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 46.99 Batches/s]
Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 46.98 Batches/s]<jupyter_text>Generative answers<jupyter_code>faiss_document_store = FAISSDocumentStore(
    faiss_index_factory_str="Flat",
    return_embedding=True
)
retriever = DensePassageRetriever(
    document_store=faiss_document_store,
    query_embedding_model="facebook/dpr-question_encoder-single-nq-base",
    passage_embedding_model="facebook/dpr-ctx_encoder-single-nq-base",
    use_gpu=True,
    embed_title=True,
)

generator = RAGenerator(
    model_name_or_path="facebook/rag-token-nq",
    use_gpu=True,
    top_k_answers=1,
    max_length=200,
    min_length=2,
    embed_title=True,
    num_beams=2,
)
docs = [{"text": row["review"], "meta":{"name": row["item_id"], "is_answerable": row["is_answerable"]}} for _, row in dfs["test"].iterrows()]
docs[0]
faiss_document_store.delete_all_documents()

faiss_document_store.write_documents(docs)

faiss_document_store.update_embeddings(
    retriever=retriever)

from haystack.pipeline import GenerativeQAPipeline

query= "does it work with ps4?"
pipe = GenerativeQAPipeline(generator=generator, retriever=retriever)
prediction = pipe.run(query=query, top_k_generator=1, top_k_retriever=5)
answers = prediction["answers"]
print()
print(f"Question: {query}")
print(f"RAG answer: {answers[0]['answer'].lower()}")<jupyter_output>Question: does it work with ps4?
RAG answer: there were a lot of whining about defective units
{'fraction_non_alphanumeric': 0.14839879997209238, 'fraction_numerical': 0.044652201213981724, 'mean_word_length': 4.458492003046459, 'pattern_counts': {'":': 38, '<': 75, '<?xml version=': 0, '>': 70, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 16, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '590084', 'license': 'Apache-2.0', 'n_tokens_mistral': 5548, 'n_tokens_neox': 4977, 'n_words': 948, 'path': 'experiments.haystack-subjqa.ipynb', 'repo_name': 'lewtun/transformerlab', 'size': '14333'}