<jupyter_start><jupyter_text>Notebook [2]: Using the PDF converter This notebook shows how to use the PDF converter to create an input dataframe for the cdQA pipeline from a directory of PDF files. ***Note:*** *To run this notebook you will need to have access to GPU. If you are using colab, you will need to install `cdQA` by executing `!pip install cdqa` in a cell.*<jupyter_code>!pip install cdqa
import os
import pandas as pd
from ast import literal_eval

from cdqa.utils.converters import pdf_converter
from cdqa.utils.filters import filter_paragraphs
from cdqa.pipeline import QAPipeline
from cdqa.utils.download import download_model<jupyter_output>/usr/local/lib/python3.6/dist-packages/tqdm/autonotebook/__init__.py:18: TqdmExperimentalWarning: Using `tqdm.autonotebook.tqdm` in notebook mode. Use `tqdm.tqdm` instead to force console mode (e.g. in jupyter console)
  " (e.g. in jupyter console)", TqdmExperimentalWarning)<jupyter_text>Download pre-trained reader model and PDF files<jupyter_code># Download model
download_model(model='bert-squad_1.1', dir='./models')
# Download pdf files from BNP Paribas public news
def download_pdf():
    import os
    import wget
    directory = './data/pdf/'
    models_url = [
      'https://invest.bnpparibas.com/documents/1q19-pr-12648',
      'https://invest.bnpparibas.com/documents/4q18-pr-18000',
      'https://invest.bnpparibas.com/documents/4q17-pr'
    ]

    print('\nDownloading PDF files...')

    if not os.path.exists(directory):
        os.makedirs(directory)
    for url in models_url:
        wget.download(url=url, out=directory)

download_pdf()<jupyter_output>Downloading PDF files...<jupyter_text>Convert the PDF files into a DataFrame for cdQA pipeline<jupyter_code>df = pdf_converter(directory_path='./data/pdf/')
df.head()<jupyter_output>2019-12-27 19:11:04,422 [MainThread  ] [INFO ]  Retrieving http://search.maven.org/remotecontent?filepath=org/apache/tika/tika-server/1.19/tika-server-1.19.jar to /tmp/tika-server.jar.
2019-12-27 19:11:04,971 [MainThread  ] [INFO ]  Retrieving http://search.maven.org/remotecontent?filepath=org/apache/tika/tika-server/1.19/tika-server-1.19.jar.md5 to /tmp/tika-server.jar.md5.
2019-12-27 19:11:05,359 [MainThread  ] [WARNI]  Failed to see startup log message; retrying...<jupyter_text>Instantiate the cdQA pipeline from a pre-trained reader model<jupyter_code>cdqa_pipeline = QAPipeline(reader='./models/bert_qa.joblib', max_df=1.0)

# Fit Retriever to documents
cdqa_pipeline.fit_retriever(df=df)<jupyter_output>100%|██████████| 231508/231508 [00:00<00:00, 1245724.61B/s]<jupyter_text>Execute a query<jupyter_code>query = 'How many contracts did BNP Paribas Cardif sell in 2019?'
prediction = cdqa_pipeline.predict(query)<jupyter_output><empty_output><jupyter_text>Explore predictions<jupyter_code>print('query: {}'.format(query))
print('answer: {}'.format(prediction[0]))
print('title: {}'.format(prediction[1]))
print('paragraph: {}'.format(prediction[2]))<jupyter_output>query: How many contracts did BNP Paribas Cardif sell in 2019?
answer: 140,000
title: 1q19-pr-12648
paragraph: 3 Excluding PEL/CEL effects of +2 million euros compared to +1 million euros in the first quarter 2018    5 RESULTS AS AT 31 MARCH 2019   The new property and casualty offering launched in May 2018 as part of the partnership between  BNP Paribas Cardif and Matmut (Cardif IARD) recorded good growth with already  140,000 contracts sold as at 31 March 2019.   The business is accelerating individual customers’ mobile uses and developing self-care features with the roll-out of the conversational chatbots Telmi in the Mes comptes BNP Paribas app and Helloïz at Hello bank!.
{'fraction_non_alphanumeric': 0.10971957527906344, 'fraction_numerical': 0.049006261911244214, 'mean_word_length': 5.133555926544241, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 20, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 11, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '482832', 'license': 'Apache-2.0', 'n_tokens_mistral': 1368, 'n_tokens_neox': 1227, 'n_words': 343, 'path': 'examples/tutorial-use-pdf-converter-rh.ipynb', 'repo_name': 'swaption2009/cdQA', 'size': '3673'}