<jupyter_start><jupyter_text>cdQA: Closed Domain Question Answering!An End-To-End Closed Domain Question Answering System. Built on top of the HuggingFace transformers library.(though it not maintained now, they have shiften development to [Haystack](https://github.com/deepset-ai/haystack).<jupyter_code># importing the necessary libraries
import os
import pandas as pd
from ast import literal_eval
import urllib.request

from cdqa.utils.converters import pdf_converter
from cdqa.utils.filters import filter_paragraphs
from cdqa.pipeline import QAPipeline
from cdqa.utils.download import download_model
download_model(model='bert-squad_1.1', dir='./models')   # distilbert-squad model can also be downloaded as per requirement.<jupyter_output>/opt/conda/lib/python3.7/site-packages/tqdm/autonotebook/__init__.py:18: TqdmExperimentalWarning: Using `tqdm.autonotebook.tqdm` in notebook mode. Use `tqdm.tqdm` instead to force console mode (e.g. in jupyter console)
  " (e.g. in jupyter console)", TqdmExperimentalWarning)<jupyter_text>Downloading Tourist Guide files from different URLs<jupyter_code>urllib.request.urlretrieve('https://www.traveliteindia.com/uimages/downloads/45.pdf', "1.pdf")
urllib.request.urlretrieve('https://www.approachguides.com/ebook-previews/Preview-ApproachGuides-India-Delhi-Agra-Architecture.pdf', "2.pdf")
urllib.request.urlretrieve('http://ficci.in/Sector/Report/20294/Unexplored-tourism-destinations-of-India.pdf', "3.pdf")
urllib.request.urlretrieve('https://pubs.iied.org/sites/default/files/pdfs/migrate/17646IIED.pdf')<jupyter_output><empty_output><jupyter_text>**pdf_converter** can create a cdqa dataframe from a directory containing .pdf files<jupyter_code>df = pdf_converter(directory_path='./')<jupyter_output>2021-01-27 14:48:20,668 [MainThread  ] [INFO ]  Retrieving http://search.maven.org/remotecontent?filepath=org/apache/tika/tika-server/1.19/tika-server-1.19.jar to /tmp/tika-server.jar.
2021-01-27 14:48:24,885 [MainThread  ] [INFO ]  Retrieving http://search.maven.org/remotecontent?filepath=org/apache/tika/tika-server/1.19/tika-server-1.19.jar.md5 to /tmp/tika-server.jar.md5.
2021-01-27 14:48:25,976 [MainThread  ] [WARNI]  Failed to see startup log message; retrying...<jupyter_text>Fitting the pipeline on corpus using the pre-trained reader<jupyter_code>cdqa_pipeline = QAPipeline(reader='./models/bert_qa.joblib', max_df=1.0, )
cdqa_pipeline.fit_retriever(df=df)<jupyter_output>100%|██████████| 231508/231508 [00:00<00:00, 893024.43B/s]<jupyter_text>Predications/QA!!<jupyter_code>query = 'when Jama Masjid was built?'
prediction = cdqa_pipeline.predict(query)
print('query: {}'.format(query))
print('answer: {}'.format(prediction[0]))
print('title: {}'.format(prediction[1]))
print('paragraph: {}'.format(prediction[2]))
query = 'Rajasthan is famous forts?'
prediction = cdqa_pipeline.predict(query)
print('query: {}'.format(query))
print('answer: {}'.format(prediction[0]))
print('title: {}'.format(prediction[1]))
print('paragraph: {}'.format(prediction[2]))
query = 'Where is Hampi?'
prediction = cdqa_pipeline.predict(query)
print('query: {}'.format(query))
print('answer: {}'.format(prediction[0]))
print('title: {}'.format(prediction[1]))
print('paragraph: {}'.format(prediction[2]))<jupyter_output>query: Where is Hampi?
answer: Karnataka
title: 3
paragraph: 1.7 Hampi (Karnataka) ...................................................................19
{'fraction_non_alphanumeric': 0.1675454012888108, 'fraction_numerical': 0.03895723491505566, 'mean_word_length': 5.969387755102041, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 17, 'https://': 4, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '373043', 'license': 'MIT', 'n_tokens_mistral': 1285, 'n_tokens_neox': 1199, 'n_words': 214, 'path': 'question-answer-bot-cdqa-w-india-tourist-guides.ipynb', 'repo_name': 'Ajax0564/TextSummary', 'size': '3414'}