<jupyter_start><jupyter_text>MVP Cognitive Search Application using "bertsquad" model  Importing Libraries<jupyter_code>import os
import pandas as pd
from ast import literal_eval

from cdqa.utils.converters import pdf_converter
from cdqa.pipeline import QAPipeline
from cdqa.utils.download import download_model<jupyter_output><empty_output><jupyter_text>Downloading Pre-trained model: BERT (Stanford question and answer data set)<jupyter_code>download_model(model = 'bert-squad_1.1', dir='./models')<jupyter_output>Downloading trained model...
bert_qa.joblib already downloaded<jupyter_text>Converting text from PDF into Pandas dataframe<jupyter_code>df = pdf_converter(directory_path='./PDF_docs/')
df.head(4)<jupyter_output>2021-03-05 17:50:33,772 [MainThread  ] [WARNI]  Failed to see startup log message; retrying...<jupyter_text>Using pre-trained language model: bertsquad<jupyter_code>cdqa_pipeline = QAPipeline(reader='./models/bert_qa.joblib', max_df = 1.0)
cdqa_pipeline.fit_retriever(df=df)<jupyter_output><empty_output><jupyter_text>Questions from technical information document:<jupyter_code>question_1 = 'What is the shelf life of Golpanol ALS?'
prediction = cdqa_pipeline.predict(question_1)
print('Question : {}'.format(question_1))
print('Answer: {}'.format(prediction[0]))
question_2 = 'What is the appearance of Golpanol ALS?'
prediction = cdqa_pipeline.predict(question_2)
print('Question : {}'.format(question_2))
print('Answer: {}'.format(prediction[0]))
question_3 = 'What is Golpanol ALS pH value?'
prediction = cdqa_pipeline.predict(question_3)
print('Question : {}'.format(question_3))
print('Answer: {}'.format(prediction[0]))<jupyter_output>Question : What is Golpanol ALS pH value?
Answer: 2 years<jupyter_text>Questions from product specification document:<jupyter_code>question_4 = 'What is the PRD number of Golpanol ALS?'
prediction = cdqa_pipeline.predict(question_4)
print('Question : {}'.format(question_4))
print('Answer: {}'.format(prediction[0]))
question_5 = 'What is the density value of Golpanol ALS?'
prediction = cdqa_pipeline.predict(question_5)
print('Question : {}'.format(question_5))
print('Answer: {}'.format(prediction[0]))
question_6 = 'What is the chemical nature of Golpanol ALS?'
prediction = cdqa_pipeline.predict(question_6)
print('Question : {}'.format(question_6))
print('Answer: {}'.format(prediction[0]))<jupyter_output>Question : What is the chemical nature of Golpanol ALS?
Answer: Solubility
{'fraction_non_alphanumeric': 0.1230142566191446, 'fraction_numerical': 0.019144602851323828, 'mean_word_length': 8.409961685823754, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '276246', 'license': 'MIT', 'n_tokens_mistral': 839, 'n_tokens_neox': 806, 'n_words': 184, 'path': 'MVP_cognitive_search_bertsquad.ipynb', 'repo_name': 'ajaykrsna/mvp-cognitive-search-using-nlp', 'size': '2455'}