<jupyter_start><jupyter_text>Building a document answering model In this section, let us learn how to use ktrain for building a document answering model. We know that in document answering, we will have some set of documents and we use those documents to answer a question. Let us see how we can do this using ktrain. First, let us import the necessary libraries:<jupyter_code>%%capture
!pip install ktrain==0.25.3
from ktrain import text
import os
import shutil<jupyter_output><empty_output><jupyter_text>In this exercise, we will use the BBC news dataset. The BBC news dataset consists of 2225 documents containing news from 2004 to 2005. It includes news in these five categories - business, entertainment, politics, sport, tech. First, let us download the dataset:<jupyter_code>!wget http://mlg.ucd.ie/files/datasets/bbc-fulltext.zip<jupyter_output>--2020-12-30 22:23:23--  http://mlg.ucd.ie/files/datasets/bbc-fulltext.zip
Resolving mlg.ucd.ie (mlg.ucd.ie)... 172.16.58.3
Connecting to mlg.ucd.ie (mlg.ucd.ie)|172.16.58.3|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2874078 (2.7M) [application/zip]
Saving to: ‘bbc-fulltext.zip’

bbc-fulltext.zip    100%[===================>]   2.74M  3.76MB/s    in 0.7s    

2020-12-30 22:23:24 (3.76 MB/s) - ‘bbc-fulltext.zip’ saved [2874078/2874078]<jupyter_text>Next, unzip the dataset:<jupyter_code>%%capture
!unzip bbc-fulltext.zip<jupyter_output><empty_output><jupyter_text>Now, import the necessary libraries:<jupyter_code>from ktrain import text
import os<jupyter_output><empty_output><jupyter_text>Change directory to our BBC news folder:<jupyter_code>os.chdir(os.getcwd() + '/bbc')<jupyter_output><empty_output><jupyter_text>The first step is initializing the index directory. It is used for indexing all the documents. We don't need to create any new directory manually. We just need to pass the name of the index directory to the function intialize_index and then the index directory will be created:<jupyter_code>text.SimpleQA.initialize_index('index')<jupyter_output><empty_output><jupyter_text>After initializing the index, we need to index the documents. Since we will have all the documents in the folder, we will use the function index_from_folder.  The function index_from_folder takes the folder_path where we have all the documents and the index_dir as the parameter:<jupyter_code>text.SimpleQA.index_from_folder(folder_path='entertainment',index_dir='index')<jupyter_output><empty_output><jupyter_text>The next step is creating an instance of the SimpleQA class as shown below. We need to pass the index directory as an argument:<jupyter_code>qa = text.SimpleQA('index')<jupyter_output><empty_output><jupyter_text>That's it. Now, we can use the ask function and retrieve answers from the documents for any question:<jupyter_code>answers = qa.ask('who who had a global hit with where is the love?')<jupyter_output><empty_output><jupyter_text>Let us print the top 5 answers:<jupyter_code>qa.display_answers(answers[:5])<jupyter_output><empty_output><jupyter_text>As we can observe, along with obtaining the candidate answer, we will get other information such as context, confidence, and document reference. Let us try with another question:<jupyter_code>answers = qa.ask('who win at mtv europe awards?')
qa.display_answers(answers[:5])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09624553039332538, 'fraction_numerical': 0.03128724672228844, 'mean_word_length': 6.082278481012659, 'pattern_counts': {'":': 0, '<': 44, '<?xml version=': 0, '>': 45, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '158639', 'license': 'MIT', 'n_tokens_mistral': 1146, 'n_tokens_neox': 1064, 'n_words': 354, 'path': 'Chapter09/.ipynb_checkpoints/9.08. Building a document answering model -checkpoint.ipynb', 'repo_name': 'shizukanaskytree/Getting-Started-with-Google-BERT', 'size': '3360'}