<jupyter_start><jupyter_text>Bookworm OverviewIn this project, you will build a simple question-answering agent that is able to learn from any text data you provide, and answer queries posed in natural language. You will use IBM Watson's cloud-based services to process the input text data and find relevant responses. Learning ObjectivesBy completing this project, you will learn how to:- Create a cloud-based NLP service instance and configure it.- Ingest a set of text documents using the service and analyze the results.- Accept questions in natural language and parse them.- Find relevant answers from the preprocessed text data. Getting StartedIn order to use Watson's cloud-based services, you first need to create an account on the [IBM Bluemix platform](https://console.ng.bluemix.net/).                                Then, for each service you want to use, you have to create an instance of that service. You can continue with the tasks below, and create a service instance when indicated.  1. Create and configure Discovery serviceCreate an instance of the **Discovery** service. You will use this to process a set of text documents, and _discover_ relevant facts and relationships.- Go to the [IBM Bluemix Catalog](https://console.bluemix.net/catalog/).- Select [Discovery](https://console.bluemix.net/catalog/services/discovery) service under the [AI](https://console.bluemix.net/catalog/?category=ai) category.- Enter a Service Name for that instance, e.g. `Discovery-Bookworm` and click **`Create`** button on the bottom right hand corner of the screen.- You should be able to see your newly-created service in your [Bluemix Apps Dashboard](https://console.bluemix.net/dashboard/apps).- Open the `Discovery-Bookworm` service instance and find your `Url` and `API Key` in **Credentials** section._Note: you will need the username and password when connecting to the service in the next steps shortly._  Connect to the service instanceLet's connect to the service instance you just created using IBM Watson's [Python SDK](https://github.com/watson-developer-cloud/python-sdk). You will first need to install the SDK:```bashpip install watson-developer-cloud```Now execute each code cell below using **`Shift+Enter`**, and complete any steps indicated by a **`TODO`** comment. For more information on the Discovery service, please read the [Documentation](https://www.ibm.com/watson/developercloud/doc/discovery/index.html) and look at the [API Reference](https://www.ibm.com/watson/developercloud/discovery/api/v1/?python) as needed.<jupyter_code># Install watson-developer-cloud
# This takes about a minute

!pip install --upgrade "watson-developer-cloud>=2.4.1"
# Usual Python imports
import sys
import os
import glob
import json

# BeautifulSoup, for parsing HTML
from bs4 import BeautifulSoup

# Matplotlib, for plotting
import matplotlib.pyplot as plt
%matplotlib inline

# Watson Python SDK
# If not installed run: 
# !pip install --upgrade watson-developer-cloud
import watson_developer_cloud

# Utility functions
%load_ext autoreload
%aimport helper
%autoreload 1<jupyter_output>The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload<jupyter_text>Using the Service CredentialsBefore you can connect to Watson Service, you need to copy and paste `Username` and `Password` credentials from Bluemix Service console to this notebook._Note: these credentials are different from your IBM Bluemix login, and are specific to the service instance._1. Open `service-credentials.json` file:    * from this Jupyter Notebook top navigation, click `File` &rarr; `Open` &rarr; `service-credentials.json`2. Copy your `API Key` and `Url` from Discovery service console.3. Paste the credentials into `apikey` and `url` values in `Discovery` object.<jupyter_code>discovery_creds = helper.fetch_credentials('discovery')

discovery = watson_developer_cloud.DiscoveryV1(
                        version='2018-08-01',
                        url=discovery_creds['url'],
                        iam_apikey=discovery_creds['apikey'])<jupyter_output>/home/rostam/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:6: DeprecationWarning: watson-developer-cloud moved to ibm-watson. To get updates, use the new package.<jupyter_text>Create an environmentThe Discovery service organizes everything needed for a particular application in an [_environment_](https://www.ibm.com/watson/developercloud/discovery/api/v1/curl.html?curlenvironments-api). An environment must be created before collections of private data can be created.Let's create one called `Bookworm` for this project.> _**Note**: It is okay to run this block multiple times - it will not create duplicate environments with the same name._<jupyter_code># Prepare an environment to work in
env, env_id = helper.fetch_object(
    discovery, "environment", "Bookworm",
    create=True, create_args=dict(
        description="A space to read and understand stories"  # feel free to edit
    ))<jupyter_output>Found environment: Bookworm (fefe8de6-8a37-42f8-a375-19795a58391a)<jupyter_text>Verify configuration optionsA [_configuration_](https://www.ibm.com/watson/developercloud/discovery/api/v1/curl.html?curlconfigurations-api) defines what natural language processing routines are applied to any documents that are submitted to the service. Each environment gets a default configuration when it is created.You can fetch the default configuration and view the different options using the following piece of code.<jupyter_code># Lists existing configurations for the service instance and store default configuration id
configurations = discovery.list_configurations(environment_id=env_id).get_result()
cfg_id =  configurations['configurations'][0]['configuration_id']
print(json.dumps(configurations, indent=2))
# Get default configuration details
config = discovery.get_configuration(environment_id=env_id, configuration_id=cfg_id).get_result()
print(json.dumps(config, indent=2))<jupyter_output>{
  "configuration_id": "c1b7667c-c107-483b-b47d-107efba866e4",
  "name": "Default Configuration",
  "created": "2019-12-02T12:47:59.815Z",
  "updated": "2019-12-02T12:47:59.815Z",
  "description": "The configuration used by default when creating a new collection without specifying a configuration_id.",
  "conversions": {
    "html": {
      "exclude_content": {
        "xpaths": []
      },
      "exclude_tag_attributes": [
        "EVENT_ACTIONS"
      ],
      "exclude_tags_completely": [
        "script",
        "sup"
      ],
      "exclude_tags_keep_content": [
        "font",
        "em",
        "span"
      ],
      "keep_content": {
        "xpaths": []
      }
    },
    "json_normalizations": [],
    "pdf": {
      "heading": {
        "fonts": [
          {
            "level": 1,
            "max_size": 80,
            "min_size": 24
          },
          {
            "bold": false,
            "italic": false,
            "level": 2,
            "max_size": 24,
     [...]<jupyter_text>There are 3 main configuration blocks that affect how input documents are processed:1. **conversions**: How to convert documents in various formats (Word, PDF, HTML) and extract elements that indicate some structure (e.g. headings).2. **enrichments**: What NLP output results are we interested in (keywords, entities, sentiment, etc.).3. **normalizations**: Post-processing steps to be applied to the output. This can be left empty in most cases, unless you need the output to be normalized into a very specific format._**Note**: The default configuration for an environment cannot be modified. If you need to change any of the options, you will need to create a new one, and then edit it. The easiest way to do this is using the service dashboard, which is described later._  Testing Language EnrichmentIt is a good idea to test your configuration on a small sample text before you apply it to a larger document collection._**Note**: We have supplied a sample document (`data/sample.html`) containing the opening crawl text for Star Wars: Episode IV, but you are free to use a text of your choosing._**Q**: (optional) If you use your own sample text, provide a brief title and description below.**A**:<jupyter_code># Test configuration on some sample text
data_dir = "data"
filename = os.path.join(data_dir, "sample.html")
with open(filename, "r") as f:
    res = discovery.test_configuration_in_environment(environment_id=env_id, configuration_id=cfg_id, file=f).get_result()
#print(json.dumps(res, indent=2))<jupyter_output><empty_output><jupyter_text>Analyze test output The results returned by the service contain a _snapshot_ of the information extracted at each step of processing - document conversions, enrichments and normalizations. We are interested in the output of applying enrichments ("enrichments_output") or after normalizing them ("normalizations_output"). These should be identical if no post-processing/normalizations were specified in the configuration.<jupyter_code># Take a closer look at the results from the "enrichments_output" or "normalizations_output" step
output = next((s["snapshot"] for s in res["snapshots"] if s["step"] == "enrichments_output"), None)
print(json.dumps(output, indent=2))<jupyter_output><empty_output><jupyter_text>Answer the following questions based on the output above. Note that it contains the input HTML, extracted text and metadata as well as the actual enrichment results. Sentiment**Q**: What is the overall sentiment detected in this text? Mention the `type` (positive/negative) and `score`.(_Hint: Look for the `"sentiment"` key in the output._)**A**:  Concepts**Q**: List 3 concepts that have been identified with a relevance > 0.5. Note that not all concepts here may be present directly in the text, some may have been inferred by Watson.(_Hint: Look for `"concepts"`._)**A**: Get a good sense of all the different pieces of information available in the results. Start thinking about which ones will be useful for looking up answers to questions, and how you might use them.  2. Ingest documents Create a collectionA _collection_ is used to organize documents of the same kind. For instance, you may want to create a collection of book reviews, or a collection of Wikipedia articles, but it may not make much sense to mix the two groups. This allows Watson to make meaningful inferences over the set of documents, find commonalities and identify important concepts.Let's create a collection called `Story Chunks` in the Discovery service environment.<jupyter_code># Prepare a collection of documents to use
col, col_id = helper.fetch_object(discovery, "collection", "Story Chunks", environment_id=env_id,
    create=True, create_args=dict(
        environment_id=env_id, configuration_id=cfg_id,
        description="Stories and plots split up into chunks suitable for answering")
    )<jupyter_output><empty_output><jupyter_text>Once you have created a collection, you should be able to view it using the Discovery Service tool. Select the Discovery instance from your BlueMix dashboard.  To open, click the **`Launch tool`** button. Here you should see the `Story Chunks` collection you just created.You can open the collection to view more details about it. If you need to modify configuration options, click the **Switch** link and create a new configuration (the default one cannot be changed).  Add documentsOkay, now that we have everything set up, let's add a set of "documents" we want Watson to look up answers from, using the Python SDK. Note that Watson treats each "document" as a unit of text that is returned as the result of a query. But we want to retrieve a paragraph of text for each question. So, let's split each file up into individual paragraphs. We will use the [BeautifulSoup](https://www.crummy.com/software/BeautifulSoup/bs4/doc/) library for this purpose._**Note**: You could also add and manage documents in the collection using the Discovery tool, but you would have to split paragraphs up into separate files.__**Note**: We have provided a set of files (`data/Star-Wars/*.html`) with summary plots for Star Wars movies, but you are free to use a collection of your choice. Open one of the files in a text editor to see how the paragraphs are delimited using `...` tags - this is how the code block below split paragraphs into separate "documents"._<jupyter_code># Add documents to collection
doc_ids = []  # to store the generated id for each document added
for filename in glob.glob(os.path.join(data_dir, "Star-Wars", "*.html")):
    print("Adding file:", filename)
    with open(filename, "r") as f:
        # Split each individual <p> into its own "document"
        doc = f.read()
        soup = BeautifulSoup(doc, 'html.parser')
        for i, p in enumerate(soup.find_all('p')):
            doc_info = discovery.add_document(
                environment_id=env_id, 
                collection_id=col_id,
                file=json.dumps({"text": p.get_text(strip=True)}),
                filename='n',
                file_content_type="application/json",
                metadata=json.dumps({"title": soup.title.get_text(strip=True)})
            ).get_result() # add get_result() here

            doc_ids.append(doc_info["document_id"])
print("Total", len(doc_ids), "documents added.")<jupyter_output><empty_output><jupyter_text>If you look at the collection details, you may notice that the `"document_counts"` field now shows some documents as `available` or `processing`. Once processing is complete, you should see all the documents under the `available` count.<jupyter_code># View collection details to verify all documents have been processed
col, col_id = helper.fetch_object(discovery, "collection", "Story Chunks", environment_id=env_id)<jupyter_output><empty_output><jupyter_text>So, what did the Discovery service learn? If you list the fields extracted from the set of documents in the collection as part of the enrichment process, you'll see familiar fields like `concepts`, `entities` and `keywords` that were returned in the test analysis.<jupyter_code># List all fields extracted
discovery.list_collection_fields(environment_id=env_id, collection_id=col_id).get_result()<jupyter_output><empty_output><jupyter_text>Test queryLet's perform a simple query to see if the service can fetch the proper document for us:> _Look for all paragraphs that have a relation (sentence) with "Jar Jar" as the subject, and return the title and text._<jupyter_code># A simple query
results = discovery.query(environment_id=env_id, collection_id=col_id,
    query_options={
        "query": "enriched_text.relations.subject.text:\"Jar Jar\"",
        "return": "metadata.title,text"
    }).get_result()
print(json.dumps(results, indent=2))<jupyter_output><empty_output><jupyter_text>Change the above query and see what results you get! Try to find one that returns relevant results, and keep that (along with the output) for review.> See [Query building reference](https://www.ibm.com/watson/developercloud/doc/discovery/query-reference.html) for descriptions of all possible parameters, operators and aggregations. You can also choose to build the query using the web interface (click the "Story Chunks" collection to query it), and then reproduce the query here.Then answer the questions below:**Q**: What query did you try? Express it in plain words below.**A**:**Q**: What answer did you get back from Watson? You only need to mention the relevant snippet of text fro mthe paragraph(s) returned.**A**:   3. Parse natural language questionsIn order to understand questions posed in natural language, we'll use another AI service called [Watson Assistant](https://console.bluemix.net/catalog/services/watson-assistant-formerly-conversation) (Formerly 'Conversation'). It can be used to design conversational agents or _chatbots_ that exhibit complex behavior, but for the purpose of this project, we'll only use it to parse certain kinds of queries.  Create a Assistant service instanceJust like you did for the Discovery service, create an instance of the Assistant service. Then launch the associated tool from the service dashboard.- Go to the [IBM Bluemix Catalog](https://console.bluemix.net/catalog/).- Select [Watson Assistant (formerly Conversation)](https://console.bluemix.net/catalog/services/watson-assistant-formerly-conversation) service under the [AI](https://console.bluemix.net/catalog/?category=ai) category.- Enter a Service Name for that instance, e.g. `Assistant-Bookworm` and click **`Create`** button on the bottom right hand corner of the screen.- You should be able to see your newly-created service in your [Bluemix Apps Dashboard](https://console.bluemix.net/dashboard/apps).- Open the service instance and find your `Url` and `API Key` in **Credentials** section.- Copy *API Key* and *URL* into `service-credentials.json` file in this notebook:  Create a Workspace from Watson Assistant consoleA [_workspace_](https://www.ibm.com/watson/developercloud/assistant/api/v1/python.html?pythonworkspaces-api) allows you to keep all the items you need for a particular application in one place, just like an _environment_ in case of the Discovery service. From Watson Assistant console, please follow these steps:1) Click `Launch Tool` to start Watson Assistant.2) Click `Skills` tab on the navigation menu and click `Create new` button. 3) Create new skills called `Bookworm` with a suitable description, such as "I know a lot of stories. Ask me a question!". Clicking on `Bookworm` workspace should open up a blank workspace, where you can add intents, define the entities you want the agent to idenitfy and structure the overall dialog.  Add intentsAn [_intent_](https://www.ibm.com/watson/developercloud/assistant/api/v1/python.html?pythonintents-api) is the goal or purpose of a user's input. Intent will determine the dialog flows with the users and allow Watson Assistant to provide a useful response. Please read Watson Assistant documentation on [_Planning Your Intents and Entities_](https://console.bluemix.net/docs/services/conversation/intents-entities.htmlplanning-your-entities).Your task is to create a set of intents (at least 3) that capture the different kinds of questions that you want the system to answer, e.g. _who_, _what_ and _where_. Along with each intent, add a list of user examples or _utterances_ that map to that intent.For instance, you could enter the following examples for the _where_ intent:- Where is the Jedi temple located?- Where was Luke born?_Intent user examples should represent typical sentences that end users will use to interact with the application. The more examples you can provide for each intent, the better Watson Assistant will respond to the end user._> See [**Defining intents**](https://console.bluemix.net/docs/services/conversation/intents.htmldefining-intents) for a helpful video and further instructions.**Q**: What intents did you add to the Assistant service instance?**A**:**Q**: Pick one of these intents, and list at least 5 examples for the intent that you entered.**A**:  Add entitiesOnce you have your intents set, let's tell the service what [_entities_](https://www.ibm.com/watson/developercloud/assistant/api/v1/python.html?pythonentities-api) we want it to identify. One way to do this is using the `Entities` tool on Watson Assistant console, and entering them one-by-one to the blank `My entities` page.> Go to [**Defining entities**](https://console.bluemix.net/docs/services/conversation/entities.htmldefining-entities) to see how that is done.But that can be tedious! So let's refer back to the entities that the Discovery service identified, and load them in programmatically.As before, let's connect to the Assistant service first. Remember to enter your service credentials below.<jupyter_code># Connect to the Assistant service instance
# TODO: Enter your username and password from the Service Credentials tab in service-credentials.json

assistant_cred = helper.fetch_credentials('assistant')

assistant = watson_developer_cloud.AssistantV1(
                        version='2018-08-01',
                        url=assistant_cred['url'],
                        iam_apikey=assistant_cred['apikey'])<jupyter_output><empty_output><jupyter_text>Fetch the workspace you just created called "Bookworm".<jupyter_code>wrk, wrk_id = helper.fetch_object(assistant, "workspace", "Bookworm")<jupyter_output><empty_output><jupyter_text>Exporting entities from Discovery service to Assistant serviceNext step is to collect all the entities from the `Star Wars` documents that we added to the **Discovery** service collection and group them by entities type.<jupyter_code># Get all the entities from the collection and group them by type
response = discovery.query(environment_id=env_id, collection_id=col_id,
    query_options={
        "return": "enriched_text.entities.type,enriched_text.entities.text"
    }).get_result()

# Group individual entities by type ("Person", "Location", etc.)
entities_by_type = {}
for document in response["results"]:
    for entity in document["enriched_text"]["entities"]:
        if entity["type"] not in entities_by_type:
            entities_by_type[entity["type"]] = set()
        entities_by_type[entity["type"]].add(entity["text"])

# Ignore case to avoid duplicates
for entity_type in entities_by_type:
    entities_by_type[entity_type] = {
        e.lower(): e for e in entities_by_type[entity_type]
    }.values()

# Restructure for loading into Assistant workspace
entities_grouped = [{
    "entity": entity_type,
    "values": [{"value": entity} for entity in entities]}
        for entity_type, entities in entities_by_type.items()]
entities_grouped<jupyter_output><empty_output><jupyter_text>Update the workspace with these entities and verify that have been added correctly.<jupyter_code># Add these grouped entities to the Assistant workspace
assistant.update_workspace(workspace_id=wrk_id, entities=entities_grouped).get_result()

workspace_details = assistant.get_workspace(workspace_id=wrk_id, export=True).get_result()
print(json.dumps(workspace_details["entities"], indent=2))<jupyter_output><empty_output><jupyter_text>_**Note**: Ensure that at least 3 entity types, with at least 1 example entity each have been added._Here is what the list of entities should look like through the Assistant tool.**Q**: Name 3 entity types that were added, with at least 1 example entity each (e.g. entity type: _City_, example: _Los Angeles_).**A**:   Design dialog flowAs a final step in creating the Assistant interface, let's design a typical dialog with a user. The most intuitive way to do this is to use the Dialog tab in the tool. Here, you can add _nodes_ that capture different stages in the dialog flow, and connect them in a meaningful way.Go ahead and add at least 3 dialog nodes. Specify the triggers in terms of the intents and entities that you'd like to match, and an optional intermediate response like "Let me find that out for you." The actual response will be fetched by querying the Discovery service.Here is what the dialog nodes should look like.**Q**: Specify 3 dialog nodes you added, along with the trigger (intent and/or entities) for each.**A**:   Test dialogLet's run through a test dialog to demonstrate how the system transitions to one of the nodes you defined above.<jupyter_code># Testing the dialog flow

# Start conversation with a blank message
results = assistant.message(workspace_id=wrk_id).get_result()
context = results["context"]

# Then ask a sample question
question= "Who is Luke's father?"
results = assistant.message(workspace_id=wrk_id,
                            input={'text': question}, context=context).get_result()

print(json.dumps(results, indent=2))<jupyter_output><empty_output><jupyter_text>4. Query document collection to fetch answersThe Discovery service includes a simple mechanism to make queries against your enriched collection of documents. But you have a lot of control over what fields are searched, how results are aggregated and values are returned. Process sample questionChoose a sample nautal language question to ask, and run it through the Assistant service, just like you did above when testing dialog flow.<jupyter_code># TODO: Run a sample question through Assistant service

# Then ask a sample question
question= "Where was Luke born?"
results = assistant.message(workspace_id=wrk_id,
                            input={'text': question}, context=context).get_result()

print(json.dumps(results, indent=2))<jupyter_output><empty_output><jupyter_text>Now extract the intent and entities identified in the question, and optionally what dialog node was triggered (in case you need it later to customize your response). Some sample code is provided below, but you may need to modify it.<jupyter_code># TODO: Identify the intent(s) the user expressed (typically a single one)
query_intents = [intent["intent"] for intent in results["intents"]]
print("Intent(s):", query_intents)

# TODO: Extract the entities found in the question text
query_entities = [entity["value"] for entity in results["entities"]]
print("Entities:", query_entities)

# TODO: (optional) Find out what dialog node was triggered<jupyter_output><empty_output><jupyter_text>Query the collectionDesign a query based on the information extracted above, and run it against the document collection. The sample query provided below simple looks for all the entities in the raw `text` field. Modify it to suit your needs.Take a look at the [API Reference](https://www.ibm.com/watson/developercloud/discovery/api/v1/?pythonquery-collection) to learn more about the query options available, and for more guidance see this [documentation page](https://www.ibm.com/watson/developercloud/doc/discovery/using.html)._**Note**: You may want to design different queries based on the intent / dialog node that was triggered._<jupyter_code># TODO: Query the Discovery service based on the intent and entities
query_results = discovery.query(environment_id=env_id, collection_id=col_id,
    query_options={
        "query": "text:{}".format(",".join("\"{}\"".format(e) for e in query_entities)),
        "return": "text"
    }).get_result()
print(json.dumps(query_results, indent=2))<jupyter_output><empty_output><jupyter_text>Process returned resultsIf you properly structure the query, Watson is able to do a pretty good job of finding the relevant information. But the result returned is a JSON object. Now your task is to convert that result into an appropriate response that best addresses the original natural language question that was asked.E.g. if the question was "Who saved Han Solo from Jabba the Hutt?" the answer should ideally just be "The Rebels" and not the entire paragraph describing Han Solo's rescue. But that can be a backup response if you cannot be more specific._**Note**: You may have to go back to the previous step and modify the query, especially what you want the Discovery service to return, and this may depend on the intent / dialog node triggered. E.g. study the different parts of a "relation" structure to see how you might construct queries to match them._<jupyter_code># TODO: Process returned results and compose an appropriate response<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08319284614820788, 'fraction_numerical': 0.005607271128051016, 'mean_word_length': 4.764047317279257, 'pattern_counts': {'":': 35, '<': 78, '<?xml version=': 0, '>': 85, 'https://': 25, 'lorem ipsum': 0, 'www.': 11, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '24737', 'license': 'MIT', 'n_tokens_mistral': 7363, 'n_tokens_neox': 7039, 'n_words': 3148, 'path': 'bookworm.ipynb', 'repo_name': 'rostam/AIND-NLP-Bookworm', 'size': '27286'}