<jupyter_start><jupyter_text>IntroductionFirst, we must ensure that the Watson Python SDK is installed and ready to use, we'll then import the SDK as well as the pandas library<jupyter_code>!pip install ibm-watson==5.1.0
from ibm_watson import DiscoveryV1
import pandas as pd<jupyter_output><empty_output><jupyter_text>Initialize Watson DiscoveryNow we'll initialize Watson Discovery using our login credentials. In order to obtain these, create a Watson Discovery services with your IBM Cloud account, and generate new credentials.https://cloud.ibm.com/apidocs/discovery?code=pythonauthentication<jupyter_code># TO DO: import the modules from the sdk

# TO DO: authenticate via the sdk, let it manage the token lifecycle<jupyter_output><empty_output><jupyter_text>Creating the queryThere are a few elements to querying Watson Discovery news, I'll break down each of the elements.`environment_id`: `system` just denotes that we're using the system environment`collection_id`: We want to query the news collection`natural_language_query`: string for free text search`offset`: The number of results (documents) to skip`count`: The number of results (documents) to return ***note: Count and offset is the way pagination of results is implemented, the maximum of total results (offset + count) cannot exceed 10,000***`aggregation`: This is a analytic query of the results set`filter`: The query for matching documents`return_`: What items to actually return to us for our use**For more information, check out the [query reference](https://cloud.ibm.com/docs/services/discovery/query-reference.htmlquery-reference)**We are using DEFAULT_COUNT of 50, the maximum you can query at once, and incrementing until we've captured all available documents or hit 10000 (the maximum).https://cloud.ibm.com/apidocs/discovery?code=pythonquery<jupyter_code># append the results from discovery news into the list all_results
all_results = []

# TO DO: query the service via the sdk
import json

print(json.dumps(all_results, indent=2))<jupyter_output><empty_output><jupyter_text>Add EnrichmentsNow that we've queried Watson Discovery, we want to analyze in detail each article.So we are going to use the `url` info that we obtained for each result to perform a detailed query with **Natural Language Understanding**.Note that this is different in comparison to the `enrich_text.sentiment` field that can be returned by Watson Discovery call, as it "only" analyzes what is contained within the `text` field  Initialize Watson Natural Language Understanding Now we'll initialize Watson Natural Language Understanding using our login credentials. In order to obtain these, create a Watson Natural Language Understanding services with your IBM Cloud account, and generate new credentials.https://cloud.ibm.com/apidocs/natural-language-understanding?code=pythonauthentication<jupyter_code># TO DO: import the modules from the sdk

# TO DO: authenticate via the sdk, let it manage the token lifecycle<jupyter_output><empty_output><jupyter_text>Select the features you want to extract with NLU.For example, let's activate the **sentiment** analysis and the **keywords** extraction, and pass the url extracted for each result as a query parameter to the service  Example testPerform a simple API call to Natural Language Understanding servicehttps://cloud.ibm.com/apidocs/natural-language-understanding?code=pythonanalyze<jupyter_code># TO DO: import the modules for extracting specific features from NLU

# TO DO: perform the query "analyze" with the sdk
# TO DO: for each result in all_results,
#        perform a query to NLU and add sentiment and keywords resulting from NLU to the object
print(json.dumps(all_results, indent=2))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06531049250535331, 'fraction_numerical': 0.004817987152034261, 'mean_word_length': 6.077651515151516, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '276206', 'license': 'Apache-2.0', 'n_tokens_mistral': 1010, 'n_tokens_neox': 941, 'n_words': 452, 'path': 'notebooks/discovery-news-nlu-sentiment.ipynb', 'repo_name': 'Fedacc/notebook-discovery-news-nlu-analysis', 'size': '3736'}