<jupyter_start><jupyter_text>Dialog skill analysis for Watson Assistant IntroductionDialog Skill Analysis for Watson Assistant (WA) is intended for use by chatbot designers, developers and data scientists who would like to experiment with and improve their existing dialog skill design.  This notebook assumes familiarity with the Watson Assistant product as well as concepts involved in dialog skill design such as intent, entities, and utterances.    Environment- Python version 3.6 or above is required. - Install dependencies with `pip install -r requirements.txt` and refer to `requirements.txt` Install all the required packages and filter out any warnings.<jupyter_code>from IPython.display import Markdown, display, HTML
import warnings
warnings.filterwarnings('ignore')
# Standard python libraries
import sys, os
import json
import importlib
from collections import Counter

# External python libraries
import pandas as pd
import numpy as np
import nltk
nltk.download('stopwords')
nltk.download('punkt')
import ibm_watson

# Internal python libraries
from assistant_dialog_skill_analysis.utils import skills_util
from assistant_dialog_skill_analysis.highlighting import highlighter
from assistant_dialog_skill_analysis.data_analysis import summary_generator
from assistant_dialog_skill_analysis.data_analysis import divergence_analyzer
from assistant_dialog_skill_analysis.data_analysis import similarity_analyzer
from assistant_dialog_skill_analysis.term_analysis import chi2_analyzer
from assistant_dialog_skill_analysis.term_analysis import keyword_analyzer
from assistant_dialog_skill_analysis.term_analysis import entity_analyzer
from assistant_dialog_skill_analysis.confidence_analysis import confidence_analyzer
from assistant_dialog_skill_analysis.inferencing import inferencer
from assistant_dialog_skill_analysis.experimentation import data_manipulator<jupyter_output><empty_output><jupyter_text>Table of contents1. [Part 1: Prepare the training data](part1)2. [Part 2: Prepare the test data](part2)3. [Part 3: Perform advanced analysis](part3)4. [Part 4: Summary](part4)  Part 1: Prepare the training data1.1 [Set up access to the training data](part1)1.2 [Process Dialog Skill Training Data](part1.2)1.3 [Analyze data distribution](part1.3)1.4 [Perform a correlation analysis](part1.4)1.5 [Visualize terms using a heat map](part1.5)1.6 [Ambiguity in the training data](part1.6)  1.1 Set up access to the training dataProvide access credentials for an existing dialog skill that you would like to analyze. For this you need your API Key and Workspace ID values.<jupyter_code>importlib.reload(skills_util)

# Change Assistant API version if needed
# Find Latest --> https://cloud.ibm.com/docs/services/assistant?topic=assistant-release-notes
API_VERSION = '2019-02-28'

# Change URL based on IBM Cloud datacenter you use   
URL = "https://gateway.watsonplatform.net/assistant/api" # Dallas (Default US South)

#URL = "https://gateway-s.watsonplatform.net/assistant/api" # Dallas Staging
#URL = "https://gateway-wdc.watsonplatform.net/assistant/api" # Washington, DC
#URL = "https://gateway-fra.watsonplatform.net/assistant/api" # Frankfurt
#URL = "https://gateway-syd.watsonplatform.net/assistant/api" # Sydney
#URL = "https://gateway-tok.watsonplatform.net/assistant/api" # Tokyo
#URL = "https://gateway-lon.watsonplatform.net/assistant/api" # London

# For ICP(IBM Cloud Private), you can disable SSL verification by changing this to True
DISABLE_SSL_VERTIFICATION = False 

# By default we only need the IAM API Key & the Workspace ID

# If you run the notebook regularly you can uncomment the two lines below
# & comment out the line after it
#iam_apikey = '###'
#workspace_id = '###'
#Prompt user for credentials
iam_apikey, workspace_id = skills_util.input_credentials()

conversation = skills_util.retrieve_conversation(iam_apikey=iam_apikey,
                                                 url=URL,
                                                 api_version=API_VERSION)

#If you do not have IAM based API Keys
#but have access to a Username, Password & Workspace ID
#You can comment out the two lines above & uncomment the lines below to authenticate
# username = 'apikey'
# password = '###'
# workspace_id = '###'
# conversation = skills_util.retrieve_conversation(username=username,
#                                              password=password,                                                       
#                                              url=URL,
#                                              api_version=API_VERSION)


conversation.set_disable_ssl_verification(DISABLE_SSL_VERTIFICATION)

workspace = skills_util.retrieve_workspace(workspace_id=workspace_id,
                                           conversation=conversation)
# Extract user workspace
workspace_data, workspace_vocabulary = skills_util.extract_workspace_data(workspace)
entity_dict = conversation.list_entities(workspace_id).get_result()
entities_list = [item['entity'] for item in entity_dict['entities']]
# Create workspace data frame
workspace_pd = pd.DataFrame(workspace_data)

display(Markdown("### Sample of Utterances & Intents"))
display(HTML(workspace_pd.sample(n = len(workspace_pd) if len(workspace_pd)<10 else 10)
             .to_html(index=False)))
if entities_list:
    display(Markdown("### Sample of Entities"))
    display(HTML(pd.DataFrame({"Entity":entities_list})
                 .sample(n = len(entities_list) if len(entities_list)<10 else 10)
                 .to_html(index=False)))<jupyter_output><empty_output><jupyter_text>1.2 Process the dialog skill training dataGenerate summary statistics related to the given skill and workspace.<jupyter_code>importlib.reload(summary_generator)
summary_generator.generate_summary_statistics(workspace_data, entities_list)<jupyter_output><empty_output><jupyter_text>1.3 Analyze the data distribution- [Analyze class imbalance](imbalance)- [List the distribution of user examples by intent](distribution)- [Actions for class imbalance](actionimbalance)  Analyze class imbalanceAnalyze whether the data set contains class imbalance by checking whether the largest intent contains less than double the number of user examples contained in the smallest intent. If there is an imbalance it does not necessarily indicate an issue; but you should review the [actions](actionimbalance) section below.<jupyter_code>importlib.reload(summary_generator)
class_imb_flag = summary_generator.class_imbalance_analysis(workspace_pd)<jupyter_output><empty_output><jupyter_text>List the distribution of user examples by intentDisplay the distribution of intents versus the number of examples per intent (sorted by the number of examples per intent) below. Ideally you should not have large variations in terms of number of user examples for various intents.<jupyter_code>importlib.reload(summary_generator)
summary_generator.scatter_plot_intent_dist(workspace_pd)
importlib.reload(summary_generator)
summary_generator.show_user_examples_per_intent(workspace_data)<jupyter_output><empty_output><jupyter_text>Actions for class imbalanceClass imbalance will not always lead to lower accuracy, which means that all intents (classes) do not need to have the same number of examples.Given a hypothetical chatbot related to banking:- For intents like `updateBankAccount` and `addNewAccountHolder` where the semantics difference between them is subtler, the number of examples per intent needs to be somewhat balanced otherwise the classifier might favor the intent with the higher number of examples.- For intents like `greetings` that are semantically distinct from other intents like `updateBankAccount`, it may be acceptable for it to have fewer examples per intent and still be easy for the intent detector to classify.If the intent classification accuracy is lower than expected during testing, you should re-examine the distribution analysis.  With regard to sorted distribution of examples per intent, if the sorted number of user examples varies a lot across different intents, it can be a potential source of bias for intent detection. Large imbalances in general should be avoided. This can potentially lead to lower accuracy. If your graph displays this characteristic, this could be a source of error.For further guidance on adding more examples to help balance out your distribution, refer to Intent Example Recommendation.  1.4  Perform correlation analysis- [Retrieve the most correlated unigrams and bigrams for each intent](retrieve)- [Actions for anomalous correlations](anomalous)  Retrieve the most correlated unigrams and bigrams for each intentPerform a chi square significance test using count features to determine the terms that are most correlated with each intent in the data set. A `unigram` is a single word, while a `bigram` is two consecutive words from within the training data. For example, if you have a sentence like `Thank you for your service`, each of the words in the sentence are considered unigrams while terms like `Thank you`, `your service` are considered bigrams.Terms such as `hi`, `hello` correlated with a `greeting` intent are reasonable. But terms such as `table`, `chair` correlated with the `greeting` intent are anomalous. A scan of the most correlated unigrams & bigrams for each intent can help you spot potential anomalies within your training data.**Note**: We ignore the following common words (\"stop words\") from consideration `an, a, in, on, be, or, of, a, and, can, is, to, the, i`<jupyter_code>importlib.reload(chi2_analyzer)
unigram_intent_dict, bigram_intent_dict = chi2_analyzer.get_chi2_analysis(workspace_pd)<jupyter_output><empty_output><jupyter_text>Actions for anomalous correlationsIf you identify unusual or anomalous correlated terms such as: numbers, names and so on, which should not be correlated with an intent, consider the following:  - **Case 1** : If you see names appearing amongst correlated unigrams or bigrams, add more variation of names so no specific names will be correlated  - **Case 2** : If you see specific numbers like 1234 amongst correlated unigrams or bigrams and these are not helpful to the use case, remove or mask these numbers from the examples- **Case 3** : If you see terms which should never be correlated to that specific intent, consider adding or removing terms/examples so that domain specific terms are correlated with the correct intent  1.5 Visualize terms using a heat map- [Display term analysis for a custom intent list](customintent)- [Actions for anomalous terms in the heat map](heatmap)A heat map of terms is a method to visualize terms or words that frequently occur within each intent. Rows are the terms, and columns are the intents. The code below displays the top 30 intents with the highest number of user examples in the analysis. This number can be changed if needed.<jupyter_code>importlib.reload(keyword_analyzer)

INTENTS_TO_DISPLAY = 30  # Total number of intents for display
MAX_TERMS_DISPLAY = 30  # Total number of terms to display

intent_list = []
keyword_analyzer.seaborn_heatmap(workspace_pd, INTENTS_TO_DISPLAY, MAX_TERMS_DISPLAY, intent_list)<jupyter_output><empty_output><jupyter_text>Display term analysis for a custom intent listIf you wish to see term analysis for specific intents, feel free to add those intents to the intent list. This generates a custom term heatmap. The code below displays the top 20 terms, but this can be changed if needed.<jupyter_code>importlib.reload(keyword_analyzer)
# intent_list = ['intent1','intent2','intent3'] 
intent_list = [] 


MAX_TERMS_DISPLAY = 20  # Total number of terms to display

if intent_list: 
    keyword_analyzer.seaborn_heatmap(workspace_pd, INTENTS_TO_DISPLAY, MAX_TERMS_DISPLAY, intent_list)<jupyter_output><empty_output><jupyter_text>Actions for anomalous terms in the heat mapIf you notice any terms or words which should not be frequently present within an intent, consider modifying examples in that intent.  1.6 Ambiguity in the training data- [Uncover ambiguous utterances across intents](uncover)- [Actions for ambiguity in the training data](ambiguityaction)Run the code blocks below to uncover possibly ambiguous terms based on feature correlation.Based on the chi-square analysis above, generate intent pairs which have overlapping correlated unigrams and bigrams.This allows you to get a glimpse of which unigrams or bigrams might cause potential confusion with intent detection:  A. Top intent pairs with overlapping correlated unigrams<jupyter_code>importlib.reload(chi2_analyzer)
ambiguous_unigram_df = chi2_analyzer.get_confusing_key_terms(unigram_intent_dict)<jupyter_output><empty_output><jupyter_text>B. Top intent pairs with overlapping correlated bigrams<jupyter_code>importlib.reload(chi2_analyzer)
ambiguous_bigram_df = chi2_analyzer.get_confusing_key_terms(bigram_intent_dict)<jupyter_output><empty_output><jupyter_text>C. Overlap checker for specific intents<jupyter_code># Add specific intent or intent pairs for which you would like to see overlap
importlib.reload(chi2_analyzer)
intent1 = 'Goodbye'
intent2 = ''
chi2_analyzer.chi2_overlap_check(ambiguous_unigram_df,ambiguous_bigram_df,intent1,intent2)<jupyter_output><empty_output><jupyter_text>Uncover ambiguous utterances across intentsThe following analysis shows user examples that are similar but fall under different intents.<jupyter_code>importlib.reload(similarity_analyzer)
similar_utterance_diff_intent_pd = similarity_analyzer.ambiguous_examples_analysis(workspace_pd)<jupyter_output><empty_output><jupyter_text>Actions for ambiguity in the training data**Ambiguous intent pairs**  If you see terms which are correlated with more than 1 intent, review if this seems anomalous based on the use case for that intent. If it seems reasonable, it is probably not an issue.  **Ambiguous utterances across intents** - **Duplicate utterances**: For duplicate or almost identical utterances, remove those that seem unnecessary.- **Similar utterances**: For similar utterances, review the use case for those intents and make sure that they are not accidental additions caused by human error when the training data was created.  For more information about entity, refer to the Entity Documentation.For more in-depth analysis related to possible conflicts in your training data across intents, try the conflict detection feature in Watson Assistant. Refer to  Conflict Resolution Documentation.  Part 2: Prepare the test dataAnalyze your existing Watson Assistant Dialog Skill with the help of a test set.2.1. [Obtain test data from Cloud Object Storage](cos)2.2. [Evaluate the test data](evaluate) 2.3. [Analyze the test data](testanalysis)   2.1 Obtain test data from Cloud Object StorageUpload a test set in tsv format. Each line in the file should have only `User_InputIntent`  For example:```hello how are youGreeting  I would like to talk to a humanAgentHandoff  ```<jupyter_code>importlib.reload(skills_util)

#Separator: Use '\t' for tab separated data, ',' for comma separated data
separator = '\t'

test_set_path = 'test_set.tsv'
test_df = skills_util.process_test_set(test_set_path, separator)

display(Markdown("### Random Test Sample"))
display(HTML(test_df.sample(n=10).to_html(index=False)))
display(HTML(test_df.sample(n=10).to_html(index=False)))<jupyter_output><empty_output><jupyter_text>2.2 Evaluate the test dataThese steps can take time if you have a large test set.  **Note**: You will be charged for calls made from this notebook based on your Watson Assistant plan. The user_id will be the same for all message calls.<jupyter_code># Maximum of 5 threads for faster inference
THREAD_NUM = 5
full_results = inferencer.inference(conversation,
                                    workspace_id,
                                    test_df,
                                    max_retries=10,
                                    max_thread=THREAD_NUM, 
                                    verbose=False)<jupyter_output><empty_output><jupyter_text>2.3 Analyze the test data- [Display an overview of the test data](overview)- [Compare the test data and the training data](compare)- [Determine the overall accuracy on the test set](accuracy)- [Analyze the errors](errors)  Display an overview of the test data<jupyter_code>importlib.reload(summary_generator)
summary_generator.generate_summary_statistics(test_df)
summary_generator.show_user_examples_per_intent(test_df)<jupyter_output><empty_output><jupyter_text>Compare the test data and the training dataIdeally the test and training data distributions should be similar. The following metrics can help identify gaps between the test set and the training set:**1.**  The distribution of user examples per intent for the test data should be comparable to the training data   **2.**  The average length of user examples for test and training data should be comparable to the training data **3.**  The vocabulary and phrasing of utterances in the test data should be comparable to the training dataIf your test data comprises of examples labelled from your logs, and the training data comprises of examples created by human subject matter experts, there may be discrepancies between what the virtual assistant designers thought the end users would type and the way they actually type in production. Thus, if you find discrepancies in this section, consider changing your design to resemble the way in which end users use your system more closely.**Note**: You will be charged for calls made from this notebook  based on your WA plan. The user_id will be the same for all message calls.<jupyter_code>importlib.reload(divergence_analyzer)
divergence_analyzer.analyze_train_test_diff(workspace_pd, test_df, full_results)<jupyter_output><empty_output><jupyter_text>Determine the overall accuracy on the test set<jupyter_code>importlib.reload(inferencer)
results = full_results[['correct_intent', 'top_confidence','top_intent','utterance']]
accuracy = inferencer.calculate_accuracy(results)
display(Markdown("### Accuracy on Test Data: {} %".format(accuracy)))<jupyter_output><empty_output><jupyter_text>Analyze the errorsThis section gives you an overview of the errors made by the intent classifier on the test set.  **Note**: `System Out of Domain` labels are assigned to user examples which get classified with confidence scores less than 0.2 as Watson Assistant considers them to be irrelevant.<jupyter_code>importlib.reload(inferencer)
wrongs_df = inferencer.calculate_mistakes(results)
display(Markdown("### Intent Detection Mistakes"))
display(Markdown("Number of Test Errors: {}".format(len(wrongs_df))))

with pd.option_context('max_colwidth', 250):
    if not wrongs_df.empty:
        display(wrongs_df)<jupyter_output><empty_output><jupyter_text>Part 3: Perform advanced analysis3.1 [Perform analysis using confidence thresholds](part3.1)3.2 [Analysis interpretation @ confidence level T](levelT)3.3 [Highlighting term importance](part3.2)3.4 [Analyzing abnormal confidence levels](part3.3)3.5 [Perform an analysis using correlated entities per intent](part3.4)  3.1 Perform analysis using  confidence thresholdsThis analysis illustrates how a confidence threshold is used to determine which data considered irrelevant or out of domain can be used for analysis.<jupyter_code>importlib.reload(confidence_analyzer)
analysis_df= confidence_analyzer.analysis(results,None)<jupyter_output><empty_output><jupyter_text>3.2 Analysis interpretation @ confidence level T If a certain confidence threshold T is selected, then: - The on-topic accuracy for test examples which cross the threshold is ***TOA***- The percentage of total test examples which returns confidences higher than the threshold is measured as ***Bot Coverage %***- If out of domain examples exist, falsely accept out of domain examples as on topic examples at a rate measured by ***FAR*** (False Acceptance Rate)<jupyter_code>analysis_df.index = np.arange(1, len(analysis_df)+1)
display(analysis_df)<jupyter_output><empty_output><jupyter_text>Select the threshold valueBy selecting a higher threshold, you can potentially bias your systems so that they are more accurate in terms of determining whether an utterance is on topic or out of domain. The default confidence threshold for Watson Assistance is 0.2.  **Effect on accuracy**: When you select a higher threshold T, this can result in higher accuracy (TOA) because only examples with confidences greater than the threshold T are included.**Effect on bot coverage %**: However, when you select a higher threshold T, this can also result in the virtual assistant responding to less examples.**Deflection to human agent**: In the scenarios where the virtual assistant is setup to hand off to a human agent when it is less confident, having a higher threshold T can:  - Improve end user experience when interacting with a virtual assistant, as it continues interaction only when its highly confident- Result in higher costs to the customer as this can result in more deflections to the human agents Thus, there is a trade-off and you need to decide on a threshold value on a per customer basis.  Examine the threshold selection on individual intentsThis section allows the examination of thresholds on specific intents.- Use `INTENT_LIST = []` to get analysis which averages across all intents- Use `INTENT_LIST = ['intent1', 'intent2']` to examine specific intents and threshold analysis on these intents- Use `INTENT_LIST = ['ALL_INTENTS']` to examine all intents and threshold analysis for each- Use `INTENT_LIST = [MOST_FREQUENT_INTENT]` to get analysis on the intent with the most test examples (DEFAULT)**False Acceptance Rate (FAR) for specific intents**  When we calculate FAR across all intents (as in previous section) we calculate fraction of out of domain examples falsely considered on topic. When we calculate FAR for specific intents, we calculate the fraction of examples which were falsely predicted to be that specific intent.<jupyter_code>importlib.reload(confidence_analyzer)

# Calculate intent with most test examples
for label in list(test_df['intent'].value_counts().index):
    if label != skills_util.OFFTOPIC_LABEL:
        MOST_FREQUENT_INTENT = label 
        break
        
# Specify intents of interest for analysis      
INTENT_LIST = [MOST_FREQUENT_INTENT]  

analysis_df_list = confidence_analyzer.analysis(results, INTENT_LIST)<jupyter_output><empty_output><jupyter_text>3.3 Highlight term importanceThis intent can be ground-truth or an incorrectly predicted intent. It provides term level insights about which terms the classifier thought were important in relation to that specific intent.Even if the system predicts an intent correctly, the terms which the intent classifier thought were important may not be as expected by human insight. Human insight might suggest that the intent classifier is focusing on the wrong terms.  The score of each term in the following highlighted images can be viewed as importance factor of that term for that specific intent. The larger the score, the more important the term. You can get the highlighted images for either wrongly-predicted utterances or utterances where the classifier returned a low confidence.   **Note**: You will be charged for calls made from this notebook  based on your WA plan. The user_id will be the same for all message calls.<jupyter_code>importlib.reload(highlighter)

# Pick an example from section 1 which was misclassified
# Add the example and correct intent for the example
utterance = "Where is the closest agent?"  # input example
intent = "General_Connect_to_Agent"  # input an intent in your workspace which you are interested in.


inference_results = inferencer.inference(conversation=conversation, 
                                    workspace_id=workspace_id, 
                                    test_data=pd.DataFrame({'utterance':[utterance], 
                                                            'intent':[intent]}), 
                                    max_retries = 10, 
                                    max_thread = 1, 
                                    verbose = False)

highlighter.get_highlights_in_batch_multi_thread(conversation, 
                                                 workspace_id,
                                                 inference_results, 
                                                 None,
                                                 1,
                                                 1)<jupyter_output><empty_output><jupyter_text>In the section below you analyze your test results and produce highlighting for the top 25 problematic utterances which were either mistakes or had confidences below the threshold that was set.    **Note**: You will be charged for calls made from this notebook  based on your WA plan. The user_id will be the same for all message calls.<jupyter_code>importlib.reload(highlighter)

# The output folder for generated images
# Note modify this if you want the generated images to be stored in a different directory

highlighting_output_folder = './highlighting_images/'
if not os.path.exists(highlighting_output_folder):
    os.mkdir(highlighting_output_folder)

# The threshold the prediction needs to achieve below which  
# it will be considered as `out of domain` or `offtopic` utterances. 
threshold = 0.2

# Maximum number of test set examples whose highlighting analysis will be conducted
K=25
highlighter.get_highlights_in_batch_multi_thread(conversation, 
                                                 workspace_id,full_results, 
                                                 highlighting_output_folder,
                                                 threshold,
                                                 K)<jupyter_output><empty_output><jupyter_text>3.4 Analyze abnormal confidence levelsEvery test utterance is classified as a specific intent with a specific confidence by the Watson Assistant intent classifier. It is expected that model would be confident when it correctly predicts examples and not highly confident when it incorrectly predicts examples. But this is not always true. This can be because there are anomalies in the design. Examples that are predicted correctly with low confidence and the examples that are predicted incorrectly with high confidence are cases which need to be reviewed.<jupyter_code>importlib.reload(confidence_analyzer)
correct_thresh, wrong_thresh = 0.3, 0.7
correct_with_low_conf_list, incorrect_with_high_conf_list = confidence_analyzer.abnormal_conf(
    full_results, correct_thresh, wrong_thresh)
if len(correct_with_low_conf_list) > 0:
    display(Markdown("#### Examples correctedly predicted with low confidence"))
    with pd.option_context('max_colwidth', 250):
        display(HTML(correct_with_low_conf_list.to_html(index=False)))
if len(incorrect_with_high_conf_list) > 0:
    display(Markdown("#### Examples incorrectedly predicted with high confidence"))
    with pd.option_context('max_colwidth', 250):
        display(HTML(incorrect_with_high_conf_list.to_html(index=False)))<jupyter_output><empty_output><jupyter_text>Actions to take when you have examples of abnormal confidenceIf there are examples which are incorrectly classified with high confidence for specific intents, it may indicate an issue in the design of those specific intents because the user examples provided for that intent may be overlapping with the design of other intents.If intent A seems to always get misclassified as intent B with high confidence or gets correctly predicted with low confidence, consider using intent conflict detection. For more information, refer to the Conflict Resolution Documentation.Also consider whether those two intents need to be two separate intents or whether they need to be merged. If they can't be merged, then consider adding more user examples which distinguish intent A specifically from intent B.  3.5 Perform an analysis using correlated entities per intentPerform a chi square significance test for entities such as we or you for unigrams and bigrams in the previous section. For each utterance in the training data, this analysis will call the message API for entity detection on each utterance and find the most correlated entities for each intent.**Note**: You will be charged for calls made from this notebook  based on your Watson Assistant plan. The user_id will be the same for all message calls.<jupyter_code>importlib.reload(entity_analyzer)
importlib.reload(inferencer)
if entities_list:
    THREAD_NUM = 5# we allow a maximum of 5 threads for faster inference
    train_full_results = inferencer.inference(conversation,
                                              workspace_id,
                                              workspace_pd,
                                              max_retries=10, 
                                              max_thread=THREAD_NUM,
                                              verbose=False)
    entity_label_correlation_df = entity_analyzer.entity_label_correlation_analysis(
        train_full_results, entities_list)
    with pd.option_context('display.max_colwidth', 200):
        entity_label_correlation_df.index = np.arange(1, len(entity_label_correlation_df) + 1)
        display(entity_label_correlation_df)
else:
    display(Markdown("### Target workspace has no entities."))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.05784870861265185, 'fraction_numerical': 0.006771701772892776, 'mean_word_length': 4.468552288797916, 'pattern_counts': {'":': 1, '<': 103, '<?xml version=': 0, '>': 104, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 57, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '25', 'dir': 'jupyter-structured-clean-dedup', 'id': '497340', 'license': 'Apache-2.0', 'n_tokens_mistral': 7542, 'n_tokens_neox': 7212, 'n_words': 3141, 'path': 'skill_analysis.ipynb', 'repo_name': 'GunnarHorve-ibm/assistant-dialog-skill-analysis', 'size': '29387'}