<jupyter_start><jupyter_text>Document Information Extraction Showcase This notebook is designed to demonstrate how to easily consume the SAP AI Business Services - Document Information Extraction service. In this demo, we first create a new client to then use the service to extract data from an example invoice.  Extract credentials from service key You require a valid service key for the Document Information Extraction service on the SAP Business Technology Platform. For the detailed setup steps see: https://help.sap.com/viewer/5fa7265b9ff64d73bac7cec61ee55ae6/SHIP/en-US/0d68dc0002f0484ba25f85f3170166e0.html  The necessary credentials are the following:- url: The URL of the service deployment provided in the outermost hierarchy of the service key json file- uaa_url: The URL of the UAA server used for authentication provided in the uaa part of the service key json file- clientid: The clientid used for authentication to the UAA server provided in the uaa part of the service key json file- clientsecret: The clientsecret used for authentication to the UAA server provided in the uaa part of the service key json file<jupyter_code># Please insert your copied service key json here
service_key = {
  "url": "*******",
  "uaa": {
    "tenantmode": "*******",
    "sburl": "*******",
    "subaccountid": "*******",
    "clientid": "*******",
    "xsappname": "*******",
    "clientsecret": "*******",
    "url": "*******",
    "uaadomain": "*******",
    "verificationkey": "*******",
    "apiurl": "*******",
    "identityzone": "*******",
    "identityzoneid": "*******",
    "tenantid": "*******",
    "zoneid": "*******"
  },
  "swagger": "/document-information-extraction/v1/"
}
url = service_key['url']
uaa_url = service_key['uaa']['url']
client_id = service_key['uaa']['clientid']
client_secret = service_key['uaa']['clientsecret']<jupyter_output><empty_output><jupyter_text>Initialize DoxApiClient<jupyter_code># Import DOX API client
from sap_business_document_processing import DoxApiClient
# Instantiate object used to communicate with Document Information Extraction REST API
api_client = DoxApiClient(url, client_id, client_secret, uaa_url)<jupyter_output><empty_output><jupyter_text>(optional) Display access token<jupyter_code># Token can be used to interact with e.g. swagger UI to explore DOX API
print(api_client.session.token)
print(f"\nYou can use this token to authorize here and explore the API via Swagger UI: \n{api_client.base_url}")<jupyter_output><empty_output><jupyter_text>See list of document fields you can extract<jupyter_code># Get the available document types and corresponding extraction fields
from utils import display_capabilities
capabilities = api_client.get_capabilities()
display_capabilities(capabilities)<jupyter_output><empty_output><jupyter_text>(optional) Create a Client To use Document Information Extraction, you need a client. This client is used to distinguish and separate data. You can create a new client if you wish to perform the information extraction with a separate client. One 'default' client already exists.<jupyter_code># Check which clients exist for this tenant
api_client.get_clients()
# Create a new client with the id 'c_00' and name 'Client 00'
api_client.create_client(client_id='c_00', client_name='Client 00')<jupyter_output><empty_output><jupyter_text>Upload a document and retrieve the extracted result<jupyter_code># Specify the fields that should be extracted
header_fields = [
         "documentNumber",
         "taxId",
         "purchaseOrderNumber", 
         "shippingAmount",
         "netAmount",
         "senderAddress",
         "senderName",
         "grossAmount",
         "currencyCode",
         "receiverContact",
         "documentDate",
         "taxAmount",
         "taxRate",
         "receiverName",
         "receiverAddress"
    ]
line_item_fields = [
         "description",
         "netAmount",
         "quantity",
         "unitPrice",
         "materialNumber"
    ]

# Extract information from invoice document
document_result = api_client.extract_information_from_document(document_path='sample-invoice-1.pdf', 
                                                               client_id='default', 
                                                               document_type='invoice', 
                                                               header_fields=header_fields, 
                                                               line_item_fields=line_item_fields)
# Check the extracted data
import json
print(json.dumps(document_result, indent=2))
# Let's visualize the extracted values on the invoice document
from utils import display_extraction
display_extraction(document_result, 'sample-invoice-1.pdf')<jupyter_output><empty_output><jupyter_text>(optional) Upload Ground Truth Ground truth values can be uploaded to evaluate the results of the Document Information Extraction<jupyter_code># Load ground truth values from json file
with open('gt-sample-invoice-1.json') as ground_truth_file:
    ground_truth = json.load(ground_truth_file)
# Add ground truth values to the uploaded invoice
api_client.post_ground_truth_for_document(document_id=document_result['id'], ground_truth=ground_truth)
# You can now also retrieve the uploaded ground truth values by setting extracted_values to False
api_client.get_extraction_for_document(document_id=document_result['id'], extracted_values=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09845984598459846, 'fraction_numerical': 0.009534286762009535, 'mean_word_length': 4.032287822878229, 'pattern_counts': {'":': 17, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '102672', 'license': 'Apache-2.0', 'n_tokens_mistral': 1522, 'n_tokens_neox': 1467, 'n_words': 473, 'path': 'examples/document_information_extraction_examples/information_extraction_showcase.ipynb', 'repo_name': 'tillgeissler/business-document-processing', 'size': '5454'}