<jupyter_start><jupyter_text>API Management operation statisticsWith this notebook API Management all defined operations are fetched and matched with their request counts fetched from Application Insights. Requirements- API Management integration to Application Insights- environment variables set to your values (see _environment variables_)- Service Principal in AAD with (see _creating a Service Principal for managing ARM resources_)- adal package installed into current jupyter kernel (see _Install the adap pip packagel in the current Jupyter kernel_) environment variablesThese environment variables need to be set with correct values | name | purpose | source || --- | --- | --- || AzMgmtAppId | AppId in AAD to allow access to ARM resources in your subscription | see 'creating a Service Principal for managing ARM resources' || AzMgmtPassword | Password for AppId above | see 'creating a Service Principal for managing ARM resources' || AzMgmtTenant | Tentant Id of AAD used to manage Azure subscription  | Azure Portal; AAD used to manage Azure subscription || AzMgmtSubscription | Id of subscription containing Api Management and Application Insights | Azure Portal || AzApiMgmtResourceGroup | Resource Group containing Api Management | Azure Portal || AzApiMgmtServiceName | API Management Service Name | Azure Portal || AzAppInsightsAppId | Application Insights AppId | Azure Portal || AzAppInsightsApiKey | Application Insights ApiKey used for REST API access | Azure Portal |This can be achieved with a PowerShell setting the environment variables for the current session and for the user persistently:```Write-Host "setting Azure Environment secrets"$env:AzMgmtAppId = "{your-value}"[Environment]::SetEnvironmentVariable("AzMgmtAppId",$env:AzMgmtAppId,"User")$env:AzMgmtPassword = "{your-value}"[Environment]::SetEnvironmentVariable("AzMgmtPassword",$env:AzMgmtPassword,"User")$env:AzMgmtTenant = "{your-value}"[Environment]::SetEnvironmentVariable("AzMgmtTenant",$env:AzMgmtTenant,"User")$env:AzMgmtSubscription = "{your-value}"[Environment]::SetEnvironmentVariable("AzMgmtSubscription",$env:AzMgmtSubscription,"User")$env:AzApiMgmtResourceGroup = "{your-value}"[Environment]::SetEnvironmentVariable("AzApiMgmtResourceGroup",$env:AzApiMgmtResourceGroup,"User")$env:AzApiMgmtServiceName = "{your-value}"[Environment]::SetEnvironmentVariable("AzApiMgmtServiceName",$env:AzApiMgmtServiceName,"User")$env:AzAppInsightsAppId = "{your-value}"[Environment]::SetEnvironmentVariable("AzAppInsightsAppId",$env:AzAppInsightsAppId,"User")$env:AzAppInsightsApiKey = "{your-value}"[Environment]::SetEnvironmentVariable("AzAppInsightsApiKey",$env:AzAppInsightsApiKey,"User")``` creating a Service Principal for managing ARM resourcesTo create a Service Principal with PowerShell login to your AAD/Azure subscription with AzureRM, switch to the target subscription (containing API Management and Application Insights resources) and execute these steps:```$sp = New-AzureRmADServicePrincipal -DisplayName myAzureManagement -Password "<PASSWORD>"New-AzureRmRoleAssignment -RoleDefinitionName Contributor -ServicePrincipalName $sp.ApplicationId```Note ```ApplicationId``` and put it into the environment variables. Install the adal pip packagel in the current Jupyter kernelTo install adal package into your kernel, copy these code lines into a newly inserted cell and execute.```pythonimport sys!{sys.executable} -m pip install adal```<jupyter_code># obtain authorization token to be used in API Management REST API
import adal
import os
import json

def getToken():
    AppId = os.environ.get('AzMgmtAppId')
    AppPassword = os.environ.get('AzMgmtPassword')
    tenantID = os.environ.get('AzMgmtTenant')
    authURL = "https://login.windows.net/" + tenantID
    resource = "https://management.azure.com/"

    context = adal.AuthenticationContext(
        authURL, validate_authority=tenantID, api_version=None)

    token = context.acquire_token_with_client_credentials(
        resource,
        AppId,
        AppPassword)

    return token

def getAuthorizationHeader():
    token = getToken()

    header = token['tokenType'] + ' ' + token['accessToken']

    return header


# ----------------------------------------------------------------------

authHeader = getAuthorizationHeader()

print(authHeader)

# use API Management REST API to obtain APIs
import requests

resource = "https://management.azure.com"
resourceGroupName = os.environ.get('AzApiMgmtResourceGroup')
serviceName = os.environ.get('AzApiMgmtServiceName')
subscriptionId = os.environ.get('AzMgmtSubscription')
apiVersionSuffix = "?api-version=2017-03-01"
        
url = '{}/subscriptions/{}/resourceGroups/{}/providers/Microsoft.ApiManagement/service/{}/apis{}'.format(resource,subscriptionId,resourceGroupName,serviceName,apiVersionSuffix)
headers = {"Content-Type": "application/json","Authorization":authHeader}

response = requests.get(url,headers=headers)

print(response.json()['value'][:2])
# transform API result into dataframe
import pandas as pd
apis = pd.DataFrame.from_dict(response.json()['value'])
apis.head(n=2)
# pull path from properties into data frame
path = apis.properties.apply(lambda x : x["path"])
apis = apis.assign(path=path)
apis.head(n=2)

# create a data frame with all operations in all APIs
import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry
import urllib

def requests_retry_session(
    retries=3,
    backoff_factor=0.3,
    status_forcelist=(500, 502, 504),
    session=None,
):
    session = session or requests.Session()
    retry = Retry(
        total=retries,
        read=retries,
        connect=retries,
        backoff_factor=backoff_factor,
        status_forcelist=status_forcelist,
    )
    adapter = HTTPAdapter(max_retries=retry)
    session.mount('http://', adapter)
    session.mount('https://', adapter)
    return session

# get operations for a single API
def getOperations(id):
    url = '{}{}/operations{}'.format(resource,urllib.parse.quote(id),apiVersionSuffix)
    print(url)
    headers = {"Content-Type": "application/json","Authorization":authHeader}
    response = requests_retry_session().get(url,headers=headers)
    print(response.status_code)
    return response.json()["value"]

# loop thru all APIs, fetch operations
for index, api in apis.iterrows():
    print(index,api.id)
    apiOperations = pd.DataFrame.from_dict(getOperations(api.id))
    apiOperations['apiPath'] = api.path
    apiOperations['operation'] = apiOperations.apply(lambda x:'%s %s' % (x.properties['method'],x.properties['urlTemplate']),axis=1)
    if index == 0:
        operations = apiOperations
    else:
        operations = operations.append(apiOperations,ignore_index=True)
    
operations.head()
import requests
import urllib
import pandas as pd

appInsightsAppId = os.environ.get('AzAppInsightsAppId')
appInsightsApiKey = os.environ.get('AzAppInsightsApiKey')
appInsightsQuery = 'requests | where timestamp >= ago(7d) and tostring(customDimensions.ApimanagementServiceName) != "" and name !startswith "OPTIONS" and success == true | project apiOperation=tostring(customDimensions["Operation Name"]),itemCount | summarize calls=count() by  apiOperation'
appInsightsUrl = "https://api.applicationinsights.io/v1/apps/{}".format(appInsightsAppId)

url = '{}/query?query={}'.format(appInsightsUrl,urllib.parse.quote(appInsightsQuery))
headers = {"Content-Type": "application/json","x-api-key":appInsightsApiKey}

response = requests.get(url,headers=headers)

stats = pd.DataFrame.from_dict(response.json()["tables"][0]["rows"])
stats.columns = ['name','counter']
stats.head()
operationResults = pd.merge(operations,stats,on='name',how='left')[['apiPath','operation','counter']]
operationResults['counter'].fillna(0,inplace=True)
operationResults.head()
operationResults.sort_values(by='counter', ascending=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12113564668769716, 'fraction_numerical': 0.0037854889589905363, 'mean_word_length': 5.862337662337662, 'pattern_counts': {'":': 6, '<': 6, '<?xml version=': 0, '>': 7, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '137639', 'license': 'Apache-2.0', 'n_tokens_mistral': 2242, 'n_tokens_neox': 2197, 'n_words': 560, 'path': 'analyze.ipynb', 'repo_name': 'KaiWalter/api-management-helpers', 'size': '7926'}