<jupyter_start><jupyter_text>An Example Python Client for the demo-echo APIWhen using [OAuth 2.0](https://tools.ietf.org/html/rfc6749) (no matter what the flow), the client app (this notebook) gets an *access token* (and optionally some other tokens like *refresh\_token* and even an OpenID Connect *id\_token* which identifies the end user to the client app -- when using the *openid* scope). The only token we are currently concerned with is the **access token**. Once the client app has the access token, it then adds it to the HTTP request that it sends to the Resource Server in one of two ways, either1. in an HTTP header: `Authorization: Bearer ` (preferred) or,2. as a query parameter: `https://example.com/resource?access_token=`.But not both!You can get a valid access token by, for example, using Postman, and then jump down to where the token is added to the Authorization header a few lines below if you want to initially skip over the OAuth 2.0 dancing.Let's try out the [oauth2-client](https://github.com/antechrestos/OAuth2Client) Python library. You can install this with:```shpip install oauth2-client``` Upon successfully authentication and authorizing in the new tab, you'll see this message: ```Response received ({"state": "state_test", "code": ""}). Result was transmitted to the original thread. You can close this window.```<jupyter_code>import requests
import webbrowser
from oauth2_client.credentials_manager import CredentialManager, ServiceInformation, OAuthError
import json
import base64
import time
from pprint import pprint, pformat
from jwcrypto import jwt, jwk
import requests
import logging
_logger = logging.getLogger(__name__)
_logger.setLevel('DEBUG')<jupyter_output><empty_output><jupyter_text>The following gets used near the end. We're basically subclassing CredentialManager to add OpenID Connect functionality.<jupyter_code># subclass CredentialManager to get the id_token which is used later.
class OpenIdCredentialManager(CredentialManager):
    def __init__(self, service_information, proxies=None):
        super(OpenIdCredentialManager, self).__init__(service_information, proxies)
        self.id_token = None

    def _process_token_response(self,  token_response, refresh_token_mandatory):
        id_token = token_response.get('id_token')
        super(OpenIdCredentialManager, self)._process_token_response(token_response, refresh_token_mandatory)
        self.id_token = id_token<jupyter_output><empty_output><jupyter_text>Composing the OAuth 2.0 Authorization Code RequestThe request consists of a few pieces of information: service endpoints, credentials that identify this registered client app, a redirect_uri to be called back to, and a list of scopes that the Resource Server requires for the given resource and method. EndpointsThe OAuth 2.0 service endpoints needed for Authorization Code flow are `authorize` and `token`. The particular required URLs for Columbia's PingFederate-based OAuth 2.0 services are show below and can be found in the RAML securitySchemes:```yaml%RAML 1.0 SecuritySchemetype: OAuth 2.0description: |  This API supports OAuth 2.0 for authorizing requests using PingFederate.  Please note that MuleSoft will not actually implement any OAuth 2.0 scope enforcement  as declared with a resource & method's `securedBy` unless you apply an one or more  relevant API Manager Policies:    - One of the `OAuth 2.0 protected` PingFederate policies.    - The `OAuth 2.0 scope enforcement` custom policy.describedBy:  headers:    Authorization?:      description: OAuth 2 access token. Use EITHER this or the access_token, not both.      type: string  queryParameters:    access_token?:      description: OAuth 2 access token. Use EITHER this or the Authorization header, not both.      type: string  responses:    401:      description: Bad or expired token.    403:      description: Bad OAuth requestsettings:  authorizationUri: https://oauth.cc.columbia.edu/as/authorization.oauth2  accessTokenUri: https://oauth.cc.columbia.edu/as/token.oauth2  authorizationGrants:     - authorization_code    - client_credentials  scopes:     !include scopes.raml ``` Client CredentialsThe `client_id` and `client_secret` were obtained from **Request API Access** in the AnyPoint API Developer console. These uniquely identify the client app to the Authorization Server (AS). Redirect URIThe OAuth 2.0 Authorization Code protocol "returns" data to the requestor by a GET of it's request\_uri with some query parameter which communicate back the code. This is part of the OAuth 2.0 security "magic". Redirect URIs have to be:- Registered in advance with the Authorization Server- Specified by the client in the "authorize" requestThey must match or the request is denied.Redirect URIs for clients are established by AnyPoint API Manager as part of the setup of new client apps along with other settings such as which grant types are allowed for this client and whether it should return a refresh token: ScopesThe required scopes for a particular endpoint can be found in the API's RAML description. For this example, a GET of the /things endpoint, the RAML shows that scopes of auth-columbia, -google or -facebook and read are required. Let's also add "openid" so that we get an id_token back for this client app to identify who the authorizing user is.```yaml/things:  displayName: Things  description: some things  get:    securedBy:  allow selection of one of columbia, facebook, or google login.       - oauth_2_0: { scopes: [ auth-columbia, read ] }      - oauth_2_0: { scopes: [ auth-google, read ] }      - oauth_2_0: { scopes: [ auth-facebook, read ] }    responses:      200:        body:          application/json:            schema: Thing  ...  post:    securedBy:      - oauth_2_0: { scopes: [ auth-columbia, demo-netphone-admin, create ] }      - oauth_2_0: { scopes: [ auth-google, create ] }      - oauth_2_0: { scopes: [ auth-facebook, create ] }```  My client credentials and OAuth 2 server<jupyter_code>creds = {'id':'demo_client','secret':'<KEY>'}
oauth_prod = 'https://oauth.cc.columbia.edu'
oauth_test = 'https://oauth-test.cc.columbia.edu'
oauth_dev  = 'https://localhost:8443'
oauth_server = oauth_dev<jupyter_output><empty_output><jupyter_text>Example of scopes: auth-google, read, openid<jupyter_code>service_information = ServiceInformation(
    authorize_service = oauth_server + '/as/authorization.oauth2',
    token_service = oauth_server + '/as/token.oauth2',
    client_id=creds['id'],
    client_secret=creds['secret'],
    scopes=['auth-columbia', 'read', 'openid'],
    skip_ssl_verifications=True if oauth_server == oauth_dev else False)
manager = OpenIdCredentialManager(service_information) # initialize the OAuth 2.0 client<jupyter_output><empty_output><jupyter_text>The redirect\_uri must:1. Match one of the redirect URIs that were registered in AnyPoint API Manager for this client ("External API Tester") w/client\_id and client\_secret, above.2. Actually have a listener on that URI -- which the `manager.init_authorize_code_process()` launches for you.<jupyter_code>redirect_uri = 'http://localhost:5432/oauth2client'<jupyter_output><empty_output><jupyter_text>The Authorization Code flow does a bunch of browser redirects so that the Resource Owner (end user) credentials never flow through the client app itself. As you can see, you must click on the URL which opens another browser tab where the user login flow happens.<jupyter_code>authUrl = manager.init_authorize_code_process(redirect_uri, 'state_test')
print('Opening this url in your browser: %s'%authUrl)
webbrowser.open_new(authUrl)
code = manager.wait_and_terminate_authorize_code_process()
print('code = %s'%code)<jupyter_output><empty_output><jupyter_text>The Authorization code flow gets the code via a the request\_uri callback and then sends the code to the AS which returns back the access token.<jupyter_code>manager.init_with_authorize_code(redirect_uri, code)<jupyter_output><empty_output><jupyter_text>And here's the access token:<jupyter_code>manager._access_token<jupyter_output><empty_output><jupyter_text>Do the GET with the Authorization Bearer TokenFinally, after this brief amount of basically one-time setup. Now you are read to actually issue the HTTP request to the Resource Server. This part is really easy (`manager.get()` is just `requests.get()` with the headers already set up for you): **NOTE** - We are no longer using Mulesoft, so this code below won't work:<jupyter_code>url = "https://test-columbia-demo-echo.cloudhub.io/v1/api/things"

response = manager.get(url)
print('status %s'%response.status_code)
print(response.headers)
print(response.text)<jupyter_output><empty_output><jupyter_text>This is a weird API in that it is echoing back information that is not normally shared with the client app, namely, the result of validating the Bearer Token that the client provided. But let's crack open that JSON response just a little anyway. First, let's look at the definition of a Thing object from the RAML:```%RAML 1.0 DataTypetype: objectproperties:  authorization:    type: string    access_token:    type: string  user:     type: string  tokenContext:     type: string  groups:     type: string  scopes:    type: string  client_id:    type: string  client_name:    type: string  http_method:    type: string  http_request_uri:    type: string  x_forwarded_for:    type: stringexample:  authorization: Bearer abcdefghi<PASSWORD>  access_token: NONE  user: <EMAIL>  tokenContext: foo bar  groups: g1 g2demo-echo  scopes: a b c  client_id: 64575d23b8504c9bb1e9e7ff558c3cd3  client_name: another authz demo app  http_method: GET  http_request_uri: /v1/api/things  x_forwarded_for: 172.16.58.3```<jupyter_code>j = json.loads(response.text)
if j and 'tokenContext' in j:
    tc = json.loads(j['tokenContext'])
    if tc and 'expires_in' in tc:
        print("Access token expires in %d minutes."%(tc['expires_in']/60))<jupyter_output><empty_output><jupyter_text>Let's try a **POST** request where we don't have the required "create" scope and see what errors look like:<jupyter_code>response = manager.post(url)
print('status %s: %s'%(response.status_code,response.text))<jupyter_output><empty_output><jupyter_text>Get a new token with the required scopes: auth-columbia, create, netphone-demo-admin<jupyter_code>post_service_information = ServiceInformation(
    authorize_service= oauth_server + '/as/authorization.oauth2',
    token_service= oauth_server + '/as/token.oauth2',
    client_id=creds['id'],
    client_secret=creds['secret'],
    scopes=['auth-columbia', 'create', 'demo-netphone-admin'])
post_manager = CredentialManager(post_service_information) # initialize the OAuth 2.0 client
post_authUrl = post_manager.init_authorize_code_process(redirect_uri, 'state_test')
print('Opening this url in your browser: %s'%post_authUrl)
webbrowser.open_new(post_authUrl)<jupyter_output><empty_output><jupyter_text>This wait_and_terminate... appears to be timing out sometimes.<jupyter_code>post_code = post_manager.wait_and_terminate_authorize_code_process()
post_manager.init_with_authorize_code(redirect_uri, post_code)
post_response = post_manager.post(url)
print('status %s'%post_response.status_code)
print(post_response.headers)
print(post_response.text)<jupyter_output><empty_output><jupyter_text>Refresh TokensSee the [documentation](https://github.com/antechrestos/OAuth2Client) for how to make use of refresh tokens. If you persist the refresh token, you can continue accessing the resource server without having to bug the user, after the access token expires, by getting a new one.<jupyter_code>print(manager.refresh_token)
rt = manager.refresh_token
newManager = OpenIdCredentialManager(service_information)
newManager.init_with_token(rt)
print(newManager._session.headers)
newResponse = newManager.get(url)
print('status %s'%newResponse.status_code)
print(newResponse.headers)
print(newResponse.text)
print("We've refreshed and now the old access token (%s) is replaced by a new token (%s)"%
    (manager._session.headers['Authorization'][len('Bearer '):],
    newManager._session.headers['Authorization'][len('Bearer '):]))
resp = manager.get(url)
print('see if the old token still works: %s'%resp.status_code)
# force an rate limiting error by trying to hit the API more than 10 times in a minute:
for i in range(100):
    resp = manager.get(url)
    if resp.status_code == 200:
        print(i)
    else:
        print('status %s: %s'%(resp.status_code,resp.text))
        break<jupyter_output><empty_output><jupyter_text>Appendix - Splitting the OpenID Connect ID TokenUses pwcrypto. The public keys of the AS are at https://oauth.cc.columbia.edu/pf/JWKS which can be found via https://oauth.cc.columbia.edu/.well-known/openid-configuration along with all the other oauth2/openID connect endpoints. There are likely multiple public keys -- and they get rotated frequently. The id_token's [JWT](https://tools.ietf.org/html/rfc7519) header includes the *kid* we need to use.Note that the id_token is generally short-lived (5 minutes) so if you wait too long to run these steps, jwcrypto will throw an error. This is also the case for the keys themselves.<jupyter_code># import the list of public keys from the OAuth server
keysetText = requests.get(oauth_server + '/pf/JWKS').text
# print(keysetText)
keyset=jwk.JWKSet().import_keyset(keysetText)<jupyter_output><empty_output><jupyter_text>The id_token is a [JSON Web Token](https://tools.ietf.org/html/rfc7519) (JWT, pronounced "jot") which is basically three base64-encoded elements separated by "."'s. Let's split it open to see what it looks like:<jupyter_code>id_token = manager.id_token
# check that openid token is valid, first just unencoding and printing it, then validating it.
print("id_token: %s"%id_token)
if id_token:
    splits = id_token.split('.')
    for i in range(3):
        missing_padding = 4 - len(splits[i]) % 4 # b64 encoding needs to be padded
        if missing_padding: 
            splits[i] += '='* missing_padding
    hdr = json.loads(base64.b64decode(splits[0]))
    body = json.loads(base64.b64decode(splits[1]))
    print('Header:\n%s'%pformat(hdr))
    print('Body:\n%s'%pformat(body))
    # the signature is binary junk
if id_token:
    # now validate the token:
    if 'exp' in body and int(time.time()) > body['exp']:
        print("The id_token is expired. The following jwcrypto validation will fail.")
    try:
        et = jwt.JWT(key=keyset, jwt=id_token)
        st = jwt.JWT(key=keyset, jwt=et.serialize())
        print('This is a %s id_token'%('valid' if st.token.is_valid else 'invalid'))
        print('header:\n%s\nbody:\n%s'%(pformat(json.loads(st.header)),pformat(json.loads(st.claims))))
    except Exception as e:
        print('Exception %s: %s'%(type(e),e))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09241882130560994, 'fraction_numerical': 0.010463781813272125, 'mean_word_length': 5.088779284833539, 'pattern_counts': {'":': 2, '<': 68, '<?xml version=': 0, '>': 69, 'https://': 14, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '402113', 'license': 'Apache-2.0', 'n_tokens_mistral': 4540, 'n_tokens_neox': 4386, 'n_words': 1508, 'path': 'demo-echo-client-py.ipynb', 'repo_name': 'n2ygk/api-training', 'size': '14845'}