<jupyter_start><jupyter_text>Copyright 2020 The KNIX AuthorsLicensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at   http://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.  Migrating workflow from AWS Stepfunctions to KNIXKNIX is compatible with AWS Lambda and Step Functions with expanded support for sophisticated parallel executions. This tutorial shows how to migrate an existing AWS StepFunctions workflow from an AWS tutorial to KNIX:https://aws.amazon.com/de/getting-started/hands-on/create-a-serverless-workflow-step-functions-lambda/ Goal for this Notebook:Show a simple example of migrating sample workflows in Python, using SDKs provided for both AWS and KNIX. This is aimed for those looking to get into the field or those who are already in the field and looking to see an example how to move existing workflows between AWS and KNIX. This Notebook will show basic examples of:* Importing SDKs* Generating and using SDK objects* Converting ARNs to KNIX names * Convert AWS Lambda deployment packages* Import and export worflow and function definitions Required Libraries:* [json] (http://www.json.org/)* [zipfile] (https://docs.python.org/3/library/zipfile) Things to remember:* In Step-Functions workflow descriptions, 'Resource' ARN needs to be changed into real Lambda ARNs. This can be achieved by prepending a fixed, user-specific prefix of the form of "arn:aws:lambda:eu-central-1:123456789012:function:" . Note this step does not require any change in user code.* In Lambda,  function handler needs to be configured when creating the function. In KNIX the function handler name must always be called "handle". Note this step does not require changing user code.* KNIX users should put the libraries that they would like to be part of LD_LIBRARY_PATH in a ./lib/ folder, which is inside their deployment zip and sits parallel to their fuction code (referring to the .py file that has the 'handle' method)* In KNIX, if user's deployment zip contains ELF executable binaries that can be invoked from the python code (using the subprocess module), then these binaries should be invoked using their complete path, and not via symbolic links to them.* User code in Lambda is only allowed to create files in /tmp, whereas, in KNIX the entire filesystem is writable. Now let's start to migrate a workflow from KNIX to AWS. First, install the required AWS SDK. Please note that you need to configure your credentials for using this SDK, e.g by adding your credentials to ~/.aws/config:<jupyter_code>!pip3 install boto3<jupyter_output>Collecting boto3
  Using cached https://files.pythonhosted.org/packages/1d/3e/95edad6297483d8d4985486f54c8c5441bdd5f0726918dbfe26cd32bc6a4/boto3-1.14.3-py2.py3-none-any.whl
Collecting jmespath<1.0.0,>=0.7.1 (from boto3)
  Using cached https://files.pythonhosted.org/packages/07/cb/5f001272b6faeb23c1c9e0acc04d48eaaf5c862c17709d20e3469c6e0139/jmespath-0.10.0-py2.py3-none-any.whl
Collecting botocore<1.18.0,>=1.17.3 (from boto3)
  Using cached https://files.pythonhosted.org/packages/b7/15/fb65d9dc6194dc9f1a9cd5f4c64c884288919562fe84159641d6154318f4/botocore-1.17.3-py2.py3-none-any.whl
Collecting s3transfer<0.4.0,>=0.3.0 (from boto3)
  Using cached https://files.pythonhosted.org/packages/69/79/e6afb3d8b0b4e96cefbdc690f741d7dd24547ff1f94240c997a26fa908d3/s3transfer-0.3.3-py2.py3-none-any.whl
Collecting urllib3<1.26,>=1.20; python_version != "3.4" (from botocore<1.18.0,>=1.17.3->boto3)
  Using cached https://files.pythonhosted.org/packages/e1/e5/df302e8017440f111c11cc41a6b432838672f5a70aa29227[...]<jupyter_text>Now import the required librares:<jupyter_code>import json
import os
import urllib.request
from zipfile import ZipFile
from mfn_sdk import MfnClient
import boto3
from botocore.exceptions import ClientError
import shutil<jupyter_output><empty_output><jupyter_text>get a boto3 client object:<jupyter_code>client_sf = boto3.client('stepfunctions')
client_lambda = boto3.client('lambda')<jupyter_output><empty_output><jupyter_text>Now use your credentials to get a knix client object:<jupyter_code>client_mfn = MfnClient(
    #mfn_url="http://knix.io/mfn",
    mfn_url="http://localhost:8080",
    mfn_user="mfn@mfn",
    mfn_password="<PASSWORD>",
    mfn_name="KS",
    proxies={"http_proxy": "None", "https_proxy": "None"}
    )<jupyter_output><empty_output><jupyter_text>We need a few parameters to prepare the data access to AWS services suchas as Lambda and Stepfunctions:* lambdaprefix: the ARN name prefix for AWS Lambda functions* awsSfRoleName: the name (ARN) allowing the boto3 client to access the the AWS Stepfunctions service* knixWfName: the name of the source workflow on KNIX* sfWFName: the name of the target workflow on AWS Stepfunctions Define a function to get the correct AWS state machine arn<jupyter_code>def get_state_machine_arn(name):
        """
        Get a state machine given its name
        """
        response = client_sf.list_state_machines()
        if not response.get('stateMachines'):
            return None
        for sm in response.get('stateMachines'):
            if sm['name'] == name:
                return sm['stateMachineArn']<jupyter_output><empty_output><jupyter_text>Define a function to convert lambda deployment package and upload to KNIX<jupyter_code>def convert_lambda_and_upload_as_mfn(local, lambda_arn):
    """
    Convert lambdas into mfn deployment packages and upload them 
    """    
    response = client_lambda.get_function(FunctionName=lambda_arn)    
        
    awsfile = lambda_arn.split(":")[-1]
    print ("awsfile: " + awsfile)
    awshandler = str(response['Configuration']['Handler'].split(".")[1])
    #print ("awshandler:" +  awshandler)
    
    if "S3" in json.dumps(response['Code']['RepositoryType']): # we need to handle a deployment package
        url =  str(response['Code']['Location'])
        filein = urllib.request.urlretrieve(url)[0]  # download the Lambda deployment package from AWS
        print ("Lambda Function deployment package:  ",  filein)

        with ZipFile(filein, 'r') as zip_ref: # extract lambda_file to local dir
            zip_ref.extractall(local)
            
        fin = open(local +  "lambda_function.py", "rt") # open extracted file
        fout = open(local + awsfile+".py", "wt") # open new python file for modified source code (handler)
        mfnhandler = "handle" # Mfn grain handle needs to be "handle"

        for line in fin:
            # replace the lambda handler and write to new python source file compatible with knix
            fout.write(line.replace(awshandler, mfnhandler))
            
        fin.close() # close input file
        os.remove(local + "lambda_function.py") # cleanup lambda function file 
        fout.close() # close output file
        
        with ZipFile(local +  awsfile+".zip", 'w') as zip_ref: # generate new deployment file compatibl with knix
            zip_ref.write(local + awsfile+".py")        
                        
        mfn = client_mfn.add_function(awsfile) # add function to mfn

        zip_name = local + "%s.zip" % (awsfile) # get deployment package file name

        mfn.upload(zip_name) # upload deployment package to KNIX
        
    else: # no deployment package for this lambda
        raise Exception("Error: Non-supported lambda repository type")<jupyter_output><empty_output><jupyter_text>Now set the the AWS Step Functions source state machine name and target KNIX workflow name for migration<jupyter_code>user_id = str(boto3.resource('iam').CurrentUser().arn.split(":")[4])
sm_name = "CallCenterStateMachine" # name of the source State Machine
sm_arn = "arn:aws:states:eu-central-1:%s:stateMachine:%s" % (user_id, sm_name)
knix_wf_name = "test_wf_knix" # target state machine name on KNIX<jupyter_output><empty_output><jupyter_text>Start the conversion loop over all States Machine states<jupyter_code>local = "./deployment_packages/" #"C:\\Python36\\Scripts\\"
    
try:
    if not os.path.exists(os.path.dirname(local)):
        os.makedirs(os.path.dirname(local))
except OSError as err:
    print(err)


# query state machine at AWS SF
response = client_sf.describe_state_machine(stateMachineArn=sm_arn)
#print("Response: " + str(response))

print ("Processing %s ... " % response['name'])
awssfwf = json.loads(response['definition'])

for att, val in awssfwf['States'].items(): # loop over all states with Resources    
    print ("processing %s" % (att))
    if "Resource" in val.keys():        
        #response_l = client_lambda.get_function(FunctionName=val['Resource'])
        convert_lambda_and_upload_as_mfn(local, val['Resource'])
        val['Resource'] =  val['Resource'].split(":")[-1] # remove arn prefix        
    else:
        print("Info: Non-Task State, not converted!")
        pass
            
print ("Uploading modified workflow to KNIX: %s" % knix_wf_name)

test_wf_knix = client_mfn.add_workflow(knix_wf_name)
test_wf_knix.json = json.dumps(awssfwf, indent=2, sort_keys=True)

print ("Workflow and grains successfully uploaded to KNIX!")

# finally, let's clean up local directories
shutil.rmtree(local)
print(f"Starting workflow deployment...")
test_wf_knix.deploy(timeout=0)
print(f"Workflow deployed, sending input...")
result = test_wf_knix.execute({"inputCaseID": "001"})
"""
Success: 
{
  "output": {
    "Case": "001",
    "Status": 1,
    "Message": "Case 001: opened...assigned...closed."
  },
  "outputDetails": null
}
or
{
  "output": {
    "Case": "001",
    "Status": 0,
    "Message": 'Message': 'Case 001: opened...assigned...unresolved...escalating.'}
  },
  "outputDetails": null
}
Fail:
{
  "error": null,
  "cause": "Engage Tier 2 Support."
}
"""
print(f"Received result: {result}")<jupyter_output>Starting workflow deployment...
Workflow deployed, sending input...
Received result: {'Case': '001', 'Status': 0, 'Message': 'Case 001: opened...assigned...unresolved...escalating.'}
{'fraction_non_alphanumeric': 0.1022627949888317, 'fraction_numerical': 0.03272797902301641, 'mean_word_length': 4.214177215189873, 'pattern_counts': {'":': 18, '<': 37, '<?xml version=': 0, '>': 38, 'https://': 7, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '89440', 'license': 'Apache-2.0', 'n_tokens_mistral': 3375, 'n_tokens_neox': 3158, 'n_words': 994, 'path': 'tutorials/migration/aws_sf_cc_to_knix.ipynb', 'repo_name': 'iakkus/knix', 'size': '10290'}