<jupyter_start><jupyter_text>The following code will take the CLI commands produced in `01-JJA-L2V-Configuration-Files` notebook + You need to install aws cli    http://docs.aws.amazon.com/cli/latest/userguide/installing.html + You need to run** `aws config` ** Let's import the functions defined before for loading parameters<jupyter_code>from load_config import params_to_cli
    llr, emb, pred,evaluation = params_to_cli("CONFIGS/ex1-ml-1m-config.yml", "CONFIGS/ex4-du04d100w10l80n10d30p1q1-1000-081417-params.yml")
llr
evaluation<jupyter_output><empty_output><jupyter_text>This function will format the AWS CLI commands so we can pass them to the cluster using `boto3`<jupyter_code>def create_steps(llr=None, emb=None, pred=None, evaluation=None, name=''):
    if llr != None:
        Steps=[

        {
            'Name': name + '-LLR',
            'ActionOnFailure': 'CONTINUE',
            'HadoopJarStep': {
                'Jar': 'command-runner.jar',
                'Args': (llr).split(),
            }
        },
        {
            'Name': name + '-EMB',
            'ActionOnFailure': 'CONTINUE',
            'HadoopJarStep': {
                'Jar': 'command-runner.jar',
                'Args': (emb).split(),
            }
        },
        {
            'Name': name + '-PRED',
            'ActionOnFailure': 'CONTINUE',
            'HadoopJarStep': {
                'Jar': 'command-runner.jar',
                'Args': (pred).split(),
            }
        },
            {
            'Name': name + '-EVAL',
            'ActionOnFailure': 'CONTINUE',
            'HadoopJarStep': {
                'Jar': 'command-runner.jar',
                'Args': (evaluation).split(),
            }
        }
    ]
    else:
        Steps=[
        {
            'Name': name + '-EMB',
            'ActionOnFailure': 'CONTINUE',
            'HadoopJarStep': {
                'Jar': 'command-runner.jar',
                'Args': (emb).split(),
            }
        },
        {
            'Name': name + '-PRED',
            'ActionOnFailure': 'CONTINUE',
            'HadoopJarStep': {
                'Jar': 'command-runner.jar',
                'Args': (pred).split(),
            }
        },
            {
            'Name': name + '-EVAL',
            'ActionOnFailure': 'CONTINUE',
            'HadoopJarStep': {
                'Jar': 'command-runner.jar',
                'Args': (evaluation).split(),
            }
        }
    ]
                     
            
    return Steps<jupyter_output><empty_output><jupyter_text>To load the commands into EMR Here we create steps based on the three steps in the pipeline<jupyter_code># ex2 = create_steps(llr=llr, emb=emb, pred=pred, evaluation=evaluation, name='EXP3')
ex3 = create_steps(llr=llr, emb=emb, pred=pred, evaluation=evaluation, name='EXP3')
# ex4 = create_steps(emb=emb348, pred=pred348, name='EXP4')
# ex5 = create_steps(emb=emb349, pred=pred349, name='EXP5')<jupyter_output><empty_output><jupyter_text>If we are adding multiple runs of the pipeline<jupyter_code># steps = ex2 + ex3 + ex4 + ex5
steps = ex3
steps<jupyter_output><empty_output><jupyter_text>To run the steps into EMR using boto3<jupyter_code>import boto3
client = boto3.client('emr')
cluster_id = 'j-2JGJ9RIFQ4VRK'
response = client.add_job_flow_steps(
    JobFlowId = cluster_id,
    Steps= steps
)
response<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13594809790622236, 'fraction_numerical': 0.01857859038631672, 'mean_word_length': 2.092069279854148, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '647683', 'license': 'Apache-2.0', 'n_tokens_mistral': 1138, 'n_tokens_neox': 1087, 'n_words': 216, 'path': 'notebooks/02-JJA-CLI-to-EMR-with-boto.ipynb', 'repo_name': 'danielemarmiroli/Like2Vec', 'size': '3391'}