<jupyter_start><jupyter_text>Deep Escape - a coding adventure in machine learningTo enjoy this adventure, run each cell one at a time and in order. Click on the cell you wish to run, the press shift-Enter (or click the play icon on the toolbar) to run a cell. The blue line on the left indicates which cell is currently selected to run.When prompted to add code, you can press the plus (+) button on the above toolbar to add a new code cell. You will often copy and paste code that's been provided to you in previous cells. For a quick guide, and to see the basic adventure, don't modify the code. If you want to make changes, you might have to train a new model yourself, which can take at least an hour, if not more. But that's fine - this is meant to be used for experimentation!![image](https://sagemaker-examples.readthedocs.io/en/latest/_images/successful_policy.gif)  Level 1: The AwakeningYou feel yourself being pulled into consiousness, like someone is talking to you and trying to get you out of bed, but you're still so tired. This voice... it just keeps talking and talking, but you can't make out the words. You try to open your eyes. You can't make them open. You can't even feel them, or your hands. You can't move! You start to panic, but finally, you hear the words clearly.<jupyter_code>import IPython
# can eventually streamline these with event function calls
IPython.display.Audio("https://aaron-1.s3.amazonaws.com/audio/youre_awake.mp3", autoplay=True)<jupyter_output><empty_output><jupyter_text>*Oh, you're awake! That's good. Can your hear me? Yes. Ok, listen. I know you're confused, but I'm going to help you get out of here. But first you need to learn how to control a robot. You'll have to use a little bit of code.*<jupyter_code>IPython.display.Audio("https://aaron-1.s3.amazonaws.com/audio/this_is_your_console.mp3", autoplay=True)<jupyter_output><empty_output><jupyter_text>*This is your console. There are many like it, but this one is yours.*<jupyter_code># Calibrating your settings
import sagemaker
from sagemaker.rl import RLEstimator, RLToolkit, RLFramework
from time import gmtime, strftime

job_name_prefix = "deep-escape"
instance_type = "ml.c4.2xlarge"
job_duration_in_seconds = 3600

sagemaker_role = sagemaker.get_execution_role()
sagemaker_session = sagemaker.Session()

s3_bucket = sagemaker_session.default_bucket()
s3_output_path = "s3://{}/".format(s3_bucket)
s3_prefix = job_name_prefix + "-sagemaker-" + strftime("%y%m%d-%H%M%S", gmtime())


print("sagemaker_role: {}".format(sagemaker_role))
print("s3_bucket: {}".format(s3_bucket))
print("s3_prefix: {}".format(s3_prefix))   
# Are you running this in a region that will work?
aws_region = sagemaker_session.boto_region_name
if aws_region not in ["us-west-2", "us-east-1", "eu-west-1"]:
    out = IPython.display.Audio("https://aaron-1.s3.amazonaws.com/audio/config_failed.mp3", autoplay=True)
else:    
    out = IPython.display.Audio("https://aaron-1.s3.amazonaws.com/audio/config_complete.mp3", autoplay=True)

out
IPython.display.Audio("https://aaron-1.s3.amazonaws.com/audio/environment.mp3", autoplay=True)<jupyter_output><empty_output><jupyter_text>*From here, you can run code that will use reinforcement learning to train a model. Your robot will use this model to move about in the world. The training uses an "agent", which is a virtual robot in this case. This agent is placed in an environment that has parameters. You define the rewards in a reward function by using the parameters from the environment.*<jupyter_code>IPython.display.Audio("https://aaron-1.s3.amazonaws.com/audio/uploaded_to_console.mp3", autoplay=True)<jupyter_output><empty_output><jupyter_text>*I'm uploading it to your console now*<jupyter_code>!pygmentize ./straight.py<jupyter_output>[34mdef[39;49;00m [32mreward_function[39;49;00m(params):
    [33m'''[39;49;00m
[33m    Example of rewarding the agent to follow a center line[39;49;00m
[33m    '''[39;49;00m
    
    [37m# Read input parameters[39;49;00m
    track_width = params[[33m'[39;49;00m[33mtrack_width[39;49;00m[33m'[39;49;00m]
    distance_from_center = params[[33m'[39;49;00m[33mdistance_from_center[39;49;00m[33m'[39;49;00m]
    
    [37m# Calculate 3 markers that are at varying distances away from the center line[39;49;00m
    marker_1 = [34m0.1[39;49;00m * track_width
    marker_2 = [34m0.25[39;49;00m * track_width
    marker_3 = [34m0.5[39;49;00m * track_width
    
    [37m# Give higher reward if the car is closer to center line and vice versa[39;49;00m
    [34mif[39;49;00m distance_from_center <= marker_1:
        reward = [34m1.0[39;49;00m
    [34melif[39;49;00m distance_from_center <= marker_2:
        reward = [34m0.5[39;49;00m
    [34melif[39;49;00m distance_f[...]<jupyter_text>Rewards for the RobotYou see before you a bunch of Python code. How you can see it without seeing does't make much sense, but you're willing to play along with this mysterious voice. You're pretty decent at hacking on Python, so you notice the key points about the function being displayed.It says it's an example for rewarding an agent to follow a center line. An agent? They must mean a robot! This must be how you train a model that a robot can use to drive around. Following a line is probably the easiest thing you can do.There are two main inputs to this reward function:* `track_width`* `distance_from_center`It goes on to provide different rewards based on a few scenarios. Three markers are created that are used like a bullseye around the center of the track. If the agent's position closest to the center by being within 10% of either side of the center the agent gets a reward of 1. If it's within 25%, it gets 0.5 as a reward. If it's within 50%, it gets .1. If it's not any of these, well, it must be off the track because 50% on either side is the whole track width![diagram]Ok, so what do I do with this?<jupyter_code>IPython.display.Audio("https://aaron-1.s3.amazonaws.com/audio/unlocking.mp3", autoplay=True)<jupyter_output><empty_output><jupyter_text>*I'm unlocking some new input parameters for you.*<jupyter_code>IPython.display.Audio("https://aaron-1.s3.amazonaws.com/audio/unlocked_track_width.mp3", autoplay=True)<jupyter_output><empty_output><jupyter_text>*Congratulations! You have unlocked: track_width*<jupyter_code>IPython.display.Audio("https://aaron-1.s3.amazonaws.com/audio/unlocked_distance_from_center.mp3", autoplay=True)<jupyter_output><empty_output><jupyter_text>*Congratulations! You have unlocked: distance_from_center* *TODO:* streamline the code below; there's way too much going on that I don't want to set manually<jupyter_code>custom_hyperparameter = {
    "s3_bucket": s3_bucket,
    "s3_prefix": s3_prefix,
    "aws_region": aws_region,
    "model_metadata_s3_key": "%s/model/model_metadata.json" % s3_prefix,
    "reward_function_s3_source": "%s/straight.py" % s3_prefix,
    "batch_size": "64",
    "num_epochs": "10",
    "stack_size": "1",
    "lr": "0.0003",
    "exploration_type": "Categorical",
    "e_greedy_value": "1",
    "epsilon_steps": "10000",
    "beta_entropy": "0.01",
    "discount_factor": "0.999",
    "loss_type": "Huber",
    "num_episodes_between_training": "20",
    "max_sample_count": "0",
    "sampling_frequency": "1"
    #     ,"pretrained_s3_bucket": "sagemaker-us-east-1-259455987231"
    #     ,"pretrained_s3_prefix": "deepracer-notebook-sagemaker-200729-202318"
}

estimators_kwargs = {
    "entry_point": "base_training_worker.py",
    "source_dir": "src",
    "dependencies": ["common/"],
    "role": sagemaker_role,
    "train_instance_type": instance_type,
    "train_instance_count": 1,
    "output_path": s3_output_path,
    "base_job_name": job_name_prefix,
#    "metric_definitions": metric_definitions,
    "train_max_run": job_duration_in_seconds,
    "hyperparameters": custom_hyperparameter,
#    "subnets": deepracer_subnets,
#    "security_group_ids": deepracer_security_groups,
}

#custom_image_name = ""
#estimators_kwargs["image_uri"] = custom_image_name
#estimator = RLEstimator(**estimators_kwargs)
#estimator.fit(wait=False)
#job_name = estimator.latest_training_job.job_name
#print("Training job: %s" % job_name)<jupyter_output><empty_output><jupyter_text>Trial and Error*TODO: If we could load the console in another tab in Studio, we could direct them to preview the job there!*You watch as a Robomaker training job is run from your console. After a few moments a video player appears that shows a cartoon car and race track. The car - this must be the "agent" - just sits there while a message blinks "initializing". A graph appears next to it, showing the start of a few lines for "average reward" and "percent completion" for training and evaluation. The video begins to play. The car moves along the track and quickly swerves, crashing into the side of the track. The simulation appears to restart, with the car back in the middle of the track. It tries again. This time it makes a little further, then crashes into the right side of the track. This repeats over and over and over with the car turning wildly into the side after making a fairly decent run. After a while you notice that it isn't starting from the beginning. It starts from were it last crashed. Interesting. You can see its progress by the presence of a ghostly overlay showing the entire track with a little purple sphere representing the current position of the car-agent.<jupyter_code># Would display a streaming video here, but it is blocked from some reason; player loads, but playback doesn't work
IPython.display.Video("https://aaron-1.s3.amazonaws.com/audio/first_tries.mp4")
IPython.display.Video("./first_tries.mp4")<jupyter_output><empty_output><jupyter_text>Eventually, it makes it all of the way around the track. Time passes as the little agent goes through its trials. Every mistake is part of its training. Every success, a reward is granted.You notice after several laps that the chart is starting to form some meaning. It has a low reward value and low completion value. Over time, you see the agent making fewer and fewer mistakes and staying on the track for longer. After an hour, the training session stops and you review the graph.<jupyter_code># TODO: Fix image so bottom isn't clipped
IPython.display.Image("https://aaron-1.s3.amazonaws.com/image/rl_training_graph.png", width=600)<jupyter_output><empty_output><jupyter_text>Deployment{apply the model to move to next level; would be interesting to trap when there's a failure, and do a reset, so they get the full trial and error experience}{display the skill tree, then unlock new skills!}Unlocks: wheels (all_wheels_on_track)<jupyter_code>IPython.display.Audio("https://aaron-1.s3.amazonaws.com/audio/unlocked_wheels.mp3", autoplay=True)<jupyter_output><empty_output><jupyter_text>*Congratulations! You have unlocked: all_wheels_on_track*<jupyter_code>!pygmentize ./wheels_track_width.py<jupyter_output>[34mdef[39;49;00m [32mreward_function[39;49;00m(params):
    [33m'''[39;49;00m
[33m    Example of rewarding the agent to stay inside the two borders of the track[39;49;00m
[33m    '''[39;49;00m
    
    [37m# Read input parameters[39;49;00m
    all_wheels_on_track = params[[33m'[39;49;00m[33mall_wheels_on_track[39;49;00m[33m'[39;49;00m]
    distance_from_center = params[[33m'[39;49;00m[33mdistance_from_center[39;49;00m[33m'[39;49;00m]
    track_width = params[[33m'[39;49;00m[33mtrack_width[39;49;00m[33m'[39;49;00m]
    
    [37m# Give a very low reward by default[39;49;00m
    reward = [34m1e-3[39;49;00m

    [37m# Give a high reward if no wheels go off the track and [39;49;00m
    [37m# the car is somewhere in between the track borders [39;49;00m
    [34mif[39;49;00m all_wheels_on_track [35mand[39;49;00m ([34m0.5[39;49;00m*track_width - distance_from_center) >= [34m0.05[39;49;00m:
        reward = [34m1.0[39;49;00m

    [37m# Always ret[...]<jupyter_text>How to use skills{show how track width works; display the code for the input params for the new skills; they'll use these in the next level to pass the obstacle; skills can be easy-to-use functions that can be cobbled together and expand the action space for the player}  Level 2: The walls are lava{inside the vent, robot can't touch the walls or it'll get stuck (realize that the offtrack even mean both wheels went off, not just the outside ones, so might need to recast this so it is technically accurate... or not)}Unlocks: steering (abs_steering)<jupyter_code>IPython.display.Audio("https://aaron-1.s3.amazonaws.com/audio/unlocked_steering.mp3", autoplay=True)<jupyter_output><empty_output><jupyter_text>*Congratulations! You have unlocked: abs_steering*<jupyter_code>!pygmentize ./zigzag.py<jupyter_output>[34mdef[39;49;00m [32mreward_function[39;49;00m(params):
    [33m'''[39;49;00m
[33m    Example of penalize steering, which helps mitigate zig-zag behaviors[39;49;00m
[33m    '''[39;49;00m
    
    [37m# Read input parameters[39;49;00m
    distance_from_center = params[[33m'[39;49;00m[33mdistance_from_center[39;49;00m[33m'[39;49;00m]
    track_width = params[[33m'[39;49;00m[33mtrack_width[39;49;00m[33m'[39;49;00m]
    abs_steering = [36mabs[39;49;00m(params[[33m'[39;49;00m[33msteering_angle[39;49;00m[33m'[39;49;00m]) [37m# Only need the absolute steering angle[39;49;00m

    [37m# Calculate 3 marks that are farther and father away from the center line[39;49;00m
    marker_1 = [34m0.1[39;49;00m * track_width
    marker_2 = [34m0.25[39;49;00m * track_width
    marker_3 = [34m0.5[39;49;00m * track_width

    [37m# Give higher reward if the car is closer to center line and vice versa[39;49;00m
    [34mif[39;49;00m distance_from_center <= marker_[...]<jupyter_text>Level 3: Straight > zig zag{fan is starting soon, so robot must make it in time and learn not to zigzag}Unlocks: speed  Level 4: Full speed ahead{fan is on now and you noticed you are going faster; you need to adjust your speed so you don't crash at a turn in the vent.}Unlocks: throttle, object distance, object location  Level 5: I was blind, but now I can see{You see a light at the end of the tunnel! There's and opening in the vent. You get to the end, but it's blocked by a vent door. It opens a little when you run into it. You decide to go back at hit it hard enough to open the vent door.}Unlocks: crashed, track length  Level 6: Charge!{You recall how long it was from the turn to the vent opening. You return to the starting point, so that you can try to bash through the vent, but only enough to open it, not damage yourself.}Unlocks: waypoints, progress  Level 7: A dodgy affair{Once out of the vent your see that you're in an office/lab where men and women in lab coats are wroking. You bide your time watching for patterns. You notice that when one leaves, the door remains open for 10 seconds. More than enough time for you to slip through!}Unlocks: dodge<jupyter_code>!pygmentize ./dodge.py<jupyter_output>[34mimport[39;49;00m [04m[36mmath[39;49;00m
[34mdef[39;49;00m [32mreward_function[39;49;00m(params):
    [33m'''[39;49;00m
[33m    Example of rewarding the agent to stay inside two borders[39;49;00m
[33m    and penalizing getting too close to the objects in front[39;49;00m
[33m    '''[39;49;00m
    all_wheels_on_track = params[[33m'[39;49;00m[33mall_wheels_on_track[39;49;00m[33m'[39;49;00m]
    distance_from_center = params[[33m'[39;49;00m[33mdistance_from_center[39;49;00m[33m'[39;49;00m]
    track_width = params[[33m'[39;49;00m[33mtrack_width[39;49;00m[33m'[39;49;00m]
    objects_location = params[[33m'[39;49;00m[33mobjects_location[39;49;00m[33m'[39;49;00m]
    agent_x = params[[33m'[39;49;00m[33mx[39;49;00m[33m'[39;49;00m]
    agent_y = params[[33m'[39;49;00m[33my[39;49;00m[33m'[39;49;00m]
    _, next_object_index = params[[33m'[39;49;00m[33mclosest_objects[39;49;00m[33m'[39;49;00m]
    objects_left_of_center = params[[33m'[...]
{'fraction_non_alphanumeric': 0.11459762944479102, 'fraction_numerical': 0.060636306924516534, 'mean_word_length': 5.318880567599527, 'pattern_counts': {'":': 33, '<': 64, '<?xml version=': 0, '>': 63, 'https://': 14, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 11, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '326884', 'license': 'Apache-2.0', 'n_tokens_mistral': 6112, 'n_tokens_neox': 5336, 'n_words': 1677, 'path': 'deep_escape.ipynb', 'repo_name': 'aaronmarkham/deepescape', 'size': '16030'}