<jupyter_start><jupyter_text>Create Load Balancer Auto Scaled Apache Web ServerIn this workshop we will explore the basics of AWS with EC2, Amazon S3, and the components required for auto scaling that will provide elasticity and durability to tradtional web applications. Python is used extensively so you will need experience in or be comfortable reading python code. ![Elasticity](../../docs/assets/images/intro_load_balancing.png) Initialize notebookWe will be using the [boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html) library for creation of all resources.<jupyter_code>import boto3
import sys
import os
import json
import base64
import project_path # path to helper methods
import pprint

from lib import workshop
from botocore.exceptions import ClientError

ec2_client = boto3.client('ec2')
ec2 = boto3.resource('ec2')
elb = boto3.client("elbv2")
cloudwatch = boto3.client('cloudwatch')
asg = boto3.client('autoscaling')

session = boto3.session.Session()
region = session.region_name

alb_sec_group_name = 'alb-sg'
launch_config_name = 'web-lc'
auto_scaling_group_name = 'web-asg'
scale_up_name = 'scale_up'
scale_down_name = 'scale_down'<jupyter_output><empty_output><jupyter_text>[Create S3 Bucket](https://docs.aws.amazon.com/AmazonS3/latest/gsg/CreatingABucket.html)We will create an S3 bucket that will be used throughout the workshop for storing data.[s3.create_bucket](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.htmlS3.Client.create_bucket) boto3 documentation<jupyter_code>bucket = workshop.create_bucket_name('intro-')
session.resource('s3').create_bucket(Bucket=bucket, CreateBucketConfiguration={'LocationConstraint': region})
print(bucket)<jupyter_output><empty_output><jupyter_text>[Create VPC](https://aws.amazon.com/vpc/)Amazon Virtual Private Cloud (Amazon VPC) lets you provision a logically isolated section of the AWS Cloud where you can launch AWS resources in a virtual network that you define. You have complete control over your virtual networking environment, including selection of your own IP address range, creation of subnets, and configuration of route tables and network gateways. You can use both IPv4 and IPv6 in your VPC for secure and easy access to resources and applications.<jupyter_code>vpc, subnet, subnet2 = workshop.create_and_configure_vpc()
vpc_id = vpc.id
subnet_id = subnet.id
subnet2_id = subnet2.id
print(vpc_id)
print(subnet_id)
print(subnet2_id)<jupyter_output><empty_output><jupyter_text>Create index.html page for the web applicationWe will write out a simple html page to demo setting up the Apache web server using an Application Load Balancer and Auto Scaling to provide elasticity to your web application.<jupyter_code>%%writefile index.html

<h1>Hello from the intro to AWS workshop!!!</h1><jupyter_output><empty_output><jupyter_text>[Upload to S3](https://docs.aws.amazon.com/AmazonS3/latest/dev/Welcome.html)Next, we will upload the index.html file created above to S3 to be used later in the workshop.[s3.upload_file](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.htmlS3.Client.upload_file) boto3 documentation<jupyter_code>session.resource('s3').Bucket(bucket).Object(os.path.join('web', 'index.html')).upload_file('index.html')<jupyter_output><empty_output><jupyter_text>[Create Security Groups](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html)A security group acts as a virtual firewall for your instance to control inbound and outbound traffic. When you launch an instance in a VPC, you can assign up to five security groups to the instance. Security groups act at the instance level, not the subnet level. Therefore, each instance in a subnet in your VPC could be assigned to a different set of security groups. If you don't specify a particular group at launch time, the instance is automatically assigned to the default security group for the VPC.[ec2_client.create_security_group](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ec2.htmlEC2.Client.create_security_group) boto3 documentation<jupyter_code>sg = ec2_client.create_security_group(
    Description='security group for ALB',
    GroupName=alb_sec_group_name,
    VpcId=vpc_id
)
alb_sec_group_id=sg["GroupId"]
print('ALB security group id - ' + alb_sec_group_id)<jupyter_output><empty_output><jupyter_text>Configure available portsIn order for the ALB to communicate with the outside world, we will open port 80 and 443. As you can see in the call below we can define the `ToPort` and `FromPort` and a `CidrIp` range we want to allow.[ec2_client.authorize_security_group_ingress](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ec2.htmlEC2.Client.authorize_security_group_ingress) boto3 documentation<jupyter_code>data = ec2_client.authorize_security_group_ingress(
    GroupId=alb_sec_group_id,
    IpPermissions=[
        {'IpProtocol': 'tcp',
         'FromPort': 80,
         'ToPort': 80,
         'IpRanges': [
            {
                'CidrIp': '0.0.0.0/0',
                'Description': 'HTTP access'
            },
          ]
        },
        {'IpProtocol': 'tcp',
         'FromPort': 443,
         'ToPort': 443,
         'IpRanges': [
            {
                'CidrIp': '0.0.0.0/0',
                'Description': 'HTTPS access'
            },
          ]
        }
    ]
)<jupyter_output><empty_output><jupyter_text>[Create Application Load Balancer (ALB)](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/introduction.html)Elastic Load Balancing supports three types of load balancers: Application Load Balancers, Network Load Balancers, and Classic Load Balancers. In this example we will be using an [Application Load Balancers](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/introduction.html). For more information about Network Load Balancers, see the [User Guide for Network Load Balancers](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/). For more information about Classic Load Balancers, see the [User Guide for Classic Load Balancers](https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/).[elbv2.create_load_balancer](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/elbv2.htmlElasticLoadBalancingv2.Client.create_load_balancer) boto3 documentation<jupyter_code>alb = elb.create_load_balancer(
    Name='web-load-balancer',
    Subnets=[
        subnet_id,
        subnet2_id
    ],
    SecurityGroups=[
        alb_sec_group_id,
    ],
    Scheme='internet-facing',
    Type='application',
    IpAddressType='ipv4'
)

alb_arn = alb["LoadBalancers"][0]["LoadBalancerArn"]
alb_name = alb["LoadBalancers"][0]["LoadBalancerName"]<jupyter_output><empty_output><jupyter_text>[Create Target Group](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-target-groups.html)Each target group is used to route requests to one or more registered targets. When you create each listener rule, you specify a target group and conditions. When a rule condition is met, traffic is forwarded to the corresponding target group. You can create different target groups for different types of requests. For example, create one target group for general requests and other target groups for requests to the microservices for your application.[elbv2.create_target_group](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/elbv2.htmlElasticLoadBalancingv2.Client.create_target_group) boto3 documentation<jupyter_code>target_group = elb.create_target_group(
    Name='alb-target-group',
    Protocol='HTTP',
    Port=80,
    VpcId=vpc_id,
    HealthCheckProtocol='HTTP',
    HealthCheckPort='80',
    HealthCheckPath='/'
)

target_group_arn = target_group["TargetGroups"][0]["TargetGroupArn"]<jupyter_output><empty_output><jupyter_text>[Create Listener](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-listeners.html)Before you start using your Application Load Balancer, you must add one or more listeners. A listener is a process that checks for connection requests, using the protocol and port that you configure. The rules that you define for a listener determine how the load balancer routes requests to the targets in one or more target groups.[elbv2.create_listener](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/elbv2.htmlElasticLoadBalancingv2.Client.create_listener) boto3 documentation<jupyter_code>listener = elb.create_listener(
    DefaultActions=[
        {'TargetGroupArn': target_group_arn,
         'Type': 'forward'
        }],
    LoadBalancerArn=alb_arn,
    Port=80,
    Protocol='HTTP'
)

listener_arn = listener["Listeners"][0]["ListenerArn"]<jupyter_output><empty_output><jupyter_text>Get Latest Amazon Linux AMI<jupyter_code>ami = workshop.get_latest_amazon_linux()
print(ami)<jupyter_output><empty_output><jupyter_text>Create UserData to install Apache web server and download indexReplace the `{{bucket}}` value with the S3 bucket you created above<jupyter_code>%%writefile userdata.sh

#!/bin/bash
yum update -y
yum -y install httpd
service httpd start

usermod -a -G apache ec2-user
chown -R ec2-user:apache /var/www
chmod 2775 /var/www
find /var/www -type d -exec chmod 2775 {} \;
find /var/www -type f -exec chmod 0664 {} \;

aws s3 cp s3://{{bucket}}/web/index.html /var/www/html/index.html<jupyter_output><empty_output><jupyter_text>Load userdata.shWe will read the UserData into a local variable and base64 encode the contents of the file to be used on the EC2 instance launch configuraton.<jupyter_code>fh=open("userdata.sh")
userdata=fh.read()
fh.close()

userdataencode = base64.b64encode(userdata.encode()).decode("ascii")<jupyter_output><empty_output><jupyter_text>[Create Launch Configuration](https://docs.aws.amazon.com/autoscaling/ec2/userguide/LaunchConfiguration.html)A launch configuration is an instance configuration template that an Auto Scaling group uses to launch EC2 instances. When you create a launch configuration, you specify information for the instances. Include the ID of the Amazon Machine Image (AMI), the instance type, a key pair, one or more security groups, and a block device mapping. If you've launched an EC2 instance before, you specified the same information in order to launch the instance.[asg.create_launch_configuration](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/autoscaling.htmlAutoScaling.Client.create_launch_configuration) boto3 documentation<jupyter_code>launch_config = asg.create_launch_configuration(
    LaunchConfigurationName=launch_config_name,
    ImageId=ami,
    SecurityGroups=[alb_sec_group_id], ## change this
    InstanceType='m4.large',
    InstanceMonitoring={'Enabled': True},
    UserData=userdataencode,
)<jupyter_output><empty_output><jupyter_text>[Create Auto Scaling Group](https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html)An Auto Scaling group contains a collection of Amazon EC2 instances that share similar characteristics and are treated as a logical grouping for the purposes of instance scaling and management. For example, if a single application operates across multiple instances, you might want to increase the number of instances in that group to improve the performance of the application. Or, you can decrease the number of instances to reduce costs when demand is low. Use the Auto Scaling group to scale the number of instances automatically based on criteria that you specify. You could also maintain a fixed number of instances even if an instance becomes unhealthy. This automatic scaling and maintaining the number of instances in an Auto Scaling group is the core functionality of the Amazon EC2 Auto Scaling service.[asg.create_auto_scaling_group](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/autoscaling.htmlAutoScaling.Client.create_auto_scaling_group) boto3 documentation<jupyter_code>auto_scaling = asg.create_auto_scaling_group(
    AutoScalingGroupName=auto_scaling_group_name,
    LaunchConfigurationName=launch_config_name, 
    MinSize=1, 
    MaxSize=4, 
    AvailabilityZones=[
        subnet.availability_zone,
        subnet2.availability_zone
    ], 
    VPCZoneIdentifier=subnet_id+','+subnet2_id,
    TargetGroupARNs=[target_group_arn],
    HealthCheckType='EC2',
    HealthCheckGracePeriod=120,
    DefaultCooldown=120
)<jupyter_output><empty_output><jupyter_text>[Create Scaling Policies](https://docs.aws.amazon.com/autoscaling/ec2/userguide/scaling_plan.html)Scaling is the ability to increase or decrease the compute capacity of your application. Scaling starts with an event, or scaling action, which instructs an Auto Scaling group to either launch or terminate Amazon EC2 instances.[asg.put_scaling_policy](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/autoscaling.htmlAutoScaling.Client.put_scaling_policy) boto3 documentation<jupyter_code>def create_policy(policy_name, adjustment_value):
    asg_policy = asg.put_scaling_policy(
        AutoScalingGroupName=auto_scaling_group_name,
        PolicyName=policy_name,
        AdjustmentType='ChangeInCapacity',
        ScalingAdjustment=adjustment_value,
        Cooldown=60
    )
    return asg_policy["PolicyARN"]

scale_up_arn = create_policy(scale_up_name, 1)
scale_down_arn = create_policy(scale_down_name,-1)<jupyter_output><empty_output><jupyter_text>[Create Alarms to trigger auto-scaling groups (ASG)](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/US_AlarmAtThresholdEC2.html)You can create a CloudWatch alarm that watches a single CloudWatch metric or the result of a math expression based on CloudWatch metrics. The alarm performs one or more actions based on the value of the metric or expression relative to a threshold over a number of time periods. The action can be an Amazon EC2 action, an Amazon EC2 Auto Scaling action, or a notification sent to an Amazon SNS topic.[cloudwatch.put_metric_alarm](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/cloudwatch.htmlCloudWatch.Client.put_metric_alarm) boto3 documentation<jupyter_code>def create_alarm(alarm_name, operator, threshold, policy_arn):
    alarm = cloudwatch.put_metric_alarm(
        AlarmName=alarm_name, 
        AlarmActions=[policy_arn],
        MetricName='CPUUtilization',
        Namespace='AWS/EC2',
        Statistic='Average',
        Dimensions=[{'Name':'AutoScalingGroupName','Value': auto_scaling_group_name}],
        Period=120, 
        EvaluationPeriods=1,
        Threshold=threshold,
        ComparisonOperator=operator,
        Unit='Percent'
    )

create_alarm('High Capacity Alarm','GreaterThanOrEqualToThreshold',65,scale_up_arn)
create_alarm('Low Capacity Alarm','LessThanOrEqualToThreshold',45,scale_down_arn)<jupyter_output><empty_output><jupyter_text>Validate web serverWait for the ASG to complete launching an EC2 instance. This will take a few minutes to launch.<jupyter_code>response = asg.describe_auto_scaling_groups(
    AutoScalingGroupNames=[
        auto_scaling_group_name
    ],
)

pprint.pprint(response['AutoScalingGroups'][0]['Instances'])
print ("ASG: https://{0}.console.aws.amazon.com/ec2/autoscaling/home?region={0}#AutoScalingGroups:id={1};view=details".format(region, auto_scaling_group_name))
print("ALB {0}: https://{1}.console.aws.amazon.com/ec2/v2/home?region={1}#LoadBalancers:sort=loadBalancerName".format(alb_name, region))
print("Web App: http://{0}".format(alb["LoadBalancers"][0]["DNSName"]))<jupyter_output><empty_output><jupyter_text>Finished!!!!!!From the links above you can now click on the Web App link to launch a new tab in the browser to show the index.html page we uploaded from S3. After that, you can click the ASG linkk above and we will change the `Desired Count` attribute to 2 so we can watch the ASG launch another instance based on the launch configuration we created above and register it with the ALB. This should give you a good idea of how you would launch and Apache Web Server with the boto3 calls. If you were to create this in a production environment you could leverage [CloudFormation](https://aws.amazon.com/cloudformation/) templates that will allow you to leverage YAML or JSON templates to launch the resources. If you would like to experiment more you can launch example [CloudFormation application templates](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/deploying.applications.html) to see how you could build your [Infrastructure as Code](https://en.wikipedia.org/wiki/Infrastructure_as_code).   Clean UpIn order to remove everything created in this workshop you can run the cells below and finally remove the VPC created for this workshop. Remove CloudWatch Alarms<jupyter_code>response = cloudwatch.delete_alarms(
    AlarmNames=[
        scale_up_name,
        scale_down_name
    ]
)<jupyter_output><empty_output><jupyter_text>Remove Auto Scaling Groups<jupyter_code>response = asg.delete_auto_scaling_group(
    AutoScalingGroupName=auto_scaling_group_name,
    ForceDelete=True
)<jupyter_output><empty_output><jupyter_text>Remove Launch Configuration<jupyter_code>response = asg.delete_launch_configuration(LaunchConfigurationName=launch_config_name)<jupyter_output><empty_output><jupyter_text>Remove Listener<jupyter_code>response = elb.delete_listener(ListenerArn=listener_arn)<jupyter_output><empty_output><jupyter_text>Remove Target Groups<jupyter_code>response = elb.delete_target_group(TargetGroupArn=target_group_arn)<jupyter_output><empty_output><jupyter_text>Remove Application Load Balancer<jupyter_code>response = elb.delete_load_balancer(LoadBalancerArn=alb_arn)<jupyter_output><empty_output><jupyter_text>Remove Security Group<jupyter_code>response = ec2_client.delete_security_group(GroupId=alb_sec_group_id)<jupyter_output><empty_output><jupyter_text>Remove Virtual Private Cloud<jupyter_code>workshop.vpc_cleanup(vpc_id)<jupyter_output><empty_output><jupyter_text>Remove S3 Bucket<jupyter_code>workshop.delete_bucket_completely(bucket)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08918740365558248, 'fraction_numerical': 0.011341114292006167, 'mean_word_length': 5.673401910360029, 'pattern_counts': {'":': 0, '<': 111, '<?xml version=': 0, '>': 111, 'https://': 31, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '27', 'dir': 'jupyter-structured-clean-dedup', 'id': '186040', 'license': 'MIT-0', 'n_tokens_mistral': 5519, 'n_tokens_neox': 5316, 'n_words': 1439, 'path': 'notebooks/intro_to_aws/intro_launch_ec2_and_s3.ipynb', 'repo_name': 'josiahbjorgaard/aws-research-workshops', 'size': '18164'}