<commit_before># Running Buildkite Agent on AWS

The Buildkite Agent can be run on AWS using our Elastic CI Stack for AWS
CloudFormation template, or by installing the agent on your self-managed
instances.

{:toc}

## Using our Elastic CI Stack for AWS CloudFormation template

The [Elastic CI Stack for AWS](/docs/agent/v3/elastic_ci_aws) is a
CloudFormation template for an autoscaling Buildkite Agent cluster. The
agent instances include Docker, S3 and CloudWatch integration.

You can use an Elastic CI Stack deployment to test Linux or Windows projects,
parallelize large test suites, run Docker containers or docker-compose
integration tests, or perform any AWS ops related tasks.

You can launch an instance of the Elastic CI Stack for AWS from your
organization’s [Agents page](http://buildkite.com/organizations/-/agents) or
the [GitHub repository](elastic-ci-github).

   [elastic-ci-github]: https://github.com/buildkite/elastic-ci-stack-for-aws
   [Red Hat/CentOS installer]: /docs/agent/v3/redhat

## Installing the agent on your own AWS instances

To run the agent on your own AWS instances, use the installer that matches your
instance operating system:

* For Amazon Linux 2 use the [Red Hat/CentOS installer](/docs/agent/v3/redhat). 
* For macOS, use [installing the agent on your own AWS mac1.metal instances](#installing-the-agent-on-your-own-aws-mac1-dot-metal-instances)

## Using the experimental Elastic Mac for AWS CloudFormation template

[Elastic Mac for AWS](https://github.com/buildkite/elastic-mac-for-aws) is an
experimental CloudFormation template for an autoscaling macOS Buildkite agent
cluster.

You can use an Elastic Mac deployment to build and test macOS, iOS, iPadOS,
tvOS, and watchOS projects.

Read the [Auto Scaling mac1.metal instances](/docs/agent/v3/aws/autoscaling_mac_metal) documentation for instructions on preparing and deploying this template.

## Installing the Agent on your own AWS mac1.metal instances

Setting up a macOS AMI that starts a Buildkite Agent on launch is a multi
step process. You can start with one of the macOS AMIs from AWS, or with an AMI
you've already installed Xcode or other software on.

To use Xcode and the iOS Simulator, you must configure auto-login of a GUI
session, and launch the Buildkite Agent in an `aqua` session as a Launchd Agent:

1. Reserve a [AWS mac1.metal](https://aws.amazon.com/ec2/instance-types/mac/)
Dedicated Host.
1. Boot a macOS instance using your desired AMI on the Dedicated Host.
1. Configure instance VPC subnets, security groups, and key pairs so that you
can access the instance.
1. Using an SSH or SSM session, set a password for the ec2-user and enable
screen sharing.
1. Using a VNC session, enable auto-login for the ec2-user in System Preferences
and disable the auto screen lock and screen saver.
1. Follow the [macOS Installation Guide](/docs/agent/v3/macos#installation)
instructions to install the Buildkite agent using Homebrew and configure
starting on login.
1. Verify that the Buildkite agent has connected to buildkite.com with your
desired agent tags.
1. Create an AMI from your instance.

Your saved AMI can now be used to boot as many macOS instances as you require.

To make this process repeatable, save your instance configuration in a
[Launch Template](https://docs.aws.amazon.com/autoscaling/ec2/userguide/LaunchTemplates.html). 
To automate instance replacement, use an [Auto Scaling group](https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html)
to boot Instances and a [Host Resource Group](https://docs.aws.amazon.com/license-manager/latest/userguide/host-resource-groups.html)
to reserve Dedicated Instances.

While the use of an Auto-Scaling Group and Host Resource Group to automatically
maintain capacity in the face of hardware failures is recommended, load based
dynamic auto-scaling of macOS instances is not recommended. The instances are
currently slow to boot and slow to terminate. Use of load based auto-scaling is
likely to result in an over-provision of agents which carries a high minimum
charge per dedicated host.

There is an excellent blog post on [running iOS agents in the cloud](https://www.starkandwayne.com/blog/buildkite-2/) that goes into more detail on preparing macOS AMIs using [Packer](https://www.packer.io/).

### Known issues

* You might need to give the agent [full disk access](https://github.com/buildkite/agent/issues/1400)

## Preventing builds from accessing Amazon EC2 metadata  

If you provision infrastructure like databases, Redis, Amazon SQS, etc using AWS permission sandboxes, you might want to restrict access to those roles in your builds.

If you run your builds on an AWS EC2 permission sandbox and then allow Buildkite agents to generate and inject some sandboxed AWS credentials into the build secrets, such builds will have access to the EC2 metadata API. They will also be able to access the same permissions as your EC2. 

To avoid this, you need to prevent the builds from accessing your EC2 metadata or provide sandboxed AWS credentials for each build and restrict their permissions. There are two main ways to do it:

* Compartmentalizing your Buildkite agents
* Downgrading an instance profile role

If you run all the build steps in Docker containers, take a look at [compartmentalizing your agents](#preventing-builds-from-accessing-amazon-ec2-metadata-restricting-permissions-via-compartmentalization-of-agents). If you are using Kubernetes for your Buildkite CI, use the [same approach](#preventing-builds-from-accessing-amazon-ec2-metadata-restricting-permissions-via-compartmentalization-of-agents) and also check out [this article](https://github.com/blakestoddard/scaledkite) for more information and inspiration.  

### Restricting permissions via compartmentalization of agents<commit_msg>s/auto-scaling group/auto scaling group<commit_after># Running Buildkite Agent on AWS

The Buildkite Agent can be run on AWS using our Elastic CI Stack for AWS
CloudFormation template, or by installing the agent on your self-managed
instances.

{:toc}

## Using our Elastic CI Stack for AWS CloudFormation template

The [Elastic CI Stack for AWS](/docs/agent/v3/elastic_ci_aws) is a
CloudFormation template for an autoscaling Buildkite Agent cluster. The
agent instances include Docker, S3 and CloudWatch integration.

You can use an Elastic CI Stack deployment to test Linux or Windows projects,
parallelize large test suites, run Docker containers or docker-compose
integration tests, or perform any AWS ops related tasks.

You can launch an instance of the Elastic CI Stack for AWS from your
organization’s [Agents page](http://buildkite.com/organizations/-/agents) or
the [GitHub repository](elastic-ci-github).

   [elastic-ci-github]: https://github.com/buildkite/elastic-ci-stack-for-aws
   [Red Hat/CentOS installer]: /docs/agent/v3/redhat

## Installing the agent on your own AWS instances

To run the agent on your own AWS instances, use the installer that matches your
instance operating system:

* For Amazon Linux 2 use the [Red Hat/CentOS installer](/docs/agent/v3/redhat). 
* For macOS, use [installing the agent on your own AWS mac1.metal instances](#installing-the-agent-on-your-own-aws-mac1-dot-metal-instances)

## Using the experimental Elastic Mac for AWS CloudFormation template

[Elastic Mac for AWS](https://github.com/buildkite/elastic-mac-for-aws) is an
experimental CloudFormation template for an autoscaling macOS Buildkite agent
cluster.

You can use an Elastic Mac deployment to build and test macOS, iOS, iPadOS,
tvOS, and watchOS projects.

Read the [Auto Scaling mac1.metal instances](/docs/agent/v3/aws/autoscaling_mac_metal) documentation for instructions on preparing and deploying this template.

## Installing the Agent on your own AWS mac1.metal instances

Setting up a macOS AMI that starts a Buildkite Agent on launch is a multi
step process. You can start with one of the macOS AMIs from AWS, or with an AMI
you've already installed Xcode or other software on.

To use Xcode and the iOS Simulator, you must configure auto-login of a GUI
session, and launch the Buildkite Agent in an `aqua` session as a Launchd Agent:

1. Reserve a [AWS mac1.metal](https://aws.amazon.com/ec2/instance-types/mac/)
Dedicated Host.
1. Boot a macOS instance using your desired AMI on the Dedicated Host.
1. Configure instance VPC subnets, security groups, and key pairs so that you
can access the instance.
1. Using an SSH or SSM session, set a password for the ec2-user and enable
screen sharing.
1. Using a VNC session, enable auto-login for the ec2-user in System Preferences
and disable the auto screen lock and screen saver.
1. Follow the [macOS Installation Guide](/docs/agent/v3/macos#installation)
instructions to install the Buildkite agent using Homebrew and configure
starting on login.
1. Verify that the Buildkite agent has connected to buildkite.com with your
desired agent tags.
1. Create an AMI from your instance.

Your saved AMI can now be used to boot as many macOS instances as you require.

To make this process repeatable, save your instance configuration in a
[Launch Template](https://docs.aws.amazon.com/autoscaling/ec2/userguide/LaunchTemplates.html). 
To automate instance replacement, use an [Auto Scaling group](https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html)
to boot Instances and a [Host Resource Group](https://docs.aws.amazon.com/license-manager/latest/userguide/host-resource-groups.html)
to reserve Dedicated Instances.

While the use of an Auto Scaling group and Host Resource Group to automatically
maintain capacity in the face of hardware failures is recommended, load based
dynamic auto-scaling of macOS instances is not recommended. The instances are
currently slow to boot and slow to terminate. Use of load based auto-scaling is
likely to result in an over-provision of agents which carries a high minimum
charge per dedicated host.

There is an excellent blog post on [running iOS agents in the cloud](https://www.starkandwayne.com/blog/buildkite-2/) that goes into more detail on preparing macOS AMIs using [Packer](https://www.packer.io/).

### Known issues

* You might need to give the agent [full disk access](https://github.com/buildkite/agent/issues/1400)

## Preventing builds from accessing Amazon EC2 metadata  

If you provision infrastructure like databases, Redis, Amazon SQS, etc using AWS permission sandboxes, you might want to restrict access to those roles in your builds.

If you run your builds on an AWS EC2 permission sandbox and then allow Buildkite agents to generate and inject some sandboxed AWS credentials into the build secrets, such builds will have access to the EC2 metadata API. They will also be able to access the same permissions as your EC2. 

To avoid this, you need to prevent the builds from accessing your EC2 metadata or provide sandboxed AWS credentials for each build and restrict their permissions. There are two main ways to do it:

* Compartmentalizing your Buildkite agents
* Downgrading an instance profile role

If you run all the build steps in Docker containers, take a look at [compartmentalizing your agents](#preventing-builds-from-accessing-amazon-ec2-metadata-restricting-permissions-via-compartmentalization-of-agents). If you are using Kubernetes for your Buildkite CI, use the [same approach](#preventing-builds-from-accessing-amazon-ec2-metadata-restricting-permissions-via-compartmentalization-of-agents) and also check out [this article](https://github.com/blakestoddard/scaledkite) for more information and inspiration.  

### Restricting permissions via compartmentalization of agents
{'fraction_non_alphanumeric': 0.06259646715829188, 'fraction_numerical': 0.006345395300977534, 'mean_word_length': 4.731203931203932, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 20, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '782079', 'n_tokens_mistral': 3272, 'n_tokens_neox': 3122, 'n_words': 1507}