<commit_before># How to Setup AWS CouldTrail with Splunk

This document describes how to setup AWS Cloudtail services to audit API calls and how to setup SpunkAppForAWS app to generate reports in Splunk.

## What is CloudTrail
[AWS CloudTrail] (http://aws.amazon.com/cloudtrail/) is an AWS service. When enabled, it captures AWS API calls made by or on behalf of an AWS account and delivers log files to an Amazon S3 bucket. 

## Why use CloudTrail
Traditionally system administrators monitor a system's integrity using intrusion detection tools such as Tripwire. System logs are usually sent to 
a central log server for auditing and security analysis.

For services running on AWS, another important operation and security related auditing is to monitor API calls that can change services and environments. 
Use cases enabled by CloudTrail service:

* Security Analysis
* Track Changes to AWS Resources
* Troubleshoot Operational Issues
* Compliance Aid

Here is an example of API call recorded by CloudTrail and how Spunk reports it (what, when, who, where, etc.)

	{ [-]
	   awsRegion: us-west-2
	   eventID: 7ad60379-0e2e-4b6c-8a6f-100f00fc5df1
	   eventName: ModifyReservedInstances
	   eventSource: ec2.amazonaws.com
	   eventTime: 2014-07-14T19:29:42Z
	   eventVersion: 1.01
	   requestID: 900f0797-0f2d-43dd-bcf5-41e1cfbc5c9f
	   requestParameters: { [-]
	     clientToken: fa<PASSWORD>
	     reservedInstancesSet: { [+]
	     }
	     targetConfigurationSet: { [+]
	     }
	   }
	   responseElements: { [-]
	     reservedInstancesModificationId: rimod-f0713af8-0984-42ff-8016-3dfea813b209
	   }
	   sourceIPAddress: 171.66.221.130
	   userAgent: console.ec2.amazonaws.com
	   userIdentity: { [-]
	     accessKeyId: <KEY>
	     accountId: XXX<KEY>
	     arn: arn:aws:sts::XXXXXXXXXXX:assumed-role/admin-saml/john
	     principalId: XXXXXXXXXXXXX:jonh
	     sessionContext: { [+]
	     }
	     type: AssumedRole
	   }
	}

## Visualized reporting tools
Many tools are available to generate visualized reports using the CloudTrail files stored in S3 bucket. Here are listed 
[AWS partners](http://aws.amazon.com/cloudtrail/partners/). This documentation describes how to use [SplunkAppforAWS](http://apps.splunk.com/app/1274/) to consume Cloudtrail data and generate reports. 

## The CloudTrail and Spunk integration

In this integration, we create CloudTrail service for each region and Simple Notification Service (SNS) topic for each CloudTrail service. The reports from all regions are aggregated to one S3 bucket. One Simple Queue Service (SQS) is subscribed to all the SNS topics. 

![](./images/splunk-aws-integration.png)

## Prerequisites for this setup

All the setups can be done through AWS console, but we use a script for CloudTrail setup to make sure we have naming schema consistency cross
all regions. All AWS services (Cloudtrail, SNS, SQS) created will have a prefix __'accountNumber-'__

* Install AWSCLI 

    [AWSCLI] (https://github.com/aws/aws-cli) command line tool is used to create Cloudtrail. To install (or upgrade) the package:

```
$ brew install awscli
```

   This will install _aws_ command under /usr/local/bin. There are three ways to setup AWS CLI AWS credentials. The examples here assumes you run the Cloudtrail creation code on an on-premise system and use a configuration file for key id and key secret. If you run it on EC2, you need to create an IAM role and åthe aws cli can use role-based token automatically.
    
   To configure AWSCLI:
    
```
$ aws --profile <profile> configure
```

   The above command will generate two files, for example:
   
```
$ cat /etc/.aws/awscli.conf
# For AWS CLI
[profile mylab-cloudtrail]
region = us-west-2
...

$cat /etc/.aws/credentials
[mylab-cloudtrail]
aws_access_key_id = <key id>
aws_secret_access_key = <secrect>
```

* Create an IAM user

    You should create an IAM user with the minimum access privileges needed. 

   Fexample, you can create an IAM user _cloudtrail-splunkapp_  which has permission to read SQS, delete message from SQS, and get data from S3 bucket. The following polices should work if you use canned AWS policy generator:

* Readonly access to S3 bucket
* Readonly access to CloudTrail
* Full access to the SQS (it deletes messages after read stuff from the message queue)

## Create AWS CloudTrail

For security auditing, you should monitor all regions that CloudTrail service is available. These are the [current supported regions](http://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-supported-regions.html):

* us-east-1
* us-west-1
* us-west-2
* eu-west-1
* sa-east-1
* ap-northeast-1
* ap-southeast-1	
* ap-southeast-2

To enable CloudTrail on these regions, download and run [cloudtrail-admnin.sh](./scripts/cloudtrail-admin.sh)
```
./cloudtrail-admin.sh -a create -p <aws profile> -r <region to receive global event>
```

The script calls AWSCLI cloudtrail command to:

* Enable Cloudtrail service in each region to send reports to a *s3://<account-number>-cloudtrail* S3 bucket
* Eeach regions trail name is *s3://`<accountNumber>`-cloudtrail-`<region>`*
* Create Simple Notification Service (SNS) topic for the Cloudtrail service in each region
* (Optional) Create *`<accountNumber>`-cloudtrail* SQS in one region to receive CloudTrail report notification. You only need this for Splunk.

The script has a dryrun (-n) option to generate the commands so you can see actual commands without running it. 

To tear down the resources created:
```
./cloudtrail-admin.sh -a delete -p  <aws profile> -r <region to receive global event>
```

We consolidate all CloudTrails reports into one S3 bucket that you give at the command line. The script takes care of creating necessary access policies. Global events - generated by services that don't have a regional endpoint, e.g. Identity and Access Management (IAM) - will be logged in one region that you define at the command line. 

## Subscribe Simple Queue Service (SQS) to CloudTrail SNS topics

The SQS named _accountNumber-cloudtrail_ is creased by create-cloudtrail script. It is easiest to use AWS's SQS service console to subscribe the queue to each SNS cloudtrail SNS topics because AWS service will generate proper policy for the queue to allow messages from the SNS topics.

SQS is needed in Splunk AWS app configuration. Splunk AWS app runs at defined interval to retrieve messages from AWS SQS service. The message body contains the S3 bucket location for Cloudtrail reports. Splunk then calls S3 API to get Cloudtrail reports from the S3 bucket.

## Setup SplunkAppForAWS

The application has two major report categories - Cloudtrail monitoring for API activities, and Billing and Usage that helps budgeting and cost analysis. 

### Billing and usage module

This module requires to turn on [Detailed Billing Reports](http://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/detailed-billing-reports.html). AWS will publish detailed billing reports in CSV format to the S3 bucket that you specify. If you have consolidated billing setup, ask the payer to create an IAM user that has read access to the S3 bucket that collects billing csv files. If you want to monitor the subaccounts's running status (e.g. the break downs of running on-demand instances, reserved instances, type of instances etc.), you need to create an IAM user in each sub-account that has access to check EC2 status in that subaccount. 

To make the Billing and Usage work, you need the following:

1. If you use consolidated billing, ask the payer to create an IAM user that have read only access to the S3 bucket's billing CSV file
2. If you need to pull dynamic EC2 information, create an IAM user for each account/subaccount with the following policies:
*  Amazon EC2 Reports Access
*  Amazon EC2 Read Only Access
3. opt/splunk/etc/apps/SplunkAppforAWS/lookupsinstancetype.csv
   This file should contain all instances types that you use in your account. The list from the SplinkAppforAWS may not be complete.
4.  aws.conf file 

  The file contains the setup for Billing and Usage. Here an example:

        # more /opt/splunk/etc/apps/SplunkAppforAWS/local/aws.conf

        # All this three stanzas are required, in order to run AWS App.

		[keys]
<commit_msg>Mention cloudtrail-admin.sh can be used independent of splunk<commit_after># How to Setup AWS CouldTrail with Splunk

This document describes how to setup AWS Cloudtail services to audit API calls and how to setup SpunkAppForAWS app to generate reports in Splunk.

## What is CloudTrail
[AWS CloudTrail] (http://aws.amazon.com/cloudtrail/) is an AWS service. When enabled, it captures AWS API calls made by or on behalf of an AWS account and delivers log files to an Amazon S3 bucket. 

## Why use CloudTrail
Traditionally system administrators monitor a system's integrity using intrusion detection tools such as Tripwire. System logs are usually sent to 
a central log server for auditing and security analysis.

For services running on AWS, another important operation and security related auditing is to monitor API calls that can change services and environments. 
Use cases enabled by CloudTrail service:

* Security Analysis
* Track Changes to AWS Resources
* Troubleshoot Operational Issues
* Compliance Aid

Here is an example of API call recorded by CloudTrail and how Spunk reports it (what, when, who, where, etc.)

	{ [-]
	   awsRegion: us-west-2
	   eventID: 7ad60379-0e2e-4b6c-8a6f-100f00fc5df1
	   eventName: ModifyReservedInstances
	   eventSource: ec2.amazonaws.com
	   eventTime: 2014-07-14T19:29:42Z
	   eventVersion: 1.01
	   requestID: 900f0797-0f2d-43dd-bcf5-41e1cfbc5c9f
	   requestParameters: { [-]
	     clientToken: <PASSWORD>
	     reservedInstancesSet: { [+]
	     }
	     targetConfigurationSet: { [+]
	     }
	   }
	   responseElements: { [-]
	     reservedInstancesModificationId: rimod-f0713af8-0984-42ff-8016-3dfea813b209
	   }
	   sourceIPAddress: 172.16.58.3
	   userAgent: console.ec2.amazonaws.com
	   userIdentity: { [-]
	     accessKeyId: <KEY>
	     accountId: XXX<KEY>
	     arn: arn:aws:sts::XXXXXXXXXXX:assumed-role/admin-saml/john
	     principalId: XXXXXXXXXXXXX:jonh
	     sessionContext: { [+]
	     }
	     type: AssumedRole
	   }
	}

## Visualized reporting tools
Many tools are available to generate visualized reports using the CloudTrail files stored in S3 bucket. Here are listed 
[AWS partners](http://aws.amazon.com/cloudtrail/partners/). This documentation describes how to use [SplunkAppforAWS](http://apps.splunk.com/app/1274/) to consume Cloudtrail data and generate reports. 

## The CloudTrail and Spunk integration

In this integration, we create CloudTrail service for each region and Simple Notification Service (SNS) topic for each CloudTrail service. The reports from all regions are aggregated to one S3 bucket. One Simple Queue Service (SQS) is subscribed to all the SNS topics. 

![](./images/splunk-aws-integration.png)

## Prerequisites for this setup

All the setups can be done through AWS console, but we use a script for CloudTrail setup to make sure we have naming schema consistency cross
all regions. All AWS services (Cloudtrail, SNS, SQS) created will have a prefix __'accountNumber-'__

### Install AWSCLI and Jq

  [AWSCLI] (https://github.com/aws/aws-cli) command line tool is used to create Cloudtrail. To install (or upgrade) the package:

```
$ brew install awscli jq
```

   This will install _aws_ command under /usr/local/bin. There are three ways to setup AWS CLI AWS credentials. The examples here assumes you run the Cloudtrail creation code on an on-premise system and use a configuration file for key id and key secret. If you run it on EC2, you need to create an IAM role and åthe aws cli can use role-based token automatically.
    
   To configure AWSCLI:
    
```
$ aws --profile <profile> configure
```

   The above command will generate two files, for example:
   
```
$ cat /etc/.aws/awscli.conf
# For AWS CLI
[profile mylab-cloudtrail]
region = us-west-2
...

$cat /etc/.aws/credentials
[mylab-cloudtrail]
aws_access_key_id = <key id>
aws_secret_access_key = <secrect>
```

### Create an IAM user

   You should create an IAM user with the minimum access privileges needed. 

   Fexample, you can create an IAM user _cloudtrail-splunkapp_  which has permission to read SQS, delete message from SQS, and get data from S3 bucket. The following polices should work if you use canned AWS policy generator:

* Readonly access to S3 bucket
* Readonly access to CloudTrail
* Full access to the SQS (it deletes messages after read stuff from the message queue)

## Create AWS CloudTrail

For security auditing, you should monitor all regions that CloudTrail service is available. These are the [current supported regions](http://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-supported-regions.html):

* us-east-1
* us-west-1
* us-west-2
* eu-west-1
* sa-east-1
* ap-northeast-1
* ap-southeast-1	
* ap-southeast-2

To enable CloudTrail on these regions, download and run [cloudtrail-admnin.sh](./scripts/cloudtrail-admin.sh)
```
./cloudtrail-admin.sh -a create -p <aws profile> -r <region to receive global event>
```

This script calls awscli cloudtrail command to create or destroy cloudtrail setup. For setup, it performs the following
functinon:

* Turn on Cloudtrail service in all supported region to send reports to  *s3://<account-number>-cloudtrail* S3 bucket
* Create trail with name *s3://`<accountNumber>`-cloudtrail-`<region>`*
* Create Simple Notification Service (SNS) topic for the Cloudtrail service in each region
* (Optional) Create *`<accountNumber>`-cloudtrail* SQS in one region to receive CloudTrail report notification. You only need this for Splunk.

The script has a dryrun (-n) option to generate the commands so you can see actual commands without running it. 

To tear down the resources created:
```
./cloudtrail-admin.sh -a delete -p  <aws profile> -r <region to receive global event>
```

We consolidate all CloudTrails reports into one S3 bucket default to *s3://<account-number>-cloudtrail*. Global events - generated by services that don't have a regional endpoint, e.g. Identity and Access Management (IAM) - will be logged in one region that you define at the command line. 

NOTE: You can stop reading if all you need is to turn on AWS Cloudtrail services in all regions. The cloudtrail now is enabled and you can go to console to view logs or through API calls.

## Subscribe Simple Queue Service (SQS) to CloudTrail SNS topics

The SQS named *`<accountNumber>`-cloudtrail* is created by cloudtrail-admin.sh script. It is easiest to use AWS's SQS service console to subscribe the queue to each SNS cloudtrail SNS topics because AWS service will generate proper policy for the queue to allow messages from the SNS topics.

SQS is needed in Splunk AWS app configuration. Splunk AWS app runs at defined interval to retrieve messages from AWS SQS service. The message body contains the S3 bucket location for Cloudtrail reports. Splunk then calls S3 API to get Cloudtrail reports from the S3 bucket.

## Setup SplunkAppForAWS

The application has two major report categories - Cloudtrail monitoring for API activities, and Billing and Usage that helps budgeting and cost analysis. 

### Billing and usage module

This module requires to turn on [Detailed Billing Reports](http://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/detailed-billing-reports.html). AWS will publish detailed billing reports in CSV format to the S3 bucket that you specify. If you have consolidated billing setup, ask the payer to create an IAM user that has read access to the S3 bucket that collects billing csv files. If you want to monitor the subaccounts's running status (e.g. the break downs of running on-demand instances, reserved instances, type of instances etc.), you need to create an IAM user in each sub-account that has access to check EC2 status in that subaccount. 

To make the Billing and Usage work, you need the following:

1. If you use consolidated billing, ask the payer to create an IAM user that have read only access to the S3 bucket's billing CSV file
2. If you need to pull dynamic EC2 information, create an IAM user for each account/subaccount with the following policies:
*  Amazon EC2 Reports Access
*  Amazon EC2 Read Only Access
3. opt/splunk/etc/apps/SplunkAppforAWS/lookupsinstancetype.csv
   This file should contain all instances types that you use in your account. The list from the SplinkAppforAWS may not be complete.
4.  aws.conf file 

  The file contains the setup for Billing and Usage. Here an example:

        # more /opt/splunk/etc/apps/SplunkAppforAWS/local/aws.conf

        # All this three stanzas are required, in order to run AWS App.

		[keys]

{'fraction_non_alphanumeric': 0.06594717071134697, 'fraction_numerical': 0.015025937630433486, 'mean_word_length': 4.021556886227545, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 4, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '271308', 'n_tokens_mistral': 5051, 'n_tokens_neox': 4693, 'n_words': 2184}