<commit_before>This is based on <NAME>'s [lambda-gdalinfo](https://github.com/hectcastro/lambda-gdalinfo).
If you are new to AWS Lambda a good place to start is [here](
http://docs.aws.amazon.com/lambda/latest/dg/getting-started.html)

# lambda-gdal_translate

This project allows you to run [gdal_translate](http://www.gdal.org/gdal_translate.html) utility using the [AWS Lambda](https://aws.amazon.com/lambda/) execution environment.
Generally it allows you run something that you would traditionally run as part of a batch file like this:

```bash
gdal_translate -b 1 -b 2 -b 3 -of GTiff -outsize 50% 50% -co tiled=yes -co BLOCKXSIZE=512 -co BLOCKYSIZE=512' -co PHOTOMETRIC=YCBCR -co COMPRESS=JPEG -co JPEG_QUALITY='85' input.tif output.tif
```
But from AWS Lambda. The difference you can do so in a highly parallel way, without much more than configuring the Lambda function's memory and timeout settings. What makes this possible at scale is that you are working with data in [Amazon S3](https://aws.amazon.com/s3), rather than a traditional file system. This example uses the USDA's NAIP data set. You can read more about the NAIP data, which is available as part of the AWS Earth on AWS collection, [here](https://aws.amazon.com/public-datasets/naip/).

## Statically Linked `gdal_translate`

You should be able to use the gdal_translate binary under /bin. However if you want a more recent version you need build a statically linked one on an Amazon Linux instance for it run on AWS Lambda.

First, spin up an Amazon Linux instance on Amazon EC2. In the EC2 console it will look like "Amazon Linux AMI 2017.03.0 (HVM), SSD Volume".  SSH to the instance and run the following commands:

```bash
$ sudo yum update -y
$ sudo yum groupinstall -y "Development Tools"
$ sudo yum install -y libcurl-devel
$ wget http://download.osgeo.org/gdal/2.2.0/gdal-2.2.0.tar.gz
$ tar xzf gdal-2.2.0.tar.gz
$ cd gdal-2.2.0
$ ./configure --without-ld-shared --disable-shared --enable-static --with-curl --prefix /tmp/gdal
$ make
$ make install
$ rm -rf /tmp/gdal
```

## Usage

Runnig Lambda-gdal_translate looks like this:

```bash
aws lambda invoke --function-name gdal_translate --region us-east-1 --invocation-type Event --payload '{"sourceBucket": "aws-naip", "sourceObjectKey": "wi/2015/1m/rgbir/47090/m_4709061_sw_15_1_20150914.tif", "targetBucket": "youBucketNameHere", "targetPrefix": "temp-000"}' log
```

As you can see in this example, there are no gdal_translate arguments in the function invocation. That is because those arguments remain static over the course of a batch operation so are provided to the script as environment variables. In addition, because you often want to modify the output objects key name before you store it back to S3, you can define a find/replace string pair as environment variables to modify the output key name.

In order to process a large group of files in S3 it make sense to work off of a file list rather than repetitively listing objeccts in S3. The NAIP bucket, aws-naip, includes a manifest file at root, but lets assume you want to build your own. You can do this by using the AWS S3 CLI and the awk command. Note, this example uses "--request-payer requester" because the NAIP data is provided in a bucket that is marked [requester-pays](http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html). You would not need it for your own data in S3.

```bash
aws s3 ls --recursive --request-payer requester s3://aws-naip/ca/2014/1m/rgbir | awk -F" " '{print $4}' > mylist
```
Your resulting list should look something like this:

```bash
cat mylist
ca/2014/1m/rgbir/42122/m_4212264_se_10_1_20140718.tif
ca/2014/1m/rgbir/42122/m_4212264_sw_10_1_20140718.tif
ca/2014/1m/rgbir/42123/m_4212360_se_10_1_20140622.tif
ca/2014/1m/rgbir/42123/m_4212360_sw_10_1_20140609.tif
...
```
You can process all of your source imagery using something like this:

```bash
cat mylist | awk -F"/" '{print "lambda invoke --function-name gdal_translate --region us-east-1 --invocation-type Event --payload \x27{\"sourceBucket\": \"aws-naip\",\"sourceObjectKey\": \""$0"\", \"targetBucket\": \"yourBucketNameHere\", \"targetPrefix\": \"yourPrefixHere\"}\x27 log" }'
```

that should result in output that looks like this:

```bash
lambda invoke --function-name gdal_translate --region us-east-1 --invocation-type Event --payload '{"sourceBucket": "aws-naip","sourceObjectKey": "ca/2014/1m/rgbir/42123/m_4212362_sw_10_1_20140622.tif", "targetBucket": "yourBucketNameHere", "targetPrefix": "yourPrefixHere"}' log
lambda invoke --function-name gdal_translate --region us-east-1 --invocation-type Event --payload '{"sourceBucket": "aws-naip","sourceObjectKey": "ca/2014/1m/rgbir/42123/m_4212363_se_10_1_20140622.tif", "targetBucket": "yourBucketNameHere", "targetPrefix": "yourPrefixHere"}' log
...
```

To test what you have, try running one of those lines by prepending the aws command like this:

```bash
aws lambda invoke --function-name gdal_translate --region us-east-1 --invocation-type Event --payload '{"sourceBucket": "aws-naip","sourceObjectKey": "ca/2014/1m/rgbir/42123/m_4212362_sw_10_1_20140622.tif", "targetBucket": "youBucketNameHere", "targetPrefix": "yourPrefixHere"}' log
```

Because you invoked it using the Event type you should see a HTTP 202 get returned.

Depending on the size of the raster file it will take a few seconds to process, but confirm that you have the expected result in your S3 bucket. Once satisfied with your results, you can speed things up (a lot) by piping to the xargs command like below.

```bash
cat mylist | awk -F"/" '{print "lambda invoke --function-name gdal_translate --region us-east-1 --invocation-type Event --payload \x27{\"sourceBucket\": \"aws-naip\",\"sourceObjectKey\": \""$0"\", \"targetBucket\": \"youBucketNameHere\", \"targetPrefix\": \"yourPrefixHere\"}\x27 log" }' | xargs -n 11 -P 64 aws
```

## Updating your own Amazon Lambda function<commit_msg>update xargs part<commit_after>This is based on Hector Castro's [lambda-gdalinfo](https://github.com/hectcastro/lambda-gdalinfo).
If you are new to AWS Lambda a good place to start is [here](
http://docs.aws.amazon.com/lambda/latest/dg/getting-started.html)

# lambda-gdal_translate

This project allows you to run [gdal_translate](http://www.gdal.org/gdal_translate.html) utility using the [AWS Lambda](https://aws.amazon.com/lambda/) execution environment.
Generally it allows you run a batch operation, one line of which might look like this,

```bash
gdal_translate -b 1 -b 2 -b 3 -of GTiff -outsize 50% 50% -co tiled=yes -co BLOCKXSIZE=512 -co BLOCKYSIZE=512' -co PHOTOMETRIC=YCBCR -co COMPRESS=JPEG -co JPEG_QUALITY='85' input.tif output.tif
```
but from AWS Lambda in a highly parallel, serverless way. What makes this possible at scale is that you are working with data in [Amazon S3](https://aws.amazon.com/s3), rather than data in a traditional file system. This example uses the USDA's NAIP data set. You can read more about the NAIP data, which is available as part of the AWS Earth on AWS collection, [here](https://aws.amazon.com/public-datasets/naip/).

## Statically Linked `gdal_translate`

You can use the gdal_translate binary under /bin. However if you want a more recent version you need to build a statically linked one on an Amazon Linux instance for it run on AWS Lambda.

First, spin up an Amazon Linux instance on Amazon EC2. In the EC2 console it will look like "Amazon Linux AMI 2017.03.0 (HVM), SSD Volume".  SSH to the instance and run the following commands:

```bash
$ sudo yum update -y
$ sudo yum groupinstall -y "Development Tools"
$ sudo yum install -y libcurl-devel
$ wget http://download.osgeo.org/gdal/2.2.0/gdal-2.2.0.tar.gz
$ tar xzf gdal-2.2.0.tar.gz
$ cd gdal-2.2.0
$ ./configure --without-ld-shared --disable-shared --enable-static --with-curl --prefix /tmp/gdal
$ make
$ make install
$ rm -rf /tmp/gdal
```

## Usage

Runnig or invoking Lambda-gdal_translate looks like this:

```bash
aws lambda invoke --function-name gdal_translate --region us-east-1 --invocation-type Event --payload '{"sourceBucket": "aws-naip", "sourceObjectKey": "wi/2015/1m/rgbir/47090/m_4709061_sw_15_1_20150914.tif", "targetBucket": "yourBucketNameHere", "targetPrefix": "yourPrefixHere"}' log
```

As you can see in this example, you are providing the Lambda function information about where to get data and how to write the data, there are no gdal_translate arguments in the function invocation. That is because those arguments remain static over the course of a batch operation so are provided to the script as environment variables. In addition, because you often want to modify the output objects key name before you store it back to S3, you can define a find/replace string pair as environment variables to modify the output key name.

In order to process a large group of files in S3 it makes sense to work off of a file list rather than repetitively listing objeccts in S3. The NAIP bucket, aws-naip, includes a manifest file at root, but lets assume you want to build your own list. You can do this by using the AWS S3 CLI and the awk command. Note, this example uses "--request-payer requester" because the NAIP data is provided in a bucket that is marked that way. You can read more about requester-pays [here](http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html). 

```bash
aws s3 ls --recursive --request-payer requester s3://aws-naip/ca/2014/1m/rgbir | awk -F" " '{print $4}' > mylist
```
Your resulting list should look something like this:

```bash
cat mylist
ca/2014/1m/rgbir/42122/m_4212264_se_10_1_20140718.tif
ca/2014/1m/rgbir/42122/m_4212264_sw_10_1_20140718.tif
ca/2014/1m/rgbir/42123/m_4212360_se_10_1_20140622.tif
ca/2014/1m/rgbir/42123/m_4212360_sw_10_1_20140609.tif
...
```
You can process all of your source imagery using something like this:

```bash
cat mylist | awk -F"/" '{print "lambda invoke --function-name gdal_translate --region us-east-1 --invocation-type Event --payload \x27{\"sourceBucket\": \"aws-naip\",\"sourceObjectKey\": \""$0"\", \"targetBucket\": \"yourBucketNameHere\", \"targetPrefix\": \"yourPrefixHere\"}\x27 log" }'
```

that should result in output that looks like this:

```bash
lambda invoke --function-name gdal_translate --region us-east-1 --invocation-type Event --payload '{"sourceBucket": "aws-naip","sourceObjectKey": "ca/2014/1m/rgbir/42123/m_4212362_sw_10_1_20140622.tif", "targetBucket": "yourBucketNameHere", "targetPrefix": "yourPrefixHere"}' log
lambda invoke --function-name gdal_translate --region us-east-1 --invocation-type Event --payload '{"sourceBucket": "aws-naip","sourceObjectKey": "ca/2014/1m/rgbir/42123/m_4212363_se_10_1_20140622.tif", "targetBucket": "yourBucketNameHere", "targetPrefix": "yourPrefixHere"}' log
...
```

To test what you have, try running one of those lines by prepending the aws command like this:

```bash
aws lambda invoke --function-name gdal_translate --region us-east-1 --invocation-type Event --payload '{"sourceBucket": "aws-naip","sourceObjectKey": "ca/2014/1m/rgbir/42123/m_4212362_sw_10_1_20140622.tif", "targetBucket": "youBucketNameHere", "targetPrefix": "yourPrefixHere"}' log
```

Because you invoked it using the Event type you should see a HTTP 202 get returned.

Depending on the size of the raster file it will take a few seconds to process, but confirm that you have the expected result in your target S3 bucket. Once satisfied with your results, you can speed things up by piping to  xargs and running in parallel mode using -P nn.

```bash
cat mylist | awk -F"/" '{print "lambda invoke --function-name gdal_translate --region us-east-1 --invocation-type Event --payload \x27{\"sourceBucket\": \"aws-naip\",\"sourceObjectKey\": \""$0"\", \"targetBucket\": \"youBucketNameHere\", \"targetPrefix\": \"yourPrefixHere\"}\x27 log" }' | xargs -n 11 -P 64 aws
```

## Updating your own Amazon Lambda function
{'fraction_non_alphanumeric': 0.11544278272559234, 'fraction_numerical': 0.05057973449840363, 'mean_word_length': 4.415377616014559, 'pattern_counts': {'":': 48, '<': 4, '<?xml version=': 0, '>': 6, 'https://': 8, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5335040', 'n_tokens_mistral': 4136, 'n_tokens_neox': 3683, 'n_words': 1413}