<commit_before># cms-audio-lambda

[![Build Status](https://snap-ci.com/PRX/cms-audio-lambda/branch/master/build_image)](https://snap-ci.com/PRX/cms-audio-lambda/branch/master)
[![codecov](https://codecov.io/gh/PRX/cms-audio-lambda/branch/master/graph/badge.svg)](https://codecov.io/gh/PRX/cms-audio-lambda)

## Description

Lambda worker to process/validate for [cms.prx.org](https://github.com/PRX/cms.prx.org) audio-file uploads.

Triggered via SNS notifications (see [Announce](https://github.com/PRX/announce)), the lambda will:

#### On `create` or `update`

1. Download the `upload_path` for the audio file object (from S3 or just HTTP)
2. Run `ffprobe` against the file to verify it's an mp3 and gather other metadata
3. Copy the original audio file to a destination S3 bucket/path (`public/audio_files/1234/filename.mp3`)
4. Make an additional copy for the "broadcast" version CMS wants to use (`public/audio_files/1234/filename_broadcast.mp3`)
5. Send success/failure/invalid messages back to CMS via SQS ([Shoryuken](https://github.com/phstc/shoryuken)), including some additional identify data about the audio length/type/etc.

#### On `delete`

1. TODO: since audio-files are only soft-deleted by CMS, not sure this would ever happen.

### Callbacks

SQS callbacks contain the following JSON data:

| Key  | Sub-key    | Description |
| ---- | ---------- | ----------- |
| id   |            | ID of the SNS message that triggered this job
| path |            | Destination path in S3 the file was copied to
| name |            | File name
| mime |            | Mime-type of the file
| size |            | Size of the file in bytes
| audio |           | Detected audio stream metadata (or undefined)
|       | duration  | Duration in ms
|       | format    | Detected codec string ('mp2' / 'mp3')
|       | bitrate   | Audio bitrate (96000 / 128000)
|       | frequency | Audio frequency hz (44100 / 48000)
|       | channels  | Number of channels (1 / 2 / 4)
|       | layout    | Channel layout string ('mono' / 'stereo')
| video |           | Detected video stream metadata (or undefined)
|       | duration  | Duration in ms
|       | format    | Detected codec string ('h264' / 'theora' / 'flv1')
|       | bitrate   | Video bitrate
|       | width     | Video width in pixels
|       | height    | Video height in pixels
|       | aspect    | Aspect ratio string ('1:1' / '0:1')
|       | framerate | Frame rate string ('30:1')
| downloaded |      | Boolean if the audio download succeeded
| valid      |      | Boolean if ffprobe recognized any audio/video streams
| processed  |      | Boolean if uploading to S3 destination succeeded
| error      |      | String if any error occurred in the above 3 states

## Developing

Make sure you have AWS credentials locally (usually in `~/.aws/credentials`) that are able to access
the `TEST_BUCKET` and `DESTINATION_BUCKET` defined in `config/test.env`.  Then, just...<commit_msg>That's not actually what "id" is<commit_after># cms-audio-lambda

[![Build Status](https://snap-ci.com/PRX/cms-audio-lambda/branch/master/build_image)](https://snap-ci.com/PRX/cms-audio-lambda/branch/master)
[![codecov](https://codecov.io/gh/PRX/cms-audio-lambda/branch/master/graph/badge.svg)](https://codecov.io/gh/PRX/cms-audio-lambda)

## Description

Lambda worker to process/validate for [cms.prx.org](https://github.com/PRX/cms.prx.org) audio-file uploads.

Triggered via SNS notifications (see [Announce](https://github.com/PRX/announce)), the lambda will:

#### On `create` or `update`

1. Download the `upload_path` for the audio file object (from S3 or just HTTP)
2. Run `ffprobe` against the file to verify it's an mp3 and gather other metadata
3. Copy the original audio file to a destination S3 bucket/path (`public/audio_files/1234/filename.mp3`)
4. Make an additional copy for the "broadcast" version CMS wants to use (`public/audio_files/1234/filename_broadcast.mp3`)
5. Send success/failure/invalid messages back to CMS via SQS ([Shoryuken](https://github.com/phstc/shoryuken)), including some additional identify data about the audio length/type/etc.

#### On `delete`

1. TODO: since audio-files are only soft-deleted by CMS, not sure this would ever happen.

### Callbacks

SQS callbacks contain the following JSON data:

| Key  | Sub-key    | Description |
| ---- | ---------- | ----------- |
| id   |            | ID of the AudioFile that triggered this job
| path |            | Destination path in S3 the file was copied to
| name |            | File name
| mime |            | Mime-type of the file
| size |            | Size of the file in bytes
| audio |           | Detected audio stream metadata (or undefined)
|       | duration  | Duration in ms
|       | format    | Detected codec string ('mp2' / 'mp3')
|       | bitrate   | Audio bitrate (96000 / 128000)
|       | frequency | Audio frequency hz (44100 / 48000)
|       | channels  | Number of channels (1 / 2 / 4)
|       | layout    | Channel layout string ('mono' / 'stereo')
| video |           | Detected video stream metadata (or undefined)
|       | duration  | Duration in ms
|       | format    | Detected codec string ('h264' / 'theora' / 'flv1')
|       | bitrate   | Video bitrate
|       | width     | Video width in pixels
|       | height    | Video height in pixels
|       | aspect    | Aspect ratio string ('1:1' / '0:1')
|       | framerate | Frame rate string ('30:1')
| downloaded |      | Boolean if the audio download succeeded
| valid      |      | Boolean if ffprobe recognized any audio/video streams
| processed  |      | Boolean if uploading to S3 destination succeeded
| error      |      | String if any error occurred in the above 3 states

## Developing

Make sure you have AWS credentials locally (usually in `~/.aws/credentials`) that are able to access
the `TEST_BUCKET` and `DESTINATION_BUCKET` defined in `config/test.env`.  Then, just...
{'fraction_non_alphanumeric': 0.12209991532599491, 'fraction_numerical': 0.01998306519898391, 'mean_word_length': 2.837556855100715, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 14, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 16, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1104694', 'n_tokens_mistral': 1849, 'n_tokens_neox': 1745, 'n_words': 615}