<commit_before>Lambda
======

Overview
--------

* StreamAlert utilizes AWS Lambda for real-time data processing and alerting
* AWS Lambda lets us run code without needing to provision or manage servers
* A Lambda function is created for each `cluster <clusters.html>`_ you define

Each StreamAlert cluster creates two Lambda functions:

* ``rule_processor``: Analyze incoming logs against rules in real time
* ``alert_processor``: Deliver alerts to designated outputs

Concepts
--------
* AWS Lambda `details`_
* AWS Lambda `FAQ`_
* AWS Lambda `pricing`_

.. _details: https://docs.aws.amazon.com/lambda/latest/dg/welcome.html
.. _faq: https://aws.amazon.com/lambda/faqs/
.. _pricing: https://aws.amazon.com/lambda/pricing/


Per Cluster Lambda Settings
---------------------------

``stream_alert`` is the main configuration module which manages:

1) Lambda Functions for rule and alert processing
2) IAM roles and policies
3) SNS topic for alerts

**Template:**

.. code-block:: json
  :caption: `conf/clusters/cluster-name.json`

  {
    "stream_alert": {
      "alert_processor": {
        "timeout": 25,
        "memory": 128,
        "current_version": "$LATEST",
        "outputs": {
          "aws-s3": [],
          "aws-lambda": []
        }
      },
      "rule_processor": {
        "timeout": 10,
        "memory": 256,
        "current_version": "$LATEST"
      }
    }
  }

**Options:**

===================  ========  ===========
Key                  Required  Description
-------------------  --------  -----------
``timeout``          ``true``  The time (in seconds) the Lambda function is allowed to process an incoming record. The timeout can be set to any value between 1 and 300 seconds.
``memory``           ``true``  The amount of memory allocated for the Lambda function execution.
``current_version``  ``true``  The most current published version of the Lambda function.
``outputs``          ``true``  A collection of S3 bucket IDs or AWS Lambda function names to configure as valid outputs.  By default, ``aws-s3`` should contain the bucket created by the ``stream_alert`` module: ``prefix.cluster.streamalerts``.  Optionally, if the alert processor needs to invoke other Lambda functions from within your AWS account, specify a list of function names.
===================  ========  ===========

vpc_config
~~~~~~~~~~

To enable StreamAlert's Alert Processor to access resources inside a VPC, you must provide additional VPC-specific configuration information.

`More information<http://docs.aws.amazon.com/lambda/latest/dg/vpc.html>`_

**Template:**

.. code-block:: json
  :caption: `conf/clusters/cluster-name.json`

  {
    "alert_processor": {
      "vpc_config": {
        "subnet_ids": [],
        "security_group_ids": []
      }
    }
  }<commit_msg>[docs] fixing link in docs<commit_after>Lambda
======

Overview
--------

* StreamAlert utilizes AWS Lambda for real-time data processing and alerting
* AWS Lambda lets us run code without needing to provision or manage servers
* A Lambda function is created for each `cluster <clusters.html>`_ you define

Each StreamAlert cluster creates two Lambda functions:

* ``rule_processor``: Analyze incoming logs against rules in real time
* ``alert_processor``: Deliver alerts to designated outputs

Concepts
--------
* AWS Lambda `details`_
* AWS Lambda `FAQ`_
* AWS Lambda `pricing`_

.. _details: https://docs.aws.amazon.com/lambda/latest/dg/welcome.html
.. _faq: https://aws.amazon.com/lambda/faqs/
.. _pricing: https://aws.amazon.com/lambda/pricing/


Per Cluster Lambda Settings
---------------------------

``stream_alert`` is the main configuration module which manages:

1) Lambda Functions for rule and alert processing
2) IAM roles and policies
3) SNS topic for alerts

**Template:**

.. code-block:: json
  :caption: `conf/clusters/cluster-name.json`

  {
    "stream_alert": {
      "alert_processor": {
        "timeout": 25,
        "memory": 128,
        "current_version": "$LATEST",
        "outputs": {
          "aws-s3": [],
          "aws-lambda": []
        }
      },
      "rule_processor": {
        "timeout": 10,
        "memory": 256,
        "current_version": "$LATEST"
      }
    }
  }

**Options:**

===================  ========  ===========
Key                  Required  Description
-------------------  --------  -----------
``timeout``          ``true``  The time (in seconds) the Lambda function is allowed to process an incoming record. The timeout can be set to any value between 1 and 300 seconds.
``memory``           ``true``  The amount of memory allocated for the Lambda function execution.
``current_version``  ``true``  The most current published version of the Lambda function.
``outputs``          ``true``  A collection of S3 bucket IDs or AWS Lambda function names to configure as valid outputs.  By default, ``aws-s3`` should contain the bucket created by the ``stream_alert`` module: ``prefix.cluster.streamalerts``.  Optionally, if the alert processor needs to invoke other Lambda functions from within your AWS account, specify a list of function names.
===================  ========  ===========

vpc_config
~~~~~~~~~~

To enable StreamAlert's Alert Processor to access resources inside a VPC, you must provide additional VPC-specific configuration information.

`More information <http://docs.aws.amazon.com/lambda/latest/dg/vpc.html>`_

**Template:**

.. code-block:: json
  :caption: `conf/clusters/cluster-name.json`

  {
    "alert_processor": {
      "vpc_config": {
        "subnet_ids": [],
        "security_group_ids": []
      }
    }
  }
{'fraction_non_alphanumeric': 0.16457960644007155, 'fraction_numerical': 0.007155635062611807, 'mean_word_length': 2.937323943661972, 'pattern_counts': {'":': 32, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6400204', 'n_tokens_mistral': 1620, 'n_tokens_neox': 1528, 'n_words': 537}