<commit_before>= Kubernetes - Cluster Logging
:icons:
:linkcss:
:imagesdir: ../images

== Kubernetes - Cluster Logging

There are many options for aggregating and streaming kubernetes logs. In this example we are going to focus on an AWS based deployment which consists of the following:

1. Fluentd DaemonSet deployed with the Cloud Watch Logs plugin
2. Logs will stream to a CloudWatch Log Group
3. Lambda will stream the Log Group into a Amazon Elasticsearch cluster

=== Configure CloudWatch Log Group

    aws logs create-log-group --log-group-name kubernetes-logs --region us-east-1

=== Prepare Kubernetes manifest files

If you customized the log_group_name (from the previous command) or want to use a custom log_stream_name, you will need to update the output.conf section in ../templates/fluent-configmap.yaml

    output.conf: |
      <match **>
        # Plugin specific settings
        type cloudwatch_logs
        log_group_name kubernetes-cluster
        log_stream_name fluentd-cloudwatch-demo
        auto_create_stream true
        # Buffer settings
        buffer_chunk_limit 2M
        buffer_queue_limit 32
        flush_interval 10s
        max_retry_wait 30
        disable_retry_limit
        num_threads 8

      </match>

You will need to create an IAM user and set the AWS_ACCESS_KEY, AWS_SECRET_KEY and AWS_REGION in the ../templates/fluentd-ds.yaml file. Other options to accomplish this (in a production environment) are to use either kubernetes secrets or ec2 IAM roles.

  env:
  - name: FLUENTD_CONFIG
    value: fluentd-standalone.conf
  - name: AWS_REGION
    value: $REGION
  - name: AWS_ACCESS_KEY
    value: $ACCESS_KEY
  - name: AWS_SECRET_KEY
    value: $SECRET_KEY

=== Deploy Fluentd into Kubernetes

First create the logging namespace

    kubectl create ns logging

Create all of the necessary service accounts and roles:

    kubectl create -f ../templates/fluentd-service-account.yaml<commit_msg>finazliing logging<commit_after>= Kubernetes - Cluster Logging
:icons:
:linkcss:
:imagesdir: ../images

== Kubernetes - Cluster Logging

There are many options for aggregating and streaming kubernetes logs. In this example we are going to focus on an AWS based deployment which consists of the following:

1. Fluentd DaemonSet deployed with the Cloud Watch Logs plugin
2. Logs will stream to a CloudWatch Log Group
3. Lambda will stream the Log Group into a Amazon Elasticsearch cluster

=== Configure CloudWatch Log Group

    aws logs create-log-group --log-group-name kubernetes-logs --region us-east-1

=== Prepare Kubernetes manifest files

If you customized the log_group_name (from the previous command) or want to use a custom log_stream_name, you will need to update the output.conf section in ../templates/fluent-configmap.yaml

    output.conf: |
      <match **>
        # Plugin specific settings
        type cloudwatch_logs
        log_group_name kubernetes-cluster
        log_stream_name fluentd-cloudwatch-demo
        auto_create_stream true
        # Buffer settings
        buffer_chunk_limit 2M
        buffer_queue_limit 32
        flush_interval 10s
        max_retry_wait 30
        disable_retry_limit
        num_threads 8
      </match>

You will need to create an IAM user and set the AWS_ACCESS_KEY, AWS_SECRET_KEY and AWS_REGION in the ../templates/fluentd-ds.yaml file. Other options to accomplish this (in a production environment) are to use either kubernetes secrets or ec2 IAM roles.

  env:
  - name: FLUENTD_CONFIG
    value: fluentd-standalone.conf
  - name: AWS_REGION
    value: $REGION
  - name: AWS_ACCESS_KEY
    value: $ACCESS_KEY
  - name: AWS_SECRET_KEY
    value: $SECRET_KEY

=== Deploy Fluentd into Kubernetes

First create the logging namespace

    kubectl create ns logging

Create all of the necessary service accounts and roles:

    kubectl create -f ../templates/fluentd-service-account.yaml
{'fraction_non_alphanumeric': 0.0539568345323741, 'fraction_numerical': 0.006680369989722507, 'mean_word_length': 3.301657458563536, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3279300', 'n_tokens_mistral': 1128, 'n_tokens_neox': 1088, 'n_words': 438}