<commit_before>%Log forwarding and centralisation configuration

In this section, we install a (network listening) rsyslog server on the Logstash server,
configure all components of OpenShift to use syslog, and syslog to push to the
rsyslog server.

# Rsyslog server

We use the default RHEL 6 rsyslog package on logstash.example.com, no software installation is
necessary.

## System configuration

We store logs in `/var/log/rsyslog/` and use one file per program (same file for
all hosts). The `/var/log/rsyslog` directory must be created first and readable by
logstash (eg. `chmod 0755`).

The host firewall must also be configured to allow both UDP and TCP port 514, for example with:

    # iptables -A INPUT -m tcp -p tcp --dport 514 -j ACCEPT
    # iptables -A INPUT -m udp -p udp --dport 514 -j ACCEPT
    # service iptables save

## Rsyslog configuration

Rsyslog is configured by editing `/etc/rsyslog.conf`.

Enable network listening for the server by uncommenting the following lines:

    # Provides UDP syslog reception
    $ModLoad imudp
    $UDPServerRun 514
    
    # Provides TCP syslog reception
    $ModLoad imtcp
    $InputTCPServerRun 514

Add a rule to centralise all non-local logs to `/var/log/rsyslog/` using one
file per originator program for all hosts:

    # change default umask so that logstash can read files
    
    $umask 0000
    $DirCreateMode 0755
    $FileCreateMode 0644
    
    $template TmplMsg, "/var/log/rsyslog/%PROGRAMNAME%.log"
    :fromhost-ip,!isequal,"127.0.0.1" -?TmplMsg
    & ~

# General configuration of OpenShift servers

All OpenShift servers (brokers, broker support nodes, nodes) are configured to
forward system logs to the logstash.example.com. We deploy a newer version of
rsyslog from the OpenShift repository to allow additional metadata in the log
files provided by a plugin.

## Rsyslog7 installation and configuration

    yum install -y rsyslog7 rsyslog7-mmopenshift

Rsyslog7 is configured by editing `/etc/rsyslog7.conf`.

Disable Systemd-specific options by commenting out the following lines:

    # $ModLoad imjournal # provides access to the systemd journal

    # Turn off message reception via local log socket;
    # local messages are retrieved through imjournal now.
    #$OmitLocalLogging on
    
    # File to store the position in the journal
    #$IMJournalStateFile imjournal.state

Make sure rsyslog7 is loading additional configuration files from
`/etc/rsyslog7.d/*.conf`

    $IncludeConfig /etc/rsyslog7.d/*.conf

*On OpenShift nodes only*, enable the rsyslog plugin by changing the modules section:

    #$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
    # $ModLoad imjournal # provides access to the systemd journal
    $ModLoad imklog   # provides kernel logging support (previously done by rklogd)
    #$ModLoad immark  # provides --MARK-- message capability
    # OpenShift plugin module
    module(load="imuxsock" SysSock.Annotate="on" SysSock.ParseTrusted="on" SysSock.UsePIDFromSystem="on")
    module(load="mmopenshift")

Add a forwarding rule by creating `/etc/rsyslog7.d/forward.conf` with the following content:

    $WorkDirectory /var/lib/rsyslog
    $ActionQueueFileName fwdRule1 # unique name prefix for spool files
    $ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as possible)
    $ActionQueueSaveOnShutdown on # save messages to disk on shutdown
    $ActionQueueType LinkedList   # run asynchronously
    $ActionResumeRetryCount -1    # infinite retries if host is down
    *.* @@logstash.example.com:514

*On OpenShift nodes only*, create a template for the plugin by creating
`/etc/rsyslog7.d/openshift.conf` with the following content:

    # OpenShift plugin configuration
    template(name="OpenShift" type="list")
      {
      property(name="timestamp" dateFormat="rfc3339")
      constant(value=" ")
      property(name="hostname")
      constant(value=" ")
      property(name="syslogtag")
      constant(value=" app=")
      property(name="$!OpenShift!OPENSHIFT_APP_NAME")
      constant(value=" ns=")
      property(name="$!OpenShift!OPENSHIFT_NAMESPACE")
      constant(value=" appUuid=")
      property(name="$!OpenShift!OPENSHIFT_APP_UUID")
      constant(value=" gearUuid=")
      property(name="$!OpenShift!OPENSHIFT_GEAR_UUID")
      property(name="msg" spifno1stsp="on")
      property(name="msg" droplastlf="on")
      constant(value="\n")
    }

## Replacing Rsyslog with Rsyslog7

On all OpenShift servers, the default rsyslog service must be stopped and
disabled to allow rsyslog7 to replace it.

    # service rsyslog stop
    # chkconfig rsyslog off
    # service rsyslog7 start
    # chkconfig rsyslog7 on

# Configuring OpenShift brokers to use syslog

## The Broker application

Add the following line to `/etc/openshift/broker.conf`
<commit_msg>Add rsyslog7 instructions for OSE 2.2/RHEL 6.6<commit_after>%Log forwarding and centralisation configuration

In this section, we install a (network listening) rsyslog server on the Logstash server,
configure all components of OpenShift to use syslog, and syslog to push to the
rsyslog server.

# Rsyslog server

We use the default RHEL 6 rsyslog package on logstash.example.com, no software installation is
necessary.

## System configuration

We store logs in `/var/log/rsyslog/` and use one file per program (same file for
all hosts). The `/var/log/rsyslog` directory must be created first and readable by
logstash (eg. `chmod 0755`).

The host firewall must also be configured to allow both UDP and TCP port 514, for example with:

    # iptables -A INPUT -m tcp -p tcp --dport 514 -j ACCEPT
    # iptables -A INPUT -m udp -p udp --dport 514 -j ACCEPT
    # service iptables save

## Rsyslog configuration

Rsyslog is configured by editing `/etc/rsyslog.conf`.

Enable network listening for the server by uncommenting the following lines:

    # Provides UDP syslog reception
    $ModLoad imudp
    $UDPServerRun 514
    
    # Provides TCP syslog reception
    $ModLoad imtcp
    $InputTCPServerRun 514

Add a rule to centralise all non-local logs to `/var/log/rsyslog/` using one
file per originator program for all hosts:

    # change default umask so that logstash can read files
    
    $umask 0000
    $DirCreateMode 0755
    $FileCreateMode 0644
    
    $template TmplMsg, "/var/log/rsyslog/%PROGRAMNAME%.log"
    :fromhost-ip,!isequal,"127.0.0.1" -?TmplMsg
    & ~

## Log rotation

You may want to add a specific rule for log rotation in /var/log/rsyslog, for
example by creating the /etc/logrotate.d/syslog-server file with the following
content:

    /var/log/rsyslog/*.log {
        daily
        rotate 30
        copytruncate
        compress
        missingok
        notifyempty
    }

# General configuration of OpenShift servers

All OpenShift servers (brokers, broker support nodes, nodes) are configured to
forward system logs to the logstash.example.com. We deploy a newer version of
rsyslog from the OpenShift repository to allow additional metadata in the log
files provided by a plugin.

## Rsyslog7 installation and configuration

    yum install -y rsyslog7 rsyslog7-mmopenshift

Important! For OpenShift Enterprise 2.2: OpenShift Enterprise 2.2 runs on
RHEL 6.6, which comes with its own package for rsyslog7. Because the RHEL
rsyslog7 is a replacement for the default rsyslog package, you’ll need to
use yum shell to run a transaction:

    yum shell
    erase rsyslog
    install rsyslog7 rsyslog7-mmopenshift
    transaction run

The rsyslog7 configuration will be in `/etc/rsyslog7.conf` for OpenShift
Enterprise 2.1 and `/etc/rsyslog.conf` starting with OpenShift Enterprise 2.2.

Disable Systemd-specific options by commenting out the following lines:

    # $ModLoad imjournal # provides access to the systemd journal

    # Turn off message reception via local log socket;
    # local messages are retrieved through imjournal now.
    #$OmitLocalLogging on
    
    # File to store the position in the journal
    #$IMJournalStateFile imjournal.state

Make sure rsyslog7 is loading additional configuration files from
`/etc/rsyslog.d/*.conf`

    $IncludeConfig /etc/rsyslog.d/*.conf

*On OpenShift nodes only*, enable the rsyslog plugin by changing the modules section:

    #$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
    # $ModLoad imjournal # provides access to the systemd journal
    $ModLoad imklog   # provides kernel logging support (previously done by rklogd)
    #$ModLoad immark  # provides --MARK-- message capability
    # OpenShift plugin module
    module(load="imuxsock" SysSock.Annotate="on" SysSock.ParseTrusted="on" SysSock.UsePIDFromSystem="on")
    module(load="mmopenshift")

Add a forwarding rule by creating `/etc/rsyslog7.d/forward.conf` with the following content:

    $WorkDirectory /var/lib/rsyslog
    $ActionQueueFileName fwdRule1 # unique name prefix for spool files
    $ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as possible)
    $ActionQueueSaveOnShutdown on # save messages to disk on shutdown
    $ActionQueueType LinkedList   # run asynchronously
    $ActionResumeRetryCount -1    # infinite retries if host is down
    *.* @@logstash.example.com:514

*On OpenShift nodes only*, create a template for the plugin by creating
`/etc/rsyslog7.d/openshift.conf` with the following content:

    # OpenShift plugin configuration
    template(name="OpenShift" type="list")
      {
      property(name="timestamp" dateFormat="rfc3339")
      constant(value=" ")
      property(name="hostname")
      constant(value=" ")
      property(name="syslogtag")
      constant(value=" app=")
      property(name="$!OpenShift!OPENSHIFT_APP_NAME")
      constant(value=" ns=")
      property(name="$!OpenShift!OPENSHIFT_NAMESPACE")
      constant(value=" appUuid=")
      property(name="$!OpenShift!OPENSHIFT_APP_UUID")
      constant(value=" gearUuid=")
      property(name="$!OpenShift!OPENSHIFT_GEAR_UUID")
      property(name="msg" spifno1stsp="on")
      property(name="msg" droplastlf="on")
      constant(value="\n")
    }

## Replacing Rsyslog with Rsyslog7

On OpenShift Enterprise 2.1, the rsyslog and rsyslog7 packages cohexist,
you need to stop and disable the default rsyslog service on all OpenShift
servers to allow rsyslog7 to replace it.

    # service rsyslog stop
    # chkconfig rsyslog off
    # service rsyslog7 start
    # chkconfig rsyslog7 on

Starting with OpenShift 2.2, the rsyslog7 package replaces the default rsyslog,
that is why we had to do this `yum shell` dance earlier. Just ensure that the
rsyslog service is enabled and running.

    # service rsyslog start
    # chkconfig rsyslog on

# Configuring OpenShift brokers to use syslog

## The Broker application

Add the following line to `/etc/openshift/broker.conf`

{'fraction_non_alphanumeric': 0.07524496210020336, 'fraction_numerical': 0.014143094841930116, 'mean_word_length': 3.712108013937282, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1963115', 'n_tokens_mistral': 3340, 'n_tokens_neox': 3178, 'n_words': 1195}