<commit_before>filebeat Cookbook
================

[![Cookbook](http://img.shields.io/badge/cookbook-v0.4.2-green.svg)](https://github.com/vkhatri/chef-filebeat) [![Build Status](https://travis-ci.org/vkhatri/chef-filebeat.svg?branch=master)](https://travis-ci.org/vkhatri/chef-filebeat)

This is a [Chef] cookbook to manage [Filebeat].


>> For Production environment, always prefer the [most recent release](https://supermarket.chef.io/cookbooks/filebeat).


## Most Recent Release

```ruby
cookbook 'filebeat', '~> 0.4.2'
```

## From Git

```ruby
cookbook 'filebeat', github: 'vkhatri/chef-filebeat',  tag: 'v0.4.2'
```

## Repository

```
https://github.com/vkhatri/chef-filebeat
```

## Supported OS

This cookbook was tested on Windows, Amazon & Ubuntu Linux and expected to work on other RHEL platforms.

This also works on Solaris zones given a physical Solaris 11.2 server. For that, use the .kitchen.zone.yml file. Check usage at (https://github.com/criticalmass/kitchen-zone). You will need an url to a filebeat package that works on Solaris 11.2. Checkout Building-Filebeat-On-Solaris11.md for instructions to build a filebeat package.

## Major Changes

### v0.2.5
- Removed default output configuration attributes for `elasticsearch`, `logstash` and `file`
- Removed attributed `default['filebeat']['enable_localhost_output']` as default `output` attributes are disabled

## Cookbook Dependency

- windows
- apt
- yum


## Recipes

- `filebeat::attributes` - cookbook derived default attributes

- `filebeat::config` - configure filebeat

- `filebeat::default` - default recipe (use it for run_list)

- `filebeat::install_package` - install filebeat package for linux platform

- `filebeat::install_solaris` - install filebeat package for solaris platform

- `filebeat::install_windows` - install filebeat for windows platform


## LWRP filebeat_prospector

LWRP `filebeat_prospector` creates filebeat prospector configuration yaml file under directory `node['filebeat']['prospectors_dir']` with file name `prospector-#{resource_name}.yml`.


**LWRP example**

```ruby
filebeat_prospector 'messages' do
  paths ['/var/log/messages']
  document_type 'apache'
  ignore_older '24h'
  scan_frequency '15s'
  harvester_buffer_size 16384
  fields 'type' => 'apacheLogs'
end
```

**LWRP Options**

- *action* (optional)	- default :create, options: :create, :delete
- *paths* (optional, String)	- filebeat prospector configuration attribute
- *type* (optional, String)	- filebeat prospector configuration attribute
- *encoding* (optional, String)	- filebeat prospector configuration attribute
- *fields* (optional, Hash)	- filebeat prospector configuration attribute
- *fields_under_root* (optional, TrueClass/FalseClass)	- filebeat prospector configuration attribute
- *ignore_older* (optional, String)	- filebeat prospector configuration attribute
- *document_type* (optional, String)	- filebeat prospector configuration attribute
- *input_type* (optional, String)	- filebeat prospector configuration attribute
- *close_older* (optional, String)	- filebeat prospector configuration attribute
- *scan_frequency* (optional, String)	- filebeat prospector configuration attribute
- *harvester_buffer_size* (optional, Integer)	- filebeat prospector configuration attribute
- *tail_files* (optional, TrueClass/FalseClass)	- filebeat prospector configuration attribute
- *backoff* (optional, String)	- filebeat prospector configuration attribute
- *max_backoff* (optional, String)	- filebeat prospector configuration attribute
- *backoff_factor* (optional, Integer)	- filebeat prospector configuration attribute
- *force_close_files* (optional, TrueClass/FalseClass)	- filebeat prospector configuration attribute
- *include_lines* (optional, Array)	- A list of regular expressions to match the lines that you want Filebeat to include. Filebeat exports only the lines that match a regular expression in the list. By default, all lines are exported.
- *exclude_lines* (optional, Array)	- A list of regular expressions to match the lines that you want Filebeat to exclude. Filebeat drops any lines that match a regular expression in the list. By default, no lines are dropped.
- *max_bytes* (optional, Integer)	- filebeat prospector configuration attribute
- *multiline* (optional, Hash)	- Multiline configuration hash. Options: `pattern`: <regex pattern to match>, `negate`: [true/false], `match`: [before/after]
- *exclude_files* (optional, Array)	- A list of regular expressions to match the files that you want Filebeat prospector instance to exclude.
- *spool_size* (optional, Integer)	- ilebeat prospector configuration attribute
- *publish_async* (optional, TrueClass/FalseClass)	- filebeat prospector configuration attribute
- *idle_timeout* (optional, String)	- filebeat prospector configuration attribute
- *registry_file* (optional, String)	- filebeat prospector configuration attribute

## How to Add Filebeat Output via Node Attribute


### Redis Output

```json
  "default_attributes": {
    "filebeat": {
      "config": {
        "output": {
          "redis": {
            "enable": true,
            "host": "127.0.0.1",
            "port": 6379,
            "save_topology": false,
            "index": "filebeat",
            "db": 0,
            "db_topology": 1,
            "password": "",
            "timeout": 5,
            "reconnect_interval": 1
          }
        }
      }
    }
  }
<commit_msg>added new lwrp attrs to README<commit_after>filebeat Cookbook
================

[![Cookbook](http://img.shields.io/badge/cookbook-v0.4.2-green.svg)](https://github.com/vkhatri/chef-filebeat) [![Build Status](https://travis-ci.org/vkhatri/chef-filebeat.svg?branch=master)](https://travis-ci.org/vkhatri/chef-filebeat)

This is a [Chef] cookbook to manage [Filebeat].


>> For Production environment, always prefer the [most recent release](https://supermarket.chef.io/cookbooks/filebeat).


## Most Recent Release

```ruby
cookbook 'filebeat', '~> 0.4.2'
```

## From Git

```ruby
cookbook 'filebeat', github: 'vkhatri/chef-filebeat',  tag: 'v0.4.2'
```

## Repository

```
https://github.com/vkhatri/chef-filebeat
```

## Supported OS

This cookbook was tested on Windows, Amazon & Ubuntu Linux and expected to work on other RHEL platforms.

This also works on Solaris zones given a physical Solaris 11.2 server. For that, use the .kitchen.zone.yml file. Check usage at (https://github.com/criticalmass/kitchen-zone). You will need an url to a filebeat package that works on Solaris 11.2. Checkout Building-Filebeat-On-Solaris11.md for instructions to build a filebeat package.

## Major Changes

### v0.2.5
- Removed default output configuration attributes for `elasticsearch`, `logstash` and `file`
- Removed attributed `default['filebeat']['enable_localhost_output']` as default `output` attributes are disabled

## Cookbook Dependency

- windows
- apt
- yum


## Recipes

- `filebeat::attributes` - cookbook derived default attributes

- `filebeat::config` - configure filebeat

- `filebeat::default` - default recipe (use it for run_list)

- `filebeat::install_package` - install filebeat package for linux platform

- `filebeat::install_solaris` - install filebeat package for solaris platform

- `filebeat::install_windows` - install filebeat for windows platform


## LWRP filebeat_prospector

LWRP `filebeat_prospector` creates filebeat prospector configuration yaml file under directory `node['filebeat']['prospectors_dir']` with file name `prospector-#{resource_name}.yml`.


**LWRP example**

```ruby
filebeat_prospector 'messages' do
  paths ['/var/log/messages']
  document_type 'apache'
  ignore_older '24h'
  scan_frequency '15s'
  harvester_buffer_size 16384
  fields 'type' => 'apacheLogs'
end
```

**LWRP Options**

- *action* (optional)	- default :create, options: :create, :delete
- *paths* (optional, String)	- filebeat prospector configuration attribute
- *type* (optional, String)	- filebeat prospector configuration attribute
- *encoding* (optional, String)	- filebeat prospector configuration attribute
- *fields* (optional, Hash)	- filebeat prospector configuration attribute
- *fields_under_root* (optional, TrueClass/FalseClass)	- filebeat prospector configuration attribute
- *ignore_older* (optional, String)	- filebeat prospector configuration attribute
- *document_type* (optional, String)	- filebeat prospector configuration attribute
- *input_type* (optional, String)	- filebeat prospector configuration attribute
- *close_older* (optional, String)	- filebeat prospector configuration attribute
- *scan_frequency* (optional, String)	- filebeat prospector configuration attribute
- *harvester_buffer_size* (optional, Integer)	- filebeat prospector configuration attribute
- *tail_files* (optional, TrueClass/FalseClass)	- filebeat prospector configuration attribute
- *backoff* (optional, String)	- filebeat prospector configuration attribute
- *max_backoff* (optional, String)	- filebeat prospector configuration attribute
- *backoff_factor* (optional, Integer)	- filebeat prospector configuration attribute
- *force_close_files* (optional, TrueClass/FalseClass)	- filebeat prospector configuration attribute
- *include_lines* (optional, Array)	- A list of regular expressions to match the lines that you want Filebeat to include. Filebeat exports only the lines that match a regular expression in the list. By default, all lines are exported.
- *exclude_lines* (optional, Array)	- A list of regular expressions to match the lines that you want Filebeat to exclude. Filebeat drops any lines that match a regular expression in the list. By default, no lines are dropped.
- *max_bytes* (optional, Integer)	- filebeat prospector configuration attribute
- *multiline* (optional, Hash)	- Multiline configuration hash. Options: `pattern`: <regex pattern to match>, `negate`: [true/false], `match`: [before/after]
- *exclude_files* (optional, Array)	- A list of regular expressions to match the files that you want Filebeat prospector instance to exclude.
- *spool_size* (optional, Integer)	- ilebeat prospector configuration attribute
- *publish_async* (optional, TrueClass/FalseClass)	- filebeat prospector configuration attribute
- *idle_timeout* (optional, String)	- filebeat prospector configuration attribute
- *registry_file* (optional, String)	- filebeat prospector configuration attribute
- *json_message_key* (optional, String)	- filebeat prospector configuration attribute `json.message_key`
- *json_keys_under_root* (optional, String)	- filebeat prospector configuration attribute `json.keys_under_root`
- *json_overwrite_keys* (optional, String)	- filebeat prospector configuration attribute `json.overwrite_keys`
- *json_add_error_key* (optional, String)	- filebeat prospector configuration attribute `json.add_error_key`

## How to Add Filebeat Output via Node Attribute


### Redis Output

```json
  "default_attributes": {
    "filebeat": {
      "config": {
        "output": {
          "redis": {
            "enable": true,
            "host": "127.0.0.1",
            "port": 6379,
            "save_topology": false,
            "index": "filebeat",
            "db": 0,
            "db_topology": 1,
            "password": "",
            "timeout": 5,
            "reconnect_interval": 1
          }
        }
      }
    }
  }

{'fraction_non_alphanumeric': 0.11141568662127509, 'fraction_numerical': 0.0076045627376425855, 'mean_word_length': 4.479651162790698, 'pattern_counts': {'":': 30, '<': 5, '<?xml version=': 0, '>': 13, 'https://': 12, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2575742', 'n_tokens_mistral': 3333, 'n_tokens_neox': 3211, 'n_words': 1045}