<commit_before>== Migrating from Logstash Forwarder to Filebeat

Filebeat is based on the Logstash Forwarder source code and replaces Logstash as the method
to use for tailing log files and forwarding them to Logstash. This guide contains
information to help you migrate from
https://github.com/elastic/logstash-forwarder[Logstash Forwarder] to Filebeat.

Filebeat introduces the following major changes:

* The config file was restructured and converted from JSON to YAML.
* The registry file, which stores the state of the currently read files, was
  changed.
* Command line options were removed and moved to the configuration file.
* Configuration options for outputs are now inherited from libbeat. For details, see the {libbeat}/index.html[Beats Platform Reference].
* A new Logstash input plugin called https://www.elastic.co/guide/en/logstash/current/plugins-inputs-beats.html[logstash-input-beats] is required.


=== Migrating to the Logstash Input Beats Plugin

Filebeat requires a new input plugin in Logstash, called
https://www.elastic.co/guide/en/logstash/current/plugins-inputs-beats.html[logstash-input-beats].
For information about getting started with this plugin, see {libbeat}/getting-started.html#logstash-setup[Setting up Logstash].

In both the 1.5.x and 2.x versions of Logstash, this plugin can be loaded in
parallel with the
https://github.com/logstash-plugins/logstash-input-lumberjack[logstash-input-lumberjack]
plugin used by the Logstash Forwarder.

If you have a large number of servers that you want to migrate from
Logstash Forwarder to Filebeat, we recommend that you keep the Lumberjack plugin and load the
Beats plugin on the same Logstash instances, but set up the Beats plugin to use a different port. After you have migrated
all the machines to Filebeat, you can remove the Lumberjack plugin.

===  Updating the Registry File

The registry file stores the state and location information that Filebeat uses to track
where it was last reading. Under Logstash Forwarder, this file was called `.logstash-fowarder`. For Filebeat,
the file was renamed. The name varies depending on the package type:

 * `.filebeat` for `.tar.gz` and `.tgz` archives
 * `/usr/lib/filebeat/registry` for DEB and RPM packages
 * `c:\ProgramData\filebeat\registry` for the Windows zip file

For enhancement reasons, especially for Windows,
the structure of the registry file has changed. This makes migrating the file
complex and leads to potential errors.

Instead of migrating the registry file, we recommend that you start Filebeat on
the same host where Logstash Forwarder is running, and send the log files to a
different index.  This will start indexing from scratch. If you want to start
reading at the end of all files, you can set the `tail_files` option in the
Filebeat configuration file to true.

Using this approach allows you to keep the old Logstash Forwarder running and then
slowly migrate over to Filebeat.

=== Migrating Your Configuration

Although Filebeat is based on Logstash Forwarder, Filebeat uses YAML for its configuration
file, rather than the JSON+comments language used by Logstash Forwarder. This means that you
will need to migrate your existing configuration files to use the YAML syntax. Filebeat has a main
configuration file called `filebeat.yml`, but Filebeat also accepts reading
multiple configuration files from a `conf.d` directory and has similar restrictions to Logstash Forwarder.
If you specify additional config files, you need to place them in a directory other than the directory
where the main Filebeat config file resides. You specify the location of the config files by using the
`config_dir` option to configure the path to the directory. In most cases, you can do a one-to-one
conversion to create a Filebeat config file for each Logstash Forwarder config file.

Before migrating your config files, we recommend that you first read the <<filebeat-configuration-details>>
section to understand the Filebeat options.

NOTE: Logstash Forwarder has the option of auto completing environment variables in
the configuration file. This option currently doesn't exist in Filebeat.

==== Migrating the "files" Section

To migrate the `files` section from the Logstash Forwarder configuration, create a  `prospectors` section in the Filebeat config file. For example, assuming that you start
with this configuration in Logstash Forwarder:

[source,json]
-------------------------------------------------------------------------------------
  # The list of files configurations
  "files": [
    # An array of hashes. Each hash tells what paths to watch and
    # what fields to annotate on events from those paths.
    {
      "paths": [
        "/var/log/messages",
        "/var/log/*.log"
      ],

      # A dictionary of fields to annotate on each event.
      "fields": {
        "type": "syslog",
        "service": "example-service"
      }
    }, {
      # A path of "-" means stdin.
      "paths": [ "-" ],
      "fields": { "type": "stdin" }
    }, {
      "paths": [
        "/var/log/apache/httpd-*.log"
      ],
      "fields": { "type": "apache" }
    }
  ]
-------------------------------------------------------------------------------------

The equivalent `prospectors` section would look like this:

[source,yaml]
-------------------------------------------------------------------------------------
filebeat:
  # List of prospectors to fetch data.
  prospectors:
    # Each - is a prospector. Below are the prospector specific configurations
    -
      paths:
        - /var/log/messages
        - "/var/log/*.log"
      document_type: syslog <1>
      fields:
        service: example-service
    -
      paths:
        - "-"
      input_type: stdin <2>
      document_type: stdin
    -
      paths:
        - "/var/log/apache/httpd-*.log"
      document_type: apache
-------------------------------------------------------------------------------------

<1> The `document_type` option controls the output `type` field, which is used by the
    Elasticsearch output to determine the document type.
<2> The explicit `input_type` option was introduced to differentiate between normal files and
    stdin. In the future, additional types might be supported.

As you can see, apart from the new `document_type` and `input_type` options,
which were before implicitly defined via the `type` custom field, the remaining
options can be migrated mechanically.

The Filebeat configuration gives you more control over how each prospector behaves
by allowing you to configure options that were previously global in Logstash Forwarder
and set them separately for each prospector. See <<filebeat-configuration-details>>.

==== Migrating the "network" Section

Like Logstash Forwarder, Filebeat can communicate directly with Logstash.
Filebeat can also insert log entries directly
into Elasticsearch. This results in an `output` section that is a bit more complex, as
you can see in the following example. You'll find, however, that you can easily
translate the Logstash part of the configuration from the equivalent Logstash Forwarder
configuration.

The following snippet shows the `network` section of the Logstash Forwarder configuration:

[source,json]
-------------------------------------------------------------------------------------
  # The network section covers network configuration :)
  "network": {
    # A list of downstream servers listening for our messages.
    # logstash-forwarder will pick one at random and only switch if
    # the selected one appears to be dead or unresponsive
    "servers": [ "localhost:5043" ],

    # The path to your client ssl certificate (optional)
    "ssl certificate": "./logstash-forwarder.crt",
    # The path to your client ssl key (optional)
    "ssl key": "./logstash-forwarder.key",

    # The path to your trusted ssl CA file. This is used
    # to authenticate your downstream server.
    "ssl ca": "./logstash-forwarder.crt",

    # Network timeout in seconds. This is most important for
    # logstash-forwarder determining whether to stop waiting for an
    # acknowledgement from the downstream server. If an timeout is reached,
    # logstash-forwarder will assume the connection or server is bad and
    # will connect to a server chosen at random from the servers list.
    "timeout": 15
  }
-------------------------------------------------------------------------------------

The equivalent in Filebeat would look like this:


[source,yaml]
-------------------------------------------------------------------------------------
output:
  logstash:
    enabled: true

    # The list of downstream Logstash servers. <1>
    hosts:
      - localhost:5043

    tls: <2>
      # The path to your SSL client certificate.
      certificate: ./logstash-forwarder.crt

      # The path to your SSL client certificate key.
      certificate_key: ./logstash-forwarder.key

      # The path to your trusted SSL CA file. This is used
      # to authenticate your downstream server.
      certificate_authorities:
        - ./logstash-forwarder.crt

      # Network timeout in seconds.
      timeout: 15
-------------------------------------------------------------------------------------

<1> When multiple hosts are defined, the default behavior in Filebeat is to
    pick a random host for new connections, similar to the Logstash Forwarder
    behavior. Filebeat can optionally do load balancing. For more details, see the
    {libbeat}/configuration.html#loadbalance[loadbalance] configuration option.
<2> Note that if the `tls` section is missing, then TLS is disabled. TLS is
    automatically enabled when you add the `tls` section. For more information about
    specific configuration options, see
    {libbeat}/configuration.html#configuration-output-tls[TLS Options].


[[changed-configuration-options]]
==== Changed Configuration File Options

With the refactoring of the configuration file, the following options were removed or renamed:

[cols="2*", options="header"]
|===
|Config Option
|Action

|`deadTime`
|`deadTime` was renamed to `ignore_older`. Filebeat keeps the files that it’s reading open until they are older than the timespan specified by `ignore_older`. If a file is changed, Filebeat reopens it.

|`netTimeout`
|`netTimeout` was removed and is replaced by the `timeout` option in libbeat.

|`log-to-syslog` and `syslog`
|Both options were removed and replaced by logging options in libbeat.
|===

For more information about these options, see {libbeat}/configuration.html[Beats configuration options].


==== A Complete Example

Let's see a simple, but complete example of a Logstash Forwarder configuration
and its equivalent for Filebeat.

Logstash Forwarder configuration:

[source,json]
-------------------------------------------------------------------------------------
{
  "files": [
    {
      "paths": [
        "/var/log/*.log"
      ],
      "fields": {
        "type": "syslog",
        "service": "test01"
      }
    }
  ],
  "network": {
    "servers": [ "localhost:5043" ],
  }
}
-------------------------------------------------------------------------------------

Filebeat configuration:

[source,yaml]
-------------------------------------------------------------------------------------
filebeat:
  prospectors:
    -
      paths:
        - "/var/log/*.log"
      document_type: syslog
      fields:
        service: test01
output:
  elasticsearch:
    enabled: true
    hosts: ["http://localhost:5043"]
-------------------------------------------------------------------------------------

=== Changes to Command Line Options

Most command line options available in Logstash Forwarder have been removed and
migrated to config file options. The only mandatory command line option for
running  Filebeat is  `-c` followed by the path to the config file. If you used command line
options with Logstash Forwarder, make sure that you add your options to the
configuration file. For naming changes, see <<renamed-options>>.

Filebeat does provide command line options that are common to all Beats. For more details about
these options, see {libbeat}/command-line-options.html[Beats Command Line Options].

[[renamed-options]]
==== Renamed Options

The following command line options have been renamed and moved to the config file. Also see
<<changed-configuration-options>> for a list of configuration file options that were completely removed or
replaced by options specified in libbeat.

[cols="3*", options="header"]
|===
|Command Line Option
|Config File Option
|Description

|`-config`
|`-c` command line option and `config_dir`
|The config option was split into two parts. You use the `-c` command line option to specify the location of the base (required) config file when you start Filebeat. To use additional config files, you specify the `config_dir` configuration option.

The `config_dir` option specifies the path to the directory that contains additional configuration files. This option MUST point to a directory other than the directory where the main Filebeat config file resides.

|`-idle-timeout`
|`idle_timeout`
|`idle_timeout` was moved to the config file and removed as a flag.

|`-spool-size`
|`spool_size`
|`spool_size` was moved to the config file and removed as a flag.

|`-harvester-buff-size`
|`harvester_buffer_size`
|`harvester_buffer_size` was moved to the config file and removed as a flag. You can now configure the buffer size separately for each harvester.

|`-tail`
|`tail_files`
|`tail_files` was moved to the config file and removed as a flag. You can now configure this option separately for each prospector.

|`-cpuProfileFile`
|
|`cpuProfileFile` option was removed. You can use the profiling options of libbeat instead. For more details on profiling, see https://github.com/elastic/libbeat/issues/122.

|`-quiet`
|
|The `quiet` option was removed. Libbeat is now used for logging, so you must use the libbeat {libbeat}/configuration.html#configuration-logging[logging options] instead.


|===


=== Changes in the output fields

In the default configuration, Filebeat structures its output documents a little
different from the Logstash Forwarder. This section discusses the differences
and the options you have in case you want compatibility with the Logstash
Forwarder.

One such difference is that the custom fields (added from the configuration
file) are set top level in Logstash Forwarder but are grouped together under a
`fields` dictionary in Filebeat. If you need the old behavior during the
migration phase, you can use the <<fields-under-root>> configuration option:

[source,yaml]
-------------------------------------------------------------------------------------
filebeat:
  prospectors:
    -
      paths:
        - "/var/log/*.log"
      document_type: syslog
      fields:
        service: test01
      fields_under_root: true
-------------------------------------------------------------------------------------

Another difference is that while the Logstash Forwarder sends the hostname of
the server it is running on in the `host` field, Filebeat uses the
`beat.hostname` field for the same purpose. Because `host` is commonly used in
the Logstash plugin ecosystem, the Beats input plugin automatically copies
`beat.hostname` into `host`.

Similarly, the `file` field was renamed to `source`. If you rely on this field
being named `file` you can rename it by using the mutate filter in Logstash. For
example:

[source,plain]
-------------------------------------------------------------------------------------
filter {
    mutate {
        rename => {
            "source" => "file"
        }
    }
}
-------------------------------------------------------------------------------------

Finally, the `line` field was removed. This field wasn't correct in case of
restarts and making it correct would require a performance penalty. We recommend
using the `offset` field instead.

=== Other Changes

The following list of implementation changes should not affect your experience migrating
from Logstash Forwarder, but you should be aware of the changes. Please post GitHub issues
if you notice any regressions from Logstash Forwarder.

==== Packaging

The packaging process for Filebeat uses the Beats infrastructure, so some
aspects of packaging, such as the init scripts, are different from
Logstash Forwarder. Please post GitHub issues if you hit any issues
with the new packages.

One notable change is the name of the registry file. The name varies depending on the package
type:

 * `.filebeat` for `.tar.gz` and `.tgz` archives
 * `/usr/lib/filebeat/registry` for DEB and RPM packages
 * `c:\ProgramData\filebeat\registry` for the Windows zip file

==== Publisher Improvements

Behind the scenes, Filebeat uses a sightly improved protocol for communicating
with Logstash.

==== TLS Is Off by Default

If you follow the section on migrating the configuration, you will have TLS
enabled. However, you must be aware that if the `tls` section is missing from the
configuration file, Filebeat uses an unencrypted connection to talk to Logstash.

==== Logging

Filebeat uses libbeat logging and can also log to rotating files instead of syslog.
<commit_msg>Added a second field to make the example more clear<commit_after>== Migrating from Logstash Forwarder to Filebeat

Filebeat is based on the Logstash Forwarder source code and replaces Logstash as the method
to use for tailing log files and forwarding them to Logstash. This guide contains
information to help you migrate from
https://github.com/elastic/logstash-forwarder[Logstash Forwarder] to Filebeat.

Filebeat introduces the following major changes:

* The config file was restructured and converted from JSON to YAML.
* The registry file, which stores the state of the currently read files, was
  changed.
* Command line options were removed and moved to the configuration file.
* Configuration options for outputs are now inherited from libbeat. For details, see the {libbeat}/index.html[Beats Platform Reference].
* A new Logstash input plugin called https://www.elastic.co/guide/en/logstash/current/plugins-inputs-beats.html[logstash-input-beats] is required.


=== Migrating to the Logstash Input Beats Plugin

Filebeat requires a new input plugin in Logstash, called
https://www.elastic.co/guide/en/logstash/current/plugins-inputs-beats.html[logstash-input-beats].
For information about getting started with this plugin, see {libbeat}/getting-started.html#logstash-setup[Setting up Logstash].

In both the 1.5.x and 2.x versions of Logstash, this plugin can be loaded in
parallel with the
https://github.com/logstash-plugins/logstash-input-lumberjack[logstash-input-lumberjack]
plugin used by the Logstash Forwarder.

If you have a large number of servers that you want to migrate from
Logstash Forwarder to Filebeat, we recommend that you keep the Lumberjack plugin and load the
Beats plugin on the same Logstash instances, but set up the Beats plugin to use a different port. After you have migrated
all the machines to Filebeat, you can remove the Lumberjack plugin.

===  Updating the Registry File

The registry file stores the state and location information that Filebeat uses to track
where it was last reading. Under Logstash Forwarder, this file was called `.logstash-fowarder`. For Filebeat,
the file was renamed. The name varies depending on the package type:

 * `.filebeat` for `.tar.gz` and `.tgz` archives
 * `/usr/lib/filebeat/registry` for DEB and RPM packages
 * `c:\ProgramData\filebeat\registry` for the Windows zip file

For enhancement reasons, especially for Windows,
the structure of the registry file has changed. This makes migrating the file
complex and leads to potential errors.

Instead of migrating the registry file, we recommend that you start Filebeat on
the same host where Logstash Forwarder is running, and send the log files to a
different index.  This will start indexing from scratch. If you want to start
reading at the end of all files, you can set the `tail_files` option in the
Filebeat configuration file to true.

Using this approach allows you to keep the old Logstash Forwarder running and then
slowly migrate over to Filebeat.

=== Migrating Your Configuration

Although Filebeat is based on Logstash Forwarder, Filebeat uses YAML for its configuration
file, rather than the JSON+comments language used by Logstash Forwarder. This means that you
will need to migrate your existing configuration files to use the YAML syntax. Filebeat has a main
configuration file called `filebeat.yml`, but Filebeat also accepts reading
multiple configuration files from a `conf.d` directory and has similar restrictions to Logstash Forwarder.
If you specify additional config files, you need to place them in a directory other than the directory
where the main Filebeat config file resides. You specify the location of the config files by using the
`config_dir` option to configure the path to the directory. In most cases, you can do a one-to-one
conversion to create a Filebeat config file for each Logstash Forwarder config file.

Before migrating your config files, we recommend that you first read the <<filebeat-configuration-details>>
section to understand the Filebeat options.

NOTE: Logstash Forwarder has the option of auto completing environment variables in
the configuration file. This option currently doesn't exist in Filebeat.

==== Migrating the "files" Section

To migrate the `files` section from the Logstash Forwarder configuration, create a  `prospectors` section in the Filebeat config file. For example, assuming that you start
with this configuration in Logstash Forwarder:

[source,json]
-------------------------------------------------------------------------------------
  # The list of files configurations
  "files": [
    # An array of hashes. Each hash tells what paths to watch and
    # what fields to annotate on events from those paths.
    {
      "paths": [
        "/var/log/messages",
        "/var/log/*.log"
      ],

      # A dictionary of fields to annotate on each event.
      "fields": {
        "type": "syslog",
        "service": "apache",
        "zone": "us-east-1"
      }
    }, {
      # A path of "-" means stdin.
      "paths": [ "-" ],
      "fields": { "type": "stdin" }
    }, {
      "paths": [
        "/var/log/apache/httpd-*.log"
      ],
      "fields": { "type": "apache" }
    }
  ]
-------------------------------------------------------------------------------------

The equivalent `prospectors` section would look like this:

[source,yaml]
-------------------------------------------------------------------------------------
filebeat:
  # List of prospectors to fetch data.
  prospectors:
    # Each - is a prospector. Below are the prospector specific configurations
    -
      paths:
        - /var/log/messages
        - "/var/log/*.log"
      document_type: syslog <1>
      fields:
        service: apache
        zone: us-east-1
    -
      paths:
        - "-"
      input_type: stdin <2>
      document_type: stdin
    -
      paths:
        - "/var/log/apache/httpd-*.log"
      document_type: apache
-------------------------------------------------------------------------------------

<1> The `document_type` option controls the output `type` field, which is used by the
    Elasticsearch output to determine the document type.
<2> The explicit `input_type` option was introduced to differentiate between normal files and
    stdin. In the future, additional types might be supported.

As you can see, apart from the new `document_type` and `input_type` options,
which were before implicitly defined via the `type` custom field, the remaining
options can be migrated mechanically.

The Filebeat configuration gives you more control over how each prospector behaves
by allowing you to configure options that were previously global in Logstash Forwarder
and set them separately for each prospector. See <<filebeat-configuration-details>>.

==== Migrating the "network" Section

Like Logstash Forwarder, Filebeat can communicate directly with Logstash.
Filebeat can also insert log entries directly
into Elasticsearch. This results in an `output` section that is a bit more complex, as
you can see in the following example. You'll find, however, that you can easily
translate the Logstash part of the configuration from the equivalent Logstash Forwarder
configuration.

The following snippet shows the `network` section of the Logstash Forwarder configuration:

[source,json]
-------------------------------------------------------------------------------------
  # The network section covers network configuration :)
  "network": {
    # A list of downstream servers listening for our messages.
    # logstash-forwarder will pick one at random and only switch if
    # the selected one appears to be dead or unresponsive
    "servers": [ "localhost:5043" ],

    # The path to your client ssl certificate (optional)
    "ssl certificate": "./logstash-forwarder.crt",
    # The path to your client ssl key (optional)
    "ssl key": "./logstash-forwarder.key",

    # The path to your trusted ssl CA file. This is used
    # to authenticate your downstream server.
    "ssl ca": "./logstash-forwarder.crt",

    # Network timeout in seconds. This is most important for
    # logstash-forwarder determining whether to stop waiting for an
    # acknowledgement from the downstream server. If an timeout is reached,
    # logstash-forwarder will assume the connection or server is bad and
    # will connect to a server chosen at random from the servers list.
    "timeout": 15
  }
-------------------------------------------------------------------------------------

The equivalent in Filebeat would look like this:


[source,yaml]
-------------------------------------------------------------------------------------
output:
  logstash:
    enabled: true

    # The list of downstream Logstash servers. <1>
    hosts:
      - localhost:5043

    tls: <2>
      # The path to your SSL client certificate.
      certificate: ./logstash-forwarder.crt

      # The path to your SSL client certificate key.
      certificate_key: ./logstash-forwarder.key

      # The path to your trusted SSL CA file. This is used
      # to authenticate your downstream server.
      certificate_authorities:
        - ./logstash-forwarder.crt

      # Network timeout in seconds.
      timeout: 15
-------------------------------------------------------------------------------------

<1> When multiple hosts are defined, the default behavior in Filebeat is to
    pick a random host for new connections, similar to the Logstash Forwarder
    behavior. Filebeat can optionally do load balancing. For more details, see the
    {libbeat}/configuration.html#loadbalance[loadbalance] configuration option.
<2> Note that if the `tls` section is missing, then TLS is disabled. TLS is
    automatically enabled when you add the `tls` section. For more information about
    specific configuration options, see
    {libbeat}/configuration.html#configuration-output-tls[TLS Options].


[[changed-configuration-options]]
==== Changed Configuration File Options

With the refactoring of the configuration file, the following options were removed or renamed:

[cols="2*", options="header"]
|===
|Config Option
|Action

|`deadTime`
|`deadTime` was renamed to `ignore_older`. Filebeat keeps the files that it’s reading open until they are older than the timespan specified by `ignore_older`. If a file is changed, Filebeat reopens it.

|`netTimeout`
|`netTimeout` was removed and is replaced by the `timeout` option in libbeat.

|`log-to-syslog` and `syslog`
|Both options were removed and replaced by logging options in libbeat.
|===

For more information about these options, see {libbeat}/configuration.html[Beats configuration options].


==== A Complete Example

Let's see a simple, but complete example of a Logstash Forwarder configuration
and its equivalent for Filebeat.

Logstash Forwarder configuration:

[source,json]
-------------------------------------------------------------------------------------
{
  "files": [
    {
      "paths": [
        "/var/log/*.log"
      ],
      "fields": {
        "type": "syslog",
        "service": "test01"
      }
    }
  ],
  "network": {
    "servers": [ "localhost:5043" ],
  }
}
-------------------------------------------------------------------------------------

Filebeat configuration:

[source,yaml]
-------------------------------------------------------------------------------------
filebeat:
  prospectors:
    -
      paths:
        - "/var/log/*.log"
      document_type: syslog
      fields:
        service: test01
output:
  elasticsearch:
    enabled: true
    hosts: ["http://localhost:5043"]
-------------------------------------------------------------------------------------

=== Changes to Command Line Options

Most command line options available in Logstash Forwarder have been removed and
migrated to config file options. The only mandatory command line option for
running  Filebeat is  `-c` followed by the path to the config file. If you used command line
options with Logstash Forwarder, make sure that you add your options to the
configuration file. For naming changes, see <<renamed-options>>.

Filebeat does provide command line options that are common to all Beats. For more details about
these options, see {libbeat}/command-line-options.html[Beats Command Line Options].

[[renamed-options]]
==== Renamed Options

The following command line options have been renamed and moved to the config file. Also see
<<changed-configuration-options>> for a list of configuration file options that were completely removed or
replaced by options specified in libbeat.

[cols="3*", options="header"]
|===
|Command Line Option
|Config File Option
|Description

|`-config`
|`-c` command line option and `config_dir`
|The config option was split into two parts. You use the `-c` command line option to specify the location of the base (required) config file when you start Filebeat. To use additional config files, you specify the `config_dir` configuration option.

The `config_dir` option specifies the path to the directory that contains additional configuration files. This option MUST point to a directory other than the directory where the main Filebeat config file resides.

|`-idle-timeout`
|`idle_timeout`
|`idle_timeout` was moved to the config file and removed as a flag.

|`-spool-size`
|`spool_size`
|`spool_size` was moved to the config file and removed as a flag.

|`-harvester-buff-size`
|`harvester_buffer_size`
|`harvester_buffer_size` was moved to the config file and removed as a flag. You can now configure the buffer size separately for each harvester.

|`-tail`
|`tail_files`
|`tail_files` was moved to the config file and removed as a flag. You can now configure this option separately for each prospector.

|`-cpuProfileFile`
|
|`cpuProfileFile` option was removed. You can use the profiling options of libbeat instead. For more details on profiling, see https://github.com/elastic/libbeat/issues/122.

|`-quiet`
|
|The `quiet` option was removed. Libbeat is now used for logging, so you must use the libbeat {libbeat}/configuration.html#configuration-logging[logging options] instead.


|===


=== Changes in the output fields

In the default configuration, Filebeat structures its output documents a little
different from the Logstash Forwarder. This section discusses the differences
and the options you have in case you want compatibility with the Logstash
Forwarder.

One such difference is that the custom fields (added from the configuration
file) are set top level in Logstash Forwarder but are grouped together under a
`fields` dictionary in Filebeat. If you need the old behavior during the
migration phase, you can use the <<fields-under-root>> configuration option:

[source,yaml]
-------------------------------------------------------------------------------------
filebeat:
  prospectors:
    -
      paths:
        - "/var/log/*.log"
      document_type: syslog
      fields:
        service: test01
      fields_under_root: true
-------------------------------------------------------------------------------------

Another difference is that while the Logstash Forwarder sends the hostname of
the server it is running on in the `host` field, Filebeat uses the
`beat.hostname` field for the same purpose. Because `host` is commonly used in
the Logstash plugin ecosystem, the Beats input plugin automatically copies
`beat.hostname` into `host`.

Similarly, the `file` field was renamed to `source`. If you rely on this field
being named `file` you can rename it by using the mutate filter in Logstash. For
example:

[source,plain]
-------------------------------------------------------------------------------------
filter {
    mutate {
        rename => {
            "source" => "file"
        }
    }
}
-------------------------------------------------------------------------------------

Finally, the `line` field was removed. This field wasn't correct in case of
restarts and making it correct would require a performance penalty. We recommend
using the `offset` field instead.

=== Other Changes

The following list of implementation changes should not affect your experience migrating
from Logstash Forwarder, but you should be aware of the changes. Please post GitHub issues
if you notice any regressions from Logstash Forwarder.

==== Packaging

The packaging process for Filebeat uses the Beats infrastructure, so some
aspects of packaging, such as the init scripts, are different from
Logstash Forwarder. Please post GitHub issues if you hit any issues
with the new packages.

One notable change is the name of the registry file. The name varies depending on the package
type:

 * `.filebeat` for `.tar.gz` and `.tgz` archives
 * `/usr/lib/filebeat/registry` for DEB and RPM packages
 * `c:\ProgramData\filebeat\registry` for the Windows zip file

==== Publisher Improvements

Behind the scenes, Filebeat uses a sightly improved protocol for communicating
with Logstash.

==== TLS Is Off by Default

If you follow the section on migrating the configuration, you will have TLS
enabled. However, you must be aware that if the `tls` section is missing from the
configuration file, Filebeat uses an unencrypted connection to talk to Logstash.

==== Logging

Filebeat uses libbeat logging and can also log to rotating files instead of syslog.

{'fraction_non_alphanumeric': 0.1424922353487562, 'fraction_numerical': 0.0024962990914632376, 'mean_word_length': 2.714901876213069, 'pattern_counts': {'":': 49, '<': 39, '<?xml version=': 0, '>': 43, 'https://': 10, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '776371', 'n_tokens_mistral': 8682, 'n_tokens_neox': 8380, 'n_words': 4101}