<commit_before>:plugin: protobuf
:type: codec

///////////////////////////////////////////
START - GENERATED VARIABLES, DO NOT EDIT!
///////////////////////////////////////////
:version: %VERSION%
:release_date: %RELEASE_DATE%
:changelog_url: %CHANGELOG_URL%
:include_path: ../../../../logstash/docs/include
///////////////////////////////////////////
END - GENERATED VARIABLES, DO NOT EDIT!
///////////////////////////////////////////

[id="plugins-{type}s-{plugin}"]

=== Protobuf codec plugin

include::{include_path}/plugin_header.asciidoc[]

==== Description

This codec converts protobuf encoded messages into logstash events and vice versa. It supports the protobuf versions 2 and 3.

The plugin requires the protobuf definitions to be compiled to ruby files.
For protobuf 2 use the ruby-protoc compiler[https://github.com/codekitchen/ruby-protocol-buffers].
For protobuf 3 use the official google protobuf compiler[https://developers.google.com/protocol-buffers/docs/reference/ruby-generated].

The following shows a usage example (protobuf v2) for decoding events from a kafka stream:
[source,ruby]
kafka 
{
 topic_id => "..."
 key_deserializer_class => "org.apache.kafka.common.serialization.ByteArrayDeserializer"
 value_deserializer_class => "org.apache.kafka.common.serialization.ByteArrayDeserializer"
 codec => protobuf 
 {
   class_name => "Animals::Mammals::Unicorn"
   include_path => ['/path/to/protobuf/definitions/UnicornProtobuf.pb.rb']
 }
}

Usage example for protobuf v3:
[source,ruby]
kafka 
{
  topic_id => "..."
  key_deserializer_class => "org.apache.kafka.common.serialization.ByteArrayDeserializer"
  value_deserializer_class => "org.apache.kafka.common.serialization.ByteArrayDeserializer"
  codec => protobuf 
  {
    class_name => "Animals.Mammals.Unicorn"
    include_path => ['/path/to/pb_definitions/Animal_pb.rb', '/path/to/pb_definitions/Unicorn_pb.rb']
    protobuf_version => 3
  }
}


The codec can be used in input and output plugins.
When using the codec in the kafka input plugin please set the deserializer classes as shown above.
When using the codec in an output plugin:

* make sure to include all the desired fields in the protobuf definition, including timestamp. 
  Remove fields that are not part of the protobuf definition from the event by using the mutate filter.
* the @ symbol is currently not supported in field names when loading the protobuf definitions for encoding. Make sure to call the timestamp field "timestamp" 
  instead of "@timestamp" in the protobuf file. Logstash event fields will be stripped of the leading @ before conversion.


[id="plugins-{type}s-{plugin}-options"]
==== Protobuf Codec Configuration Options

[cols="<,<,<",options="header",]
|=======================================================================
|Setting |Input type|Required
| <<plugins-{type}s-{plugin}-class_name>> |<<string,string>>|Yes
| <<plugins-{type}s-{plugin}-include_path>> |<<array,array>>|Yes
|=======================================================================

&nbsp;

[id="plugins-{type}s-{plugin}-class_name"]
===== `class_name` 

  * This is a required setting.
  * Value type is <<string,string>>
  * There is no default value for this setting.

Fully qualified name of the class to decode.
Please note that the module delimiter is different depending on the protobuf version. For protobuf v2, use double colons:
[source,ruby]
class_name => "Animals::Mammals::Unicorn"

For protobuf v3, use single dots:
[source,ruby]
class_name => "Animals.Mammals.Unicorn"

For protobuf v3, you can copy the class name from the Descriptorpool registrations at the bottom of the generated protobuf ruby file. It contains lines like this:
[source,ruby]<commit_msg>Unify documentation; add hints for protobuf version 3<commit_after>:plugin: protobuf
:type: codec

///////////////////////////////////////////
START - GENERATED VARIABLES, DO NOT EDIT!
///////////////////////////////////////////
:version: %VERSION%
:release_date: %RELEASE_DATE%
:changelog_url: %CHANGELOG_URL%
:include_path: ../../../../logstash/docs/include
///////////////////////////////////////////
END - GENERATED VARIABLES, DO NOT EDIT!
///////////////////////////////////////////

[id="plugins-{type}s-{plugin}"]

=== Protobuf codec plugin

include::{include_path}/plugin_header.asciidoc[]

==== Description

This codec converts protobuf encoded messages into logstash events and vice versa. It supports the protobuf versions 2 and 3.

The plugin requires the protobuf definitions to be compiled to ruby files.
For protobuf 2 use the ruby-protoc compiler[https://github.com/codekitchen/ruby-protocol-buffers].
For protobuf 3 use the official google protobuf compiler[https://developers.google.com/protocol-buffers/docs/reference/ruby-generated].

The following shows a usage example (protobuf v2) for decoding events from a kafka stream:
[source,ruby]
kafka 
{
 topic_id => "..."
 key_deserializer_class => "org.apache.kafka.common.serialization.ByteArrayDeserializer"
 value_deserializer_class => "org.apache.kafka.common.serialization.ByteArrayDeserializer"
 codec => protobuf 
 {
   class_name => "Animals::Mammals::Unicorn"
   include_path => ['/path/to/protobuf/definitions/UnicornProtobuf.pb.rb']
 }
}

Usage example for protobuf v3:
[source,ruby]
kafka 
{
  topic_id => "..."
  key_deserializer_class => "org.apache.kafka.common.serialization.ByteArrayDeserializer"
  value_deserializer_class => "org.apache.kafka.common.serialization.ByteArrayDeserializer"
  codec => protobuf 
  {
    class_name => "Animals.Mammals.Unicorn"
    include_path => ['/path/to/pb_definitions/Animal_pb.rb', '/path/to/pb_definitions/Unicorn_pb.rb']
    protobuf_version => 3
  }
}


The codec can be used in input and output plugins.
When using the codec in the kafka input plugin please set the deserializer classes as shown above.
When using the codec in an output plugin:

* make sure to include all the desired fields in the protobuf definition, including timestamp. 
  Remove fields that are not part of the protobuf definition from the event by using the mutate filter.
* the @ symbol is currently not supported in field names when loading the protobuf definitions for encoding. Make sure to call the timestamp field "timestamp" 
  instead of "@timestamp" in the protobuf file. Logstash event fields will be stripped of the leading @ before conversion.


[id="plugins-{type}s-{plugin}-options"]
==== Protobuf Codec Configuration Options

[cols="<,<,<",options="header",]
|=======================================================================
|Setting |Input type|Required
| <<plugins-{type}s-{plugin}-class_name>> |<<string,string>>|Yes
| <<plugins-{type}s-{plugin}-include_path>> |<<array,array>>|Yes
| <<plugins-{type}s-{plugin}-protobuf_version>> |<<number,number>>|Yes
|=======================================================================

&nbsp;

[id="plugins-{type}s-{plugin}-class_name"]
===== `class_name` 

  * This is a required setting.
  * Value type is <<string,string>>
  * There is no default value for this setting.

Fully qualified name of the class to decode.
Please note that the module delimiter is different depending on the protobuf version. For protobuf v2, use double colons:
[source,ruby]
class_name => "Animals::Mammals::Unicorn"

For protobuf v3, use single dots:
[source,ruby]
class_name => "Animals.Mammals.Unicorn"

For protobuf v3, you can copy the class name from the Descriptorpool registrations at the bottom of the generated protobuf ruby file. It contains lines like this:
[source,ruby]
{'fraction_non_alphanumeric': 0.18422796554009277, 'fraction_numerical': 0.002783300198807157, 'mean_word_length': 4.044117647058823, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 57, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3055503', 'n_tokens_mistral': 2207, 'n_tokens_neox': 2070, 'n_words': 680}