<commit_before>IronBee Changes                                                   {#CHANGES}
===============

IronBee v0.8.0
--------------

**Deprecations**

* The 'ac' module (deprecated in 0.7.0) has been removed.

**Build**

* Modules and plugins are now installed into libexec instead of lib.

**Predicate**

* A new rule injection system, Predicate, was added.  Predicate provides a
  functional approach to writing rules and is designed to make rule logic
  composition and reuse easier and provide performance benefits.  See
  `predicate/predicate.md` for an overview.

**Engine**

* Operators have been overhauled.  They are now entirely independent of the
  rule engine and can be called by any code.  The API has been significantly
  simplified as well.

* Added `ib_module_config_initialize()`.  This function provides an
  alternative approach to initializing module configuration data.  The
  original (still existent) method is to store an initial configuration data
  value and length in the module structure.  The new approach is to call
  this function in the module initialization handler.

* Modules now provide their static `ib_module_t` as a `const ib_module_t *`
  instead of an `ib_module_t *`.  The engine makes its own copy rather than
  reusing the static.  This change allows simultaneous use of modules by
  multiple engines.

* The context hook functions have been removed from the module initialization
  structures, and have been replaced with context hook registration functions.

* As part of the provider removal project, the matcher provider was
  removed -- nothing was using it; the parser provider was removed -- modhtp
  now provides parsing via engine hooks; and the audit log provider was
  removed -- audit logging is now contained entirely within core.

* Added indexed data fields which allows modules to register data field keys
  that are known at configuration time for rapid lookup.  Most pre-defined
  fields have been set as indexed.  Module authors that create fields should
  consider registering those keys as indexed during initialization via:
  `ib_data_register_indexed(ib_engine_data_configuration_get(ib), "my key")`.
  Custom data fields can be indexed via the `InitVarIndexed` and
  `InitCollectionIndexed` directives.

* Transformations have been overhauled: output flags have been removed;
  callback data is now the final argument; input flags have been changed to
  a single bool; added accessors; `ib_tfn_transform()` has been renamed to `ib_tfn_execute()` and now handles lists properly; separated creation and
  registration similar to operators.

**Util**

* Add external iterator support for hash.  See `ib_hash_iterator*`.
  `ib_hash_get()` and `ib_hash_get_ex()` now support NULL for the value
  argument to allow for membership tests.

* Hash keys are now consistently `const char *` instead of a mix of
  `const char *` and `const void *`.

* Hash now supports callback data for key hashing and equality.

**IronBee++**

* IronBee++ includes full support for operators and adds an optional
  functional based interface that can significantly simplify operator
  definitions, especially in C++11.

* Module delegates are now constructed on module initialization rather than
  load.  As a result, the `initialize()` method is no longer called.  This
  change makes it easier to write modules that function in multiple engine
  environments.

* Added static `Module::with_name(engine, name)` to acquire a module of a
  given name, i.e., `ib_engine_module_get()`.

* `convert_exception()` now only requires a ConstEngine instead of an Engine.

* Added `IronBee::Hash<T>`.

* Exceptions can now have a transaction or configuration parser attached to
  them (`errinfo_configuration_parser` and `errinfo_transaction`) which will
  be used to improve the log message.  Also, logging can be prevented by not
  attaching an `errinfo_what`.

**CLIPP**

* Added `-e path` which causes `clipp` to handle consumer errors differently.
  On the first error, `clipp` will write the last input to `path` in protobuf
  format and exit.

* Added `@clipp_print` operator to IronBee modifier and consumer which outputs
  its argument and input to standard out.

**Other**

* Added `example_modules` directory with example modules.

* Major test organization overhaul.  The `tests` directory now holds only
  common test code. Module tests now in `modules/tests`, engine tests in
  `engine/tests`, and utility tests in `util/tests`.

* CLIPP based tests now use more meaningful filenames.  Filenames for the
  same test now use the same identifier.  Numbers in identifiers are
  incremental rather than random and identifiers now include the name of the
  test.

* CLIPP based tests no longer require modhtp.

* Various clean up and bug fixes.

IronBee v0.7.0
--------------

**Deprecations**

* The `ac` module is deprecated.  It will emit a warning if loaded.

**Documentation**

* Syntax added to all operators.

* Preface added.

**Build**

* libhtp is now configured as part of configure stage rather than build
  stage.  In addition, libhtp will make use of any configure options.  Use
  ``./configure --help=recursive`` to see libhtp specific configure options.

* Extensive cleanup regarding use of `CFLAGS`, `CXXFLAGS`, etc.  Those
  variables are now respected and may be specified at configure or make time.
  Several configure options used to control those variables have been removed
  in favor of directly setting them.

* Warning settings changed to `-Wall -Wextra`.  `-Werror` will be enabled on
  newer compilers (any clang or gcc 4.6 or later).

* Build system now compatible with automake 1.13.  In addition, IronBee will
  take advantage of the new parallel test harness if automake 1.13 is used.

* Configure now checks for `ruby`, `gem`, and `ruby_protobuf` gem if C++ code
  is enabled.

* Configure now checks for `libcurl` and `yajl` and only enabled RIAK support
  if present.

* The Clang Thread Sanitizer is now supported.  However, a few tests cause
  false positives or break the thread sanitizer.  Pass
  `--enable-thread-sanitizer-workaround` to `configure` to disable these
  tests.  See the thread sanitizer documentation for how to enable it.

* Several unneeded checks removed.

**Configuration**

* Added `InspectionEngineOptions` to set the defaults for the inspection
  engine.

* Added `IncludeIfExists` directive to include a file only if it exists and is
  accessible.  This allows for inclusion of optional files.

**Engine**

* `ib_tx_t::data` has changed from a generic hash to an array indexed by
  module index.  This change puts it in line with per-module engine data and
  per-module context data.  `ib_tx_data_set()` and `ib_tx_data_get()` can be
  used by modules to read/write this data.

* Added RIAK kvstore.

* Several fixes to dynamic collections in the DPI.

* Lua rule support moved from the rule component to the Lua module.  The rules
  component gained support for modules to register arbitrary external rule
  drivers (see `ib_rule_register_external_driver()`), which the Lua module
  now uses.

* Data fields were cleaned up and refactored.  Notable changes to the public
  API include:

  * All capture related data routines have been moved to capture.h and begin
    `ib_capture` instead of `ib_data`.
  * Several transformation functions have been moved to transformation.h and
	to `ib_tfn` from `ib_data`.
  * All remaining data routines are now in `data.h` instead of `engine.h`.
  * All public `dpi` fields are now `data`.
  * To disambiguate, previous module data code has moved from `data` to
    `module_data`.

* Added managed collections which allow TX collections to be automatically
  populated / persisted.

* Added a core collection manager which takes one or more name=value pairs,
  and will automatically populate a collection with the specified name/value
  pairs.

* Added a core collection manager which takes a JSON formatted file,
  will automatically populate a collection from the content of the file.
  Optionally, the collection can persist to the collection, as well.

* Removed backward compatibility support for the `ip=` and `port=` parameters
  to the Hostname directive.

* Removed backward compatibility support for `=+` to the `SetVar` action.

* Logging overhaul.

  * For servers, use `ib_log_set_logger` and `ib_log_set_loglevel` to setup
    custom loggers.  Provider interface is gone.
  * For configuration writers, use `Log` and `LogLevel`; `DebugLog` and
    `DebugLogLevel` are gone.  `LogHandler` is also gone.
  * For module writers, use `ib_log_vex` instead of `ib_log_vex_ex`.  Include
    `log.h` for logging routines.
  * For engine developers, logging code is now in `log.c` and `log.h`.

* LogEvents has been refactored to use a direct API rather than a provider.

* Added utility functions that wrap YAJL, using it to decode JSON into an
  `ib_list_t` of `ib_field_t` pointers, and to encode an `ib_list_t` of
  `ib_field_t` pointers into JSON.

* Added `@match` and `@imatch` operators to do string-in-set calculations.

* Added `@istreq`, a string insensitive version of `@streq`.

* Support for unparsed data has been removed from IronBee.

  * The `ib_conndata_t` type has been removed.
  * `ib_conn_data_create()` has been removed.
  * The `ib_state_conndata_hook_fn_t` function typedef has been removed.
  * The `ib_hook_conndata_register()` and `ib_hook_conndata_unregister()
`    functions have been removed.
  * The `ib_state_notify_conn_data_in()` and `ib_state_notify_conn_data_out()
`    functions have been removed.

* The libhtp library has been updated to 0.5.

* All memory pool routines now assert fail instead of returning `EINVAL` when
  passed NULLs for require arguments.

**Modules**

* The `pcre` module has been updated to use the new transaction data API.

* The `pcre` module `dfa` operator now supports captures.

* Added a 'persist' module, which implements a collection manager that can
  populate and persist a collection using a file-system kvstore.

* Added a 'fast' module which supports rapid selection of rules to evaluate
  based on Aho-Corasick patterns.  See below and `fast/fast.html`.

* Added a module implementing libinjection functionality to aid in detecting
  SQL injection. This module exposes the `normalizeSqli` and the
  `normalizeSqliFold` transformations as well as the `is_sqli` operator.

* Added a module implementing <NAME>'s sqltfn library for normalizing
  SQL to aid in detecting SQL injection. This module exposes the
  `normalizeSqlPg` transformation.

* The `htp` module has been vastly reworked to work properly with libhtp 0.5.

**Fast**

* Added a variety of support for the fast rule system (the fast module
  described above is the runtime component).  Support includes utilities to
  suggest fast patterns for rules and for easy generation of the fast automata
  needed by the fast module.  See above and `fast/fast.html`.

**IronBee++**

* Moved catch, throw, and data support from internals to public.  These
  routines are not needed if you only use IronBee++ APIs but are very useful
  when accessing the IronBee C API from C++.

* Fixed bug with adding to `List<T>` where `T` was a `ConstX` IronBee++ class.

**Automata**

* Intermediate format and Eudoxus now support arbitrary automata metadata in
  the form of key-value pairs.  All command line generators include an
  `Output-Type` metadata key with value set to the output type as defined by
  `ee`.  `ee` now defaults to using this metadata to determine output type.
  This changes increments the Eudoxus format version and, as such, is not
  compatible with compiled automata from earlier versions.

* Eudoxus output callbacks are now passed the engine.

* Added `ia_eudoxus_all_outputs()` to iterate through every output in an
  automata.  `ee -L` can be used to do this from the command line.

* Added '\\iX' to Aho Corasick patterns which matches upper case of X and
  lower case of X for any X in A-Za-z.

* Added '\$' to Aho Corasick patterns which matches CR or NL.

* Added union support to Aho Corasick patterns, e.g., `[A-Q0-5]`.

**Clipp**

* All generators except `pb` now produced parsed events.  Use `@unparse` to
  get the previous behavior.  But note that IronBee no longer supports
  unparsed events.

* Several tests have been added, including a randomized test of IronBee in
  both single and multithreaded mode (`test_holistic`).

* The parse modifier now generates a complete set of events even if some of them are data less.  For example, if no headers are present provided in
  connection data, clipp will still produce a `REQUEST_HEADERS` event; before
  this change it would not.

**Other**

* The old CLI (ibcli) has been removed.

* Removed FTRACE code.

* Various bug fixes and cleanup.

IronBee v0.6.0
--------------

**Build**

* IronBee++ and CLIPP are now built by default.  Use `--disable-cpp` to
  prevent.

* Build system now handles boost and libnids libraries better.  New
  `--with-boost-suffix` configuration option.

* Removed a number of unnecessary checks in configure.

* Included libhtp source, so this is no longer required.

**Engine**

* Enhanced support for buffering request/response data, including
  runtime support via the setflag action.

* Added initial support for persistent data. (see:
  `include/ironbee/kvstore.h`)

* Partial progress towards rework of configuration state transitions.
  Currently implicit.  Next version should be gone completely.

* Events can now be suppressed by setting the `suppress` field.

* Directory creation (`ib_util_mkpath`) rewritten.

**Rules, Configuration and Logging**

* Enhanced rule engine diagnostics logging (`RuleEngineLogData`,
  `RuleEngineLogLevel`).

* Simplified Hostname directive by moving IP/Port to a new
  Service directive.  For 0.6.x only, support the "ip=" and "port="
  parameters to the Hostname directive for backward compatibility with 0.5.x.

* Enhanced configuration context selection, which now takes Site,
  Service, Hostname and Location into account.

* Added an `InitVar` directive to set custom fields at config time.

* `SetVar` `=+` operator changed to `+=`.  Also added `-=` and `*=`.  For
  0.6.x only, support `=+` for backward compatibility with 0.5.x.

* Added floating point field type; removed unsigned field type.  Note that
  floating point values do not support eq and ne.

* The `ne` operator now correctly compares numbers.

* Initial support for implicit type conversions in operators.

* Fixed `pmf` operator so that relative filenames are based on
  config file location vs CWD.

* Enhanced PCRE matching to support setting limits.

* `AuditLogFileMode` now works.

* Default of `AuditEngine` is now `RelevantOnly`.

* Cleaned up audit log format, removing event level action and adding
  transaction level action, message, tags and threat level.

**Lua**

* Updated luajit code to v2.0.0.

* Enhanced Lua rule API with more access to internals.

**Modules**

* Enhanced GeoIP module to use O1/O01 country codes when
  lookups fail.

**Servers**

* Added support for regexp based header editing.

* Rewrote Apache httpd server module for httpd 2.4.

**Automata**

* Added IronAutomata framework for building, modifying, and executing automata
  (see: `automata/doc/example.md`).  Currently works as stand alone library
  but is not integrated into IronBee.

**CLIPP**

* CLIPP manual updated. (see: `clipp/clipp.md`)

* CLIPP tests now provide more information about failures.

**IronBee++**

* Support for new site API.

* Support for new float field type.

**Documentation**

* Added CHANGES file.

* Many manual updates.

* Doxygen dependency calculation fixed.  `make doxygen` in `docs` should now
  run only if files have changed.

* Removed long deprecated `fulldocs` doxygen.  Use `external` or `internal`
  instead.

* Updated to doxygen 1.8.1.

**Other**

* Various bug fixes and code cleanup.


<commit_msg>CHANGES.md: Update.<commit_after>IronBee Changes                                                   {#CHANGES}
===============

IronBee v0.8.0
--------------

**Deprecations**

* The 'ac' module (deprecated in 0.7.0) has been removed.

**Build**

* Modules and plugins are now installed into libexec instead of lib.

**Predicate**

* A new rule injection system, Predicate, was added.  Predicate provides a
  functional approach to writing rules and is designed to make rule logic
  composition and reuse easier and provide performance benefits.  See
  `predicate/predicate.md` for an overview.

**Engine**

* Operators have been overhauled.  They are now entirely independent of the
  rule engine and can be called by any code.  The API has been significantly
  simplified as well.

* Added `ib_module_config_initialize()`.  This function provides an
  alternative approach to initializing module configuration data.  The
  original (still existent) method is to store an initial configuration data
  value and length in the module structure.  The new approach is to call
  this function in the module initialization handler.

* Modules now provide their static `ib_module_t` as a `const ib_module_t *`
  instead of an `ib_module_t *`.  The engine makes its own copy rather than
  reusing the static.  This change allows simultaneous use of modules by
  multiple engines.

* The context hook functions have been removed from the module initialization
  structures, and have been replaced with context hook registration functions.

* As part of the provider removal project, the matcher provider was
  removed -- nothing was using it; the parser provider was removed -- modhtp
  now provides parsing via engine hooks; and the audit log provider was
  removed -- audit logging is now contained entirely within core.

* Added indexed data fields which allows modules to register data field keys
  that are known at configuration time for rapid lookup.  Most pre-defined
  fields have been set as indexed.  Module authors that create fields should
  consider registering those keys as indexed during initialization via:
  `ib_data_register_indexed(ib_engine_data_configuration_get(ib), "my key")`.
  Custom data fields can be indexed via the `InitVarIndexed` and
  `InitCollectionIndexed` directives.

* Transformations have been overhauled: output flags have been removed;
  callback data is now the final argument; input flags have been changed to
  a single bool; added accessors; `ib_tfn_transform()` has been renamed to `ib_tfn_execute()` and now handles lists properly; separated creation and
  registration similar to operators.

**Util**

* Add external iterator support for hash.  See `ib_hash_iterator*`.
  `ib_hash_get()` and `ib_hash_get_ex()` now support NULL for the value
  argument to allow for membership tests.

* Hash keys are now consistently `const char *` instead of a mix of
  `const char *` and `const void *`.

* Hash now supports callback data for key hashing and equality.

**IronBee++**

* IronBee++ includes full support for operators and adds an optional
  functional based interface that can significantly simplify operator
  definitions, especially in C++11.

* Module delegates are now constructed on module initialization rather than
  load.  As a result, the `initialize()` method is no longer called.  This
  change makes it easier to write modules that function in multiple engine
  environments.

* Added static `Module::with_name(engine, name)` to acquire a module of a
  given name, i.e., `ib_engine_module_get()`.

* `convert_exception()` now only requires a ConstEngine instead of an Engine.

* Added `IronBee::Hash<T>`.

* Exceptions can now have a transaction or configuration parser attached to
  them (`errinfo_configuration_parser` and `errinfo_transaction`) which will
  be used to improve the log message.  Also, logging can be prevented by not
  attaching an `errinfo_what`.

* IronBee++ includes full support for transformations.

**CLIPP**

* Added `-e path` which causes `clipp` to handle consumer errors differently.
  On the first error, `clipp` will write the last input to `path` in protobuf
  format and exit.

* Added `@clipp_print` operator to IronBee modifier and consumer which outputs
  its argument and input to standard out.

**Other**

* Added `example_modules` directory with example modules.

* Major test organization overhaul.  The `tests` directory now holds only
  common test code. Module tests now in `modules/tests`, engine tests in
  `engine/tests`, and utility tests in `util/tests`.

* CLIPP based tests now use more meaningful filenames.  Filenames for the
  same test now use the same identifier.  Numbers in identifiers are
  incremental rather than random and identifiers now include the name of the
  test.

* CLIPP based tests no longer require modhtp.

* Various clean up and bug fixes.

IronBee v0.7.0
--------------

**Deprecations**

* The `ac` module is deprecated.  It will emit a warning if loaded.

**Documentation**

* Syntax added to all operators.

* Preface added.

**Build**

* libhtp is now configured as part of configure stage rather than build
  stage.  In addition, libhtp will make use of any configure options.  Use
  ``./configure --help=recursive`` to see libhtp specific configure options.

* Extensive cleanup regarding use of `CFLAGS`, `CXXFLAGS`, etc.  Those
  variables are now respected and may be specified at configure or make time.
  Several configure options used to control those variables have been removed
  in favor of directly setting them.

* Warning settings changed to `-Wall -Wextra`.  `-Werror` will be enabled on
  newer compilers (any clang or gcc 4.6 or later).

* Build system now compatible with automake 1.13.  In addition, IronBee will
  take advantage of the new parallel test harness if automake 1.13 is used.

* Configure now checks for `ruby`, `gem`, and `ruby_protobuf` gem if C++ code
  is enabled.

* Configure now checks for `libcurl` and `yajl` and only enabled RIAK support
  if present.

* The Clang Thread Sanitizer is now supported.  However, a few tests cause
  false positives or break the thread sanitizer.  Pass
  `--enable-thread-sanitizer-workaround` to `configure` to disable these
  tests.  See the thread sanitizer documentation for how to enable it.

* Several unneeded checks removed.

**Configuration**

* Added `InspectionEngineOptions` to set the defaults for the inspection
  engine.

* Added `IncludeIfExists` directive to include a file only if it exists and is
  accessible.  This allows for inclusion of optional files.

**Engine**

* `ib_tx_t::data` has changed from a generic hash to an array indexed by
  module index.  This change puts it in line with per-module engine data and
  per-module context data.  `ib_tx_data_set()` and `ib_tx_data_get()` can be
  used by modules to read/write this data.

* Added RIAK kvstore.

* Several fixes to dynamic collections in the DPI.

* Lua rule support moved from the rule component to the Lua module.  The rules
  component gained support for modules to register arbitrary external rule
  drivers (see `ib_rule_register_external_driver()`), which the Lua module
  now uses.

* Data fields were cleaned up and refactored.  Notable changes to the public
  API include:

  * All capture related data routines have been moved to capture.h and begin
    `ib_capture` instead of `ib_data`.
  * Several transformation functions have been moved to transformation.h and
	to `ib_tfn` from `ib_data`.
  * All remaining data routines are now in `data.h` instead of `engine.h`.
  * All public `dpi` fields are now `data`.
  * To disambiguate, previous module data code has moved from `data` to
    `module_data`.

* Added managed collections which allow TX collections to be automatically
  populated / persisted.

* Added a core collection manager which takes one or more name=value pairs,
  and will automatically populate a collection with the specified name/value
  pairs.

* Added a core collection manager which takes a JSON formatted file,
  will automatically populate a collection from the content of the file.
  Optionally, the collection can persist to the collection, as well.

* Removed backward compatibility support for the `ip=` and `port=` parameters
  to the Hostname directive.

* Removed backward compatibility support for `=+` to the `SetVar` action.

* Logging overhaul.

  * For servers, use `ib_log_set_logger` and `ib_log_set_loglevel` to setup
    custom loggers.  Provider interface is gone.
  * For configuration writers, use `Log` and `LogLevel`; `DebugLog` and
    `DebugLogLevel` are gone.  `LogHandler` is also gone.
  * For module writers, use `ib_log_vex` instead of `ib_log_vex_ex`.  Include
    `log.h` for logging routines.
  * For engine developers, logging code is now in `log.c` and `log.h`.

* LogEvents has been refactored to use a direct API rather than a provider.

* Added utility functions that wrap YAJL, using it to decode JSON into an
  `ib_list_t` of `ib_field_t` pointers, and to encode an `ib_list_t` of
  `ib_field_t` pointers into JSON.

* Added `@match` and `@imatch` operators to do string-in-set calculations.

* Added `@istreq`, a string insensitive version of `@streq`.

* Support for unparsed data has been removed from IronBee.

  * The `ib_conndata_t` type has been removed.
  * `ib_conn_data_create()` has been removed.
  * The `ib_state_conndata_hook_fn_t` function typedef has been removed.
  * The `ib_hook_conndata_register()` and `ib_hook_conndata_unregister()
`    functions have been removed.
  * The `ib_state_notify_conn_data_in()` and `ib_state_notify_conn_data_out()
`    functions have been removed.

* The libhtp library has been updated to 0.5.

* All memory pool routines now assert fail instead of returning `EINVAL` when
  passed NULLs for require arguments.

**Modules**

* The `pcre` module has been updated to use the new transaction data API.

* The `pcre` module `dfa` operator now supports captures.

* Added a 'persist' module, which implements a collection manager that can
  populate and persist a collection using a file-system kvstore.

* Added a 'fast' module which supports rapid selection of rules to evaluate
  based on Aho-Corasick patterns.  See below and `fast/fast.html`.

* Added a module implementing libinjection functionality to aid in detecting
  SQL injection. This module exposes the `normalizeSqli` and the
  `normalizeSqliFold` transformations as well as the `is_sqli` operator.

* Added a module implementing <NAME>'s sqltfn library for normalizing
  SQL to aid in detecting SQL injection. This module exposes the
  `normalizeSqlPg` transformation.

* The `htp` module has been vastly reworked to work properly with libhtp 0.5.

**Fast**

* Added a variety of support for the fast rule system (the fast module
  described above is the runtime component).  Support includes utilities to
  suggest fast patterns for rules and for easy generation of the fast automata
  needed by the fast module.  See above and `fast/fast.html`.

**IronBee++**

* Moved catch, throw, and data support from internals to public.  These
  routines are not needed if you only use IronBee++ APIs but are very useful
  when accessing the IronBee C API from C++.

* Fixed bug with adding to `List<T>` where `T` was a `ConstX` IronBee++ class.

**Automata**

* Intermediate format and Eudoxus now support arbitrary automata metadata in
  the form of key-value pairs.  All command line generators include an
  `Output-Type` metadata key with value set to the output type as defined by
  `ee`.  `ee` now defaults to using this metadata to determine output type.
  This changes increments the Eudoxus format version and, as such, is not
  compatible with compiled automata from earlier versions.

* Eudoxus output callbacks are now passed the engine.

* Added `ia_eudoxus_all_outputs()` to iterate through every output in an
  automata.  `ee -L` can be used to do this from the command line.

* Added '\\iX' to Aho Corasick patterns which matches upper case of X and
  lower case of X for any X in A-Za-z.

* Added '\$' to Aho Corasick patterns which matches CR or NL.

* Added union support to Aho Corasick patterns, e.g., `[A-Q0-5]`.

**Clipp**

* All generators except `pb` now produced parsed events.  Use `@unparse` to
  get the previous behavior.  But note that IronBee no longer supports
  unparsed events.

* Several tests have been added, including a randomized test of IronBee in
  both single and multithreaded mode (`test_holistic`).

* The parse modifier now generates a complete set of events even if some of them are data less.  For example, if no headers are present provided in
  connection data, clipp will still produce a `REQUEST_HEADERS` event; before
  this change it would not.

**Other**

* The old CLI (ibcli) has been removed.

* Removed FTRACE code.

* Various bug fixes and cleanup.

IronBee v0.6.0
--------------

**Build**

* IronBee++ and CLIPP are now built by default.  Use `--disable-cpp` to
  prevent.

* Build system now handles boost and libnids libraries better.  New
  `--with-boost-suffix` configuration option.

* Removed a number of unnecessary checks in configure.

* Included libhtp source, so this is no longer required.

**Engine**

* Enhanced support for buffering request/response data, including
  runtime support via the setflag action.

* Added initial support for persistent data. (see:
  `include/ironbee/kvstore.h`)

* Partial progress towards rework of configuration state transitions.
  Currently implicit.  Next version should be gone completely.

* Events can now be suppressed by setting the `suppress` field.

* Directory creation (`ib_util_mkpath`) rewritten.

**Rules, Configuration and Logging**

* Enhanced rule engine diagnostics logging (`RuleEngineLogData`,
  `RuleEngineLogLevel`).

* Simplified Hostname directive by moving IP/Port to a new
  Service directive.  For 0.6.x only, support the "ip=" and "port="
  parameters to the Hostname directive for backward compatibility with 0.5.x.

* Enhanced configuration context selection, which now takes Site,
  Service, Hostname and Location into account.

* Added an `InitVar` directive to set custom fields at config time.

* `SetVar` `=+` operator changed to `+=`.  Also added `-=` and `*=`.  For
  0.6.x only, support `=+` for backward compatibility with 0.5.x.

* Added floating point field type; removed unsigned field type.  Note that
  floating point values do not support eq and ne.

* The `ne` operator now correctly compares numbers.

* Initial support for implicit type conversions in operators.

* Fixed `pmf` operator so that relative filenames are based on
  config file location vs CWD.

* Enhanced PCRE matching to support setting limits.

* `AuditLogFileMode` now works.

* Default of `AuditEngine` is now `RelevantOnly`.

* Cleaned up audit log format, removing event level action and adding
  transaction level action, message, tags and threat level.

**Lua**

* Updated luajit code to v2.0.0.

* Enhanced Lua rule API with more access to internals.

**Modules**

* Enhanced GeoIP module to use O1/O01 country codes when
  lookups fail.

**Servers**

* Added support for regexp based header editing.

* Rewrote Apache httpd server module for httpd 2.4.

**Automata**

* Added IronAutomata framework for building, modifying, and executing automata
  (see: `automata/doc/example.md`).  Currently works as stand alone library
  but is not integrated into IronBee.

**CLIPP**

* CLIPP manual updated. (see: `clipp/clipp.md`)

* CLIPP tests now provide more information about failures.

**IronBee++**

* Support for new site API.

* Support for new float field type.

**Documentation**

* Added CHANGES file.

* Many manual updates.

* Doxygen dependency calculation fixed.  `make doxygen` in `docs` should now
  run only if files have changed.

* Removed long deprecated `fulldocs` doxygen.  Use `external` or `internal`
  instead.

* Updated to doxygen 1.8.1.

**Other**

* Various bug fixes and code cleanup.



{'fraction_non_alphanumeric': 0.07053487644666875, 'fraction_numerical': 0.002940256490459806, 'mean_word_length': 3.9822346891070595, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1907686', 'n_tokens_mistral': 9129, 'n_tokens_neox': 8762, 'n_words': 4329}