<commit_before># About ConfigKit

ConfigKit is a configuration management system that lets you define configuration keys and store configuration values, and plays well with JSON.

**Table of contents:**

<!-- MarkdownTOC depth=3 autolink="true" bracket="round" -->

- [Motivations](#motivations)
  - [Configuration flow from high-level to low-level with a minimum of repeated code](#configuration-flow-from-high-level-to-low-level-with-a-minimum-of-repeated-code)
  - [Unifying configuration management](#unifying-configuration-management)
- [Status inside the Passenger codebase](#status-inside-the-passenger-codebase)
- [Features and class overview](#features-and-class-overview)
  - [ConfigKit::Schema](#configkitschema)
  - [ConfigKit::Store](#configkitstore)
  - [Translators](#translators)
- [Using the schema](#using-the-schema)
  - [Defining the schema](#defining-the-schema)
  - [Defining default values](#defining-default-values)
  - [Defining custom validators](#defining-custom-validators)
  - [Inspecting the schema](#inspecting-the-schema)
- [Using the store](#using-the-store)
  - [Putting data in the store](#putting-data-in-the-store)
    - [Updating data](#updating-data)
  - [Unregistered keys are ignored](#unregistered-keys-are-ignored)
  - [Deleting data](#deleting-data)
  - [Fetching data](#fetching-data)
  - [Default values](#default-values)
  - [Inspecting all data](#inspecting-all-data)
- [Putting it all together: synchronous version](#putting-it-all-together-synchronous-version)
  - [SecurityChecker example: a configurable, low-level component](#securitychecker-example-a-configurable-low-level-component)
  - [DnsQuerier example: a low-level component with post-configuration application operations](#dnsquerier-example-a-low-level-component-with-post-configuration-application-operations)
  - [Downloader example: a high-level component that combines subcomponents](#downloader-example-a-high-level-component-that-combines-subcomponents)
    - [The special problem of conflicting overlapping configuration names and translation](#the-special-problem-of-conflicting-overlapping-configuration-names-and-translation)
    - [Code example](#code-example)
  - [Main function example](#main-function-example)
- [Putting it all together: asynchronous version](#putting-it-all-together-asynchronous-version)

<!-- /MarkdownTOC -->

## Motivations

### Configuration flow from high-level to low-level with a minimum of repeated code

Passenger is architected as a tree of components. At the top of the tree there are high-level components that implement Passenger business logic, while at the bottom of the tree there are low-level, generic components that are reusable outside Passenger. Higher-level components can encapsulate lower-level components, but not the other way around.

Almost every component can be configured individually. This is fairly easy, but when you put all the components together in Passenger's tree-like architecture, you end up with the following challenge: how do higher-level components expose the configuration of their encapsulated subcomponents, and how do higher-level components allow configuring their encapsulated subcomponents, without repeating a lot of code? Suppose that you have a class A that encapsulates B:

~~~c++
class B {
public:
    int option1;
    int option2;
    int option3;
};

class A {
private:
    B b;
    int option4;

public:
    void setOption1(int val) { b.option1 = val; }
    void setOption2(int val) { b.option2 = val; }
    void setOption3(int val) { b.option3 = val; }
    void setOption4(int val) { b.option4 = val; }
};
~~~

If B has a lot of configuration options, then writing all these setters (and we haven't even looked at getters yet) becomes tiresome quickly. Worse, if there is a component X that encapsulates A, that all the setters in A have to be repeated in X too. And every time you add an option to B, you have to modify A and X too.

How do you prevent having to manually write so much code for exposing and configuring all the options exposed by lower-level components? One possible answer consists of the following aspects:

 - Configuration should be stored in some kind of key-value data structure, e.g. a Json::Value, instead of in individual class members for each option.
 - There must exist the concept of a configuration schema, and this must be introspectable during runtime, so that higher-level components can manage subcomponents' configuration through automation.

ConfigKit implements these aspects, and also provides various other useful features in the area of configuration management. ConfigKit is built around the concept of introspection during *runtime*. This is not the only approach: static introspection and code generation is another valid approach (which is used by some components in Passenger), but this is outside ConfigKit's scope.

### Unifying configuration management

Another challenge pertains having different ways to configure a component. Should components be configured using getters and setters for each option? Or should they have a single method that accepts a struct (or some kind of key-value map) that specifies multiple options? Should components be configurable at all after construction. It would be great if we have a unified answer for all components in Passenger.

ConfigKit provides this unified answer:

 * Components should be configurable at any time after construction. Motivation: we want to allow configuration reloads without having to restart Passenger.
 * Components should be configurable through a single method that accepts a JSON object. Motivations:

    - Applying a configuration change is potentially expensive, so you want to batch multiple configuration changes into a single transaction.
    - Configuration parameters often originate from an I/O channel. Using a struct to store all the different configuration options results in a lot of repeated code, like this:

          config.foo1 = userInput["foo1"];
          config.foo2 = userInput["foo2"];
          config.foo3 = userInput["foo3"];

      So we want some kind of key-value data structure. JSON is a pretty popular format nowadays, and is likely to be the format used in the I/O channel, so may as well optimize for JSON.

ConfigKit backs these answers with code that helps you implement these principles, as well as with documented design patterns that provide guidance.

## Status inside the Passenger codebase

At the time of writing (25 Feb 2017), ConfigKit was just introduced, so these practices and patterns documented in this README aren't yet used everywhere throughout the Passenger codebase, but the long-term plan is to adopt these practices/patterns throughout the entire codebase.

## Features and class overview

### ConfigKit::Schema

Everything starts with `ConfigKit::Schema`. This is a class that lets you define a schema of supported configuration keys, their types and other properties like default values. Default values may either be static or dynamically calculated. `ConfigKit::Schema` also allows data validation against the schema.

### ConfigKit::Store

There is also `ConfigKit::Store`. This is a class that stores configuration values in such a way that it respects a schema.  The values supplied to and stored in `ConfigKit::Store` are JSON values (i.e. of the `Json::Value` type), although Store uses the schema to validate that you are actually putting the right JSON types in the Store.

`ConfigKit::Store` also keeps track of which values are explicitly supplied and which ones are not.

### Translators

And finally there is a "translator" class: `ConfigKit::TableTranslator`. The role of translators are described in the section "The special problem of conflicting overlapping configuration names and translation".

## Using the schema

### Defining the schema

Start using ConfigKit by defining a schema. There are two ways to do this. The first one is to simply create ConfigKit::Schema object and adding definitions to it:

~~~c++
ConfigKit::Schema schema;

// A required string key.
schema.add("foo", ConfigKit::STRING_TYPE, ConfigKit::REQUIRED);

// An optional integer key without default value.
schema.add("bar", ConfigKit::FLOAT_TYPE, ConfigKit::OPTIONAL);

// An optional integer key, with default value 123.
schema.add("baz", ConfigKit::INTEGER_TYPE, ConfigKit::OPTIONAL, 123);

// Call this when done, otherwise the object cannot be used yet.
schema.finalize();
~~~

The second one, and the one we recommend, is to subclass ConfigKit::Schema and to add the definitions inside the subclass's constructor. It will become apparent in the section "Putting it all together: synchronous version" why this is the recommended approach.

~~~c++
struct YourSchema: public ConfigKit::Schema {
    YourSchema() {
        using namespace ConfigKit;
        add("foo", STRING_TYPE, REQUIRED);
        add("bar", FLOAT_TYPE, OPTIONAL);
        add("baz", INT_TYPE, OPTIONAL, 123);
        finalize();
    }
};

// Instantiate it:
YourSchema schema;
~~~

Tip: see ConfigKit/Common.h for all supported types and flags.

### Defining default values<commit_msg>ConfigKit: document types and flags<commit_after># About ConfigKit

ConfigKit is a configuration management system that lets you define configuration keys and store configuration values, and plays well with JSON.

**Table of contents:**

<!-- MarkdownTOC depth=3 autolink="true" bracket="round" -->

- [Motivations](#motivations)
  - [Configuration flow from high-level to low-level with a minimum of repeated code](#configuration-flow-from-high-level-to-low-level-with-a-minimum-of-repeated-code)
  - [Unifying configuration management](#unifying-configuration-management)
- [Status inside the Passenger codebase](#status-inside-the-passenger-codebase)
- [Features and class overview](#features-and-class-overview)
  - [ConfigKit::Schema](#configkitschema)
  - [ConfigKit::Store](#configkitstore)
  - [Translators](#translators)
- [Using the schema](#using-the-schema)
  - [Defining the schema](#defining-the-schema)
    - [Types](#types)
    - [Flags](#flags)
  - [Defining default values](#defining-default-values)
  - [Defining custom validators](#defining-custom-validators)
  - [Inspecting the schema](#inspecting-the-schema)
- [Using the store](#using-the-store)
  - [Putting data in the store](#putting-data-in-the-store)
    - [Updating data](#updating-data)
  - [Unregistered keys are ignored](#unregistered-keys-are-ignored)
  - [Deleting data](#deleting-data)
  - [Fetching data](#fetching-data)
  - [Default values](#default-values)
  - [Inspecting all data](#inspecting-all-data)
- [Putting it all together: synchronous version](#putting-it-all-together-synchronous-version)
  - [SecurityChecker example: a configurable, low-level component](#securitychecker-example-a-configurable-low-level-component)
  - [DnsQuerier example: a low-level component with post-configuration application operations](#dnsquerier-example-a-low-level-component-with-post-configuration-application-operations)
  - [Downloader example: a high-level component that combines subcomponents](#downloader-example-a-high-level-component-that-combines-subcomponents)
    - [The special problem of conflicting overlapping configuration names and translation](#the-special-problem-of-conflicting-overlapping-configuration-names-and-translation)
    - [Code example](#code-example)
  - [Main function example](#main-function-example)
- [Putting it all together: asynchronous version](#putting-it-all-together-asynchronous-version)

<!-- /MarkdownTOC -->

## Motivations

### Configuration flow from high-level to low-level with a minimum of repeated code

Passenger is architected as a tree of components. At the top of the tree there are high-level components that implement Passenger business logic, while at the bottom of the tree there are low-level, generic components that are reusable outside Passenger. Higher-level components can encapsulate lower-level components, but not the other way around.

Almost every component can be configured individually. This is fairly easy, but when you put all the components together in Passenger's tree-like architecture, you end up with the following challenge: how do higher-level components expose the configuration of their encapsulated subcomponents, and how do higher-level components allow configuring their encapsulated subcomponents, without repeating a lot of code? Suppose that you have a class A that encapsulates B:

~~~c++
class B {
public:
    int option1;
    int option2;
    int option3;
};

class A {
private:
    B b;
    int option4;

public:
    void setOption1(int val) { b.option1 = val; }
    void setOption2(int val) { b.option2 = val; }
    void setOption3(int val) { b.option3 = val; }
    void setOption4(int val) { b.option4 = val; }
};
~~~

If B has a lot of configuration options, then writing all these setters (and we haven't even looked at getters yet) becomes tiresome quickly. Worse, if there is a component X that encapsulates A, that all the setters in A have to be repeated in X too. And every time you add an option to B, you have to modify A and X too.

How do you prevent having to manually write so much code for exposing and configuring all the options exposed by lower-level components? One possible answer consists of the following aspects:

 - Configuration should be stored in some kind of key-value data structure, e.g. a Json::Value, instead of in individual class members for each option.
 - There must exist the concept of a configuration schema, and this must be introspectable during runtime, so that higher-level components can manage subcomponents' configuration through automation.

ConfigKit implements these aspects, and also provides various other useful features in the area of configuration management. ConfigKit is built around the concept of introspection during *runtime*. This is not the only approach: static introspection and code generation is another valid approach (which is used by some components in Passenger), but this is outside ConfigKit's scope.

### Unifying configuration management

Another challenge pertains having different ways to configure a component. Should components be configured using getters and setters for each option? Or should they have a single method that accepts a struct (or some kind of key-value map) that specifies multiple options? Should components be configurable at all after construction. It would be great if we have a unified answer for all components in Passenger.

ConfigKit provides this unified answer:

 * Components should be configurable at any time after construction. Motivation: we want to allow configuration reloads without having to restart Passenger.
 * Components should be configurable through a single method that accepts a JSON object. Motivations:

    - Applying a configuration change is potentially expensive, so you want to batch multiple configuration changes into a single transaction.
    - Configuration parameters often originate from an I/O channel. Using a struct to store all the different configuration options results in a lot of repeated code, like this:

          config.foo1 = userInput["foo1"];
          config.foo2 = userInput["foo2"];
          config.foo3 = userInput["foo3"];

      So we want some kind of key-value data structure. JSON is a pretty popular format nowadays, and is likely to be the format used in the I/O channel, so may as well optimize for JSON.

ConfigKit backs these answers with code that helps you implement these principles, as well as with documented design patterns that provide guidance.

## Status inside the Passenger codebase

At the time of writing (25 Feb 2017), ConfigKit was just introduced, so these practices and patterns documented in this README aren't yet used everywhere throughout the Passenger codebase, but the long-term plan is to adopt these practices/patterns throughout the entire codebase.

## Features and class overview

### ConfigKit::Schema

Everything starts with `ConfigKit::Schema`. This is a class that lets you define a schema of supported configuration keys, their types and other properties like default values. Default values may either be static or dynamically calculated. `ConfigKit::Schema` also allows data validation against the schema.

### ConfigKit::Store

There is also `ConfigKit::Store`. This is a class that stores configuration values in such a way that it respects a schema.  The values supplied to and stored in `ConfigKit::Store` are JSON values (i.e. of the `Json::Value` type), although Store uses the schema to validate that you are actually putting the right JSON types in the Store.

`ConfigKit::Store` also keeps track of which values are explicitly supplied and which ones are not.

### Translators

And finally there is a "translator" class: `ConfigKit::TableTranslator`. The role of translators are described in the section "The special problem of conflicting overlapping configuration names and translation".

## Using the schema

### Defining the schema

Start using ConfigKit by defining a schema. There are two ways to do this. The first one is to simply create ConfigKit::Schema object and adding definitions to it with `schema.add(name, type, flags, [default value])`:

~~~c++
ConfigKit::Schema schema;

// A required string key.
schema.add("foo", ConfigKit::STRING_TYPE, ConfigKit::REQUIRED);

// An optional integer key without default value.
schema.add("bar", ConfigKit::FLOAT_TYPE, ConfigKit::OPTIONAL);

// An optional integer key, with default value 123.
schema.add("baz", ConfigKit::INTEGER_TYPE, ConfigKit::OPTIONAL, 123);

// Call this when done, otherwise the object cannot be used yet.
schema.finalize();
~~~

The second one, and the one we recommend, is to subclass ConfigKit::Schema and to add the definitions inside the subclass's constructor. It will become apparent in the section "Putting it all together: synchronous version" why this is the recommended approach.

~~~c++
struct YourSchema: public ConfigKit::Schema {
    YourSchema() {
        using namespace ConfigKit;
        add("foo", STRING_TYPE, REQUIRED);
        add("bar", FLOAT_TYPE, OPTIONAL);
        add("baz", INT_TYPE, OPTIONAL, 123);
        finalize();
    }
};

// Instantiate it:
YourSchema schema;
~~~

#### Types

The following types are available:

 * `STRING_TYPE` -- a string.
 * `PASSWORD_TYPE` -- a password string. Unlike `STRING_TYPE`, the value of password fields will be filtered out in the output generated by `ConfigKit::Store::inspect()`. Learn more about `inspect()` in [Using the store -- Inspecting all data](#inspecting-all-data).
 * `INT_TYPE` -- a signed integer.
 * `UINT_TYPE` -- an unsigned integer.
 * `FLOAT_TYPE` -- a floating point number.
 * `BOOL_TYPE` -- a boolean.

#### Flags

 * `REQUIRED` -- this field is required. Mutually exclusive with `OPTIONAL`.
 * `OPTIONAL` -- this field is optional. Mutually exclusive with `REQUIRED`.
 * `CACHE_DEFAULT_VALUE` -- use in combination with [dynamic default values](#defining-default-values). When this flag is set, the value returned by the dynamic value function is cached so that the function won't be called over and over again.
 * `READ_ONLY` -- this field can only be set once. Only the first `ConfigKit::Store::update()` call actually updates the value; subsequent calls won't. Learn more about `update()` in [Using the store -- Putting data in the store](#putting-data-in-the-store).

### Defining default values
{'fraction_non_alphanumeric': 0.08050695263995074, 'fraction_numerical': 0.00348914772435733, 'mean_word_length': 4.584527220630372, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2218811', 'n_tokens_mistral': 5055, 'n_tokens_neox': 4839, 'n_words': 2313}