<commit_before># CT Log Deployment (Manual)

This document describes the individual steps and components involved in the
deployment of a Trillian-based CT Log.  These steps will gradually build up a
system as shown in the diagram below.

<img src="images/DeploymentFull.png" width="650">

The text here describes the general approach, and details key options for
the various binaries involved, but does not give full command-lines.
To see complete details in a machine-executable format (which is therefore
less likely to fall out of date), please consult the various testing
shell scripts:

 - [`trillian/integration/demo-script.sh`](../integration/demo-script.sh)
   runs a simple single-instance test, corresponding to the first few sections
   of this document.
 - [`trillian/integration/ct_integration_test.sh`](../integration/ct_integration_test.sh)
   runs a short integration test against a CT Log system.
 - [`trillian/integration/ct_hammer_test.sh`](../integration/ct_hammer_test.sh)
   runs a continuous integration test against a CT Log system.
 - Both of the previous two tests allow for a more complex CT Log system, using
   the shell functions in
   [`trillian/integration/ct_functions.sh`](../integration/ct_functions.sh)
   and
   [github.com/google/trillian/integration/functions.sh](https://github.com/google/trillian/blob/master/integration/functions.sh).
     - Multiple instances of each component can be invoked.
     - A simple etcd instance can be enabled (by setting the `ETCD_DIR`
       environment variable to the location of etcd binaries)
     - A Prometheus instance can be enabled (by setting the `PROMETHEUS_DIR`
       environment variable to the location of the Prometheus binary).

**Cross-check**s are given throughout the document to allow
confirmation of successful setup.

 - [Data Storage](#data-storage)
 - [Trillian Services](#trillian-services)
 - [Tree Provisioning](#tree-provisioning)
 - [CT Personality](#ct-personality)
    - [Key Generation](#key-generation)
    - [CA Certificates](#ca-certificates)
    - [CTFE Configuration](#ctfe-configuration)
    - [CTFE Start-up](#ctfe-start-up)
 - [Distribution](#distribution)
 - [Primary Signer Election](#primary-signer-election)
 - [Load Balancing](#load-balancing)
 - [Monitoring](#monitoring)
 - [DoS Protection](#dos-protection)
 - [Service Discovery](#service-discovery)

## Data Storage

Data storage for the logged certificates is the heart of a CT Log.  The Trillian
project has an internal storage interface that allows a variety of different
implementations.

This document uses the
[MySQL storage implementation](https://github.com/google/trillian/blob/master/storage/mysql),
which is set up according to the
[instructions in the Trillian repo](https://github.com/google/trillian#mysql-setup);
these instructions configure the Trillian database according to its
[core schema file](http://github.com/google/trillian/blob/master/storage/mysql/schema/storage.sql).


**Cross-check**: At this point, manually connecting to the MySQL database should
succeed:
```
% mysql --host=127.0.0.1 --port=3306 --user=root --database=test
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 764
Server version: 10.1.29-MariaDB-6 Debian rodete

Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [test]> show tables;
+-------------------+
| Tables_in_test    |
+-------------------+
| LeafData          |
| MapHead           |
| MapLeaf           |
| SequencedLeafData |
| Subtree           |
| TreeControl       |
| TreeHead          |
| Trees             |
| Unsequenced       |
+-------------------+
9 rows in set (0.00 sec)

MariaDB [test]> exit
Bye
```

The setup so far is shown as:

<img src="images/Deployment1DB.png" width="650">


## Trillian Services

The next step is to deploy two Trillian processes, the log server and the log
signer.  These binaries are not specific to CT or to WebPKI certificates; they
provide a general mechanism for transparently recording data in a Merkle tree.

The log server (`github.com/google/trillian/server/trillian_log_server`) exposes
a gRPC interface that allows various primitives for querying and adding to the
underlying Merkle tree.  These operations are translated into operations on the
storage layer, which are SQL operations in this example.

 - The `--mysql_uri` option indicates where the MySQL database is available.
 - The `--rpc_endpoint` option for the log server indicates the port that the
   gRPC methods are available on.

However, add operations are not immediately incorporated into the Merkle tree.
Instead, pending add operations are queued up and a separate process, the log
signer (`github.com/google/trillian/server/trillian_log_signer`) periodically
reads pending entries from the queue.  The signer gives these entries unique,
monotonicallly increasing, sequence numbers and incorporates them into the
Merkle tree.

 - The `--mysql_uri` option indicates where the MySQL database is available.
 - The `--sequencer_interval`, `--batch_size` and `--num_sequencers` options
   provide control over the timing and batching of sequencing operations.
 - The `--force-master` option allows the signer to assume that it is the only
   instance running (more on this [later](#primary-signer-election)).
 - The `--logtostderr` option emits more debug logging, which is helpful while
   getting a deployment running.

<img src="images/Deployment2Trillian.png" width="650">


## Tree Provisioning

The Trillian system is *multi-tenant*: a single Trillian system can support
multiple independent Merkle trees.  However, this means that our particular tree
for holding Web PKI certificates needs to be provisioned in the system.

The `github.com/google/trillian/cmd/createtree` tool performs this provisioning
operation, and emits a **tree ID** that needs to be recorded for later in the
deployment process.

 - The `--admin_server` option for `createtree` indicates the address
   (host:port) that tree creation gRPC requests should be sent to; it should
   match the `--rpc_endpoint` for the log server.
 - Other options (`--pem_key_path`, `--pem_key_password`,
   `--signature_algorithm`) allow control of the private keys used for
   *Trillian*'s signatures over the log's contents.  **Note** that this
   is not the private key used for signing externally-visible content of the CT
   log (see [below](#key-generation)). (Instead, these signatures allow for
   future deployment scenarios where the Trillian services are operated by a
   different entity than the CT personality, and so a trust boundary between
   the two is needed.)


**Cross-check**: Once a new tree has been provisioned, the debug logging for the running
`trillian_log_signer` should include a mention of the new tree.
```
I1011 16:44:16.160069  176101 log_operation_manager.go:210] create master election goroutine for 2385931157013381257
I1011 16:44:17.160875  176101 log_operation_manager.go:246] now acting as master for 1 / 1, master for: <log-2385931157013381257>
```

<commit_msg>docs: Describe the max_root_duration flag (#684)<commit_after># CT Log Deployment (Manual)

This document describes the individual steps and components involved in the
deployment of a Trillian-based CT Log.  These steps will gradually build up a
system as shown in the diagram below.

<img src="images/DeploymentFull.png" width="650">

The text here describes the general approach, and details key options for
the various binaries involved, but does not give full command-lines.
To see complete details in a machine-executable format (which is therefore
less likely to fall out of date), please consult the various testing
shell scripts:

 - [`trillian/integration/demo-script.sh`](../integration/demo-script.sh)
   runs a simple single-instance test, corresponding to the first few sections
   of this document.
 - [`trillian/integration/ct_integration_test.sh`](../integration/ct_integration_test.sh)
   runs a short integration test against a CT Log system.
 - [`trillian/integration/ct_hammer_test.sh`](../integration/ct_hammer_test.sh)
   runs a continuous integration test against a CT Log system.
 - Both of the previous two tests allow for a more complex CT Log system, using
   the shell functions in
   [`trillian/integration/ct_functions.sh`](../integration/ct_functions.sh)
   and
   [github.com/google/trillian/integration/functions.sh](https://github.com/google/trillian/blob/master/integration/functions.sh).
     - Multiple instances of each component can be invoked.
     - A simple etcd instance can be enabled (by setting the `ETCD_DIR`
       environment variable to the location of etcd binaries)
     - A Prometheus instance can be enabled (by setting the `PROMETHEUS_DIR`
       environment variable to the location of the Prometheus binary).

**Cross-check**s are given throughout the document to allow
confirmation of successful setup.

 - [Data Storage](#data-storage)
 - [Trillian Services](#trillian-services)
 - [Tree Provisioning](#tree-provisioning)
 - [CT Personality](#ct-personality)
    - [Key Generation](#key-generation)
    - [CA Certificates](#ca-certificates)
    - [CTFE Configuration](#ctfe-configuration)
    - [CTFE Start-up](#ctfe-start-up)
 - [Distribution](#distribution)
 - [Primary Signer Election](#primary-signer-election)
 - [Load Balancing](#load-balancing)
 - [Monitoring](#monitoring)
 - [DoS Protection](#dos-protection)
 - [Service Discovery](#service-discovery)

## Data Storage

Data storage for the logged certificates is the heart of a CT Log.  The Trillian
project has an internal storage interface that allows a variety of different
implementations.

This document uses the
[MySQL storage implementation](https://github.com/google/trillian/blob/master/storage/mysql),
which is set up according to the
[instructions in the Trillian repo](https://github.com/google/trillian#mysql-setup);
these instructions configure the Trillian database according to its
[core schema file](http://github.com/google/trillian/blob/master/storage/mysql/schema/storage.sql).


**Cross-check**: At this point, manually connecting to the MySQL database should
succeed:
```
% mysql --host=127.0.0.1 --port=3306 --user=root --database=test
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 764
Server version: 10.1.29-MariaDB-6 Debian rodete

Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [test]> show tables;
+-------------------+
| Tables_in_test    |
+-------------------+
| LeafData          |
| MapHead           |
| MapLeaf           |
| SequencedLeafData |
| Subtree           |
| TreeControl       |
| TreeHead          |
| Trees             |
| Unsequenced       |
+-------------------+
9 rows in set (0.00 sec)

MariaDB [test]> exit
Bye
```

The setup so far is shown as:

<img src="images/Deployment1DB.png" width="650">


## Trillian Services

The next step is to deploy two Trillian processes, the log server and the log
signer.  These binaries are not specific to CT or to WebPKI certificates; they
provide a general mechanism for transparently recording data in a Merkle tree.

The log server (`github.com/google/trillian/server/trillian_log_server`) exposes
a gRPC interface that allows various primitives for querying and adding to the
underlying Merkle tree.  These operations are translated into operations on the
storage layer, which are SQL operations in this example.

 - The `--mysql_uri` option indicates where the MySQL database is available.
 - The `--rpc_endpoint` option for the log server indicates the port that the
   gRPC methods are available on.

However, add operations are not immediately incorporated into the Merkle tree.
Instead, pending add operations are queued up and a separate process, the log
signer (`github.com/google/trillian/server/trillian_log_signer`) periodically
reads pending entries from the queue.  The signer gives these entries unique,
monotonicallly increasing, sequence numbers and incorporates them into the
Merkle tree.

 - The `--mysql_uri` option indicates where the MySQL database is available.
 - The `--sequencer_interval`, `--batch_size` and `--num_sequencers` options
   provide control over the timing and batching of sequencing operations.
 - The `--force-master` option allows the signer to assume that it is the only
   instance running (more on this [later](#primary-signer-election)).
 - The `--logtostderr` option emits more debug logging, which is helpful while
   getting a deployment running.

<img src="images/Deployment2Trillian.png" width="650">


## Tree Provisioning

The Trillian system is *multi-tenant*: a single Trillian system can support
multiple independent Merkle trees.  However, this means that our particular tree
for holding Web PKI certificates needs to be provisioned in the system.

The `github.com/google/trillian/cmd/createtree` tool performs this provisioning
operation, and emits a **tree ID** that needs to be recorded for later in the
deployment process.

 - The `--admin_server` option for `createtree` indicates the address
   (host:port) that tree creation gRPC requests should be sent to; it should
   match the `--rpc_endpoint` for the log server.
 - Other options (`--pem_key_path`, `--pem_key_password`,
   `--signature_algorithm`) allow control of the private keys used for
   *Trillian*'s signatures over the log's contents.  **Note** that this
   is not the private key used for signing externally-visible content of the CT
   log (see [below](#key-generation)). (Instead, these signatures allow for
   future deployment scenarios where the Trillian services are operated by a
   different entity than the CT personality, and so a trust boundary between
   the two is needed.)
 - The `--max_root_duration` option should be set to less than the log's MMD.
   This ensures that the log periodically produces a fresh STH even if there are
   no updates. Make sure to leave a reasonable safety margin (e.g., 23h59m seems
   risky for MMD=24h, while 1h or 12h feels safe).


**Cross-check**: Once a new tree has been provisioned, the debug logging for the running
`trillian_log_signer` should include a mention of the new tree.
```
I1011 16:44:16.160069  176101 log_operation_manager.go:210] create master election goroutine for 2385931157013381257
I1011 16:44:17.160875  176101 log_operation_manager.go:246] now acting as master for 1 / 1, master for: <log-2385931157013381257>
```


{'fraction_non_alphanumeric': 0.0915175522469608, 'fraction_numerical': 0.018849883895642672, 'mean_word_length': 4.019883441892355, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 15, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7228735', 'n_tokens_mistral': 4311, 'n_tokens_neox': 3919, 'n_words': 1673}