<commit_before># Trillian: General Transparency

[![Go Report Card](https://goreportcard.com/badge/github.com/google/trillian)](https://goreportcard.com/report/github.com/google/trillian)
[![codecov](https://codecov.io/gh/google/trillian/branch/master/graph/badge.svg?token=QwofUwmvAs)](https://codecov.io/gh/google/trillian)
[![GoDoc](https://godoc.org/github.com/google/trillian?status.svg)](https://godoc.org/github.com/google/trillian)
[![Slack Status](https://img.shields.io/badge/Slack-Chat-blue.svg)](https://gtrillian.slack.com/)

 - [Overview](#overview)
 - [Support](#support)
 - [Using the Code](#using-the-code)
     - [MySQL Setup](#mysql-setup)
     - [Integration Tests](#integration-tests)
 - [Working on the Code](#working-on-the-code)
     - [Rebuilding Generated Code](#rebuilding-generated-code)
     - [Updating Dependencies](#updating-dependencies)
     - [Running Codebase Checks](#running-codebase-checks)
 - [Design](#design)
     - [Design Overview](#design-overview)
     - [Personalities](#personalities)
     - [Log Mode](#log-mode)
 - [Use Cases](#use-cases)
     - [Certificate Transparency Log](#certificate-transparency-log)


## Overview

Trillian is an implementation of the concepts described in the
[Verifiable Data Structures](docs/papers/VerifiableDataStructures.pdf) white paper,
which in turn is an extension and generalisation of the ideas which underpin
[Certificate Transparency](https://certificate-transparency.org).

Trillian implements a [Merkle tree](https://en.wikipedia.org/wiki/Merkle_tree)
whose contents are served from a data storage layer, to allow scalability to
extremely large trees.  On top of this Merkle tree, Trillian provides the
following:

 - An append-only **Log** mode, analogous to the original
   [Certificate Transparency](https://certificate-transparency.org) logs.  In
   this mode, the Merkle tree is effectively filled up from the left, giving a
   *dense* Merkle tree.

Note that Trillian requires particular applications to provide their own
[personalities](#personalities) on top of the core transparent data store
functionality.

[Certificate Transparency (CT)](https://tools.ietf.org/html/rfc6962)
is the most well-known and widely deployed transparency application, and an implementation of CT as a Trillian personality is available in the
[certificate-transparency-go repo](https://github.com/google/certificate-transparency-go/blob/master/trillian).

Other examples of Trillian personalities are available in the
[trillian-examples](https://github.com/google/trillian-examples) repo.


## Support

- Mailing list: https://groups.google.com/forum/#!forum/trillian-transparency
- Slack: https://gtrillian.slack.com/ ([invitation](https://join.slack.com/t/gtrillian/shared_invite/<KEY>))


## Using the Code

**WARNING**: The Trillian codebase is still under development, but the Log mode
is now being used in production by several organizations. We will try to avoid
any further incompatible code and schema changes but cannot guarantee that they
will never be necessary.

The current state of feature implementation is recorded in the
[Feature implementation matrix](docs/Feature_Implementation_Matrix.md).

To build and test Trillian you need:

 - Go 1.17 or later (go 1.17 matches cloudbuild, and is preferred for developers
   that will be submitting PRs to this project).

To run many of the tests (and production deployment) you need:

 - [MySQL](https://www.mysql.com/) or [MariaDB](https://mariadb.org/) to provide
   the data storage layer; see the [MySQL Setup](#mysql-setup) section.

Note that this repository uses Go modules to manage dependencies; Go will fetch
and install them automatically upon build/test.

To fetch the code, dependencies, and build Trillian, run the following:

```bash
export GO111MODULE=auto

git clone https://github.com/google/trillian.git
cd trillian

go build ./...
```

To build and run tests, use:

```bash
go test ./...
```


The repository also includes multi-process integration tests, described in the
[Integration Tests](#integration-tests) section below.


### MySQL Setup

To run Trillian's integration tests you need to have an instance of MySQL
running and configured to:
<commit_msg>Remove export GO111MODULE=auto in README.md (#2773)<commit_after># Trillian: General Transparency

[![Go Report Card](https://goreportcard.com/badge/github.com/google/trillian)](https://goreportcard.com/report/github.com/google/trillian)
[![codecov](https://codecov.io/gh/google/trillian/branch/master/graph/badge.svg?token=QwofUwmvAs)](https://codecov.io/gh/google/trillian)
[![GoDoc](https://godoc.org/github.com/google/trillian?status.svg)](https://godoc.org/github.com/google/trillian)
[![Slack Status](https://img.shields.io/badge/Slack-Chat-blue.svg)](https://gtrillian.slack.com/)

 - [Overview](#overview)
 - [Support](#support)
 - [Using the Code](#using-the-code)
     - [MySQL Setup](#mysql-setup)
     - [Integration Tests](#integration-tests)
 - [Working on the Code](#working-on-the-code)
     - [Rebuilding Generated Code](#rebuilding-generated-code)
     - [Updating Dependencies](#updating-dependencies)
     - [Running Codebase Checks](#running-codebase-checks)
 - [Design](#design)
     - [Design Overview](#design-overview)
     - [Personalities](#personalities)
     - [Log Mode](#log-mode)
 - [Use Cases](#use-cases)
     - [Certificate Transparency Log](#certificate-transparency-log)


## Overview

Trillian is an implementation of the concepts described in the
[Verifiable Data Structures](docs/papers/VerifiableDataStructures.pdf) white paper,
which in turn is an extension and generalisation of the ideas which underpin
[Certificate Transparency](https://certificate-transparency.org).

Trillian implements a [Merkle tree](https://en.wikipedia.org/wiki/Merkle_tree)
whose contents are served from a data storage layer, to allow scalability to
extremely large trees.  On top of this Merkle tree, Trillian provides the
following:

 - An append-only **Log** mode, analogous to the original
   [Certificate Transparency](https://certificate-transparency.org) logs.  In
   this mode, the Merkle tree is effectively filled up from the left, giving a
   *dense* Merkle tree.

Note that Trillian requires particular applications to provide their own
[personalities](#personalities) on top of the core transparent data store
functionality.

[Certificate Transparency (CT)](https://tools.ietf.org/html/rfc6962)
is the most well-known and widely deployed transparency application, and an implementation of CT as a Trillian personality is available in the
[certificate-transparency-go repo](https://github.com/google/certificate-transparency-go/blob/master/trillian).

Other examples of Trillian personalities are available in the
[trillian-examples](https://github.com/google/trillian-examples) repo.


## Support

- Mailing list: https://groups.google.com/forum/#!forum/trillian-transparency
- Slack: https://gtrillian.slack.com/ ([invitation](https://join.slack.com/t/gtrillian/<KEY>jMWJkNzc0MGY1Y2QyNWQyMWM4NzJlOGMxNTZkZGU))


## Using the Code

**WARNING**: The Trillian codebase is still under development, but the Log mode
is now being used in production by several organizations. We will try to avoid
any further incompatible code and schema changes but cannot guarantee that they
will never be necessary.

The current state of feature implementation is recorded in the
[Feature implementation matrix](docs/Feature_Implementation_Matrix.md).

To build and test Trillian you need:

 - Go 1.17 or later (go 1.17 matches cloudbuild, and is preferred for developers
   that will be submitting PRs to this project).

To run many of the tests (and production deployment) you need:

 - [MySQL](https://www.mysql.com/) or [MariaDB](https://mariadb.org/) to provide
   the data storage layer; see the [MySQL Setup](#mysql-setup) section.

Note that this repository uses Go modules to manage dependencies; Go will fetch
and install them automatically upon build/test.

To fetch the code, dependencies, and build Trillian, run the following:

```bash
git clone https://github.com/google/trillian.git
cd trillian

go build ./...
```

To build and run tests, use:

```bash
go test ./...
```


The repository also includes multi-process integration tests, described in the
[Integration Tests](#integration-tests) section below.


### MySQL Setup

To run Trillian's integration tests you need to have an instance of MySQL
running and configured to:

{'fraction_non_alphanumeric': 0.11339233038348083, 'fraction_numerical': 0.004011799410029498, 'mean_word_length': 4.754243041412084, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 40, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6656323', 'n_tokens_mistral': 2593, 'n_tokens_neox': 2371, 'n_words': 789}