<commit_before>Trillian: General Transparency
==============================

[![Build Status](https://travis-ci.org/google/trillian.svg?branch=master)](https://travis-ci.org/google/trillian)
[![Go Report Card](https://goreportcard.com/badge/github.com/google/trillian)](https://goreportcard.com/report/github.com/google/trillian)
[![GoDoc](https://godoc.org/github.com/google/trillian?status.svg)](https://godoc.org/github.com/google/trillian)
[![Slack Status](https://img.shields.io/badge/Slack-Chat-blue.svg)](https://gtrillian.slack.com/)

 - [Overview](#overview)
 - [Support](#support)
 - [Using the Code](#using-the-code)
     - [MySQL Setup](#mysql-setup)
     - [Integration Tests](#integration-tests)
 - [Working on the Code](#working-on-the-code)
     - [Rebuilding Generated Code](#rebuilding-generated-code)
     - [Updating Vendor Code](#updating-vendor-code)
     - [Running Codebase Checks](#running-codebase-checks)
 - [Design](#design)
     - [Design Overview](#design-overview)
     - [Map Mode](#map-mode)
     - [Log Mode](#log-mode)
     - [Personalities](#personalities)
 - [Use Cases](#use-cases)
     - [Certificate Transparency Log](#certificate-transparency-log)
     - [Verifiable Log-Derived Map](#verifiable-log-derived-map)


Overview
--------

Trillian is an implementation of the concepts described in the
[Verifiable Data Structures](docs/VerifiableDataStructures.pdf) white paper,
which in turn is an extension and generalisation of the ideas which underpin
[Certificate Transparency](https://certificate-transparency.org).

Trillian implements a [Merkle tree](https://en.wikipedia.org/wiki/Merkle_tree)
whose contents are served from a data storage layer, to allow scalability to
extremely large trees.  On top of this Merkle tree, Trillian provides two
modes:

 - An append-only **Log** mode, analogous to the original
   [Certificate Transparency](https://certificate-transparency.org) logs.  In
   this mode, the Merkle tree is effectively filled up from the left, giving a
   *dense* Merkle tree.
 - A **Map** mode that allows transparent storage of arbitrary key:value pairs.
   In this mode, the key's hash is used to designate a particular leaf of a deep
   Merkle tree, giving a *sparse* Merkle tree.  (A Trillian Map is an *unordered*
   map; it does not allow enumeration of the Map's keys.)

Note that Trillian requires particular applications to provide their own
[personalities](#personalities) on top of the core transparent data store
functionality; example code for a certificate transparency log and for a
[log-derived map](#verifiable-log-derived-map) are included to help with this.

The code for the CT personality has now been moved to a separate repository and
can be obtained from
[certificate-transparency-go](https://github.com/google/certificate-transparency-go).

Support
-------

- Mailing list: https://groups.google.com/forum/#!forum/trillian-transparency
- Slack: https://gtrillian.slack.com/ ([invitation](https://join.slack.com/t/gtrillian/shared_invite/enQtMzE2MTkxMTQ5ODQxLTY0YTQwMjlmZDNjNWViMTAzZjJjYmVmNDk4ZjQzMDNhMjIxMzMyZTcxOWMwYTRmOGJiOTljMWMzY2EzMTljYmY) - ask on the mailing list if expired)

Using the Code
--------------

**WARNING**: The Trillian codebase is still under development but is now being
used in production by several organizations. We will try to avoid any
further incompatible code and schema changes but cannot guarantee that they
will never be necessary.

To build and test Trillian you need:

 - Go 1.9 or later.

To run many of the tests (and production deployment) you need:

 - [MySQL](https://www.mysql.com/) or [MariaDB](https://mariadb.org/) to provide
   the data storage layer; see the [MySQL Setup](#mysql-setup) section.

Use the standard Go tools to install other dependencies.

```bash
go get github.com/google/trillian
cd $GOPATH/src/github.com/google/trillian
go get -t -u -v ./...
```

To build and run tests, use:

```bash
go test ./...
```

The repository also includes multi-process integration tests, described in the
[Integration Tests](#integration-tests) section below.

### MySQL Setup

To run Trillian's integration tests you need to have an instance of MySQL
running and configured to:

 - listen on the standard MySQL port 3306 (so `mysql --host=127.0.0.1
   --port=3306` connects OK)
 - not require a password for the `root` user

You can then set up the [expected tables](storage/mysql/storage.sql) in a `test`
database like so:<commit_msg>Update README.md (#1122)<commit_after>Trillian: General Transparency
==============================

[![Build Status](https://travis-ci.org/google/trillian.svg?branch=master)](https://travis-ci.org/google/trillian)
[![Go Report Card](https://goreportcard.com/badge/github.com/google/trillian)](https://goreportcard.com/report/github.com/google/trillian)
[![GoDoc](https://godoc.org/github.com/google/trillian?status.svg)](https://godoc.org/github.com/google/trillian)
[![Slack Status](https://img.shields.io/badge/Slack-Chat-blue.svg)](https://gtrillian.slack.com/)

 - [Overview](#overview)
 - [Support](#support)
 - [Using the Code](#using-the-code)
     - [MySQL Setup](#mysql-setup)
     - [Integration Tests](#integration-tests)
 - [Working on the Code](#working-on-the-code)
     - [Rebuilding Generated Code](#rebuilding-generated-code)
     - [Updating Vendor Code](#updating-vendor-code)
     - [Running Codebase Checks](#running-codebase-checks)
 - [Design](#design)
     - [Design Overview](#design-overview)
     - [Map Mode](#map-mode)
     - [Log Mode](#log-mode)
     - [Personalities](#personalities)
 - [Use Cases](#use-cases)
     - [Certificate Transparency Log](#certificate-transparency-log)
     - [Verifiable Log-Derived Map](#verifiable-log-derived-map)


Overview
--------

Trillian is an implementation of the concepts described in the
[Verifiable Data Structures](docs/VerifiableDataStructures.pdf) white paper,
which in turn is an extension and generalisation of the ideas which underpin
[Certificate Transparency](https://certificate-transparency.org).

Trillian implements a [Merkle tree](https://en.wikipedia.org/wiki/Merkle_tree)
whose contents are served from a data storage layer, to allow scalability to
extremely large trees.  On top of this Merkle tree, Trillian provides two
modes:

 - An append-only **Log** mode, analogous to the original
   [Certificate Transparency](https://certificate-transparency.org) logs.  In
   this mode, the Merkle tree is effectively filled up from the left, giving a
   *dense* Merkle tree.
 - A **Map** mode that allows transparent storage of arbitrary key:value pairs.
   In this mode, the key's hash is used to designate a particular leaf of a deep
   Merkle tree, giving a *sparse* Merkle tree.  (A Trillian Map is an *unordered*
   map; it does not allow enumeration of the Map's keys.)

Note that Trillian requires particular applications to provide their own
[personalities](#personalities) on top of the core transparent data store
functionality; example code for a certificate transparency log and for a
[log-derived map](#verifiable-log-derived-map) are included to help with this.

The code for the CT personality has now been moved to a separate repository and
can be obtained from
[certificate-transparency-go](https://github.com/google/certificate-transparency-go).

Support
-------

- Mailing list: https://groups.google.com/forum/#!forum/trillian-transparency
- Slack: https://gtrillian.slack.com/ ([invitation](https://join.slack.com/t/gtrillian/shared_invite/<KEY>) - ask on the mailing list if expired)

Using the Code
--------------

**WARNING**: The Trillian codebase is still under development but is now being
used in production by several organizations. We will try to avoid any
further incompatible code and schema changes but cannot guarantee that they
will never be necessary.

To build and test Trillian you need:

 - Go 1.9 or later.

To run many of the tests (and production deployment) you need:

 - [MySQL](https://www.mysql.com/) or [MariaDB](https://mariadb.org/) to provide
   the data storage layer; see the [MySQL Setup](#mysql-setup) section.

Use the standard Go tools to install other dependencies.

```bash
go get github.com/google/trillian
cd $GOPATH/src/github.com/google/trillian
go get -t -u -v ./...
```

To build and run tests, use:

```bash
go test ./...
```

Note that go seems to sometimes fail to fetch or update all dependencies (as of
v1.10.2), so you may need to manually fetch missing ones, or update all Go
source with:

```bash
go get -u -v all
```

The repository also includes multi-process integration tests, described in the
[Integration Tests](#integration-tests) section below.

### MySQL Setup

To run Trillian's integration tests you need to have an instance of MySQL
running and configured to:

 - listen on the standard MySQL port 3306 (so `mysql --host=127.0.0.1
   --port=3306` connects OK)
 - not require a password for the `root` user

You can then set up the [expected tables](storage/mysql/storage.sql) in a `test`
database like so:
{'fraction_non_alphanumeric': 0.12511061946902655, 'fraction_numerical': 0.00497787610619469, 'mean_word_length': 4.250290360046457, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 4, 'https://': 34, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5637815', 'n_tokens_mistral': 2841, 'n_tokens_neox': 2618, 'n_words': 914}