<commit_before>---
title: Redis Streams and the Unified Log
published_at: 2017-11-01T17:24:20Z
location: San Francisco
hook: TODO
---

Years ago, LinkedIn [wrote an article about the unified
log][thelog], a useful architectural pattern for services
in a distributed system converge state with one another. It
was a refreshingly novel idea at the time, and still is:
Kafka may be more prevalent in 2017, but most of us are
still gluing components together with little more than
patches and duct tape.

In the log's design, services emit state changes into an
ordered data structure where each new record gets a unique
ID. Unlike a queue, a log is durable across any number of
reads until the log is explicitly truncated.

TODO: Diagram

Consumers track changes in the wider system by consuming
the log. Each one maintains the ID of the last record it
successfully consumed and aims to consume every record at
least once -- no records should be missed. When a consumer
is knocked offline, it looks up the last ID that it
consumed, and continues reading the log from there.

The article above points out that this design is nothing
new -- we've been using logs in various forms in computer
science for decades. A common example is how a Postgres
installation streams changes to its replicas over the WAL
(write-ahead log). Changes in the database's physical
structure are written as records to the WAL, and each
replica reads records and applies them to their own state.
Tuple `123` was added; tuple `124` was updated; tuple `125`
was deleted. The WAL is saved in segments and in a
production environment often uploaded to a service like S3
for durable access and high availability.

## At-least once design (#at-least-once)

On systems powered by a unified log, resilience and
correctness are the name of the game. Consumers should get
every message that a producer sends, and to that end
processes are built to guarantee ***at-least once***
delivery semantics. Messages are usually sent once, but in
cases where there's uncertainty around whether the
transmission occurred, a message will be send as many times
as necessary to be sure.

At-least once delivery is opposed to ***best-effort
delivery*** where messages will be received once under
normal conditions, but may be dropped in extraordinary
cases. It's also opposed by ***exactly-once delivery***; a
classic panacea of distributed systems. Exactly-once
delivery is a difficult guarantee to make, and even if
possible, would add costly overhead to transmission. In
practice, at-least once semantics are fine to handle as
long as consumers are built with consideration for it from
the beginning.

## Redis streams (#redis-streams)

I was happy to hear recently that Redis will soon [1] be
shipping with a new data primitive that will make an
admirable backend for unified logs: [streams][streams].
They're distinct from Redis lists in that records in them
are assigned with addressable IDs, and they lend themselves
well to be being read more than once. They're a
near-perfect analog for what we've been talking about so
far.

about so far. Records are added to one with the new `XADD`
command:

```
> XADD rocket-rides-log * id 123 distance 456.7
1506871964177.0
```

This adds a record to the `rocket-rides-log` stream with
the fields `id = 123` and `distance = 456.7`. Redis
responds with a unique ID for the record within the stream
that's made up of a timestamp and a sequence number (`.0`)
to disambiguate within the millisecond.

The other important command is `XRANGE`, which allows us to
read records from the stream:

```
> XRANGE rocket-rides-log - + COUNT 2
1) 1) 1506871964177.0
   2) 1) "id"
      2) "123"
      3) "distance"
      4) "456.7"
2) 1) 1506872463535.0
   2) 1) "id"
      2) "124"
      3) "distance"
      4) "89.0"
```

The tokens `-` and `+` are special in that they tell Redis
to read from the first available record in the stream and
up to the last available record in the stream respectively.
Either one can be replaced with an ID like
`1506871964177.0` to read from or up to a specific record.
Using this capability allows us to slice out just records
that we haven't consumed yet.

Streams have a variety of other useful features that I
won't cover here, but [the original blog post covers
everything pretty well][streams].

### Versus Kafka (#kakfa)

Kafka is a popular system component that also makes a great
backend for a unified log implementation, and once
everything is in place, probably a better one than Redis
because of its design around high availability and advanced
features.

What's exciting about Redis streams is that they bring an
easy way of building a unified log in a small and/or
inexpensive app. Kafka is infamously difficult to configure
and get running, and is fairly expensive to operation once
you do. Pricing for a small Kafka cluster on Heroku costs
$100 a month and climbs steeply from there. It's temping to
think you can do it more cheaply yourself, but after
factoring in server and personnel costs along with the time
it takes to build a working expertise in the system, it'll
probably cost more.
<commit_msg>Word tweaking<commit_after>---
title: Redis Streams and the Unified Log
published_at: 2017-11-01T17:24:20Z
location: San Francisco
hook: TODO
---

Years ago, LinkedIn [wrote an article about the unified
log][thelog], a useful architectural pattern for services
in a distributed system converge state with one another. It
was a refreshingly novel idea at the time, and still is:
Kafka may be more prevalent in 2017, but most of us are
still gluing components together with little more than
patches and duct tape.

In the log's design, services emit state changes into an
ordered data structure where each new record gets a unique
ID. Unlike a queue, a log is durable across any number of
reads until the log is explicitly truncated.

TODO: Diagram

Consumers track changes in the wider system by consuming
the log. Each one maintains the ID of the last record it
successfully consumed and aims to consume every record at
least once -- no records should be missed. When a consumer
is knocked offline, it looks up the last ID that it
consumed, and continues reading the log from there.

The article above points out that this design is nothing
new -- we've been using logs in various forms in computer
science for decades. A common example is how a Postgres
installation streams changes to its replicas over the WAL
(write-ahead log). Changes in the database's physical
structure are written as records to the WAL, and each
replica reads records and applies them to their own state.
Tuple `123` was added; tuple `124` was updated; tuple `125`
was deleted. The WAL is saved in segments and in a
production environment often uploaded to a service like S3
for durable access and high availability.

## At-least once design (#at-least-once)

On systems powered by a unified log, resilience and
correctness are the name of the game. Consumers should get
every message that a producer sends, and to that end
processes are built to guarantee ***at-least once***
delivery semantics. Messages are usually sent once, but in
cases where there's uncertainty around whether the
transmission occurred, a message will be send as many times
as necessary to be sure.

At-least once delivery is opposed to ***best-effort
delivery*** where messages will be received once under
normal conditions, but may be dropped in extraordinary
cases. It's also opposed by ***exactly-once delivery***; a
classic panacea of distributed systems. Exactly-once
delivery is a difficult guarantee to make, and even if
possible, would add costly overhead to transmission. In
practice, at-least once semantics are fine to handle as
long as consumers are built with consideration for it from
the beginning.

## Redis streams (#redis-streams)

I was happy to hear recently that Redis will soon [1] be
shipping with a new data primitive that will make an
admirable backend for unified logs: [streams][streams].
They're distinct from Redis lists in that records in them
are assigned with addressable IDs, and they lend themselves
well to be being read more than once. They're a
near-perfect analog for what we've been talking about so
far.

about so far. Records are added to one with the new `XADD`
command:

```
> XADD rocket-rides-log * id 123 distance 456.7
1506871964177.0
```

This adds a record to the `rocket-rides-log` stream with
the fields `id = 123` and `distance = 456.7`. Redis
responds with a unique ID for the record within the stream
that's made up of a timestamp and a sequence number (`.0`)
to disambiguate within the millisecond.

The other important command is `XRANGE`, which allows us to
read records from the stream:

```
> XRANGE rocket-rides-log - + COUNT 2
1) 1) 1506871964177.0
   2) 1) "id"
      2) "123"
      3) "distance"
      4) "456.7"
2) 1) 1506872463535.0
   2) 1) "id"
      2) "124"
      3) "distance"
      4) "89.0"
```

The tokens `-` and `+` are special in that they tell Redis
to read from the first available record in the stream and
up to the last available record in the stream respectively.
Either one can be replaced with an ID like
`1506871964177.0` to read from or up to a specific record.
Using this capability allows us to slice out just records
that we haven't consumed yet.

Streams have a variety of other useful features that I
won't cover here, but [the original blog post covers
everything pretty well][streams].

### Versus Kafka (#kakfa)

Kafka is a popular system component that also makes a great
backend for a unified log implementation, and once
everything is in place, probably a better one than Redis
because of its design around high availability and advanced
features.

What's exciting about Redis streams is that they bring an
easy way of building a unified log in a small and/or
inexpensive app. Kafka is infamously difficult to configure
and get running, and is expensive to operate once you do.
Pricing for a small Kafka cluster on Heroku costs $100 a
month and climbs steeply from there. It's temping to think
you can do it more cheaply yourself, but after factoring in
server and personnel costs along with the time it takes to
build a working expertise in the system, it'll probably
cost more.

{'fraction_non_alphanumeric': 0.05029729993176723, 'fraction_numerical': 0.025538551515742275, 'mean_word_length': 4.21606507371632, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1186385', 'n_tokens_mistral': 2945, 'n_tokens_neox': 2647, 'n_words': 1652}