<commit_before>---
title: Redis Streams and the Unified Log
published_at: 2017-11-08T15:15:31Z
location: San Francisco
hook: Building a log-based architecture that's fast,
  efficient, and resilient on the new stream data structure
  in Redis.
hn_link: https://news.ycombinator.com/item?id=15653544
---

Years ago an article came out of LinkedIn [about the
unified log][thelog], a useful architectural pattern for
services in a distributed system share state with one
another. In the log's design, services emit state changes
into an ordered data structure in which each new record
gets a unique ID. Unlike a queue, a log is durable across
any number of reads until it's explicitly truncated.

Consumers track changes in the wider system by reading the
log. Each one maintains the ID of the last record it
successfully consumed and aims to read every record at
least once -- nothing should ever be missed. When a
consumer is knocked offline, it restarts and looks up the
last ID that it saw, and continues reading the log from
there.

The log is ***unified*** because it acts as a master ledger
of state changes in a wider system. All components that are
making critical changes write to it, and all components
that need to track distributed state are subscribed.

LinkedIn's article is sober enough to point out that this
design is nothing new: we've been using logs in various
forms in computer science for decades. [Journaling file
systems][journalfs] use them to protect data against
corruption. Databases use them in places like the
[write-ahead log (WAL)][wal] in Postgres as they stream
changes to their read replicas.

!fig src="/assets/redis-streams/unified-log.svg" caption="The unified log: a producer emits to the stream and consumers read from it."

Even so, the unified log was a refreshingly novel idea when
the article was written, and still is. File systems and
databases use the structure because it's effective, and it
lends itself just as well to distributed architectures.
Kafka is more prevalent in 2017, but most of us are still
gluing components together with patches and duct tape.

Chatty services exchange high-frequency messages back and
forth in a way that's slow (they rely on synchrony),
inefficient (single messages are passed around), and
fragile (every individual message introduces some
possibility of failure). In contrast, the log is
asynchronous, its records are produced and consumed in
batches, and its design builds in resilience at every turn.

## Redis streams (#redis-streams)

This brings us to Redis. I was happy to hear recently that
the project will soon [1] be shipping with a new data
structure that's a perfect foundation for a unified log:
[streams][streams]. Unlike a Redis list, records in a
stream are assigned with addressable IDs and are indexed or
sliced with those IDs instead than a relative offset (i.e.
like `0` or `len() - 1`).This lends itself well to having
multiple consumers reading out of a single stream and
tracking their position within it by persisting the ID of
the last record they read.

The new `XADD` command appends to one:

```
> XADD rocket-rides-log * id 123 distance 456.7
1506871964177.0
```

A record with `id = 123` and `distance = 456.7` is appended
to the stream `rocket-rides-log`. Redis responds with a
unique ID for the record that's made up of a timestamp and
a sequence number (`.0`) to disambiguate entries created
within the same millisecond.

`XRANGE` is `XADD`'s counterpart. It reads a set of records
from a stream:

```
> XRANGE rocket-rides-log - + COUNT 2
1) 1) 1506871964177.0
   2) 1) "id"
      2) "123"
      3) "distance"
      4) "456.7"
2) 1) 1506872463535.0
   2) 1) "id"
      2) "124"
      3) "distance"
      4) "89.0"
```

The tokens `-` and `+` are special in that they tell Redis
to read from the first available record in the stream and
up to the last available record in the stream respectively.
Either one can be replaced with an ID like
`1506871964177.0` to read from or up to a specific record.
Using this capability allows us to slice out just records
that we haven't consumed yet. Specifying `COUNT 2` lets us
bound the number of records read so that we can process the
stream in efficient batches.

### Versus Kafka (#kakfa)

Kafka is a popular system component that also makes a nice
alternative for a unified log implementation; and once
everything is in place, probably a better one compared to
Redis thanks to its sophisticated design around high
availability and other advanced features.

Redis streams aren't exciting for their innovativeness, but
rather than they bring building a unified log architecture
within reach of a small and/or inexpensive app. Kafka is
infamously difficult to configure and get running, and is
expensive to operate once you do. Pricing for a small Kafka
cluster on Heroku costs $100 a month and climbs steeply
from there. It's temping to think you can do it more
cheaply yourself, but after factoring in server and
personnel costs along with the time it takes to build
working expertise in the system, it'll cost more.

Redis on the other hand is probably already in your stack.
Being the Swiss army knife of cloud persistence, it's
useful for a multitude of things including caching, rate
limiting, storing user sessions, etc. Even if you don't
already have it, you can compile it from source and get it
configured and running in all of about thirty seconds.
Dozens of cloud providers (including big ones like AWS)
offer a hosted version.

Once you're operating at serious scale, Kafka might be the
right fit. In the meantime, Redis streams make a great (and
economic) alternative.

### Configuring Redis for durability (#redis-durability)

One highly desirable property of a unified log is that it's
***durable***, meaning that even if its host crashes or
something terrible happens, it doesn't lose information
that producers think they had persisted.

By default Redis is not durable; a sane configuration
choice when it's been used for caching or rate limiting,
but not when it's being used for a log. To make Redis fully<commit_msg>Fix typo in redis-streams.md<commit_after>---
title: Redis Streams and the Unified Log
published_at: 2017-11-08T15:15:31Z
location: San Francisco
hook: Building a log-based architecture that's fast,
  efficient, and resilient on the new stream data structure
  in Redis.
hn_link: https://news.ycombinator.com/item?id=15653544
---

Years ago an article came out of LinkedIn [about the
unified log][thelog], a useful architectural pattern for
services in a distributed system share state with one
another. In the log's design, services emit state changes
into an ordered data structure in which each new record
gets a unique ID. Unlike a queue, a log is durable across
any number of reads until it's explicitly truncated.

Consumers track changes in the wider system by reading the
log. Each one maintains the ID of the last record it
successfully consumed and aims to read every record at
least once -- nothing should ever be missed. When a
consumer is knocked offline, it restarts and looks up the
last ID that it saw, and continues reading the log from
there.

The log is ***unified*** because it acts as a master ledger
of state changes in a wider system. All components that are
making critical changes write to it, and all components
that need to track distributed state are subscribed.

LinkedIn's article is sober enough to point out that this
design is nothing new: we've been using logs in various
forms in computer science for decades. [Journaling file
systems][journalfs] use them to protect data against
corruption. Databases use them in places like the
[write-ahead log (WAL)][wal] in Postgres as they stream
changes to their read replicas.

!fig src="/assets/redis-streams/unified-log.svg" caption="The unified log: a producer emits to the stream and consumers read from it."

Even so, the unified log was a refreshingly novel idea when
the article was written, and still is. File systems and
databases use the structure because it's effective, and it
lends itself just as well to distributed architectures.
Kafka is more prevalent in 2017, but most of us are still
gluing components together with patches and duct tape.

Chatty services exchange high-frequency messages back and
forth in a way that's slow (they rely on synchrony),
inefficient (single messages are passed around), and
fragile (every individual message introduces some
possibility of failure). In contrast, the log is
asynchronous, its records are produced and consumed in
batches, and its design builds in resilience at every turn.

## Redis streams (#redis-streams)

This brings us to Redis. I was happy to hear recently that
the project will soon [1] be shipping with a new data
structure that's a perfect foundation for a unified log:
[streams][streams]. Unlike a Redis list, records in a
stream are assigned with addressable IDs and are indexed or
sliced with those IDs instead than a relative offset (i.e.
like `0` or `len() - 1`).This lends itself well to having
multiple consumers reading out of a single stream and
tracking their position within it by persisting the ID of
the last record they read.

The new `XADD` command appends to one:

```
> XADD rocket-rides-log * id 123 distance 456.7
1506871964177.0
```

A record with `id = 123` and `distance = 456.7` is appended
to the stream `rocket-rides-log`. Redis responds with a
unique ID for the record that's made up of a timestamp and
a sequence number (`.0`) to disambiguate entries created
within the same millisecond.

`XRANGE` is `XADD`'s counterpart. It reads a set of records
from a stream:

```
> XRANGE rocket-rides-log - + COUNT 2
1) 1) 1506871964177.0
   2) 1) "id"
      2) "123"
      3) "distance"
      4) "456.7"
2) 1) 1506872463535.0
   2) 1) "id"
      2) "124"
      3) "distance"
      4) "89.0"
```

The tokens `-` and `+` are special in that they tell Redis
to read from the first available record in the stream and
up to the last available record in the stream respectively.
Either one can be replaced with an ID like
`1506871964177.0` to read from or up to a specific record.
Using this capability allows us to slice out just records
that we haven't consumed yet. Specifying `COUNT 2` lets us
bound the number of records read so that we can process the
stream in efficient batches.

### Versus Kafka (#kakfa)

Kafka is a popular system component that also makes a nice
alternative for a unified log implementation; and once
everything is in place, probably a better one compared to
Redis thanks to its sophisticated design around high
availability and other advanced features.

Redis streams aren't exciting for their innovativeness, but
rather than they bring building a unified log architecture
within reach of a small and/or inexpensive app. Kafka is
infamously difficult to configure and get running, and is
expensive to operate once you do. Pricing for a small Kafka
cluster on Heroku costs $100 a month and climbs steeply
from there. It's tempting to think you can do it more
cheaply yourself, but after factoring in server and
personnel costs along with the time it takes to build
working expertise in the system, it'll cost more.

Redis on the other hand is probably already in your stack.
Being the Swiss army knife of cloud persistence, it's
useful for a multitude of things including caching, rate
limiting, storing user sessions, etc. Even if you don't
already have it, you can compile it from source and get it
configured and running in all of about thirty seconds.
Dozens of cloud providers (including big ones like AWS)
offer a hosted version.

Once you're operating at serious scale, Kafka might be the
right fit. In the meantime, Redis streams make a great (and
economic) alternative.

### Configuring Redis for durability (#redis-durability)

One highly desirable property of a unified log is that it's
***durable***, meaning that even if its host crashes or
something terrible happens, it doesn't lose information
that producers think they had persisted.

By default Redis is not durable; a sane configuration
choice when it's been used for caching or rate limiting,
but not when it's being used for a log. To make Redis fully
{'fraction_non_alphanumeric': 0.05112239882025234, 'fraction_numerical': 0.02162870719318368, 'mean_word_length': 4.321272885789015, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 7, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3377051', 'n_tokens_mistral': 3493, 'n_tokens_neox': 3149, 'n_words': 1938}