<commit_before>
# Talk architecture

This document is complemented by talk's source code, but doesn't require it. Definitions of data structures, code snippets and diagrams will be provided where useful.


## Overview

Talk was Urbit's first big user-facing application, and continues to enjoy a prominent role in the Urbit landscape as the primary messaging bus.

To be clear, we make a distinction between "talk the platform" and "talk the chat application". The latter makes use of the former for most of its functionality, and serves as an example of things that can be built on top of the platform. (Also see the small Twitter clone, [feed](http://urbit.org/fora/posts/~2017.4.12..21.14.00..fe17~/).)  
This document, however, focusses on the platform. In talk's case, we refer to this "server" (or `guardian`, in Urbit terminology) as the *broker*. It manages an identity's messages, subscriptions, and more.


## Core concepts & functionality

To begin to gain an understanding of how the broker functions, let's look at its state mold.

```
++  state                                               :>  broker state
  $:  stories/(map knot story)                          :<  conversations
      readers/(map bone (set knot))                     :<  our message readers
      ::  ...                                           ::
  ==                                                    ::
++  story                                               :>  wire content
  $:  grams/(list telegram)                             :<  all messages
      locals/group                                      :<  local presence
      remotes/(map partner group)                       :<  remote presence
      shape/config                                      :<  configuration
      mirrors/(map circle config)                       :<  remote config
      followers/(map bone river)                        :<  subscribers
      ::  ...                                           ::
  ==                                                    ::
```

Aside from keeping track of `readers`, the different applications that use the broker, it also stores a map of stories.

### Stories

Stories are the primary driver behind the broker. Whether you're hosting a group chat, or just want a place to aggregate messages into, a `story` will be created for it. They contain all data they need to be complete, from messages to metadata.

Looking carefully at that metadata, stories seem to store both "local" and "remote" versions. This is to facilitate the "aggregate messages" use-case mentioned above, and core to how subscriptions function. Looking at the `config` structure, we see the following attribute:

```
src/(set partner)                                       :<  pulls from
```

Every story has a set of sources from which it receives data. If there's nothing in this set, the story will only receive messages people explicitly target towards it. If there *are* sources in that set, then their message will get forwarded to the story automatically. Presences and configurations for those sources are kept track of as well.<commit_msg>Added table of contents<commit_after>
# Talk architecture

This document is complemented by talk's source code, but doesn't require it. Definitions of data structures, code snippets and diagrams will be provided where useful.

## Table of contents

* [Overview](#overview)
* [Core concepts & functionality](#core-concepts--functionality)
  * [Stories](#stories)
  * [Partners](#partners)
  * [Access control](#access-control)
  * [Federation](#federation)
* [Interfaces for applications](#interfaces-for-applications)
  * [Interactions](#interactions)
  * [Subscriptions](#subscriptions)
* [Communication between brokers](#communication-between-brokers)
  * [Interactions](#interactions-1)
  * [Subscriptions](#subscriptions-1)
* [Broker implementation](#broker-implementation)
  * [Subscriptions](#subscriptions-2)
  * [Reports](#reports)
  * [Messaging](#messaging)
* [The future](#the-future)
  * [Architectural issues](#architectural-issues)
  * [Features and functionality](#features-and-functionality)
* [Further reading](#further-reading)

## Overview

Talk was Urbit's first big user-facing application, and continues to enjoy a prominent role in the Urbit landscape as the primary messaging bus.

To be clear, we make a distinction between "talk the platform" and "talk the chat application". The latter makes use of the former for most of its functionality, and serves as an example of things that can be built on top of the platform. (Also see the small Twitter clone, [feed](http://urbit.org/fora/posts/~2017.4.12..21.14.00..fe17~/).)  
This document, however, focusses on the platform. In talk's case, we refer to this "server" (or `guardian`, in Urbit terminology) as the *broker*. It manages an identity's messages, subscriptions, and more.


## Core concepts & functionality

To begin to gain an understanding of how the broker functions, let's look at its state model.

```
++  state                                               :>  broker state
  $:  stories/(map knot story)                          :<  conversations
      readers/(map bone (set knot))                     :<  our message readers
      ::  ...                                           ::
  ==                                                    ::
++  story                                               :>  wire content
  $:  grams/(list telegram)                             :<  all messages
      locals/group                                      :<  local presence
      remotes/(map partner group)                       :<  remote presence
      shape/config                                      :<  configuration
      mirrors/(map circle config)                       :<  remote config
      followers/(map bone river)                        :<  subscribers
      ::  ...                                           ::
  ==                                                    ::
```

Aside from keeping track of `readers`, the different applications that use the broker, it also stores a map of stories.

### Stories

Stories are the primary driver behind the broker. Whether you're hosting a group chat, or just want a place to aggregate messages into, a `story` will be created for it. They contain all data they need to be complete, from messages to metadata.

Looking carefully at that metadata, stories seem to store both "local" and "remote" versions. This is to facilitate the "aggregate messages" use-case mentioned above, and core to how subscriptions function. Looking at the `config` structure, we see the following attribute:

```
src/(set partner)                                       :<  pulls from
```

Every story has a set of sources from which it receives data. If there's nothing in this set, the story will only receive messages people explicitly target towards it. If there *are* sources in that set, then their message will get forwarded to the story automatically. Presences and configurations for those sources are kept track of as well.
{'fraction_non_alphanumeric': 0.07770556654422153, 'fraction_numerical': 0.004662333992653292, 'mean_word_length': 2.165921288014311, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 38, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7428337', 'n_tokens_mistral': 1691, 'n_tokens_neox': 1569, 'n_words': 758}