<commit_before>----
authors: <NAME> <<EMAIL>>
state: predraft
----

<!--
    This Source Code Form is subject to the terms of the Mozilla Public
    License, v. 2.0. If a copy of the MPL was not distributed with this
    file, You can obtain one at http://mozilla.org/MPL/2.0/.
-->

<!--
    Copyright 2015 Joyent, Inc.
-->

# RFD 16 Manta Metering

## Motivation

Manta performs a number of periodic "housekeeping" operations.  These include
daily jobs for:

* metering: reporting how much storage, compute, bandwidth, and other resources
  are used, on a per-user basis, and delivering per-user reports and activity
  logs
* garbage collection: cleaning up objects no longer referenced
* auditing: checking for Manta-level data corruption

Additional jobs that run on-demand are:

* cruft: identify objects unexpectedly left around that are no longer referenced
* rebalance: move object copies between storage nodes

There are a number of problems with the current implementation that require a
lot time spent on a daily basis by both engineering and operations to keep the
system functioning.  This RFD describes those issues, benefits for the current
system, and a suggested approach for resolving them.  The focus is on metering,
which is the source of most issues today, but several other components are
affected.


## Background

The jobs described above are all driven hourly or daily by the "ops" zone, which
is built from the [manta-mola](https://github.com/joyent/manta-mola) repository,
with metering implemented by the
[manta-mackerel](https://github.com/joyent/manta-mackerel) by reference.  The
various jobs and their relationships are described in the [documentation in that
repository](https://github.com/joyent/manta-mola/blob/6f3b46703d9c906ee76ae884755acd377c815b1f/docs/index.md).

The jobs are much more interdependent than it might seem.  The daily process
broadly works like this:

* At 0000Z each day, the async peer in each PostgreSQL shard takes a ZFS
  snapshot of the current dataset, starts a separate PostgreSQL database
  instance using the copy in the snapshot, and dumps this data to a local file.
  The file is then uploaded to Manta for use by the various jobs.  This object
  represents a backup of the whole database for that shard at 0000Z that day.
* Some time later, each shard's backup object is "unpacked": it's transformed
  into a group of objects, one for each database table.  These objects represent
  a backup of the corresponding _table_ for that shard at 0000Z that day.
* Some time later, the daily metering, auditing, and GC jobs kick off, using
  these per-table backups as input.  The processes diverage at this point.

There are a few other jobs that go through different processes:

* There are also a number of hourly metering jobs that do not use these daily
  backups, but rather use log files uploaded hourly from services like Muskie
  and Marlin.<commit_msg>RFD 16: fix typo<commit_after>----
authors: <NAME> <<EMAIL>>
state: predraft
----

<!--
    This Source Code Form is subject to the terms of the Mozilla Public
    License, v. 2.0. If a copy of the MPL was not distributed with this
    file, You can obtain one at http://mozilla.org/MPL/2.0/.
-->

<!--
    Copyright 2015 Joyent, Inc.
-->

# RFD 16 Manta Metering

## Motivation

Manta performs a number of periodic "housekeeping" operations.  These include
daily jobs for:

* metering: reporting how much storage, compute, bandwidth, and other resources
  are used, on a per-user basis, and delivering per-user reports and activity
  logs
* garbage collection: cleaning up objects no longer referenced
* auditing: checking for Manta-level data corruption

Additional jobs that run on-demand are:

* cruft: identify objects unexpectedly left around that are no longer referenced
* rebalance: move object copies between storage nodes

There are a number of problems with the current implementation that require a
lot time spent on a daily basis by both engineering and operations to keep the
system functioning.  This RFD describes those issues, benefits for the current
system, and a suggested approach for resolving them.  The focus is on metering,
which is the source of most issues today, but several other components are
affected.


## Background

The jobs described above are all driven hourly or daily by the "ops" zone, which
is built from the [manta-mola](https://github.com/joyent/manta-mola) repository,
with metering implemented by the
[manta-mackerel](https://github.com/joyent/manta-mackerel) by reference.  The
various jobs and their relationships are described in the [documentation in that
repository](https://github.com/joyent/manta-mola/blob/6f3b46703d9c906ee76ae884755acd377c815b1f/docs/index.md).

The jobs are much more interdependent than it might seem.  The daily process
broadly works like this:

* At 0000Z each day, the async peer in each PostgreSQL shard takes a ZFS
  snapshot of the current dataset, starts a separate PostgreSQL database
  instance using the copy in the snapshot, and dumps this data to a local file.
  The file is then uploaded to Manta for use by the various jobs.  This object
  represents a backup of the whole database for that shard at 0000Z that day.
* Some time later, each shard's backup object is "unpacked": it's transformed
  into a group of objects, one for each database table.  These objects represent
  a backup of the corresponding _table_ for that shard at 0000Z that day.
* Some time later, the daily metering, auditing, and GC jobs kick off, using
  these per-table backups as input.  The processes diverge at this point.

There are a few other jobs that go through different processes:

* There are also a number of hourly metering jobs that do not use these daily
  backups, but rather use log files uploaded hourly from services like Muskie
  and Marlin.
{'fraction_non_alphanumeric': 0.05561259411362081, 'fraction_numerical': 0.01676933607118412, 'mean_word_length': 4.109265734265734, 'pattern_counts': {'":': 2, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4505317', 'n_tokens_mistral': 1642, 'n_tokens_neox': 1535, 'n_words': 840}