<commit_before>---
authors: <NAME> <<EMAIL>>
state: draft
---

# RFD 47 Retention policy for Joyent engineering data in Manta

Joyent engineering puts a lot of stuff in Manta through automated processes
(builds, images, logs, crash dumps, etc.). Currently that data is growing
forever. This can prove to be untenable. Let's get a policy for what to
retain and for how long.


## Current Status

Dave Pacheco has worked on Manta (poseidon) usage.

Trent has worked on Triton/Manta [builds](#builds) usage. There is a
`purge-mg-builds` script and Jenkins job to handle daily removal of old builds.

See the [Tickets](#tickets) section below for work that has been done cleaning
up some usage.


## mdu

Having `mdu` per <https://github.com/joyent/node-manta/issues/143> would be
really helpful to understand how much is used where.

Status:
- Dave has a prototype here: <https://github.com/davepacheco/manta-mdu>
  Importantly, Dave's does real `stat` calls on the storage CNs, so the
  numbers more accurately reflect disk usage, rather than file size.
- Trent has a couple poorman's prototypes here:
  <https://mo.joyent.com/trentops/blob/master/bin/poormans-mdu-subdirs>,
  <https://mo.joyent.com/trentops/blob/master/bin/poormans-mdu>
- Trent had a start at one which involved changes to node-manta. This
  is currently uncommited.


## builds

Automatic builds of Triton and Manta bits are done for various reasons
(commits to #master of top-level repos, commits to `release-YYYYMMDD` release
branches, a regular schedule for `headnode*` builds, etc.). In general the
builds all get uploaded to one of:

```
/Joyent_Dev/public/builds/$name/$branch-$buildstamp
/Joyent_Dev/stor/builds/$name/$branch-$buildstamp
```

Retention policy for builds:

- Have a whitelist of build `name`s that will be deleted. This means new ones
  won't get cleaned out by default -- i.e. safe by default.
- Delete anything older than:
    - for 'headnode*' builds: a month
      A year of headnode* builds is too much. At ~1 per day * 4 build flavours
      (`headnode[-joyent][-debug]`) * 6 GiB per build, that is 8 TiB of space.
      At one month retention we are down to 744 GiB of space.
    - for other builds: a year
- Ensure that the `$branch-latest` "link" files are removed if the last<commit_msg>RFD 47: markdown syntax fix<commit_after>---
authors: <NAME> <<EMAIL>>
state: draft
---

# RFD 47 Retention policy for Joyent engineering data in Manta

Joyent engineering puts a lot of stuff in Manta through automated processes
(builds, images, logs, crash dumps, etc.). Currently that data is growing
forever. This can prove to be untenable. Let's get a policy for what to
retain and for how long.


## Current Status

<NAME> has worked on Manta (poseidon) usage.

Trent has worked on Triton/Manta [builds](#builds) usage. There is a
`purge-mg-builds` script and Jenkins job to handle daily removal of old builds.

See the [Tickets](#tickets) section below for work that has been done cleaning
up some usage.


## mdu

Having `mdu` per <https://github.com/joyent/node-manta/issues/143> would be
really helpful to understand how much is used where.

Status:
- Dave has a prototype here: <https://github.com/davepacheco/manta-mdu>
  Importantly, Dave's does real `stat` calls on the storage CNs, so the
  numbers more accurately reflect disk usage, rather than file size.
- Trent has a couple poorman's prototypes here:
  <https://mo.joyent.com/trentops/blob/master/bin/poormans-mdu-subdirs>,
  <https://mo.joyent.com/trentops/blob/master/bin/poormans-mdu>
- Trent had a start at one which involved changes to node-manta. This
  is currently uncommited.


## builds

Automatic builds of Triton and Manta bits are done for various reasons
(commits to #master of top-level repos, commits to `release-YYYYMMDD` release
branches, a regular schedule for `headnode*` builds, etc.). In general the
builds all get uploaded to one of:

```
/Joyent_Dev/public/builds/$name/$branch-$buildstamp
/Joyent_Dev/stor/builds/$name/$branch-$buildstamp
```

Retention policy for builds:

- Have a whitelist of build `name`s that will be deleted. This means new ones
  won't get cleaned out by default -- i.e. safe by default.
- Delete anything older than:
    - for `headnode*` builds: a month
      A year of `headnode*` builds is too much. At ~1 per day \* 4 build
      flavours (`headnode[-joyent][-debug]`) \* 6 GiB per build, that is
      8 TiB of space. At one month retention we are down to 744 GiB of space.
    - for other builds: a year
- Ensure that the `$branch-latest` "link" files are removed if the last
{'fraction_non_alphanumeric': 0.09306887532693985, 'fraction_numerical': 0.005666957279860506, 'mean_word_length': 3.810272536687631, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 18, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7176739', 'n_tokens_mistral': 1485, 'n_tokens_neox': 1403, 'n_words': 618}