<commit_before># Organizing data in S3 buckets

You can think of your data buckets as a grid. The horizontal dimension reflects
the data lifecycle phase (raw, refined, curated). The veritcal dimension reflects
topic or domain.

As discussed under [mental model](../MentalModel.md#buckets-are-branches), S3
buckets in Quilt are like branches in git; they reflect how "done" data are. 

## Don't rely on folders

There's no one folder structure that works for everyone. If you organize experiments
by `department/date` then looking at everything that happened on given date
becomes a bear. For this reason, packages offer both metadata tags and infinite
_logical_ views atop _physical_ locations in S3. With packages you can have
one file in a thousand packages without ever copying that file. Or you can "reorganize" S3 without moving any objects, with custom views of S3 for each team,
by using Quilt packages and the `.set*()` API calls.

## S3 data lifecycle
Below is an example of how you might organize three data domains according to a three-phase data lifecycle.

| Domain / Lifecycle phase | Raw | Refined | Curated |
|---|---|---|---|
| Domain1 | s3://domain1-raw | s3://domain1-refined | s3://domain1-curated |
| Domain2 | s3://domain2-raw | s3://domain2-refined | s3://domain2-curated |
| Domain3 | s3://domain3-raw | s3://domain3-refined | s3://domain3-curated |

## How enterprises organize intrument, scientist, ELN data

See [Bringing Scientific Data Management Systems Into the Cloud](https://blog.quiltdata.com/bringing-scientific-data-management-systems-into-the-cloud-video-41be228a41b9)
from Bio-IT world for a in-depth view of organizing scientific data in the cloud
from instrument, to scientist, to ELN.<commit_msg>Improve description of liberation from tyranny of files and folders (#2926)<commit_after># Organizing data in S3 buckets

You can think of your data buckets as a grid. The horizontal dimension reflects
the data lifecycle phase (raw, refined, curated). The veritcal dimension reflects
topic or domain.

As discussed under [mental model](../MentalModel.md#buckets-are-branches), S3
buckets in Quilt are like branches in git; they reflect how "done" data are. 

## Don't rely on folder structure

There's no one folder structure that works for everyone. For instance,
if you organize experiments by `department/date` then it's tedious for users to
look at everything that happened on a given data. And vice versa.

For this reason, packages offer both metadata tags and infinite
_logical_ views atop _physical_ locations in S3. Metadata tags prevent file names
from getting longer and longer to hackily include metadata.

With Quilt packages you can place one S3 object in a thousand packages without ever copying that object. Or you can "reorganize" S3 without moving any objects, with custom views of S3 for each team,
by using Quilt packages and the `.set*()` API calls.

## S3 data lifecycle
Below is an example of how you might organize three data domains according to a three-phase data lifecycle.

| Domain / Lifecycle phase | Raw | Refined | Curated |
|---|---|---|---|
| Domain1 | s3://domain1-raw | s3://domain1-refined | s3://domain1-curated |
| Domain2 | s3://domain2-raw | s3://domain2-refined | s3://domain2-curated |
| Domain3 | s3://domain3-raw | s3://domain3-refined | s3://domain3-curated |

## How enterprises organize intrument, scientist, ELN data

See [Bringing Scientific Data Management Systems Into the Cloud](https://blog.quiltdata.com/bringing-scientific-data-management-systems-into-the-cloud-video-41be228a41b9)
from Bio-IT world for a in-depth view of organizing scientific data in the cloud
from instrument, to scientist, to ELN.
{'fraction_non_alphanumeric': 0.08635625338386572, 'fraction_numerical': 0.020303194369247428, 'mean_word_length': 4.308908045977011, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4932235', 'n_tokens_mistral': 1087, 'n_tokens_neox': 1006, 'n_words': 472}