<commit_before># A package manager for data

Quilt is a data package manager, inspired by the likes of `pip` and `npm`. Just as software package managers provide versioned, reusable building blocks for execution, Quilt provides versioned, reusable building blocks for analysis.

# Motivations

1. **Reproducibility** - Imagine source code without versions. Ouch. Why live with un-versioned data? Versioned data makes analysis reproducible by creating unambiguous references to potentially complex data dependencies.
2. **Less data cleaning** - Finding, cleaning, and organizing data consumes 79% of the average data scientist's time. If data is cleaned _once_ and packaged for posterity, it frees up time for analysis. Quilt further makes it possible to _import_ data and start working immediately. Users can skip data preparation scripts for downloading, cleaning, and parsing data.
1. **De-duplication** - Data fragments are hashed with `SHA256`. Duplicate data fragments are written to disk once globally per user. As a result, large, repeated data fragments consume less disk and network bandwidth.
3. **Faster analysis** - Serialized data loads 5 to 20 times faster than files. Moreover, specialized storage formats like Apache Parquet minimize I/O bottlenecks so that tools like Presto DB and Hive run faster.
4. **Collaboration and transparency** - Data likes to be shared. Quilt offers a centralized data warehouse for finding and sharing data sets.

# Demo
<iframe width="560" height="315" src="https://www.youtube.com/embed/bKIV1GUVLPc" frameborder="0" allowfullscreen></iframe>

# Learn more on the Quilt blog
* [Manage data like code](https://blog.quiltdata.com/its-time-to-manage-data-like-source-code-3df04cd312b8)
* [Data packages for fast, reproducible analysis](https://blog.quiltdata.com/data-packages-for-fast-reproducible-python-analysis-c74b78015c7f)

***


<commit_msg>Remove boring stat [ci skip]<commit_after># A package manager for data

Quilt is a data package manager, inspired by the likes of `pip` and `npm`. Just as software package managers provide versioned, reusable building blocks for execution, Quilt provides versioned, reusable building blocks for analysis.

# Motivations

1. **Reproducibility** - Imagine source code without versions. Ouch. Why live with un-versioned data? Versioned data makes analysis reproducible by creating unambiguous references to potentially complex data dependencies.
2. **Less data prep** - Quilt abstracts away network, storage, and file format so that users can focus on what they wish to do with the data.
1. **De-duplication** - Data fragments are hashed with `SHA256`. Duplicate data fragments are written to disk once globally per user. As a result, large, repeated data fragments consume less disk and network bandwidth.
3. **Faster analysis** - Serialized data loads 5 to 20 times faster than files. Moreover, specialized storage formats like Apache Parquet minimize I/O bottlenecks so that tools like Presto DB and Hive run faster.
4. **Collaboration and transparency** - Data likes to be shared. Quilt offers a centralized data warehouse for finding and sharing data sets.

# Demo
<iframe width="560" height="315" src="https://www.youtube.com/embed/bKIV1GUVLPc" frameborder="0" allowfullscreen></iframe>

# Learn more on the Quilt blog
* [Manage data like code](https://blog.quiltdata.com/its-time-to-manage-data-like-source-code-3df04cd312b8)
* [Data packages for fast, reproducible analysis](https://blog.quiltdata.com/data-packages-for-fast-reproducible-python-analysis-c74b78015c7f)

***



{'fraction_non_alphanumeric': 0.07784934192103052, 'fraction_numerical': 0.019602352282273874, 'mean_word_length': 4.973244147157191, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 6, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4574925', 'n_tokens_mistral': 1000, 'n_tokens_neox': 916, 'n_words': 440}