<commit_before># Spindle

![](https://git.corp.adobe.com/amos/spindle/raw/master/images/architecture.png)

Analytics platforms such as [Adobe Analytics][adobe-analytics]
are growing to process petabytes of data in real-time.
Delivering responsive interfaces over this much data is difficult,
and there are many distributed data processing technologies such
as [Hadoop MapReduce][mapreduce], [Apache Spark][spark],
[Apache Drill][drill], and [Cloudera Impala][impala].

**Spindle is a research-based distributed analytics query
engine built with Spark.**
Spark claims 100x speedups over MapReduce for in-memory processing.
This project contains the full Spindle implementation with
benchmarking scripts to tune Spindle and Spark for maximum performance.

# Demo
We used Spindle to generate static webpages that are hosted
statically [here][demo].
Unfortunately, the demo is only meant for illustrative purposes and
is not running Spindle in real-time.

![](https://git.corp.adobe.com/amos/spindle/raw/master/images/page-views.png)
![](https://git.corp.adobe.com/amos/spindle/raw/master/images/revenue.png)
![](https://git.corp.adobe.com/amos/spindle/raw/master/images/top-pages-by-browser.png)

[Grunt][grunt] is used to deploy `demo` to [Github pages][ghp]
in the [gh-pages][ghp] branch with the [grunt-build-control][gbc] plugin.
The [npm][npm] dependencies are managed in [package.json][pjson]
and can be installed with `npm install`.

# Data Format
Adobe Analytics events data have 250 columns, and
most queries use less than 7 columns.
Spindle stores event data in the [Parquet][parquet] columnar store
on the [Hadoop Distributed File System][hdfs] (HDFS).

Parquet can be used with [Avro][avro] or [Thrift][thrift] schemas.
[<NAME>'s article][spark-parquet-avro] provides an example of
using Parquet with Avro.
[adobe-research/spark-parquet-thrift-example][spark-parquet-thrift-example]
is a complete [Scala][scala]/[sbt][sbt] project
using Thrift for data serialization and shows how to only load the
specified columnar subset.
<commit_msg>Only show one screenshot for the demo.<commit_after># Spindle

![](https://git.corp.adobe.com/amos/spindle/raw/master/images/architecture.png)

Analytics platforms such as [Adobe Analytics][adobe-analytics]
are growing to process petabytes of data in real-time.
Delivering responsive interfaces over this much data is difficult,
and there are many distributed data processing technologies such
as [Hadoop MapReduce][mapreduce], [Apache Spark][spark],
[Apache Drill][drill], and [Cloudera Impala][impala].

**Spindle is a research-based distributed analytics query
engine built with Spark.**
Spark claims 100x speedups over MapReduce for in-memory processing.
This project contains the full Spindle implementation with
benchmarking scripts to tune Spindle and Spark for maximum performance.

# Demo
We used Spindle to generate static webpages that are hosted
statically [here][demo].
Unfortunately, the demo is only meant for illustrative purposes and
is not running Spindle in real-time.

![](https://git.corp.adobe.com/amos/spindle/raw/master/images/top-pages-by-browser.png)

[Grunt][grunt] is used to deploy `demo` to [Github pages][ghp]
in the [gh-pages][ghp] branch with the [grunt-build-control][gbc] plugin.
The [npm][npm] dependencies are managed in [package.json][pjson]
and can be installed with `npm install`.

# Data Format
Adobe Analytics events data have 250 columns, and
most queries use less than 7 columns.
Spindle stores event data in the [Parquet][parquet] columnar store
on the [Hadoop Distributed File System][hdfs] (HDFS).

Parquet can be used with [Avro][avro] or [Thrift][thrift] schemas.
[<NAME>'s article][spark-parquet-avro] provides an example of
using Parquet with Avro.
[adobe-research/spark-parquet-thrift-example][spark-parquet-thrift-example]
is a complete [Scala][scala]/[sbt][sbt] project
using Thrift for data serialization and shows how to only load the
specified columnar subset.

{'fraction_non_alphanumeric': 0.09939455095862765, 'fraction_numerical': 0.0035317860746720484, 'mean_word_length': 5.468189233278956, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6971393', 'n_tokens_mistral': 1251, 'n_tokens_neox': 1155, 'n_words': 444}