<commit_before>---
title: How Hazelcast Jet Compares to Apache Spark
author: <NAME>
authorURL: https://twitter.com/voloda
authorImageURL: https://3l0wd94f0qdd10om8642z9se-wpengine.netdna-ssl.com/wp-content/uploads/2018/10/speaker-vladimir-schreiner-e1551380845855-170x170.jpg
---

“How Jet compares to Spark” and “why should I choose Jet over Spark” are
arguably the most frequent questions I’ve been asked during the talks
and workshops. While it is hard to assess the product fit without
focusing on a concrete use-case, I’d still like to compare concepts and
architecture used under the hood of both frameworks.

Versions considered: [Hazelcast Jet 4.1](https://jet-start.sh/download)
and [Apache Spark 2.4.5](https://spark.apache.org/downloads.html).

## Computations Modeled as Graphs

Apache Spark and Hazelcast Jet (referred to as “frameworks”) are both
tools for clustered computing. They are applicable mostly for analytical
(OLAP) applications, including those that apply a series of processing
steps to many uniform data records (such as lines in a file, rows in a
table or records appended to a stream), as one example.

Both frameworks build the on principles of dataflow programming: a user
builds an application by chaining high-level coarse-grained operators
such as map, join or aggregate. The operators form a network that can be
modeled as a graph ([directed acyclic
graphs](https://en.wikipedia.org/wiki/Directed_acyclic_graph) or DAG to
be specific) where nodes represent steps in the computation and edges
represent data exchange.

The dataflow model has some important properties that both frameworks
use for scaling and fault-tolerance:

- Pipeline Parallelism: operators can work independently, in parallel.
- Data Parallelism: a single operator can run in multiple instances,
  each instance processing a particular data partition
- No Shared State: each operator instance manages its state exclusively.
  There is no shared state to coordinate access to or to replicate.
  Moreover, the state is only determined by the input data. As a result,
  the operator can be recovered by replaying the input data.

Spark and Jet differ in how they use and execute the DAG as explained in
the next section but fundamentally: no matter which API you use (RDDs,<commit_msg>Typo fix (#2274)<commit_after>---
title: How Hazelcast Jet Compares to Apache Spark
author: <NAME>
authorURL: https://twitter.com/voloda
authorImageURL: https://3l0wd94f0qdd10om8642z9se-wpengine.netdna-ssl.com/wp-content/uploads/2018/10/speaker-vladimir-schreiner-e1551380845855-170x170.jpg
---

“How Jet compares to Spark” and “why should I choose Jet over Spark” are
arguably the most frequent questions I’ve been asked during the talks
and workshops. While it is hard to assess the product fit without
focusing on a concrete use-case, I’d still like to compare concepts and
architecture used under the hood of both frameworks.

Versions considered: [Hazelcast Jet 4.1](https://jet-start.sh/download)
and [Apache Spark 2.4.5](https://spark.apache.org/downloads.html).

## Computations Modeled as Graphs

Apache Spark and Hazelcast Jet (referred to as “frameworks”) are both
tools for clustered computing. They are applicable mostly for analytical
(OLAP) applications, including those that apply a series of processing
steps to many uniform data records (such as lines in a file, rows in a
table or records appended to a stream), as one example.

Both frameworks build on the principles of dataflow programming: a user
builds an application by chaining high-level coarse-grained operators
such as map, join or aggregate. The operators form a network that can be
modeled as a graph ([directed acyclic
graphs](https://en.wikipedia.org/wiki/Directed_acyclic_graph) or DAG to
be specific) where nodes represent steps in the computation and edges
represent data exchange.

The dataflow model has some important properties that both frameworks
use for scaling and fault-tolerance:

- Pipeline Parallelism: operators can work independently, in parallel.
- Data Parallelism: a single operator can run in multiple instances,
  each instance processing a particular data partition
- No Shared State: each operator instance manages its state exclusively.
  There is no shared state to coordinate access to or to replicate.
  Moreover, the state is only determined by the input data. As a result,
  the operator can be recovered by replaying the input data.

Spark and Jet differ in how they use and execute the DAG as explained in
the next section but fundamentally: no matter which API you use (RDDs,
{'fraction_non_alphanumeric': 0.056562568246341996, 'fraction_numerical': 0.019218169906093034, 'mean_word_length': 4.761006289308176, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 10, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7308382', 'n_tokens_mistral': 1298, 'n_tokens_neox': 1195, 'n_words': 621}