<commit_before>---
layout: post
title:  "Stateful processing with Apache Beam"
date:   2017-02-13 00:00:01 -0800
excerpt_separator: <!--more-->
categories: blog
authors:
  - klk
---

Beam lets you process unbounded, out-of-order, global-scale data with portable
high-level pipelines. Stateful processing is a new feature of the Beam model
that expands the capabilities of Beam, unlocking new use cases and new
efficiencies. In this post, I will guide you through stateful processing in
Beam: how it works, how it fits in with the other features of the Beam model,
what you might use it for, and what it looks like in code.

<!--more-->

> **Warning: new features ahead!**: This is a very new aspect of the Beam
> model. Runners are still adding support. You can try it out today on multiple
> runners, but do check the [runner capability
> matrix]({{ site.baseurl }}/documentation/runners/capability-matrix/) for
> the current status in each runner.

First, a quick recap: In Beam, a big data processing _pipeline_ is a directed,
acyclic graph of parallel operations called _`PTransforms`_ processing data
from _`PCollections`_ I'll expand on that by walking through this illustration:

<img class="center-block" 
    src="{{ site.baseurl }}/images/blog/stateful-processing/pipeline.png" 
    alt="A Beam Pipeline - PTransforms are boxes - PCollections are arrows" 
    width="300">

The boxes are `PTransforms` and the edges represent the data in `PCollections`
flowing from one `PTransform` to the next. A `PCollection` may be _bounded_ (which
means it is finite and you know it) or _unbounded_ (which means you don't know if
it is finite or not - basically, it is like an incoming stream of data that may
or may not ever terminate). The cylinders are the data sources and sinks at the
edges of your pipeline, such as bounded collections of log files or unbounded
data streaming over a Kafka topic. This blog post isn't about sources or sinks,
but about what happens in between - your data processing.

There are two main building blocks for processing your data in Beam: _`ParDo`_,
for performing an operation in parallel across all elements, and _`GroupByKey`_
(and the closely related `CombinePerKey` that I will talk about quite soon)
for aggregating elements to which you have assigned the same key. In the
picture below (featured in many of our presentations) the color indicates the
key of the element. Thus the `GroupByKey`/`CombinePerKey` transform gathers all the
green squares to produce a single output element.

<img class="center-block" <commit_msg>This closes https://github.com/apache/beam-site/pull/155: Fix typo in stateful processing post<commit_after>---
layout: post
title:  "Stateful processing with Apache Beam"
date:   2017-02-13 00:00:01 -0800
excerpt_separator: <!--more-->
categories: blog
authors:
  - klk
---

Beam lets you process unbounded, out-of-order, global-scale data with portable
high-level pipelines. Stateful processing is a new feature of the Beam model
that expands the capabilities of Beam, unlocking new use cases and new
efficiencies. In this post, I will guide you through stateful processing in
Beam: how it works, how it fits in with the other features of the Beam model,
what you might use it for, and what it looks like in code.

<!--more-->

> **Warning: new features ahead!**: This is a very new aspect of the Beam
> model. Runners are still adding support. You can try it out today on multiple
> runners, but do check the [runner capability
> matrix]({{ site.baseurl }}/documentation/runners/capability-matrix/) for
> the current status in each runner.

First, a quick recap: In Beam, a big data processing _pipeline_ is a directed,
acyclic graph of parallel operations called _`PTransforms`_ processing data
from _`PCollections`_. I'll expand on that by walking through this illustration:

<img class="center-block" 
    src="{{ site.baseurl }}/images/blog/stateful-processing/pipeline.png" 
    alt="A Beam Pipeline - PTransforms are boxes - PCollections are arrows" 
    width="300">

The boxes are `PTransforms` and the edges represent the data in `PCollections`
flowing from one `PTransform` to the next. A `PCollection` may be _bounded_ (which
means it is finite and you know it) or _unbounded_ (which means you don't know if
it is finite or not - basically, it is like an incoming stream of data that may
or may not ever terminate). The cylinders are the data sources and sinks at the
edges of your pipeline, such as bounded collections of log files or unbounded
data streaming over a Kafka topic. This blog post isn't about sources or sinks,
but about what happens in between - your data processing.

There are two main building blocks for processing your data in Beam: _`ParDo`_,
for performing an operation in parallel across all elements, and _`GroupByKey`_
(and the closely related `CombinePerKey` that I will talk about quite soon)
for aggregating elements to which you have assigned the same key. In the
picture below (featured in many of our presentations) the color indicates the
key of the element. Thus the `GroupByKey`/`CombinePerKey` transform gathers all the
green squares to produce a single output element.

<img class="center-block" 
{'fraction_non_alphanumeric': 0.06696085955487337, 'fraction_numerical': 0.008633921719109747, 'mean_word_length': 4.418918918918919, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 19, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6642231', 'n_tokens_mistral': 1467, 'n_tokens_neox': 1374, 'n_words': 764}