<commit_before>---
title: 'Flink Operations Playground'
nav-title: 'Flink Operations Playground'
weight: 6
type: docs
aliases:
  - /try-flink/flink-operations-playground.html
  - /getting-started/playground/flink-operations-playground.html
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.  See the NOTICE file
distributed with this work for additional information
regarding copyright ownership.  The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License.  You may obtain a copy of the License at

  http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.
-->

# Flink Operations Playground

There are many ways to deploy and operate Apache Flink in various environments. Regardless of this
variety, the fundamental building blocks of a Flink Cluster remain the same, and similar
operational principles apply.

In this playground, you will learn how to manage and run Flink Jobs. You will see how to deploy and 
monitor an application, experience how Flink recovers from Job failure, and perform everyday 
operational tasks like upgrades and rescaling.

{{< unstable >}}
{{< hint warning >}}
**Attention**: The Apache Flink Docker images used for this playground are only available for released versions of Apache Flink.

Since you are currently looking at the latest SNAPSHOT
version of the documentation, all version references below will not work.
Please switch the documentation to the latest released version via the release picker which you
find on the left side below the menu.
{{< /hint  >}}
{{< /unstable >}}

## Anatomy of this Playground

This playground consists of a long living
[Flink Session Cluster]({{< ref "docs/concepts/glossary" >}}#flink-session-cluster) and a Kafka
Cluster.

A Flink Cluster always consists of a 
[JobManager]({{< ref "docs/concepts/glossary" >}}#flink-jobmanager) and one or more 
[Flink TaskManagers]({{< ref "docs/concepts/glossary" >}}#flink-taskmanager). The JobManager 
is responsible for handling [Job]({{< ref "docs/concepts/glossary" >}}#flink-job) submissions, 
the supervision of Jobs as well as resource management. The Flink TaskManagers are the worker 
processes and are responsible for the execution of the actual 
[Tasks]({{< ref "docs/concepts/glossary" >}}#task) which make up a Flink Job. In this 
playground you will start with a single TaskManager, but scale out to more TaskManagers later. 
Additionally, this playground comes with a dedicated *client* container, which we use to submit the 
Flink Job initially and to perform various operational tasks later on. The *client* container is not
needed by the Flink Cluster itself but only included for ease of use.

The Kafka Cluster consists of a Zookeeper server and a Kafka Broker.

{{< img src="/fig/flink-docker-playground.svg" alt="Flink Docker Playground" width="80%" >}}

When the playground is started a Flink Job called *Flink Event Count* will be submitted to the 
JobManager. Additionally, two Kafka Topics *input* and *output* are created.

{{< img src="/fig/click-event-count-example.svg" alt="Click Event Count Example" width="80%" >}}

The Job consumes `ClickEvent`s from the *input* topic, each with a `timestamp` and a `page`. The 
events are then keyed by `page` and counted in 15 second
[windows]({{< ref "docs/dev/datastream/operators/windows" >}}). The results are written to the 
*output* topic. 

There are six different pages and we generate 1000 click events per page and 15 seconds. Hence, the 
output of the Flink job should show 1000 views per page and window.

{{< top >}}

## Starting the Playground

The playground environment is set up in just a few steps. We will walk you through the necessary 
commands and show how to validate that everything is running correctly.

We assume that you have [Docker](https://docs.docker.com/) (1.12+) and
[docker-compose](https://docs.docker.com/compose/) (2.1+) installed on your machine.

The required configuration files are available in the 
[flink-playgrounds](https://github.com/apache/flink-playgrounds) repository. Check it out and spin
up the environment:

```bash
git clone --branch release-{{ site.version_title }} https://github.com/apache/flink-playgrounds.git
cd flink-playgrounds/operations-playground
docker-compose build
docker-compose up -d
```

Afterwards, you can inspect the running Docker containers with the following command:

```bash
docker-compose ps

                    Name                                  Command               State                   Ports                
-----------------------------------------------------------------------------------------------------------------------------
operations-playground_clickevent-generator_1   /docker-entrypoint.sh java ...   Up       6123/tcp, 8081/tcp                  
operations-playground_client_1                 /docker-entrypoint.sh flin ...   Exit 0                                       
operations-playground_jobmanager_1             /docker-entrypoint.sh jobm ...   Up       6123/tcp, 0.0.0.0:8081->8081/tcp    
operations-playground_kafka_1                  start-kafka.sh                   Up       0.0.0.0:9094->9094/tcp              
operations-playground_taskmanager_1            /docker-entrypoint.sh task ...   Up       6123/tcp, 8081/tcp                  
operations-playground_zookeeper_1              /bin/sh -c /usr/sbin/sshd  ...   Up       2181/tcp, 22/tcp, 2888/tcp, 3888/tcp
```

This indicates that the client container has successfully submitted the Flink Job (`Exit 0`) and all 
cluster components as well as the data generator are running (`Up`).

You can stop the playground environment by calling:

```bash
docker-compose down -v
```

## Entering the Playground

There are many things you can try and check out in this playground. In the following two sections we <commit_msg>[FLINK-23059][docs] describe how to create ckp and savepoint dirs in operations playground docs<commit_after>---
title: 'Flink Operations Playground'
nav-title: 'Flink Operations Playground'
weight: 6
type: docs
aliases:
  - /try-flink/flink-operations-playground.html
  - /getting-started/playground/flink-operations-playground.html
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.  See the NOTICE file
distributed with this work for additional information
regarding copyright ownership.  The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License.  You may obtain a copy of the License at

  http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.
-->

# Flink Operations Playground

There are many ways to deploy and operate Apache Flink in various environments. Regardless of this
variety, the fundamental building blocks of a Flink Cluster remain the same, and similar
operational principles apply.

In this playground, you will learn how to manage and run Flink Jobs. You will see how to deploy and 
monitor an application, experience how Flink recovers from Job failure, and perform everyday 
operational tasks like upgrades and rescaling.

{{< unstable >}}
{{< hint warning >}}
**Attention**: The Apache Flink Docker images used for this playground are only available for released versions of Apache Flink.

Since you are currently looking at the latest SNAPSHOT
version of the documentation, all version references below will not work.
Please switch the documentation to the latest released version via the release picker which you
find on the left side below the menu.
{{< /hint  >}}
{{< /unstable >}}

## Anatomy of this Playground

This playground consists of a long living
[Flink Session Cluster]({{< ref "docs/concepts/glossary" >}}#flink-session-cluster) and a Kafka
Cluster.

A Flink Cluster always consists of a 
[JobManager]({{< ref "docs/concepts/glossary" >}}#flink-jobmanager) and one or more 
[Flink TaskManagers]({{< ref "docs/concepts/glossary" >}}#flink-taskmanager). The JobManager 
is responsible for handling [Job]({{< ref "docs/concepts/glossary" >}}#flink-job) submissions, 
the supervision of Jobs as well as resource management. The Flink TaskManagers are the worker 
processes and are responsible for the execution of the actual 
[Tasks]({{< ref "docs/concepts/glossary" >}}#task) which make up a Flink Job. In this 
playground you will start with a single TaskManager, but scale out to more TaskManagers later. 
Additionally, this playground comes with a dedicated *client* container, which we use to submit the 
Flink Job initially and to perform various operational tasks later on. The *client* container is not
needed by the Flink Cluster itself but only included for ease of use.

The Kafka Cluster consists of a Zookeeper server and a Kafka Broker.

{{< img src="/fig/flink-docker-playground.svg" alt="Flink Docker Playground" width="80%" >}}

When the playground is started a Flink Job called *Flink Event Count* will be submitted to the 
JobManager. Additionally, two Kafka Topics *input* and *output* are created.

{{< img src="/fig/click-event-count-example.svg" alt="Click Event Count Example" width="80%" >}}

The Job consumes `ClickEvent`s from the *input* topic, each with a `timestamp` and a `page`. The 
events are then keyed by `page` and counted in 15 second
[windows]({{< ref "docs/dev/datastream/operators/windows" >}}). The results are written to the 
*output* topic. 

There are six different pages and we generate 1000 click events per page and 15 seconds. Hence, the 
output of the Flink job should show 1000 views per page and window.

{{< top >}}

## Starting the Playground

The playground environment is set up in just a few steps. We will walk you through the necessary 
commands and show how to validate that everything is running correctly.

We assume that you have [Docker](https://docs.docker.com/) (1.12+) and
[docker-compose](https://docs.docker.com/compose/) (2.1+) installed on your machine.

The required configuration files are available in the 
[flink-playgrounds](https://github.com/apache/flink-playgrounds) repository. First checkout the code and build the docker image:

```bash
git clone https://github.com/apache/flink-playgrounds.git
cd flink-playgrounds/operations-playground
docker-compose build
```

Then before starting the playground, create the checkpoint and savepoint directories on the Docker host machine (these volumes are mounted by the jobmanager and taskmanager, as specified in docker-compose.yaml):

```bash
mkdir -p /tmp/flink-checkpoints-directory
mkdir -p /tmp/flink-savepoints-directory
```

Then start the playground:

```bash
docker-compose up -d
```

Afterwards, you can inspect the running Docker containers with the following command:

```bash
docker-compose ps

                    Name                                  Command               State                   Ports                
-----------------------------------------------------------------------------------------------------------------------------
operations-playground_clickevent-generator_1   /docker-entrypoint.sh java ...   Up       6123/tcp, 8081/tcp                  
operations-playground_client_1                 /docker-entrypoint.sh flin ...   Exit 0                                       
operations-playground_jobmanager_1             /docker-entrypoint.sh jobm ...   Up       6123/tcp, 0.0.0.0:8081->8081/tcp    
operations-playground_kafka_1                  start-kafka.sh                   Up       0.0.0.0:9094->9094/tcp              
operations-playground_taskmanager_1            /docker-entrypoint.sh task ...   Up       6123/tcp, 8081/tcp                  
operations-playground_zookeeper_1              /bin/sh -c /usr/sbin/sshd  ...   Up       2181/tcp, 22/tcp, 2888/tcp, 3888/tcp
```

This indicates that the client container has successfully submitted the Flink Job (`Exit 0`) and all 
cluster components as well as the data generator are running (`Up`).

You can stop the playground environment by calling:

```bash
docker-compose down -v
```

## Entering the Playground

There are many things you can try and check out in this playground. In the following two sections we 
{'fraction_non_alphanumeric': 0.09682193488082255, 'fraction_numerical': 0.014721919302071973, 'mean_word_length': 3.245701058201058, 'pattern_counts': {'":': 0, '<': 31, '<?xml version=': 0, '>': 35, 'https://': 8, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 4, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4552670', 'n_tokens_mistral': 3623, 'n_tokens_neox': 3314, 'n_words': 1519}