<commit_before>---
title: SQL Editor for Apache Flink SQL
author: Romain
type: post
date: 2020-05-06T00:00:00+00:00
url: /blog/sql-editor-for-apache-flink-sql/
sf_thumbnail_type:
  - none
sf_thumbnail_link_type:
  - link_to_post
sf_detail_type:
  - none
sf_page_title:
  - 1
sf_page_title_style:
  - standard
sf_no_breadcrumbs:
  - 1
sf_page_title_bg:
  - none
sf_page_title_text_style:
  - light
sf_background_image_size:
  - cover
sf_social_sharing:
  - 1
sf_related_articles:
  - 1
sf_sidebar_config:
  - left-sidebar
sf_left_sidebar:
  - Sidebar-2
sf_right_sidebar:
  - Sidebar-1
sf_caption_position:
  - caption-right
sf_remove_promo_bar:
  - 1
ampforwp-amp-on-off:
  - default
categories:
  - Querying
  - Version 4.8

---

# Flink SQL Editor

This is the very first version of the SQL Editor for Flink.

The goal is to demo how to execute Flink SQL queries. We use the new Flink SQL gateway project and point to a Flink cluster with live data in a docker container. Hue is used as the SQL Editor for querying Flink tables.

Feel free to read more about Flink [SQL](https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql/queries.html#queries) and [continuous](https://ci.apache.org/projects/flink/flink-docs-master/dev/table/streaming/dynamic_tables.html#continuous-queries) queries.

{{< youtube fKHD-fOdDY0 >}}

## Setup

Any Flink 1.10 cluster would work and the demo is based on [Ververica SQL Training](https://github.com/ververica/sql-training) which has some easy [setup instructions](https://github.com/ververica/sql-training/wiki/Setting-up-the-Training-Environment):

    git clone https://github.com/ververica/sql-training.git
    cd sql-training
    docker-compose up -d

Then [http://localhost:8081](http://localhost:8081) should be up.

![Flink Dashboard](https://cdn.gethue.com/uploads/2020/05/flink_dashboard.png)

Here we start a SQL client container and install the gateway inside (to avoid installing a local Flink as the gateway needs a FLINK_HOME) but this could be done locally or in another containers.

    docker-compose exec sql-client bash

We grab a [release](https://github.com/ververica/flink-sql-gateway/releases) of the gateway:

    cd /opt
    wget https://github.com/ververica/flink-sql-gateway/releases/download/v0.1-snapshot/flink-sql-gateway-0.1-SNAPSHOT-bin.zip
    unzip flink-sql-gateway-0.1-SNAPSHOT-bin.zip
    cd flink-sql-gateway-0.1-SNAPSHOT

    echo $FLINK_HOME

Then from another shell we copy the Flink SQL config to the gateway so that we get the demo tables by default:

    wget https://raw.githubusercontent.com/romainr/flink-sql-gateway/master/docs/demo/sql-gateway-defaults.yaml

    docker cp sql-gateway-defaults.yaml sql-training_sql-client_1:/opt/flink-sql-gateway-0.1-SNAPSHOT/conf/

Now we can go back to the shell in the container and are ready to start it:

    cd bin
    ./sql-gateway.sh --library /opt/sql-client/lib

Putting the server in the background with `CTRL-Z` and then:

    bg

And now we can issue a few commands to validate the setup:

    curl localhost:8083/v1/info
    > {"product_name":"Apache Flink","version":"1.10.0"}

    curl -X POST localhost:8083/v1/sessions -d '{"planner":"blink","execution_type":"streaming"}'
    > {"session_id":"7eea0827c249e5a8fcbe129422f049e8"}


**Note**

It "should" also be possible to deploy the SQL gateway not in the SQL client container by:

1. Having a local Flink [binary package](https://www.apache.org/dyn/closer.lua/flink/flink-1.10.0/flink-1.10.0-bin-scala_2.11.tgz) with FLINK_HOME configured

2. Updating the `jobmanager.rpc.address` to the real jobmanager address in `$FLINK_HOME/conf/flink-conf.yaml`

3. Changing the two address properties in `sql-gateway-defaults.yaml`

<pre>
    server:
      # The address that the gateway binds itself.
      bind-address: 172.18.0.7
      # The address that should be used by clients to connect to the gateway.
      address: 172.18.0.7
</pre>

## Query Editor

As detailed in the [connector](https://docs.gethue.com/administrator/configuration/connectors/) section of Hue, we add a Flink interpreter:

    [notebook]
<commit_msg>HUE-8888 [blog] Updating the Flink SQL gateway setup instruction<commit_after>---
title: SQL Editor for Apache Flink SQL
author: Romain
type: post
date: 2020-05-06T00:00:00+00:00
url: /blog/sql-editor-for-apache-flink-sql/
sf_thumbnail_type:
  - none
sf_thumbnail_link_type:
  - link_to_post
sf_detail_type:
  - none
sf_page_title:
  - 1
sf_page_title_style:
  - standard
sf_no_breadcrumbs:
  - 1
sf_page_title_bg:
  - none
sf_page_title_text_style:
  - light
sf_background_image_size:
  - cover
sf_social_sharing:
  - 1
sf_related_articles:
  - 1
sf_sidebar_config:
  - left-sidebar
sf_left_sidebar:
  - Sidebar-2
sf_right_sidebar:
  - Sidebar-1
sf_caption_position:
  - caption-right
sf_remove_promo_bar:
  - 1
ampforwp-amp-on-off:
  - default
categories:
  - Querying
  - Version 4.8

---

# Flink SQL Editor

This is the very first version of the SQL Editor for Flink.

The goal is to demo how to execute Flink SQL queries. We use the new Flink SQL gateway project and point to a Flink cluster with live data in a docker container. Hue is used as the SQL Editor for querying Flink tables.

Feel free to read more about Flink [SQL](https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql/queries.html#queries) and [continuous](https://ci.apache.org/projects/flink/flink-docs-master/dev/table/streaming/dynamic_tables.html#continuous-queries) queries.

{{< youtube fKHD-fOdDY0 >}}

## Setup

Any Flink 1.10 cluster would work and the demo is based on [Ververica SQL Training](https://github.com/ververica/sql-training) which has some easy [setup instructions](https://github.com/ververica/sql-training/wiki/Setting-up-the-Training-Environment):

    git clone https://github.com/ververica/sql-training.git
    cd sql-training
    docker-compose up -d

Then [http://localhost:8081](http://localhost:8081) should be up.

![Flink Dashboard](https://cdn.gethue.com/uploads/2020/05/flink_dashboard.png)

Here we start a SQL client container and install the gateway inside (to avoid installing a local Flink as the gateway needs a FLINK_HOME) but this could be done locally or in another containers.

    docker-compose exec sql-client bash

We grab a [release](https://github.com/ververica/flink-sql-gateway/releases) of the gateway:

    cd /opt
    wget https://github.com/ververica/flink-sql-gateway/releases/download/flink-1.11.1/flink-sql-gateway-0.2-SNAPSHOT-bin.zip
    unzip flink-sql-gateway-0.2-SNAPSHOT-bin.zip
    cd flink-sql-gateway-0.2-SNAPSHOT

    echo $FLINK_HOME

Then from another shell we copy the Flink SQL config to the gateway so that we get the demo tables by default:

    wget https://raw.githubusercontent.com/romainr/flink-sql-gateway/master/docs/demo/sql-gateway-defaults.yaml

    docker cp sql-gateway-defaults.yaml sql-training_sql-client_1:/opt/flink-sql-gateway-0.2-SNAPSHOT/conf/

Now we can go back to the shell in the container and are ready to start it:

    cd bin
    ./sql-gateway.sh --library /opt/sql-client/lib

Putting the server in the background with `CTRL-Z` and then:

    bg

And now we can issue a few commands to validate the setup:

    curl sql-training_sql-client_1:8083/v1/info
    > {"product_name":"Apache Flink","version":"1.10.0"}

    curl -X POST sql-training_sql-client_1:8083/v1/sessions -d '{"planner":"blink","execution_type":"streaming"}'
    > {"session_id":"7eea0827c249e5a8fcbe129422f049e8"}


**Note**

It "should" also be possible to deploy the SQL gateway not in the SQL client container by:

1. Having a local Flink [binary package](https://www.apache.org/dyn/closer.lua/flink/flink-1.10.0/flink-1.10.0-bin-scala_2.11.tgz) with FLINK_HOME configured

2. Updating the `jobmanager.rpc.address` to the real jobmanager address in `$FLINK_HOME/conf/flink-conf.yaml`

3. Changing the two address properties in `sql-gateway-defaults.yaml`

<pre>
    server:
      # The address that the gateway binds itself.
      bind-address: 172.18.0.7
      # The address that should be used by clients to connect to the gateway.
      address: 172.18.0.7
</pre>

## Query Editor

As detailed in the [connector](https://docs.gethue.com/administrator/configuration/connectors/) section of Hue, we add a Flink interpreter:

    [notebook]

{'fraction_non_alphanumeric': 0.10571840461316674, 'fraction_numerical': 0.029312830370014416, 'mean_word_length': 3.7927461139896375, 'pattern_counts': {'":': 10, '<': 9, '<?xml version=': 0, '>': 13, 'https://': 22, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 4, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1375367', 'n_tokens_mistral': 3043, 'n_tokens_neox': 2845, 'n_words': 790}