<commit_before>---
title: "Data Sources"
weight: 11
type: docs
aliases:
  - /dev/stream/sources.html
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.  See the NOTICE file
distributed with this work for additional information
regarding copyright ownership.  The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License.  You may obtain a copy of the License at

  http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.
-->

# Data Sources

This page describes Flink's Data Source API and the concepts and architecture behind it.
**Read this, if you are interested in how data sources in Flink work, or if you want to implement a new Data Source.**

If you are looking for pre-defined source connectors, please check the [Connector Docs]({{< ref "docs/connectors/datastream/overview" >}}).

## Data Source Concepts

**Core Components**

A Data Source has three core components: *Splits*, the *SplitEnumerator*, and the *SourceReader*.

  - A **Split** is a portion of data consumed by the source, like a file or a log partition. Splits are granularity by which the source distributes the work and parallelizes the data reading.

  - The **SourceReader** requests *Splits* and processes them, for example by reading the file or log partition represented by the *Split*. The *SourceReader* run in parallel on the Task Managers in the `SourceOperators` and produces the parallel stream of events/records.

  - The **SplitEnumerator** generates the *Splits* and assigns them to the *SourceReaders*. It runs as a single instance on the Job Manager and is responsible for maintaining the backlog of pending *Splits* and assigning them to the readers in a balanced manner.
  
The [Source](https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/connector/source/Source.java) class is API entry point that ties the above three components together.

{{< img src="/fig/source_components.svg" alt="Illustration of SplitEnumerator and SourceReader interacting" width="70%" >}}


**Unified Across Streaming and Batch**

The Data Source API supports both unbounded streaming sources and bounded batch sources, in a unified way.

The difference between both cases is minimal: In the bounded/batch case, the enumerator generates a fix set of splits, and each split is necessarily finite. In the unbounded streaming case, one of the two is not true (splits are not finite, or the enumerator keep generating new splits).

#### Examples

Here are some simplified conceptual examples to illustrate how the data source components interact, in streaming and batch cases.

*Note that this does not the accurately describe how the Kafka and File source implementations work; parts are simplified, for illustrative purposes.*

**Bounded File Source**

The source has the URI/Path of a directory to read, and a *Format* that defines how to parse the files.

  - A *Split* is a file, or a region of a file (if the data format supports splitting the file).
  - The *SplitEnumerator* lists all files under the given directory path. It assigns Splits to the next reader that requests a Split. Once all Splits are assigned, it responds to requests with *NoMoreSplits*.
  - The *SourceReader* requests a Split and reads the assigned Split (file or file region) and parses it using the given Format. If it does not get another Split, but a *NoMoreSplits* message, it finishes.

**Unbounded Streaming File Source**

This source works the same way as described above, except that the *SplitEnumerator* never responds with *NoMoreSplits* and periodically lists the contents under the given URI/Path to check for new files. Once it finds new files, it generates new Splits for them and can assign them to the available SourceReaders.

**Unbounded Streaming Kafka Source**

The source has a Kafka Topic (or list of Topics or Topic regex) and a *Deserializer* to parse the records.
<commit_msg>[hotfix][docs] Fix typo in sources doc<commit_after>---
title: "Data Sources"
weight: 11
type: docs
aliases:
  - /dev/stream/sources.html
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.  See the NOTICE file
distributed with this work for additional information
regarding copyright ownership.  The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License.  You may obtain a copy of the License at

  http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.
-->

# Data Sources

This page describes Flink's Data Source API and the concepts and architecture behind it.
**Read this, if you are interested in how data sources in Flink work, or if you want to implement a new Data Source.**

If you are looking for pre-defined source connectors, please check the [Connector Docs]({{< ref "docs/connectors/datastream/overview" >}}).

## Data Source Concepts

**Core Components**

A Data Source has three core components: *Splits*, the *SplitEnumerator*, and the *SourceReader*.

  - A **Split** is a portion of data consumed by the source, like a file or a log partition. Splits are granularity by which the source distributes the work and parallelizes the data reading.

  - The **SourceReader** requests *Splits* and processes them, for example by reading the file or log partition represented by the *Split*. The *SourceReader* run in parallel on the Task Managers in the `SourceOperators` and produces the parallel stream of events/records.

  - The **SplitEnumerator** generates the *Splits* and assigns them to the *SourceReaders*. It runs as a single instance on the Job Manager and is responsible for maintaining the backlog of pending *Splits* and assigning them to the readers in a balanced manner.
  
The [Source](https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/connector/source/Source.java) class is API entry point that ties the above three components together.

{{< img src="/fig/source_components.svg" alt="Illustration of SplitEnumerator and SourceReader interacting" width="70%" >}}


**Unified Across Streaming and Batch**

The Data Source API supports both unbounded streaming sources and bounded batch sources, in a unified way.

The difference between both cases is minimal: In the bounded/batch case, the enumerator generates a fix set of splits, and each split is necessarily finite. In the unbounded streaming case, one of the two is not true (splits are not finite, or the enumerator keep generating new splits).

#### Examples

Here are some simplified conceptual examples to illustrate how the data source components interact, in streaming and batch cases.

*Note that this does not accurately describe how the Kafka and File source implementations work; parts are simplified, for illustrative purposes.*

**Bounded File Source**

The source has the URI/Path of a directory to read, and a *Format* that defines how to parse the files.

  - A *Split* is a file, or a region of a file (if the data format supports splitting the file).
  - The *SplitEnumerator* lists all files under the given directory path. It assigns Splits to the next reader that requests a Split. Once all Splits are assigned, it responds to requests with *NoMoreSplits*.
  - The *SourceReader* requests a Split and reads the assigned Split (file or file region) and parses it using the given Format. If it does not get another Split, but a *NoMoreSplits* message, it finishes.

**Unbounded Streaming File Source**

This source works the same way as described above, except that the *SplitEnumerator* never responds with *NoMoreSplits* and periodically lists the contents under the given URI/Path to check for new files. Once it finds new files, it generates new Splits for them and can assign them to the available SourceReaders.

**Unbounded Streaming Kafka Source**

The source has a Kafka Topic (or list of Topics or Topic regex) and a *Deserializer* to parse the records.

{'fraction_non_alphanumeric': 0.061917366785590976, 'fraction_numerical': 0.0018414086776383934, 'mean_word_length': 4.517460317460317, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4430231', 'n_tokens_mistral': 2243, 'n_tokens_neox': 2121, 'n_words': 1260}