<commit_before>To provide flexibility in managing snapshots, {prodname} includes a supplementary snapshot mechanism, known as _incremental snapshotting_.
Incremental snapshots rely on the {prodname} mechanism for xref:{link-signalling}#sending-signals-to-a-debezium-connector[sending signals to a {prodname} connector].
ifdef::community[]
Incremental snapshots are based on the link:https://github.com/debezium/debezium-design-documents/blob/main/DDD-3.md[DDD-3] design document.
endif::community[]

In an incremental snapshot, instead of capturing the full state of a database all at once, as in an initial snapshot, {prodname} captures each {data-collection} in phases, in a series of configurable chunks.
You can specify the {data-collection}s that you want the snapshot to capture and the xref:{context}-property-incremental-snapshot-chunk-size[size of each chunk].
The chunk size determines the number of rows that the snapshot collects during each fetch operation on the database.
The default chunk size for incremental snapshots is 1 KB.

As an incremental snapshot proceeds, {prodname} uses watermarks to track its progress, maintaining a record of each {data-collection} row that it captures.
This phased approach to capturing data provides the following advantages over the standard initial snapshot process:

* You can run incremental snapshots in parallel with streamed data capture, instead of postponing streaming until the snapshot completes.
  The connector continues to capture near real-time events from the change log throughout the snapshot process, and neither operation blocks the other.
* If the progress of an incremental snapshot is interrupted, you can resume it without losing any data.
  After the process resumes, the snapshot begins at the point where it stopped, rather than recapturing the {data-collection} from the beginning.
* You can run an incremental snapshot on demand at any time, and repeat the process as needed to adapt to database updates.
  For example, you might re-run a snapshot after you modify the connector configuration to add a {data-collection} to its xref:{context}-property-{data-collection}-include-list[`{data-collection}.include.list`] property.

.Incremental snapshot process
When you run an incremental snapshot, {prodname} sorts each {data-collection} by primary key and then splits the {data-collection} into chunks based on the xref:{context}-property-incremental-snapshot-chunk-size[configured chunk size].
Working chunk by chunk, it then captures each {data-collection} row in a chunk.
For each row that it captures, the snapshot emits a `READ` event.
That event represents the value of the row when the snapshot for the chunk began.

As a snapshot proceeds, it’s likely that other processes continue to access the database, potentially modifying {data-collection} records.
To reflect such changes, `INSERT`, `UPDATE`, or `DELETE` operations are committed to the transaction log as per usual.
Similarly, the ongoing {prodname} streaming process continues to detect these change events and emits corresponding change event records to Kafka.

.How {prodname} resolves collisions among records with the same primary key
In some cases, the `UPDATE` or `DELETE` events that the streaming process emits are received out of sequence.
That is, the streaming process might emit an event that modifies a {data-collection} row before the snapshot captures the chunk that contains the `READ` event for that row.
When the snapshot eventually emits the corresponding `READ` event for the row, its value is already superseded.
To ensure that incremental snapshot events that arrive out of sequence are processed in the correct logical order, {prodname} employs a buffering scheme for resolving collisions.
Only after collisions between the snapshot events and the streamed events are resolved does {prodname} emit an event record to Kafka.

.Snapshot window
To assist in resolving collisions between late-arriving `READ` events and streamed events that modify the same {data-collection} row, {prodname} employs a so-called _snapshot window_.
The snapshot windows demarcates the interval during which an incremental snapshot captures data for a specified {data-collection} chunk.
Before the snapshot window for a chunk opens, {prodname} follows its usual behavior and emits events from the transaction log directly downstream to the target Kafka topic.
But from the moment that the snapshot for a particular chunk opens, until it closes, {prodname} performs a de-duplication step to resolve collisions between events that have the same primary key..

For each data collection, the {prodname} emits two types of events, and stores the records for them both in a single destination Kafka topic.
The snapshot records that it  captures directly from a table are emitted as `READ` operations.
Meanwhile, as users continue to update records in the data collection, and the transaction log is updated to reflect each commit, {prodname} emits `UPDATE` or `DELETE` operations for each change.

As the snapshot window opens, and {prodname} begins processing a snapshot chunk, it delivers snapshot records to a memory buffer.
During the snapshot windows, the primary keys of the `READ` events in the buffer are compared to the primary keys of the incoming streamed events.
If no match is found, the streamed event record is sent directly to Kafka.
If {prodname} detects a match, it discards the buffered `READ` event, and writes the streamed record to the destination topic, because the streamed event logically supersede the static snapshot event.
After the snapshot window for the chunk closes, the buffer contains only `READ` events for which no related transaction log events exist.
{prodname} emits these remaining `READ` events to the {data-collection}'s Kafka topic.

The connector repeats the process for each snapshot chunk.

.Triggering an incremental snapshot

Currently, the only way to initiate an incremental snapshot is to send an xref:{link-signalling}#debezium-signaling-ad-hoc-snapshots[ad hoc snapshot signal] to the signaling {data-collection} on the source database.
You submit signals to the {data-collection} as SQL `INSERT` queries.
After {prodname} detects the change in the signaling {data-collection}, it reads the signal, and runs the requested snapshot operation.

The query that you submit specifies the {data-collection}s to include in the snapshot, and, optionally, specifies the kind of snapshot operation.
Currently, the only valid option for snapshots operations is the default value, `incremental`.

To specify the {data-collection}s to include in the snapshot, provide a `data-collections` array that lists the {data-collection}s or an array of regular expressions used to match {data-collection}s, for example, +
`{"data-collections": ["public.MyFirstTable", "public.MySecondTable"]}` +

The `data-collections` array for an incremental snapshot signal has no default value.
If the `data-collections` array is empty, {prodname} detects that no action is required and does not perform a snapshot.

.Prerequisites

* xref:{link-signalling}#debezium-signaling-enabling-signaling[Signaling is enabled]. +
** A signaling data collection exists on the source database and the connector is configured to capture it.
** The signaling data collection is specified in the xref:{context}-property-signal-data-collection[`signal.data.collection`] property.

.Procedure

. Send a SQL query to add the ad hoc incremental snapshot request to the signaling {data-collection}:
+
[source,sql,indent=0,subs="+attributes"]
----
INSERT INTO _<signalTable>_ (id, type, data) VALUES (_'<id>'_, _'<snapshotType>'_, '{"data-collections": ["_<tableName>_","_<tableName>_"],"type":"_<snapshotType>_"}');
----
+
For example,
+
[source,sql,indent=0,subs="+attributes"]
----
INSERT INTO myschema.debezium_signal (id, type, data) VALUES('ad-hoc-1', 'execute-snapshot', '{"data-collections": ["schema1.table1", "schema2.table2"],"type":"incremental"}');
----
The values of the `id`,`type`, and `data` parameters in the command correspond to the xref:{link-signalling}#debezium-signaling-required-structure-of-a-signaling-data-collection[fields of the signaling {data-collection}].
+
The following {data-collection} describes the these parameters:
+
.Descriptions of fields in a SQL command for sending an incremental snapshot signal to the signaling {data-collection}
[cols="1,4",options="header"]
|===
|Value |Description

|`myschema.debezium_signal`
|Specifies the fully-qualified name of the signaling {data-collection} on the source database

|`ad-hoc-1`
| The `id` parameter specifies an arbitrary string that is assigned as the `id` identifier for the signal request. +
Use this string to identify logging messages to entries in the signaling {data-collection}.
{prodname} does not use this string.
Rather, during the snapshot, {prodname} generates its own `id` string as a watermarking signal.

|`execute-snapshot`
| Specifies `type` parameter specifies the operation that the signal is intended to trigger. +

|`data-collections`
|A required component of the `data` field of a signal that specifies an array of {data-collection} names or regular expressions to match {data-collection} names to include in the snapshot. +
The array lists regular expressions which match {data-collection}s by their fully-qualified names, using the same format as you use to specify the name of the connector's signaling {data-collection} in the xref:{context}-property-signal-data-collection[`signal.data.collection`] configuration property.

|`incremental`
|An optional `type` component of the `data` field of a signal that specifies the kind of snapshot operation to run. +
Currently, the only valid option is the default value, `incremental`. +
Specifying a `type` value in the SQL query that you submit to the signaling {data-collection} is optional. +
If you do not specify a value, the connector runs an incremental snapshot.
|===

The following example, shows the JSON for an incremental snapshot event that is captured by a connector.

.Example: Incremental snapshot event message
[source,json,index=0]
----
{
    "before":null,
    "after": {
        "pk":"1",
        "value":"New data"
    },
    "source": {
        ...
        "snapshot":"incremental" <1>
    },
    "op":"r", <2>
    "ts_ms":"1620393591654",
    "transaction":null
}
----
[cols="1,1,4",options="header"]
|===
|Item |Field name |Description
|1
|`snapshot`
|Specifies the type of snapshot operation to run. +
Currently, the only valid option is the default value, `incremental`. +
Specifying a `type` value in the SQL query that you submit to the signaling {data-collection} is optional. +
If you do not specify a value, the connector runs an incremental snapshot.

|2
|`op`
|Specifies the event type. +
The value for snapshot events is `r`, signifying a `READ` operation.

|===<commit_msg>DBZ-4251 Add stopping signal to documentation<commit_after>To provide flexibility in managing snapshots, {prodname} includes a supplementary snapshot mechanism, known as _incremental snapshotting_.
Incremental snapshots rely on the {prodname} mechanism for xref:{link-signalling}#sending-signals-to-a-debezium-connector[sending signals to a {prodname} connector].
ifdef::community[]
Incremental snapshots are based on the link:https://github.com/debezium/debezium-design-documents/blob/main/DDD-3.md[DDD-3] design document.
endif::community[]

In an incremental snapshot, instead of capturing the full state of a database all at once, as in an initial snapshot, {prodname} captures each {data-collection} in phases, in a series of configurable chunks.
You can specify the {data-collection}s that you want the snapshot to capture and the xref:{context}-property-incremental-snapshot-chunk-size[size of each chunk].
The chunk size determines the number of rows that the snapshot collects during each fetch operation on the database.
The default chunk size for incremental snapshots is 1 KB.

As an incremental snapshot proceeds, {prodname} uses watermarks to track its progress, maintaining a record of each {data-collection} row that it captures.
This phased approach to capturing data provides the following advantages over the standard initial snapshot process:

* You can run incremental snapshots in parallel with streamed data capture, instead of postponing streaming until the snapshot completes.
  The connector continues to capture near real-time events from the change log throughout the snapshot process, and neither operation blocks the other.
* If the progress of an incremental snapshot is interrupted, you can resume it without losing any data.
  After the process resumes, the snapshot begins at the point where it stopped, rather than recapturing the {data-collection} from the beginning.
* You can run an incremental snapshot on demand at any time, and repeat the process as needed to adapt to database updates.
  For example, you might re-run a snapshot after you modify the connector configuration to add a {data-collection} to its xref:{context}-property-{data-collection}-include-list[`{data-collection}.include.list`] property.

.Incremental snapshot process
When you run an incremental snapshot, {prodname} sorts each {data-collection} by primary key and then splits the {data-collection} into chunks based on the xref:{context}-property-incremental-snapshot-chunk-size[configured chunk size].
Working chunk by chunk, it then captures each {data-collection} row in a chunk.
For each row that it captures, the snapshot emits a `READ` event.
That event represents the value of the row when the snapshot for the chunk began.

As a snapshot proceeds, it’s likely that other processes continue to access the database, potentially modifying {data-collection} records.
To reflect such changes, `INSERT`, `UPDATE`, or `DELETE` operations are committed to the transaction log as per usual.
Similarly, the ongoing {prodname} streaming process continues to detect these change events and emits corresponding change event records to Kafka.

.How {prodname} resolves collisions among records with the same primary key
In some cases, the `UPDATE` or `DELETE` events that the streaming process emits are received out of sequence.
That is, the streaming process might emit an event that modifies a {data-collection} row before the snapshot captures the chunk that contains the `READ` event for that row.
When the snapshot eventually emits the corresponding `READ` event for the row, its value is already superseded.
To ensure that incremental snapshot events that arrive out of sequence are processed in the correct logical order, {prodname} employs a buffering scheme for resolving collisions.
Only after collisions between the snapshot events and the streamed events are resolved does {prodname} emit an event record to Kafka.

.Snapshot window
To assist in resolving collisions between late-arriving `READ` events and streamed events that modify the same {data-collection} row, {prodname} employs a so-called _snapshot window_.
The snapshot windows demarcates the interval during which an incremental snapshot captures data for a specified {data-collection} chunk.
Before the snapshot window for a chunk opens, {prodname} follows its usual behavior and emits events from the transaction log directly downstream to the target Kafka topic.
But from the moment that the snapshot for a particular chunk opens, until it closes, {prodname} performs a de-duplication step to resolve collisions between events that have the same primary key..

For each data collection, the {prodname} emits two types of events, and stores the records for them both in a single destination Kafka topic.
The snapshot records that it  captures directly from a table are emitted as `READ` operations.
Meanwhile, as users continue to update records in the data collection, and the transaction log is updated to reflect each commit, {prodname} emits `UPDATE` or `DELETE` operations for each change.

As the snapshot window opens, and {prodname} begins processing a snapshot chunk, it delivers snapshot records to a memory buffer.
During the snapshot windows, the primary keys of the `READ` events in the buffer are compared to the primary keys of the incoming streamed events.
If no match is found, the streamed event record is sent directly to Kafka.
If {prodname} detects a match, it discards the buffered `READ` event, and writes the streamed record to the destination topic, because the streamed event logically supersede the static snapshot event.
After the snapshot window for the chunk closes, the buffer contains only `READ` events for which no related transaction log events exist.
{prodname} emits these remaining `READ` events to the {data-collection}'s Kafka topic.

The connector repeats the process for each snapshot chunk.

.Triggering an incremental snapshot

Currently, the only way to initiate an incremental snapshot is to send an xref:{link-signalling}#debezium-signaling-ad-hoc-snapshots[ad hoc snapshot signal] to the signaling {data-collection} on the source database.
You submit signals to the {data-collection} as SQL `INSERT` queries.
After {prodname} detects the change in the signaling {data-collection}, it reads the signal, and runs the requested snapshot operation.

The query that you submit specifies the {data-collection}s to include in the snapshot, and, optionally, specifies the kind of snapshot operation.
Currently, the only valid option for snapshots operations is the default value, `incremental`.

To specify the {data-collection}s to include in the snapshot, provide a `data-collections` array that lists the {data-collection}s or an array of regular expressions used to match {data-collection}s, for example, +
`{"data-collections": ["public.MyFirstTable", "public.MySecondTable"]}` +

The `data-collections` array for an incremental snapshot signal has no default value.
If the `data-collections` array is empty, {prodname} detects that no action is required and does not perform a snapshot.

.Prerequisites

* xref:{link-signalling}#debezium-signaling-enabling-signaling[Signaling is enabled]. +
** A signaling data collection exists on the source database and the connector is configured to capture it.
** The signaling data collection is specified in the xref:{context}-property-signal-data-collection[`signal.data.collection`] property.

.Procedure

. Send a SQL query to add the ad hoc incremental snapshot request to the signaling {data-collection}:
+
[source,sql,indent=0,subs="+attributes"]
----
INSERT INTO _<signalTable>_ (id, type, data) VALUES (_'<id>'_, _'<snapshotType>'_, '{"data-collections": ["_<tableName>_","_<tableName>_"],"type":"_<snapshotType>_"}');
----
+
For example,
+
[source,sql,indent=0,subs="+attributes"]
----
INSERT INTO myschema.debezium_signal (id, type, data) VALUES('ad-hoc-1', 'execute-snapshot', '{"data-collections": ["schema1.table1", "schema2.table2"],"type":"incremental"}');
----
The values of the `id`,`type`, and `data` parameters in the command correspond to the xref:{link-signalling}#debezium-signaling-required-structure-of-a-signaling-data-collection[fields of the signaling {data-collection}].
+
The following {data-collection} describes the these parameters:
+
.Descriptions of fields in a SQL command for sending an incremental snapshot signal to the signaling {data-collection}
[cols="1,4",options="header"]
|===
|Value |Description

|`myschema.debezium_signal`
|Specifies the fully-qualified name of the signaling {data-collection} on the source database

|`ad-hoc-1`
| The `id` parameter specifies an arbitrary string that is assigned as the `id` identifier for the signal request. +
Use this string to identify logging messages to entries in the signaling {data-collection}.
{prodname} does not use this string.
Rather, during the snapshot, {prodname} generates its own `id` string as a watermarking signal.

|`execute-snapshot`
| Specifies `type` parameter specifies the operation that the signal is intended to trigger. +

|`data-collections`
|A required component of the `data` field of a signal that specifies an array of {data-collection} names or regular expressions to match {data-collection} names to include in the snapshot. +
The array lists regular expressions which match {data-collection}s by their fully-qualified names, using the same format as you use to specify the name of the connector's signaling {data-collection} in the xref:{context}-property-signal-data-collection[`signal.data.collection`] configuration property.

|`incremental`
|An optional `type` component of the `data` field of a signal that specifies the kind of snapshot operation to run. +
Currently, the only valid option is the default value, `incremental`. +
Specifying a `type` value in the SQL query that you submit to the signaling {data-collection} is optional. +
If you do not specify a value, the connector runs an incremental snapshot.
|===

The following example, shows the JSON for an incremental snapshot event that is captured by a connector.

.Example: Incremental snapshot event message
[source,json,index=0]
----
{
    "before":null,
    "after": {
        "pk":"1",
        "value":"New data"
    },
    "source": {
        ...
        "snapshot":"incremental" <1>
    },
    "op":"r", <2>
    "ts_ms":"1620393591654",
    "transaction":null
}
----
[cols="1,1,4",options="header"]
|===
|Item |Field name |Description
|1
|`snapshot`
|Specifies the type of snapshot operation to run. +
Currently, the only valid option is the default value, `incremental`. +
Specifying a `type` value in the SQL query that you submit to the signaling {data-collection} is optional. +
If you do not specify a value, the connector runs an incremental snapshot.

|2
|`op`
|Specifies the event type. +
The value for snapshot events is `r`, signifying a `READ` operation.

|===

.Stopping an incremental snapshot

Incremental snapshots can also be stopped by sending a signal to the {data-collection} on the source database.
You submit signals to the {data-collection} as SQL `INSERT` queries.
After {prodname} detects the change in the signaling {data-collection}, it reads the signal, and stops the incremental snapshot operation if it's in progress.

The query that you submit specifies the snapshot operation of `incremental`, and, optionally, the {data-collection}s of the current running snapshot to be removed.

.Prerequisites

* xref:{link-signalling}#debezium-signaling-enabling-signaling[Signaling is enabled]. +
** A signaling data collection exists on the source database and the connector is configured to capture it.
** The signaling data collection is specified in the xref:{context}-property-signal-data-collection[`signal.data.collection`] property.

.Procedure

. Send a SQL query to stop the ad hoc incremental snapshot to the signaling {data-collection}:
+
[source,sql,indent=0,subs="+attributes"]
----
INSERT INTO _<signalTable>_ (id, type, data) values (_'<id>'_, 'stop-snapshot', '{"data-collections": ["_<tableName>_","_<tableName>_"],"type":"incremental"}');
----
+
For example,
+
[source,sql,indent=0,subs="+attributes"]
----
INSERT INTO myschema.debezium_signal (id, type, dat) values ('ad-hoc-1', 'stop-snapshot', '{"data-collections": ["schema1.table1", "schema2.table2"],"type":"incremental"}');
----
The values of the `id`, `type`, and `data` parameters in the command correspond to the xref:{link-signaling}#debezium-signaling-required-structure-of-a-signaling-data-collection[fields of the signaling {data-collection}].
+
The following {data-collection} describes these parameters:
+
.Descriptions of fields in a SQL command for sending a stop incremental snapshot signal to the signaling {data-collection}
[cols="1,4",options="header"]
|===
|Value |Description

|`myschema.debezium_signal`
|Specifies the fully-qualified name of the signaling {data-collection} on the source database

|`ad-hoc-1`
| The `id` parameter specifies an arbitrary string that is assigned as the `id` identifier for the signal request. +
Use this string to identify logging messages to entries in the signaling {data-collection}.
{prodname} does not use this string.

|`stop-snapshot`
| Specifies `type` parameter specifies the operation that the signal is intended to trigger. +

|`data-collections`
|An optional component of the `data` field of a signal that specifies an array of {data-collection} names or regular expressions to match {data-collection} names to remove from the snapshot. +
The array lists regular expressions which match {data-collection}s by their fully-qualified names, using the same format as you use to specify the name of the connector's signaling {data-collection} in the xref:{context}-property-signal-data-collection[`signal.data.collection`] configuration property.
If this component of the `data` field is omitted, the signal stops the entire incremental snapshot that is in progress.

|`incremental`
|A required component of the `data` field of a signal that specifies the kind of snapshot operation that is to be stopped. +
Currently, the only valid option is `incremental`. +
Specifying a `type` value in the SQL query that you submit to the signaling {data-collection} is required. +
If you do not specify a value, the signal will not stop the incremental snapshot.
|===
{'fraction_non_alphanumeric': 0.07975630019385212, 'fraction_numerical': 0.003323179174743838, 'mean_word_length': 5.133948070856588, 'pattern_counts': {'":': 32, '<': 23, '<?xml version=': 0, '>': 23, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4675126', 'n_tokens_mistral': 6554, 'n_tokens_neox': 6288, 'n_words': 3241}