<commit_before>= Deploying the Debezium 1.0 Developer Preview with AMQ Streams
include::../_attributes.adoc[]
:linkattrs:
:icons: font
:toc:
:toc-placement: macro

toc::[]

The following describes how to set up the https://developers.redhat.com/products/amq/download/[Debezium 1.0.0 Developer Preview connectors] for change data capture on Red Hat's https://www.openshift.com/[OpenShift] container platform.

[NOTE]
====
This and other documentation pages mention ```Debezium 1.0.0 Developer Preview```, please note that this version corresponds to version ```0.10.0.Beta2``` of upstream https://github.com/debezium/debezium/tree/v0.10.0.Beta2[Debezium project].
====

== Apache Kafka Deployment

For setting up Apache Kafka and Apache Kafka Connect on OpenShift, https://access.redhat.com/products/red-hat-amq#streams[Red Hat AMQ Streams] can be used, which offers "Kafka as a Service".
It consists of an enterprise grade operator and images that bring Kafka to OpenShift.

Follow these steps to prepare your Kafka cluster:

* Install the AMQ Streams operator by following steps in https://access.redhat.com/documentation/en-us/red_hat_amq/7.3/html/using_amq_streams_on_openshift_container_platform/getting-started-str#downloads-str[AMQ docs].
* Select the desired configuration and https://access.redhat.com/documentation/en-us/red_hat_amq/7.3/html/using_amq_streams_on_openshift_container_platform/getting-started-str#kafka-cluster-str[deploy your Kafka Cluster].
* Deploy https://access.redhat.com/documentation/en-us/red_hat_amq/7.3/html/using_amq_streams_on_openshift_container_platform/getting-started-str#using-kafka-connect-with-plug-ins-str[Kafka Connect s2i].

We now have a working ```Kafka cluster``` running within ```OpenShift``` with ```Kafka Connect s2i```.

After a while and depending on your Kafka configuration, these pods should be running (assuming you used "debezium-kafka-cluster" as the AMQ Streams Kafka cluster name):

[source%nowrap,bash]
----
$ oc get pods

NAME                                                      READY     STATUS      RESTARTS   AGE
debezium-kafka-cluster-entity-operator-7b6b9d4c5f-k7b92   3/3       Running     0          5m
debezium-kafka-cluster-kafka-0                            2/2       Running     0          4m
debezium-kafka-cluster-zookeeper-0                        2/2       Running     0          4m
strimzi-cluster-operator-97cd5cf7b-l58bq                  1/1       Running     0          6m
----

In addition to running pods you should have a ```DeploymentConfig``` associated with your Connect ```s2i```.

[NOTE]
====
Instead of utilising the ```KafkaConnectS2I``` you can create a new ```Dockerfile``` based on AMQ's Kafka image, including the required Debezium connectors.
The precise instruction can also be found in https://access.redhat.com/documentation/en-us/red_hat_amq/7.3/html/using_amq_streams_on_openshift_container_platform/getting-started-str#using-kafka-connect-with-plug-ins-str[AMQ Streams documentation].
In this case the last item in the list above can be omitted.
====

== Debezium Deployment

To deploy a Kafka Cluster with the Debezium Developer Preview connectors we need to

* Download the connector archives
* Download the associated database drivers
* Prepare the plugin directory structure
* Start the OpenShift s2i build to deploy the final Kafka Connect cluster

=== Downloading Database Drivers

The Debezium Developer Preview comes without database drivers included. The table below links to driver dependencies which need to be included in a directory of each repsective connector prior to triggering the ```s2i``` build.

[cols="25%a,75%a", options="header", role="table table-bordered table-striped"]
|===
| Connector
| Driver

| MySql
| http://central.maven.org/maven2/mysql/mysql-connector-java/8.0.16/mysql-connector-java-8.0.16.jar[mysql-connector-java-8.0.16.jar]

| PostgreSQL
| http://central.maven.org/maven2/org/postgresql/postgresql/42.2.5/postgresql-42.2.5.jar[postgresql-42.2.5.jar]

| SQL Server
| http://central.maven.org/maven2/com/microsoft/sqlserver/mssql-jdbc/7.2.2.jre8/mssql-jdbc-7.2.2.jre8.jar[mssql-jdbc-7.2.2.jre8.jar]

| MongoDB
| https://repo1.maven.org/maven2/org/mongodb/mongodb-driver-core/3.9.1/mongodb-driver-core-3.9.1.jar[mongodb-driver-core-3.9.1.jar] +
  https://repo1.maven.org/maven2/org/mongodb/mongodb-driver/3.9.1/mongodb-driver-3.9.1.jar[mongodb-driver-3.9.1.jar] +
  https://repo1.maven.org/maven2/org/mongodb/bson/3.9.1/bson-3.9.1.jar[bson-3.9.1.jar]
|===

=== Preparing Directory Structure

Download the connector archives from https://developers.redhat.com/products/amq/download[Red Hat Developer Portal], extract the archives, and download the required driver denendencies.
Once complete you should end with the following directory structure:

[source%nowrap,bash]
----

$ tree ./plugins
plugins
├── debezium-connector-mongodb
│   ├── CHANGELOG.md
│   ├── CONTRIBUTE.md
│   ├── COPYRIGHT.txt
│   ├── LICENSE-3rd-PARTIES.txt
│   ├── LICENSE.txt
│   ├── README.md
│   ├── bson-3.9.1.jar
│   ├── debezium-connector-mongodb-0.10.0.Beta1-redhat-00001.jar
│   ├── debezium-core-0.10.0.Beta1-redhat-00001.jar
│   ├── mongodb-driver-3.9.1.jar
│   └── mongodb-driver-core-3.9.1.jar
├── debezium-connector-mysql
│   ├── CHANGELOG.md
│   ├── CONTRIBUTE.md
│   ├── COPYRIGHT.txt
│   ├── LICENSE-3rd-PARTIES.txt
│   ├── LICENSE.txt
│   ├── README.md
│   ├── antlr4-runtime-4.7.0.redhat-00007.jar
│   ├── debezium-connector-mysql-0.10.0.Beta1-redhat-00001.jar
│   ├── debezium-core-0.10.0.Beta1-redhat-00001.jar
│   ├── debezium-ddl-parser-0.10.0.Beta1-redhat-00001.jar
│   ├── mysql-binlog-connector-java-0.19.1.redhat-00002.jar
│   └── mysql-connector-java-8.0.16.jar
├── debezium-connector-postgres
│   ├── CHANGELOG.md
│   ├── CONTRIBUTE.md
│   ├── COPYRIGHT.txt
│   ├── LICENSE-3rd-PARTIES.txt
│   ├── LICENSE.txt
│   ├── README.md
│   ├── debezium-connector-postgres-0.10.0.Beta1-redhat-00001.jar
│   ├── debezium-core-0.10.0.Beta1-redhat-00001.jar
│   ├── postgresql-42.2.5.jar
│   └── protobuf-java-2.6.1.redhat-1.jar
└── debezium-connector-sqlserver
    ├── CHANGELOG.md
    ├── CONTRIBUTE.md
    ├── COPYRIGHT.txt
    ├── LICENSE-3rd-PARTIES.txt
    ├── LICENSE.txt
    ├── README.md
    ├── debezium-connector-sqlserver-0.10.0.Beta1-redhat-00001.jar
    ├── debezium-core-0.10.0.Beta1-redhat-00001.jar
    └── mssql-jdbc-7.2.2.jre8.jar
----

=== Deploying Kafka Connect with the Debezium Developer Preview

With plugin directory structure ready, all we have to do now in order to deploy our Kafka Connect cluster is triggering the ```s2i``` build.

[source%nowrap,bash]
----
# Check the name of our build config
$ oc get buildconfigs

NAME                                     TYPE     FROM     LATEST
debezium-kafka-connect-cluster-connect   Source   Binary   2

# Build and deploy Kafka Connect cluster with Debezium
$ oc start-build debezium-kafka-connect-cluster-connect --from-dir=./plugins
----

After a while all parts should be up and running:

[source%nowrap,bash]
----
oc get pods

NAME                                                      READY     STATUS      RESTARTS   AGE
debezium-kafka-cluster-entity-operator-7b6b9d4c5f-k7b92   3/3       Running     0          10m
debezium-kafka-cluster-kafka-0                            2/2       Running     0          9m
debezium-kafka-cluster-zookeeper-0                        2/2       Running     0          9m
debezium-kafka-connect-cluster-connect-2-jw695            1/1       Running     0          1m
debezium-kafka-connect-cluster-connect-2-deploy           0/1       Completed   0          3m
strimzi-cluster-operator-97cd5cf7b-l58bq                  1/1       Running     0          11m
----

Alternatively, you can go to the "Pods" view of your OpenShift Web Console to confirm all pods are up and running:

image::/images/openshift_amq_pods.png[width=771,align="center"]

== Verifying the Deployment

Next we are going to verify whether the deployment is correct by emulating the xref:tutorial.adoc[Debezium Tutorial] and following the steps in xref:operations/openshift.adoc#verifying_the_deployment[OpenShift Installation]

[NOTE]
====
You will want to change the pod names to correspond with your AMQ Streams deployment.
====

== Questions and Resources
In case of any requests or questions related to running the Debezium Developer Preview with AMQ Streams on OpenShift,
please let us know by sending an e-mail to mailto:<EMAIL>[debezium-cdc-preview] mailing list.<commit_msg>DBZ-1543 Update documentation based on Technology Preview<commit_after>= Deploying the Debezium 1.0 Technology Preview with AMQ Streams
include::../_attributes.adoc[]
:linkattrs:
:icons: font
:toc:
:toc-placement: macro

toc::[]

The following describes how to set up the https://developers.redhat.com/products/amq/download/[Debezium 1.0.0 Technology Preview connectors] for change data capture on Red Hat's https://www.openshift.com/[OpenShift] container platform.

[NOTE]
====
This and other documentation pages mention ```Debezium 1.0.0 Technology Preview```, please note that this version corresponds to version ```1.0.0.Beta1``` of upstream https://github.com/debezium/debezium/tree/v1.0.0.Beta1[Debezium project].
====

== Apache Kafka Deployment

For setting up Apache Kafka and Apache Kafka Connect on OpenShift, https://access.redhat.com/products/red-hat-amq#streams[Red Hat AMQ Streams] can be used, which offers "Kafka as a Service".
It consists of an enterprise grade operator and images that bring Kafka to OpenShift.

Follow these steps to prepare your Kafka cluster:

* Install the AMQ Streams operator by following steps in https://access.redhat.com/documentation/en-us/red_hat_amq/7.3/html/using_amq_streams_on_openshift_container_platform/getting-started-str#downloads-str[AMQ docs].
* Select the desired configuration and https://access.redhat.com/documentation/en-us/red_hat_amq/7.3/html/using_amq_streams_on_openshift_container_platform/getting-started-str#kafka-cluster-str[deploy your Kafka Cluster].
* Deploy https://access.redhat.com/documentation/en-us/red_hat_amq/7.3/html/using_amq_streams_on_openshift_container_platform/getting-started-str#using-kafka-connect-with-plug-ins-str[Kafka Connect s2i].

We now have a working ```Kafka cluster``` running within ```OpenShift``` with ```Kafka Connect s2i```.

After a while and depending on your Kafka configuration, these pods should be running (assuming you used "debezium-kafka-cluster" as the AMQ Streams Kafka cluster name):

[source%nowrap,bash]
----
$ oc get pods

NAME                                                      READY     STATUS      RESTARTS   AGE
debezium-kafka-cluster-entity-operator-7b6b9d4c5f-k7b92   3/3       Running     0          5m
debezium-kafka-cluster-kafka-0                            2/2       Running     0          4m
debezium-kafka-cluster-zookeeper-0                        2/2       Running     0          4m
strimzi-cluster-operator-97cd5cf7b-l58bq                  1/1       Running     0          6m
----

In addition to running pods you should have a ```DeploymentConfig``` associated with your Connect ```s2i```.

[NOTE]
====
Instead of utilising the ```KafkaConnectS2I``` you can create a new ```Dockerfile``` based on AMQ's Kafka image, including the required Debezium connectors.
The precise instruction can also be found in https://access.redhat.com/documentation/en-us/red_hat_amq/7.3/html/using_amq_streams_on_openshift_container_platform/getting-started-str#using-kafka-connect-with-plug-ins-str[AMQ Streams documentation].
In this case the last item in the list above can be omitted.
====

== Debezium Deployment

To deploy a Kafka Cluster with the Debezium Technology Preview connectors we need to

* Download the connector archives
* Download the associated database drivers
* Prepare the plugin directory structure
* Start the OpenShift s2i build to deploy the final Kafka Connect cluster

=== Downloading Database Drivers

The Debezium Technology Preview comes without database drivers included. The table below links to driver dependencies which need to be included in a directory of each repsective connector prior to triggering the ```s2i``` build.

[cols="25%a,75%a", options="header", role="table table-bordered table-striped"]
|===
| Connector
| Driver

| MySql
| http://central.maven.org/maven2/mysql/mysql-connector-java/8.0.16/mysql-connector-java-8.0.16.jar[mysql-connector-java-8.0.16.jar]

| PostgreSQL
| http://central.maven.org/maven2/org/postgresql/postgresql/42.2.8/postgresql-42.2.8.jar[postgresql-42.2.8.jar]

| SQL Server
| http://central.maven.org/maven2/com/microsoft/sqlserver/mssql-jdbc/7.2.2.jre8/mssql-jdbc-7.2.2.jre8.jar[mssql-jdbc-7.2.2.jre8.jar]

| MongoDB
| https://repo1.maven.org/maven2/org/mongodb/mongodb-driver-core/3.9.1/mongodb-driver-core-3.9.1.jar[mongodb-driver-core-3.9.1.jar] +
  https://repo1.maven.org/maven2/org/mongodb/mongodb-driver/3.9.1/mongodb-driver-3.9.1.jar[mongodb-driver-3.9.1.jar] +
  https://repo1.maven.org/maven2/org/mongodb/bson/3.9.1/bson-3.9.1.jar[bson-3.9.1.jar]
|===

=== Preparing Directory Structure

Download the connector archives from https://developers.redhat.com/products/amq/download[Red Hat Developer Portal], extract the archives, and download the required driver dependencies.
Once complete you should end with the following directory structure:

[source%nowrap,bash]
----

$ tree ./plugins
plugins
├── debezium-connector-mongodb
│   ├── CHANGELOG.md
│   ├── CONTRIBUTE.md
│   ├── COPYRIGHT.txt
│   ├── LICENSE-3rd-PARTIES.txt
│   ├── LICENSE.txt
│   ├── README.md
│   ├── bson-3.9.1.jar
│   ├── debezium-connector-mongodb-1.0.0.Beta1-redhat-00001.jar
│   ├── debezium-core-1.0.0.Beta1-redhat-00001.jar
│   ├── mongodb-driver-3.9.1.jar
│   └── mongodb-driver-core-3.9.1.jar
├── debezium-connector-mysql
│   ├── CHANGELOG.md
│   ├── CONTRIBUTE.md
│   ├── COPYRIGHT.txt
│   ├── LICENSE-3rd-PARTIES.txt
│   ├── LICENSE.txt
│   ├── README.md
│   ├── antlr4-runtime-4.7.0.redhat-00007.jar
│   ├── debezium-connector-mysql-1.0.0.Beta1-redhat-00001.jar
│   ├── debezium-core-1.0.0.Beta1-redhat-00001.jar
│   ├── debezium-ddl-parser-1.0.0.Beta1-redhat-00001.jar
│   ├── mysql-binlog-connector-java-0.19.1.redhat-00002.jar
│   └── mysql-connector-java-8.0.16.jar
├── debezium-connector-postgres
│   ├── CHANGELOG.md
│   ├── CONTRIBUTE.md
│   ├── COPYRIGHT.txt
│   ├── LICENSE-3rd-PARTIES.txt
│   ├── LICENSE.txt
│   ├── README.md
│   ├── debezium-connector-postgres-1.0.0.Beta1-redhat-00001.jar
│   ├── debezium-core-1.0.0.Beta1-redhat-00001.jar
│   ├── postgresql-42.2.8.jar
│   └── protobuf-java-2.6.1.redhat-1.jar
└── debezium-connector-sqlserver
    ├── CHANGELOG.md
    ├── CONTRIBUTE.md
    ├── COPYRIGHT.txt
    ├── LICENSE-3rd-PARTIES.txt
    ├── LICENSE.txt
    ├── README.md
    ├── debezium-connector-sqlserver-1.0.0.Beta1-redhat-00001.jar
    ├── debezium-core-1.0.0.Beta1-redhat-00001.jar
    └── mssql-jdbc-7.2.2.jre8.jar
----

=== Deploying Kafka Connect with the Debezium Technology Preview

With plugin directory structure ready, all we have to do now in order to deploy our Kafka Connect cluster is triggering the ```s2i``` build.

[source%nowrap,bash]
----
# Check the name of our build config
$ oc get buildconfigs

NAME                                     TYPE     FROM     LATEST
debezium-kafka-connect-cluster-connect   Source   Binary   2

# Build and deploy Kafka Connect cluster with Debezium
$ oc start-build debezium-kafka-connect-cluster-connect --from-dir=./plugins
----

After a while all parts should be up and running:

[source%nowrap,bash]
----
oc get pods

NAME                                                      READY     STATUS      RESTARTS   AGE
debezium-kafka-cluster-entity-operator-7b6b9d4c5f-k7b92   3/3       Running     0          10m
debezium-kafka-cluster-kafka-0                            2/2       Running     0          9m
debezium-kafka-cluster-zookeeper-0                        2/2       Running     0          9m
debezium-kafka-connect-cluster-connect-2-jw695            1/1       Running     0          1m
debezium-kafka-connect-cluster-connect-2-deploy           0/1       Completed   0          3m
strimzi-cluster-operator-97cd5cf7b-l58bq                  1/1       Running     0          11m
----

Alternatively, you can go to the "Pods" view of your OpenShift Web Console to confirm all pods are up and running:

image::/images/openshift_amq_pods.png[width=771,align="center"]

== Verifying the Deployment

Next we are going to verify whether the deployment is correct by emulating the xref:tutorial.adoc[Debezium Tutorial] and following the steps in xref:operations/openshift.adoc#verifying_the_deployment[OpenShift Installation]

[NOTE]
====
You will want to change the pod names to correspond with your AMQ Streams deployment.
====

== Questions and Resources
In case of any requests or questions related to running the Debezium Technology Preview with AMQ Streams on OpenShift,
please let us know by sending an e-mail to mailto:<EMAIL>[debezium-cdc-preview] mailing list.
{'fraction_non_alphanumeric': 0.12235667293233082, 'fraction_numerical': 0.038240131578947366, 'mean_word_length': 3.189222440944882, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 24, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 41, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7588002', 'n_tokens_mistral': 6231, 'n_tokens_neox': 5922, 'n_words': 1473}