186 lines
8.6 KiB
Plaintext
186 lines
8.6 KiB
Plaintext
= Deploying the Debezium 1.0 Developer Preview with AMQ Streams
|
|
include::../_attributes.adoc[]
|
|
:linkattrs:
|
|
:icons: font
|
|
|
|
The following describes how to set up the https://developers.redhat.com/products/amq/download/[Debezium 1.0.0 Developer Preview connectors] for change data capture on Red Hat's https://www.openshift.com/[OpenShift] container platform.
|
|
|
|
[NOTE]
|
|
====
|
|
This and other documentation pages mention ```Debezium 1.0.0 Developer Preview```, please note that this version corresponds to version ```0.10.0.Beta2``` of upstream https://github.com/debezium/debezium/tree/v0.10.0.Beta2[Debezium project].
|
|
====
|
|
|
|
== Apache Kafka Deployment
|
|
|
|
For setting up Apache Kafka and Apache Kafka Connect on OpenShift, https://access.redhat.com/products/red-hat-amq#streams[Red Hat AMQ Streams] can be used, which offers "Kafka as a Service".
|
|
It consists of an enterprise grade operator and images that bring Kafka to OpenShift.
|
|
|
|
Follow these steps to prepare your Kafka cluster:
|
|
|
|
* Install the AMQ Streams operator by following steps in https://access.redhat.com/documentation/en-us/red_hat_amq/7.3/html/using_amq_streams_on_openshift_container_platform/getting-started-str#downloads-str[AMQ docs].
|
|
* Select the desired configuration and https://access.redhat.com/documentation/en-us/red_hat_amq/7.3/html/using_amq_streams_on_openshift_container_platform/getting-started-str#kafka-cluster-str[deploy your Kafka Cluster].
|
|
* Deploy https://access.redhat.com/documentation/en-us/red_hat_amq/7.3/html/using_amq_streams_on_openshift_container_platform/getting-started-str#using-kafka-connect-with-plug-ins-str[Kafka Connect s2i].
|
|
|
|
We now have a working ```Kafka cluster``` running within ```OpenShift``` with ```Kafka Connect s2i```.
|
|
|
|
After a while and depending on your Kafka configuration, these pods should be running (assuming you used "debezium-kafka-cluster" as the AMQ Streams Kafka cluster name):
|
|
|
|
[source%nowrap,bash]
|
|
----
|
|
$ oc get pods
|
|
|
|
NAME READY STATUS RESTARTS AGE
|
|
debezium-kafka-cluster-entity-operator-7b6b9d4c5f-k7b92 3/3 Running 0 5m
|
|
debezium-kafka-cluster-kafka-0 2/2 Running 0 4m
|
|
debezium-kafka-cluster-zookeeper-0 2/2 Running 0 4m
|
|
strimzi-cluster-operator-97cd5cf7b-l58bq 1/1 Running 0 6m
|
|
----
|
|
|
|
In addition to running pods you should have a ```DeploymentConfig``` associated with your Connect ```s2i```.
|
|
|
|
[NOTE]
|
|
====
|
|
Instead of utilising the ```KafkaConnectS2I``` you can create a new ```Dockerfile``` based on AMQ's Kafka image, including the required Debezium connectors.
|
|
The precise instruction can also be found in https://access.redhat.com/documentation/en-us/red_hat_amq/7.3/html/using_amq_streams_on_openshift_container_platform/getting-started-str#using-kafka-connect-with-plug-ins-str[AMQ Streams documentation].
|
|
In this case the last item in the list above can be omitted.
|
|
====
|
|
|
|
== Debezium Deployment
|
|
|
|
To deploy a Kafka Cluster with the Debezium Developer Preview connectors we need to
|
|
|
|
* Download the connector archives
|
|
* Download the associated database drivers
|
|
* Prepare the plugin directory structure
|
|
* Start the OpenShift s2i build to deploy the final Kafka Connect cluster
|
|
|
|
=== Downloading Database Drivers
|
|
|
|
The Debezium Developer Preview comes without database drivers included. The table below links to driver dependencies which need to be included in a directory of each repsective connector prior to triggering the ```s2i``` build.
|
|
|
|
[cols="25%a,75%a", options="header", role="table table-bordered table-striped"]
|
|
|===
|
|
| Connector
|
|
| Driver
|
|
|
|
| MySql
|
|
| http://central.maven.org/maven2/mysql/mysql-connector-java/8.0.16/mysql-connector-java-8.0.16.jar[mysql-connector-java-8.0.16.jar]
|
|
|
|
| PostgreSQL
|
|
| http://central.maven.org/maven2/org/postgresql/postgresql/42.2.5/postgresql-42.2.5.jar[postgresql-42.2.5.jar]
|
|
|
|
| SQL Server
|
|
| http://central.maven.org/maven2/com/microsoft/sqlserver/mssql-jdbc/7.2.2.jre8/mssql-jdbc-7.2.2.jre8.jar[mssql-jdbc-7.2.2.jre8.jar]
|
|
|
|
| MongoDB
|
|
| https://repo1.maven.org/maven2/org/mongodb/mongodb-driver-core/3.9.1/mongodb-driver-core-3.9.1.jar[mongodb-driver-core-3.9.1.jar] +
|
|
https://repo1.maven.org/maven2/org/mongodb/mongodb-driver/3.9.1/mongodb-driver-3.9.1.jar[mongodb-driver-3.9.1.jar] +
|
|
https://repo1.maven.org/maven2/org/mongodb/bson/3.9.1/bson-3.9.1.jar[bson-3.9.1.jar]
|
|
|===
|
|
|
|
=== Preparing Directory Structure
|
|
|
|
Download the connector archives from https://developers.redhat.com/products/amq/download[Red Hat Developer Portal], extract the archives, and download the required driver denendencies.
|
|
Once complete you should end with the following directory structure:
|
|
|
|
[source%nowrap,bash]
|
|
----
|
|
|
|
$ tree ./plugins
|
|
plugins
|
|
├── debezium-connector-mongodb
|
|
│ ├── CHANGELOG.md
|
|
│ ├── CONTRIBUTE.md
|
|
│ ├── COPYRIGHT.txt
|
|
│ ├── LICENSE-3rd-PARTIES.txt
|
|
│ ├── LICENSE.txt
|
|
│ ├── README.md
|
|
│ ├── bson-3.9.1.jar
|
|
│ ├── debezium-connector-mongodb-0.10.0.Beta1-redhat-00001.jar
|
|
│ ├── debezium-core-0.10.0.Beta1-redhat-00001.jar
|
|
│ ├── mongodb-driver-3.9.1.jar
|
|
│ └── mongodb-driver-core-3.9.1.jar
|
|
├── debezium-connector-mysql
|
|
│ ├── CHANGELOG.md
|
|
│ ├── CONTRIBUTE.md
|
|
│ ├── COPYRIGHT.txt
|
|
│ ├── LICENSE-3rd-PARTIES.txt
|
|
│ ├── LICENSE.txt
|
|
│ ├── README.md
|
|
│ ├── antlr4-runtime-4.7.0.redhat-00007.jar
|
|
│ ├── debezium-connector-mysql-0.10.0.Beta1-redhat-00001.jar
|
|
│ ├── debezium-core-0.10.0.Beta1-redhat-00001.jar
|
|
│ ├── debezium-ddl-parser-0.10.0.Beta1-redhat-00001.jar
|
|
│ ├── mysql-binlog-connector-java-0.19.1.redhat-00002.jar
|
|
│ └── mysql-connector-java-8.0.16.jar
|
|
├── debezium-connector-postgres
|
|
│ ├── CHANGELOG.md
|
|
│ ├── CONTRIBUTE.md
|
|
│ ├── COPYRIGHT.txt
|
|
│ ├── LICENSE-3rd-PARTIES.txt
|
|
│ ├── LICENSE.txt
|
|
│ ├── README.md
|
|
│ ├── debezium-connector-postgres-0.10.0.Beta1-redhat-00001.jar
|
|
│ ├── debezium-core-0.10.0.Beta1-redhat-00001.jar
|
|
│ ├── postgresql-42.2.5.jar
|
|
│ └── protobuf-java-2.6.1.redhat-1.jar
|
|
└── debezium-connector-sqlserver
|
|
├── CHANGELOG.md
|
|
├── CONTRIBUTE.md
|
|
├── COPYRIGHT.txt
|
|
├── LICENSE-3rd-PARTIES.txt
|
|
├── LICENSE.txt
|
|
├── README.md
|
|
├── debezium-connector-sqlserver-0.10.0.Beta1-redhat-00001.jar
|
|
├── debezium-core-0.10.0.Beta1-redhat-00001.jar
|
|
└── mssql-jdbc-7.2.2.jre8.jar
|
|
----
|
|
|
|
=== Deploying Kafka Connect with the Debezium Developer Preview
|
|
|
|
With plugin directory structure ready, all we have to do now in order to deploy our Kafka Connect cluster is triggering the ```s2i``` build.
|
|
|
|
[source%nowrap,bash]
|
|
----
|
|
# Check the name of our build config
|
|
$ oc get buildconfigs
|
|
|
|
NAME TYPE FROM LATEST
|
|
debezium-kafka-connect-cluster-connect Source Binary 2
|
|
|
|
# Build and deploy Kafka Connect cluster with Debezium
|
|
$ oc start-build debezium-kafka-connect-cluster-connect --from-dir=./plugins
|
|
----
|
|
|
|
After a while all parts should be up and running:
|
|
|
|
[source%nowrap,bash]
|
|
----
|
|
oc get pods
|
|
|
|
NAME READY STATUS RESTARTS AGE
|
|
debezium-kafka-cluster-entity-operator-7b6b9d4c5f-k7b92 3/3 Running 0 10m
|
|
debezium-kafka-cluster-kafka-0 2/2 Running 0 9m
|
|
debezium-kafka-cluster-zookeeper-0 2/2 Running 0 9m
|
|
debezium-kafka-connect-cluster-connect-2-jw695 1/1 Running 0 1m
|
|
debezium-kafka-connect-cluster-connect-2-deploy 0/1 Completed 0 3m
|
|
strimzi-cluster-operator-97cd5cf7b-l58bq 1/1 Running 0 11m
|
|
----
|
|
|
|
Alternatively, you can go to the "Pods" view of your OpenShift Web Console to confirm all pods are up and running:
|
|
|
|
image::/images/openshift_amq_pods.png[width=771,align="center"]
|
|
|
|
== Verifying the Deployment
|
|
|
|
Next we are going to verify whether the deployment is correct by emulating the xref:tutorial.adoc[Debezium Tutorial] and following the steps in xref:openshift.adoc#verifying_the_deployment[OpenShift Installation]
|
|
|
|
[NOTE]
|
|
====
|
|
You will want to change the pod names to correspond with your AMQ Streams deployment.
|
|
====
|
|
|
|
== Questions and Resources
|
|
In case of any requests or questions related to running the Debezium Developer Preview with AMQ Streams on OpenShift,
|
|
please let us know by sending an e-mail to mailto:debezium-cdc-preview@redhat.com[debezium-cdc-preview] mailing list.
|