tet123/documentation/modules/ROOT/pages/operations/amq-streams.adoc

166 lines
7.6 KiB
Plaintext
Raw Normal View History

= Deploying the Debezium 1.0 Technology Preview with AMQ Streams
include::../_attributes.adoc[]
:linkattrs:
:icons: font
:toc:
:toc-placement: macro
toc::[]
The following describes how to set up the https://developers.redhat.com/products/amq/download/[Debezium 1.0.0 Technology Preview connectors] for change data capture on Red Hat's https://www.openshift.com/[OpenShift] container platform.
[NOTE]
====
This and other documentation pages mention ```Debezium 1.0.0 Technology Preview```, please note that this version corresponds to version ```1.0.0.Beta2``` of upstream https://github.com/debezium/debezium/tree/v1.0.0.Beta2[Debezium project].
====
== Apache Kafka Deployment
For setting up Apache Kafka and Apache Kafka Connect on OpenShift, https://access.redhat.com/products/red-hat-amq#streams[Red Hat AMQ Streams] can be used, which offers "Kafka as a Service".
It consists of an enterprise grade operator and images that bring Kafka to OpenShift.
Follow these steps to prepare your Kafka cluster:
* Install the AMQ Streams operator by following steps in https://access.redhat.com/documentation/en-us/red_hat_amq/7.5/html/using_amq_streams_on_openshift/getting-started-str#downloads-str[AMQ docs].
* Select the desired configuration and https://access.redhat.com/documentation/en-us/red_hat_amq/7.5/html/using_amq_streams_on_openshift/getting-started-str#downloads-str[deploy your Kafka Cluster].
* Deploy https://access.redhat.com/documentation/en-us/red_hat_amq/7.5/html/using_amq_streams_on_openshift/getting-started-str#using-kafka-connect-with-plug-ins-str[Kafka Connect s2i].
We now have a working ```Kafka cluster``` running within ```OpenShift``` with ```Kafka Connect s2i```.
After a while and depending on your Kafka configuration, these pods should be running (assuming you used "debezium-kafka-cluster" as the AMQ Streams Kafka cluster name):
[source%nowrap,bash]
----
$ oc get pods
NAME READY STATUS RESTARTS AGE
debezium-kafka-cluster-entity-operator-7b6b9d4c5f-k7b92 3/3 Running 0 5m
debezium-kafka-cluster-kafka-0 2/2 Running 0 4m
debezium-kafka-cluster-zookeeper-0 2/2 Running 0 4m
strimzi-cluster-operator-97cd5cf7b-l58bq 1/1 Running 0 6m
----
In addition to running pods you should have a ```DeploymentConfig``` associated with your Connect ```s2i```.
[NOTE]
====
Instead of utilising the ```KafkaConnectS2I``` you can create a new ```Dockerfile``` based on AMQ's Kafka image, including the required Debezium connectors.
The precise instruction can also be found in https://access.redhat.com/documentation/en-us/red_hat_amq/7.3/html/using_amq_streams_on_openshift_container_platform/getting-started-str#using-kafka-connect-with-plug-ins-str[AMQ Streams documentation].
In this case the last item in the list above can be omitted.
====
== Debezium Deployment
To deploy a Kafka Cluster with the Debezium Technology Preview connectors we need to
* Download the connector archives
* Prepare the plugin directory structure
* Start the OpenShift s2i build to deploy the final Kafka Connect cluster
=== Preparing Directory Structure
Download the connector archives from https://developers.redhat.com/products/amq/download[Red Hat Developer Portal], extract the archives, and download the required driver dependencies.
Once complete you should end with the following directory structure:
[source%nowrap,bash]
----
$ tree ./plugins
plugins
├── debezium-connector-mongodb
│   ├── CHANGELOG.md
│   ├── CONTRIBUTE.md
│   ├── COPYRIGHT.txt
│   ├── LICENSE-3rd-PARTIES.txt
│   ├── LICENSE.txt
│   ├── README.md
│   ├── bson-3.10.1.redhat-00001.jar
│   ├── debezium-connector-mongodb-1.0.0.Beta2-redhat-00001.jar
│   ├── debezium-core-1.0.0.Beta2-redhat-00001.jar
│   ├── mongodb-driver-3.10.1.redhat-00001.jar
│   ├── mongodb-driver-core-3.10.1.redhat-00001.jar
│   └── util-3.10.1.redhat-00001.jar
├── debezium-connector-mysql
│   ├── CHANGELOG.md
│   ├── CONTRIBUTE.md
│   ├── COPYRIGHT.txt
│   ├── LICENSE-3rd-PARTIES.txt
│   ├── LICENSE.txt
│   ├── README.md
│   ├── antlr4-runtime-4.7.0.redhat-00013.jar
│   ├── debezium-connector-mysql-1.0.0.Beta2-redhat-00001.jar
│   ├── debezium-core-1.0.0.Beta2-redhat-00001.jar
│   ├── debezium-ddl-parser-1.0.0.Beta2-redhat-00001.jar
│   ├── mysql-binlog-connector-java-0.19.1.redhat-00002.jar
│   └── mysql-connector-java-8.0.16.redhat-00001.jar
├── debezium-connector-postgres
│   ├── CHANGELOG.md
│   ├── CONTRIBUTE.md
│   ├── COPYRIGHT.txt
│   ├── LICENSE-3rd-PARTIES.txt
│   ├── LICENSE.txt
│   ├── README.md
│   ├── debezium-connector-postgres-1.0.0.Beta2-redhat-00001.jar
│   ├── debezium-core-1.0.0.Beta2-redhat-00001.jar
│   ├── postgresql-42.2.8.redhat-00001.jar
│   └── protobuf-java-3.8.0.redhat-00001.jar
└── debezium-connector-sqlserver
├── CHANGELOG.md
├── CONTRIBUTE.md
├── COPYRIGHT.txt
├── LICENSE-3rd-PARTIES.txt
├── LICENSE.txt
├── README.md
├── debezium-connector-sqlserver-1.0.0.Beta2-redhat-00001.jar
├── debezium-core-1.0.0.Beta2-redhat-00001.jar
└── mssql-jdbc-7.2.2.jre8-redhat-00001.jar
----
=== Deploying Kafka Connect with the Debezium Technology Preview
With plugin directory structure ready, all we have to do now in order to deploy our Kafka Connect cluster is triggering the ```s2i``` build.
[source%nowrap,bash]
----
# Check the name of our build config
$ oc get buildconfigs
NAME TYPE FROM LATEST
debezium-kafka-connect-cluster-connect Source Binary 2
# Build and deploy Kafka Connect cluster with Debezium
$ oc start-build debezium-kafka-connect-cluster-connect --from-dir=./plugins
----
After a while all parts should be up and running:
[source%nowrap,bash]
----
oc get pods
NAME READY STATUS RESTARTS AGE
debezium-kafka-cluster-entity-operator-7b6b9d4c5f-k7b92 3/3 Running 0 10m
debezium-kafka-cluster-kafka-0 2/2 Running 0 9m
debezium-kafka-cluster-zookeeper-0 2/2 Running 0 9m
debezium-kafka-connect-cluster-connect-2-jw695 1/1 Running 0 1m
debezium-kafka-connect-cluster-connect-2-deploy 0/1 Completed 0 3m
strimzi-cluster-operator-97cd5cf7b-l58bq 1/1 Running 0 11m
----
Alternatively, you can go to the "Pods" view of your OpenShift Web Console to confirm all pods are up and running:
image::/images/openshift_amq_pods.png[width=771,align="center"]
== Verifying the Deployment
Next we are going to verify whether the deployment is correct by emulating the xref:tutorial.adoc[Debezium Tutorial] and following the steps in xref:operations/openshift.adoc#verifying_the_deployment[OpenShift Installation]
[NOTE]
====
You will want to change the pod names to correspond with your AMQ Streams deployment.
====
== Questions and Resources
In case of any requests or questions related to running the Debezium Technology Preview with AMQ Streams on OpenShift,
please let us know by sending an e-mail to mailto:debezium-cdc-preview@redhat.com[debezium-cdc-preview] mailing list.