= Deploying the Debezium 1.0 Developer Preview with AMQ Streams include::../_attributes.adoc[] :linkattrs: :icons: font The following describes how to set up the https://developers.redhat.com/products/amq/download/[Debezium 1.0.0 Developer Preview connectors] for change data capture on Red Hat's https://www.openshift.com/[OpenShift] container platform. [NOTE] ==== This and other documentation pages mention ```Debezium 1.0.0 Developer Preview```, please note that this version corresponds to version ```0.10.0.Beta2``` of upstream https://github.com/debezium/debezium/tree/v0.10.0.Beta2[Debezium project]. ==== == Apache Kafka Deployment For setting up Apache Kafka and Apache Kafka Connect on OpenShift, https://access.redhat.com/products/red-hat-amq#streams[Red Hat AMQ Streams] can be used, which offers "Kafka as a Service". It consists of an enterprise grade operator and images that bring Kafka to OpenShift. Follow these steps to prepare your Kafka cluster: * Install the AMQ Streams operator by following steps in https://access.redhat.com/documentation/en-us/red_hat_amq/7.3/html/using_amq_streams_on_openshift_container_platform/getting-started-str#downloads-str[AMQ docs]. * Select the desired configuration and https://access.redhat.com/documentation/en-us/red_hat_amq/7.3/html/using_amq_streams_on_openshift_container_platform/getting-started-str#kafka-cluster-str[deploy your Kafka Cluster]. * Deploy https://access.redhat.com/documentation/en-us/red_hat_amq/7.3/html/using_amq_streams_on_openshift_container_platform/getting-started-str#using-kafka-connect-with-plug-ins-str[Kafka Connect s2i]. We now have a working ```Kafka cluster``` running within ```OpenShift``` with ```Kafka Connect s2i```. After a while and depending on your Kafka configuration, these pods should be running (assuming you used "debezium-kafka-cluster" as the AMQ Streams Kafka cluster name): [source%nowrap,bash] ---- $ oc get pods NAME READY STATUS RESTARTS AGE debezium-kafka-cluster-entity-operator-7b6b9d4c5f-k7b92 3/3 Running 0 5m debezium-kafka-cluster-kafka-0 2/2 Running 0 4m debezium-kafka-cluster-zookeeper-0 2/2 Running 0 4m strimzi-cluster-operator-97cd5cf7b-l58bq 1/1 Running 0 6m ---- In addition to running pods you should have a ```DeploymentConfig``` associated with your Connect ```s2i```. [NOTE] ==== Instead of utilising the ```KafkaConnectS2I``` you can create a new ```Dockerfile``` based on AMQ's Kafka image, including the required Debezium connectors. The precise instruction can also be found in https://access.redhat.com/documentation/en-us/red_hat_amq/7.3/html/using_amq_streams_on_openshift_container_platform/getting-started-str#using-kafka-connect-with-plug-ins-str[AMQ Streams documentation]. In this case the last item in the list above can be omitted. ==== == Debezium Deployment To deploy a Kafka Cluster with the Debezium Developer Preview connectors we need to * Download the connector archives * Download the associated database drivers * Prepare the plugin directory structure * Start the OpenShift s2i build to deploy the final Kafka Connect cluster === Downloading Database Drivers The Debezium Developer Preview comes without database drivers included. The table below links to driver dependencies which need to be included in a directory of each repsective connector prior to triggering the ```s2i``` build. [cols="25%a,75%a", options="header", role="table table-bordered table-striped"] |=== | Connector | Driver | MySql | http://central.maven.org/maven2/mysql/mysql-connector-java/8.0.16/mysql-connector-java-8.0.16.jar[mysql-connector-java-8.0.16.jar] | PostgreSQL | http://central.maven.org/maven2/org/postgresql/postgresql/42.2.5/postgresql-42.2.5.jar[postgresql-42.2.5.jar] | SQL Server | http://central.maven.org/maven2/com/microsoft/sqlserver/mssql-jdbc/7.2.2.jre8/mssql-jdbc-7.2.2.jre8.jar[mssql-jdbc-7.2.2.jre8.jar] | MongoDB | https://repo1.maven.org/maven2/org/mongodb/mongodb-driver-core/3.9.1/mongodb-driver-core-3.9.1.jar[mongodb-driver-core-3.9.1.jar] + https://repo1.maven.org/maven2/org/mongodb/mongodb-driver/3.9.1/mongodb-driver-3.9.1.jar[mongodb-driver-3.9.1.jar] + https://repo1.maven.org/maven2/org/mongodb/bson/3.9.1/bson-3.9.1.jar[bson-3.9.1.jar] |=== === Preparing Directory Structure Download the connector archives from https://developers.redhat.com/products/amq/download[Red Hat Developer Portal], extract the archives, and download the required driver denendencies. Once complete you should end with the following directory structure: [source%nowrap,bash] ---- $ tree ./plugins plugins ├── debezium-connector-mongodb │   ├── CHANGELOG.md │   ├── CONTRIBUTE.md │   ├── COPYRIGHT.txt │   ├── LICENSE-3rd-PARTIES.txt │   ├── LICENSE.txt │   ├── README.md │   ├── bson-3.9.1.jar │   ├── debezium-connector-mongodb-0.10.0.Beta1-redhat-00001.jar │   ├── debezium-core-0.10.0.Beta1-redhat-00001.jar │   ├── mongodb-driver-3.9.1.jar │   └── mongodb-driver-core-3.9.1.jar ├── debezium-connector-mysql │   ├── CHANGELOG.md │   ├── CONTRIBUTE.md │   ├── COPYRIGHT.txt │   ├── LICENSE-3rd-PARTIES.txt │   ├── LICENSE.txt │   ├── README.md │   ├── antlr4-runtime-4.7.0.redhat-00007.jar │   ├── debezium-connector-mysql-0.10.0.Beta1-redhat-00001.jar │   ├── debezium-core-0.10.0.Beta1-redhat-00001.jar │   ├── debezium-ddl-parser-0.10.0.Beta1-redhat-00001.jar │   ├── mysql-binlog-connector-java-0.19.1.redhat-00002.jar │   └── mysql-connector-java-8.0.16.jar ├── debezium-connector-postgres │   ├── CHANGELOG.md │   ├── CONTRIBUTE.md │   ├── COPYRIGHT.txt │   ├── LICENSE-3rd-PARTIES.txt │   ├── LICENSE.txt │   ├── README.md │   ├── debezium-connector-postgres-0.10.0.Beta1-redhat-00001.jar │   ├── debezium-core-0.10.0.Beta1-redhat-00001.jar │   ├── postgresql-42.2.5.jar │   └── protobuf-java-2.6.1.redhat-1.jar └── debezium-connector-sqlserver ├── CHANGELOG.md ├── CONTRIBUTE.md ├── COPYRIGHT.txt ├── LICENSE-3rd-PARTIES.txt ├── LICENSE.txt ├── README.md ├── debezium-connector-sqlserver-0.10.0.Beta1-redhat-00001.jar ├── debezium-core-0.10.0.Beta1-redhat-00001.jar └── mssql-jdbc-7.2.2.jre8.jar ---- === Deploying Kafka Connect with the Debezium Developer Preview With plugin directory structure ready, all we have to do now in order to deploy our Kafka Connect cluster is triggering the ```s2i``` build. [source%nowrap,bash] ---- # Check the name of our build config $ oc get buildconfigs NAME TYPE FROM LATEST debezium-kafka-connect-cluster-connect Source Binary 2 # Build and deploy Kafka Connect cluster with Debezium $ oc start-build debezium-kafka-connect-cluster-connect --from-dir=./plugins ---- After a while all parts should be up and running: [source%nowrap,bash] ---- oc get pods NAME READY STATUS RESTARTS AGE debezium-kafka-cluster-entity-operator-7b6b9d4c5f-k7b92 3/3 Running 0 10m debezium-kafka-cluster-kafka-0 2/2 Running 0 9m debezium-kafka-cluster-zookeeper-0 2/2 Running 0 9m debezium-kafka-connect-cluster-connect-2-jw695 1/1 Running 0 1m debezium-kafka-connect-cluster-connect-2-deploy 0/1 Completed 0 3m strimzi-cluster-operator-97cd5cf7b-l58bq 1/1 Running 0 11m ---- Alternatively, you can go to the "Pods" view of your OpenShift Web Console to confirm all pods are up and running: image::/images/openshift_amq_pods.png[width=771,align="center"] == Verifying the Deployment Next we are going to verify whether the deployment is correct by emulating the xref:tutorial.adoc[Debezium Tutorial] and following the steps in xref:openshift.adoc#verifying_the_deployment[OpenShift Installation] [NOTE] ==== You will want to change the pod names to correspond with your AMQ Streams deployment. ==== == Questions and Resources In case of any requests or questions related to running the Debezium Developer Preview with AMQ Streams on OpenShift, please let us know by sending an e-mail to mailto:debezium-cdc-preview@redhat.com[debezium-cdc-preview] mailing list.