257 lines
8.8 KiB
Plaintext
257 lines
8.8 KiB
Plaintext
[id="deploying-debezium-on-openshift"]
|
|
= Deploying {prodname} on OpenShift
|
|
|
|
:linkattrs:
|
|
:icons: font
|
|
:toc:
|
|
:toc-placement: macro
|
|
|
|
toc::[]
|
|
|
|
This procedure is for setting up {prodname} connectors on Red Hat's link:https://www.openshift.com/[OpenShift] container platform. These instructions have been tested with the two most recent releases of OpenShift. These instructions should also work on any other Kubernetes distribution by using the `kubectl` command.
|
|
|
|
To get started more quickly, try the link:https://learn.openshift.com/middleware/debezium-getting-started/[{prodname} online learning scenario].
|
|
It starts an OpenShift cluster just for you, which lets you start using {prodname} in your browser within a few minutes.
|
|
|
|
== {prodname} Deployment
|
|
|
|
To set up Apache Kafka and Kafka Connect on OpenShift, use the set of images that are provided by the link:https://strimzi.io/[Strimzi] project. These images offer "Kafka as a Service" by providing enterprise grade configuration files and images that bring Kafka to Kubernetes and OpenShift, as well as Kubernetes operators for running Kafka there.
|
|
|
|
.Prerequisites
|
|
|
|
* The OpenShift command line interface (`oc`) is installed.
|
|
* Docker is installed.
|
|
|
|
.Procedure
|
|
|
|
. In your OpenShift project, enter the following commands to install the operators and templates for the Kafka broker and Kafka Connect:
|
|
+
|
|
[source,shell,subs="attributes",options="nowrap"]
|
|
----
|
|
export STRIMZI_VERSION={strimzi-version}
|
|
git clone -b $STRIMZI_VERSION https://github.com/strimzi/strimzi-kafka-operator
|
|
cd strimzi-kafka-operator
|
|
|
|
# Switch to an admin user to create security objects as part of installation:
|
|
oc login -u system:admin
|
|
oc create -f install/cluster-operator && oc create -f examples/templates/cluster-operator
|
|
----
|
|
+
|
|
To learn more about setting up Apache Kafka with Strimzi on Kubernetes and OpenShift, see link:https://strimzi.io/docs/operators/latest/overview.html#kafka-components_str[Strimzi deployment of Kafka].
|
|
|
|
. Deploy a Kafka broker cluster:
|
|
+
|
|
[source,shell,subs="attributes",options="nowrap"]
|
|
----
|
|
# Deploy an ephemeral single instance Kafka broker:
|
|
oc process strimzi-ephemeral -p CLUSTER_NAME=broker -p ZOOKEEPER_NODE_COUNT=1 -p KAFKA_NODE_COUNT=1 -p KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 -p KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR=1 | oc apply -f -
|
|
----
|
|
|
|
. Create a Kafka Connect image with the {prodname} connectors installed:
|
|
|
|
.. Download and extract the archive for each {prodname} connector you want to run. For example:
|
|
+
|
|
[source,subs="attributes",options="nowrap"]
|
|
----
|
|
curl https://repo1.maven.org/maven2/io/debezium/debezium-connector-mysql/{debezium-version}/debezium-connector-mysql-{debezium-version}-plugin.tar.gz tar xvz`
|
|
----
|
|
|
|
.. Create a `Dockerfile` that uses a Strimzi Kafka image as the base image. The following example creates a `plugins/debezium` directory, which would contain a directory for each {prodname} connector that you want to run. To run more than one {prodname} connector, insert a `COPY` line for each connector.
|
|
+
|
|
[source,subs=+macros,subs="attributes"]
|
|
----
|
|
FROM strimzi/kafka:{strimzi-version}-kafka-2.5.0
|
|
USER root:root
|
|
RUN mkdir -p /opt/kafka/plugins/debezium
|
|
COPY ./debezium-connector-mysql/ /opt/kafka/plugins/debezium/
|
|
USER 1001
|
|
----
|
|
+
|
|
Before Kafka Connect starts running the connector, Kafka Connect loads any third-party plug-ins that are in the `/opt/kafka/plugins` directory.
|
|
|
|
.. Build a Debezium image from your Dockerfile and push it to your preferred container registry, for example, `quay.io` or Docker Hub, by executing the following commands. Replace `debezium-community` with the name of your Docker Hub organization.
|
|
+
|
|
----
|
|
export DOCKER_ORG=debezium-community
|
|
docker build . -t ${DOCKER_ORG}/connect-debezium
|
|
docker push ${DOCKER_ORG}/connect-debezium
|
|
----
|
|
+
|
|
After a while all parts should be up and running:
|
|
+
|
|
[source%nowrap,bash]
|
|
----
|
|
oc get pods
|
|
|
|
NAME READY STATUS RESTARTS AGE
|
|
broker-entity-operator-5fb7bc8b9b-r86nz 3/3 Running 1 4m
|
|
broker-kafka-0 2/2 Running 0 4m
|
|
broker-zookeeper-0 2/2 Running 0 5m
|
|
debezium-connect-3-4sdjr 1/1 Running 0 1m
|
|
strimzi-cluster-operator-d77476b8f-rblqf 1/1 Running 0 5m
|
|
----
|
|
+
|
|
Alternatively, go to the "Pods" view of your OpenShift Web Console (https://myhost:8443/console/project/myproject/browse/pods) to confirm that all pods are up and running:
|
|
+
|
|
image::openshift_pods.png[width=771,align="center"]
|
|
|
|
== Verifying the Deployment
|
|
|
|
Verify whether the deployment is correct by emulating the xref:tutorial.adoc[{prodname} Tutorial] in the OpenShift environment.
|
|
|
|
. Start a MySQL server instance that contains some example tables:
|
|
+
|
|
[source%nowrap,bash,subs="attributes"]
|
|
----
|
|
# Deploy pre-populated MySQL instance
|
|
oc new-app --name=mysql debezium/example-mysql:{debezium-docker-label}
|
|
|
|
# Configure credentials for the database
|
|
oc set env dc/mysql MYSQL_ROOT_PASSWORD=debezium MYSQL_USER=mysqluser MYSQL_PASSWORD=mysqlpw
|
|
----
|
|
+
|
|
A new pod with MySQL server should be up and running:
|
|
+
|
|
[source%nowrap,bash]
|
|
----
|
|
oc get pods
|
|
NAME READY STATUS RESTARTS AGE
|
|
...
|
|
mysql-1-4503l 1/1 Running 0 2s
|
|
mysql-1-deploy 1/1 Running 0 4s
|
|
...
|
|
----
|
|
|
|
. Register the {prodname} MySQL connector to run against the deployed MySQL instance:
|
|
+
|
|
[source%nowrap,bash]
|
|
----
|
|
oc exec -i -c kafka broker-kafka-0 -- curl -X POST \
|
|
-H "Accept:application/json" \
|
|
-H "Content-Type:application/json" \
|
|
http://debezium-connect-api:8083/connectors -d @- <<'EOF'
|
|
|
|
{
|
|
"name": "inventory-connector",
|
|
"config": {
|
|
"connector.class": "io.debezium.connector.mysql.MySqlConnector",
|
|
"tasks.max": "1",
|
|
"database.hostname": "mysql",
|
|
"database.port": "3306",
|
|
"database.user": "debezium",
|
|
"database.password": "dbz",
|
|
"database.server.id": "184054",
|
|
"database.server.name": "dbserver1",
|
|
"database.include.list": "inventory",
|
|
"database.history.kafka.bootstrap.servers": "broker-kafka-bootstrap:9092",
|
|
"database.history.kafka.topic": "schema-changes.inventory"
|
|
}
|
|
}
|
|
EOF
|
|
----
|
|
+
|
|
Kafka Connect's log file should contain messages regarding execution of the initial snapshot:
|
|
+
|
|
[source%nowrap,bash]
|
|
----
|
|
oc logs $(oc get pods -o name -l strimzi.io/name=debezium-connect)
|
|
----
|
|
|
|
. Read change events for the `customers` table from the corresponding Kafka topic:
|
|
+
|
|
[source%nowrap,bash]
|
|
----
|
|
oc exec -it broker-kafka-0 -- /opt/kafka/bin/kafka-console-consumer.sh \
|
|
--bootstrap-server localhost:9092 \
|
|
--from-beginning \
|
|
--property print.key=true \
|
|
--topic dbserver1.inventory.customers
|
|
----
|
|
+
|
|
You should see an output like the following (formatted for the sake of readability):
|
|
+
|
|
[source%nowrap,json,subs="attributes"]
|
|
----
|
|
# Message 1
|
|
{
|
|
"id": 1001
|
|
}
|
|
|
|
# Message 1 Value
|
|
{
|
|
"before": null,
|
|
"after": {
|
|
"id": 1001,
|
|
"first_name": "Sally",
|
|
"last_name": "Thomas",
|
|
"email": "sally.thomas@acme.com"
|
|
},
|
|
"source": {
|
|
"version": "{debezium-version}",
|
|
"connector": "mysql",
|
|
"name": "dbserver1",
|
|
"server_id": 0,
|
|
"ts_sec": 0,
|
|
"gtid": null,
|
|
"file": "mysql-bin.000003",
|
|
"pos": 154,
|
|
"row": 0,
|
|
"snapshot": true,
|
|
"thread": null,
|
|
"db": "inventory",
|
|
"table": "customers"
|
|
},
|
|
"op": "c",
|
|
"ts_ms": 1509530901446
|
|
}
|
|
|
|
# Message 2 Key
|
|
{
|
|
"id": 1002
|
|
}
|
|
|
|
# Message 2 Value
|
|
{
|
|
"before": null,
|
|
"after": {
|
|
"id": 1002,
|
|
"first_name": "George",
|
|
"last_name": "Bailey",
|
|
"email": "gbailey@foobar.com"
|
|
},
|
|
"source": {
|
|
"version": "{debezium-version}",
|
|
"connector": "mysql",
|
|
"name": "dbserver1",
|
|
"server_id": 0,
|
|
"ts_sec": 0,
|
|
"gtid": null,
|
|
"file": "mysql-bin.000003",
|
|
"pos": 154,
|
|
"row": 0,
|
|
"snapshot": true,
|
|
"thread": null,
|
|
"db": "inventory",
|
|
"table": "customers"
|
|
},
|
|
"op": "c",
|
|
"ts_ms": 1509530901446
|
|
}
|
|
...
|
|
----
|
|
|
|
. Modify some records in the `customers` table of the database:
|
|
+
|
|
[source%nowrap,bash]
|
|
----
|
|
oc exec -it $(oc get pods -o custom-columns=NAME:.metadata.name --no-headers -l app=mysql) \
|
|
-- bash -c 'mysql -u $MYSQL_USER -p$MYSQL_PASSWORD inventory'
|
|
|
|
# For example, run UPDATE customers SET email="sally.thomas@example.com" WHERE ID = 1001;
|
|
----
|
|
+
|
|
You should now see additional change events in the consumer started previously.
|
|
|
|
If you have any questions or requests related to running {prodname} on Kubernetes or OpenShift,
|
|
then please let us know in our https://groups.google.com/forum/#!forum/debezium[user group] or in the {prodname} https://debezium.zulipchat.com/#narrow/stream/302533-dev[developer's chat].
|