tet123/documentation/modules/ROOT/pages/operations/openshift.adoc

[id="deploying-debezium-on-openshift"]
= Deploying {prodname} on OpenShift

:linkattrs:
:icons: font
:toc:
:toc-placement: macro

toc::[]

This procedure is for setting up {prodname} connectors on Red Hat's link:https://www.openshift.com/[OpenShift] container platform.
For development or testing on OpenShift you can use https://developers.redhat.com/products/codeready-containers/overview[CodeRady Containers].

To get started more quickly, try the link:https://learn.openshift.com/middleware/debezium-getting-started/[{prodname} online learning scenario].
It starts an OpenShift cluster just for you, which lets you start using {prodname} in your browser within a few minutes.

== Prerequisites

To keep containers separated from other workloads on the cluster, create a dedicated project for {prodname}.
In the remainder of this document, the `debezium-example` namespace will be used:

[source,subs="attributes",options="nowrap"]
----
$ oc new-project debezium-example
----

For the {prodname} deployment we will use the https://strimzi.io/[Strimzi] project, which manages the Kafka deployment on OpenShift clusters.
The simplest way for installing Strimzi is to install the Strimzi operator from https://operatorhub.io/[OperatorHub].
Navigate to the "OperatorHub" tab in the OpenShift UI,
select "Strimzi" and click the "install" button.
If you prefer command line tools, you can install the Strimzi operator this way as well:

[source,subs="attributes",options="nowrap"]
----
$ cat << EOF | oc create -f -
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: my-strimzi-kafka-operator
  namespace: openshift-operators
spec:
  channel: stable
  name: strimzi-kafka-operator
  source: operatorhubio-catalog
  sourceNamespace: olm
EOF
----

== Deploying Apache Kafka

Next, deploy a (single-node) Kafka cluster:

[source,bash]
----
$ cat << EOF | oc create -n debezium-example -f -
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: debezium-cluster
spec:
  kafka:
    replicas: 1
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
        authentication:
          type: tls
      - name: external
        port: 9094
        type: nodeport
        tls: false
    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        size: 100Gi
        deleteClaim: false
    config:
      offsets.topic.replication.factor: 1
      transaction.state.log.replication.factor: 1
      transaction.state.log.min.isr: 1
      default.replication.factor: 1
      min.insync.replicas: 1
  zookeeper:
    replicas: 1
    storage:
      type: persistent-claim
      size: 100Gi
      deleteClaim: false
  entityOperator:
    topicOperator: {}
    userOperator: {}
EOF
----

* Wait until it's ready:

[source,subs="attributes",options="nowrap"]
----
$ oc wait kafka/debezium-cluster --for=condition=Ready --timeout=300s
----

== Deploying a Data Source

As a data source, MySQL will be used in the following.
Besides running a pod with MySQL, an appropriate service which will point to the pod with DB itself is needed.
It can be created e.g. as follows:

[source,bash]
----
$ cat << EOF | oc create -f -
apiVersion: v1
kind: Service
metadata:
  name: mysql
spec:
  ports:
  - port: 3306
  selector:
    app: mysql
  clusterIP: None
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql
spec:
  selector:
    matchLabels:
      app: mysql
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - image: quay.io/debezium/example-mysql:{debezium-docker-label}
        name: mysql
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: debezium
        - name: MYSQL_USER
          value: mysqluser
        - name: MYSQL_PASSWORD
          value: mysqlpw
        ports:
        - containerPort: 3306
          name: mysql
EOF
----

== Deploying a {prodname} Connector

To deploy a {prodname} connector, you need to deploy a Kafka Connect cluster with the required connector plug-in(s), before instantiating the actual connector itself.
As the first step, a container image for Kafka Connect with the plug-in has to be created.
If you  already have a container image built and available in the registry, you can skip this step.
In this document, the MySQL connector will be used as an example.

=== Creating Kafka Connect Cluster

Again, we will use Strimzi for creating the Kafka Connect cluster.
Strimzi also can be used for building and pushing the required container image for us.
In fact, both tasks can be merged together and instructions for building the container image can be provided directly within the `KafkaConnect` object specification:

[source,bash]
----
$ cat << EOF | oc create -f -
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: debezium-connect-cluster
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  version: 3.1.0
  replicas: 1
  bootstrapServers: debezium-cluster-kafka-bootstrap:9092
  config:
    group.id: connect-cluster
    offset.storage.topic: connect-cluster-offsets
    config.storage.topic: connect-cluster-configs
    status.storage.topic: connect-cluster-status
    # -1 means it will use the default replication factor configured in the broker
    config.storage.replication.factor: -1
    offset.storage.replication.factor: -1
    status.storage.replication.factor: -1
  build:
    output:
      type: docker
      image: image-registry.openshift-image-registry.svc:5000/debezium-example/debezium-connect-mysql:latest
    plugins:
      - name: debezium-mysql-connector
        artifacts:
          - type: tgz
            url: https://repo1.maven.org/maven2/io/debezium/debezium-connector-mysql/{debezium-version}/debezium-connector-mysql-{debezium-version}-plugin.tar.gz
EOF
----

Here we took the advantage of the OpenShift built-in registry, already running as a service on the OpenShift cluster.

[NOTE]
====
For simplicity, we've skipped the checksum validation for the downloaded artifact.
If you want to be sure the artifact was correctly downloaded, specify its checksum via the `sha512sum` attribute.
See the https://strimzi.io/docs/operators/latest/deploying.html#creating-new-image-using-kafka-connect-build-str[Strimzi documentation] for more details.
====

If you already have a suitable container image either in the local or a remote registry (such as quay.io or DockerHub), you can use this simplified version:

[source,bash]
----
$ cat << EOF | oc create -f -
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: debezium-connect-cluster
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  version: 3.1.0
  image: 10.110.154.103/debezium-connect-mysql:latest
  replicas: 1
  bootstrapServers: debezium-cluster-kafka-bootstrap:9092
  config:
    group.id: connect-cluster
    offset.storage.topic: connect-cluster-offsets
    config.storage.topic: connect-cluster-configs
    status.storage.topic: connect-cluster-status
    # -1 means it will use the default replication factor configured in the broker
    config.storage.replication.factor: -1
    offset.storage.replication.factor: -1
    status.storage.replication.factor: -1
EOF
----

=== Creating a {prodname} Connector

To create a {prodname} connector, you just need to create a `KafkaConnector` with the appropriate configuration, MySQL in this case:

[source,bash]
----
$ cat << EOF | oc create -f -
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
  name: debezium-connector-mysql
  labels:
    strimzi.io/cluster: debezium-connect-cluster
spec:
  class: io.debezium.connector.mysql.MySqlConnector
  tasksMax: 1
  config:
    tasks.max: 1
    database.hostname: mysql
    database.port: 3306
    database.user: debezium
    database.password: dbz
    database.server.id: 184054
    database.server.name: mysql
    database.include.list: inventory
    database.history.kafka.bootstrap.servers: debezium-cluster-kafka-bootstrap:9092
    database.history.kafka.topic: schema-changes.inventory
EOF
----

== Verifying the Deployment

To verify the everything works fine, you can e.g. start watching `mysql.inventory.customers` Kafka topic:

[source,subs="attributes",options="nowrap"]
----
$ oc run -n debezium-example -it --rm --image=quay.io/debezium/tooling:1.2  --restart=Never watcher -- kcat -b debezium-cluster-kafka-bootstrap:9092 -C -o beginning -t mysql.inventory.customers
----

Connect to the MySQL database:

[source,subs="attributes",options="nowrap"]
----
$ oc run -n debezium-example -it --rm --image=mysql:8.0 --restart=Never --env MYSQL_ROOT_PASSWORD=debezium mysqlterm -- mysql -hmysql -P3306 -uroot -pdebezium
----

Do some changes in the `customers` table:

[source,subs="attributes",options="nowrap"]
----
sql> update customers set first_name="Sally Marie" where id=1001;
----

You now should be able to observe the change events on the Kafka topic:

[source,json]
----
{
...
  "payload": {
    "before": {
      "id": 1001,
      "first_name": "Sally",
      "last_name": "Thomas",
      "email": "sally.thomas@acme.com"
    },
    "after": {
      "id": 1001,
      "first_name": "Sally Marie",
      "last_name": "Thomas",
      "email": "sally.thomas@acme.com"
    },
    "source": {
      "version": "{debezium-version}",
      "connector": "mysql",
      "name": "mysql",
      "ts_ms": 1646300467000,
      "snapshot": "false",
      "db": "inventory",
      "sequence": null,
      "table": "customers",
      "server_id": 223344,
      "gtid": null,
      "file": "mysql-bin.000003",
      "pos": 401,
      "row": 0,
      "thread": null,
      "query": null
    },
    "op": "u",
    "ts_ms": 1646300467746,
    "transaction": null
  }
}
----

If you have any questions or requests related to running {prodname} on Kubernetes or OpenShift,
then please let us know in our https://groups.google.com/forum/#!forum/debezium[user group] or in the {prodname} https://debezium.zulipchat.com/#narrow/stream/302533-dev[developer's chat].