201 lines
9.2 KiB
Plaintext
201 lines
9.2 KiB
Plaintext
= Monitoring Debezium
|
|
include::../_attributes.adoc[]
|
|
:linkattrs:
|
|
:icons: font
|
|
:toc:
|
|
:toc-placement: macro
|
|
|
|
toc::[]
|
|
|
|
Debezium uses Kafka and Zookeeper, and all of these support monitoring via JMX. In fact, there are a lot of https://zookeeper.apache.org/doc/r3.1.2/zookeeperJMX.html[Zookeeper metrics] and even more http://docs.confluent.io/3.0.0/kafka/monitoring.html[Kafka metrics] available. But how JMX is enabled in these services depends on whether you're running them in Docker containers or using the standard installations.
|
|
|
|
[NOTE]
|
|
====
|
|
If you are running multiple services on the same machine, be sure to use distinct JMX ports for each service.
|
|
====
|
|
|
|
|
|
== Local installations
|
|
|
|
JMX can be enabled in Zookeeper, Kafka, and Kafka Connect using their standard installations and environment variables when running the startup commands.
|
|
|
|
=== Local Zookeeper
|
|
|
|
Zookeeper has built-in support for JMX. When running Zookeeper using a local installation, the `zkServer.sh` script recognizes the following environment variables:
|
|
|
|
[cols="35%a,10%a,55%a",options="header,footer",role="table table-bordered table-striped"]
|
|
|=======================
|
|
|Environment Variable
|
|
|Default
|
|
|Description
|
|
|
|
|`JMXPORT`
|
|
|
|
|
|Enables JMX and specifies the port number that will be used for JMX. The value is used to specify the JVM parameter `-Dcom.sun.management.jmxremote.port=$JMXPORT`.
|
|
|
|
|`JMXAUTH`
|
|
|`false`
|
|
|Whether JMX clients must use password authentication when connecting. Must be either `true` or `false`. The value is used to specify the JVM parameter `-Dcom.sun.management.jmxremote.authenticate=$JMXAUTH`.
|
|
|
|
|`JMXSSL`
|
|
|`false`
|
|
|Whether JMX clients connect using SSL/TLS. Must be either `true` or `false`. The value is used to specify the JVM parameter `-Dcom.sun.management.jmxremote.ssl=$JMXSSL`.
|
|
|
|
|`JMXLOG4J`
|
|
|`true`
|
|
|Whether the Log4J JMX MBeans should be disabled. Must be either `true` or `false`. The value is used to specify the JVM parameter `-Dzookeeper.jmx.log4j.disable=$JMXLOG4J`.
|
|
|=======================
|
|
|
|
=== Local Kafka
|
|
|
|
When running Kafka using a local installation, the `kafka-server-start.sh` script recognizes the following environment variables:
|
|
|
|
|
|
[cols="35%a,10%a,55%a",options="header,footer",role="table table-bordered table-striped"]
|
|
|=======================
|
|
|Environment Variable
|
|
|Default
|
|
|Description
|
|
|
|
|`JMX_PORT`
|
|
|
|
|
|Enables JMX and specifies the port number that will be used for JMX. The value is used to specify the JVM parameter `-Dcom.sun.management.jmxremote.port=$JMX_PORT`.
|
|
|
|
|`KAFKA_JMX_OPTS`
|
|
|`-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false `
|
|
|JMX options, passed directly to the JVM during startup.
|
|
|=======================
|
|
|
|
|
|
=== Local Kafka Connect
|
|
|
|
When running Kafka using a local installation, the `connect-distributed.sh` script recognizes the following environment variables:
|
|
|
|
|
|
[cols="35%a,10%a,55%a",options="header,footer",role="table table-bordered table-striped"]
|
|
|=======================
|
|
|Environment Variable
|
|
|Default
|
|
|Description
|
|
|
|
|`JMX_PORT`
|
|
|
|
|
|Enables JMX and specifies the port number that will be used for JMX. The value is used to specify the JVM parameter `-Dcom.sun.management.jmxremote.port=$JMX_PORT`.
|
|
|
|
|`KAFKA_JMX_OPTS`
|
|
|`-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false `
|
|
|JMX options, passed directly to the JVM during startup.
|
|
|=======================
|
|
|
|
|
|
== Using Docker
|
|
|
|
Enable JMX for a JVM running in a Docker container requires several additional options not normally needed when running on a local machine. This is because the JVM requires the hostname to which it will advertise itself to JMX clients. Because of this, Debezium's Docker images for Zookeeper, Kafka, and Kafka Connect use several environment variables to enable and configure JMX. Most of the environment variables are the same for all of the images, but there are some minor differences.
|
|
|
|
=== Zookeeper in Docker
|
|
|
|
The `debezium/zookeeper` image recognizes the following JMX-related environment variables:
|
|
|
|
[cols="35%a,10%a,55%a",options="header,footer",role="table table-bordered table-striped"]
|
|
|=======================
|
|
|Environment Variable
|
|
|Default
|
|
|Description
|
|
|
|
|`JMXPORT`
|
|
|
|
|
|Required. The port number that will be used for JMX. The value is used to specify the JVM parameters `-Dcom.sun.management.jmxremote.port=$JMXPORT` and `-Dcom.sun.management.jmxremote.rmi.port=$JMXPORT`.
|
|
|
|
|`JMXHOST`
|
|
|
|
|
|Required. The IP address or resolvable hostname of the Docker host, which JMX uses to construct a URL sent to the JMX client. A value of `localhost` or `127.0.0.1` will not work. The value is used to specify the JVM parameter `-Djava.rmi.server.hostname=$JMXHOST`.
|
|
|
|
|`JMXAUTH`
|
|
|`false`
|
|
|Whether JMX clients must use password authentication when connecting. Must be either `true` or `false`. The value is used to specify the JVM parameter `-Dcom.sun.management.jmxremote.authenticate=$JMXAUTH`.
|
|
|
|
|`JMXSSL`
|
|
|`false`
|
|
|Whether JMX clients connect using SSL/TLS. Must be either `true` or `false`. The value is used to specify the JVM parameter `-Dcom.sun.management.jmxremote.ssl=$JMXSSL`.
|
|
|
|
|`JMXLOG4J`
|
|
|`true`
|
|
|Whether the Log4J JMX MBeans should be disabled. Must be either `true` or `false`. The value is used to specify the JVM parameter `-Dzookeeper.jmx.log4j.disable=$JMXLOG4J`.
|
|
|=======================
|
|
|
|
The following example Docker command start a container using the `debezium/zookeeper` image with values for the `JMXPORT` and `JMXHOST` environment variables, and maps the Docker host's port 9010 to the container's JMX port:
|
|
|
|
```
|
|
docker run -it --rm --name zookeeper -p 2181:2181 -p 2888:2888 -p 3888:3888 -p 9010:9010 -e JMXPORT=9010 -e JMXHOST=10.0.1.10 debezium/zookeeper:latest
|
|
```
|
|
|
|
=== Kafka in Docker
|
|
|
|
The `debezium/kafka` image recognizes the following JMX-related environment variables:
|
|
|
|
[cols="35%a,10%a,55%a",options="header,footer",role="table table-bordered table-striped"]
|
|
|=======================
|
|
|Environment Variable
|
|
|Default
|
|
|Description
|
|
|
|
|`JMXPORT`
|
|
|
|
|
|Required. The port number that will be used for JMX. The value is used to specify the JVM parameters `-Dcom.sun.management.jmxremote.port=$JMXPORT` and `-Dcom.sun.management.jmxremote.rmi.port=$JMXPORT`.
|
|
|
|
|`JMXHOST`
|
|
|
|
|
|Required. The IP address or resolvable hostname of the Docker host, which JMX uses to construct a URL sent to the JMX client. A value of `localhost` or `127.0.0.1` will not work. The value is used to specify the JVM parameter `-Djava.rmi.server.hostname=$JMXHOST`.
|
|
|
|
|`JMXAUTH`
|
|
|`false`
|
|
|Whether JMX clients must use password authentication when connecting. Must be either `true` or `false`. The value is used to specify the JVM parameter `-Dcom.sun.management.jmxremote.authenticate=$JMXAUTH`.
|
|
|
|
|`JMXSSL`
|
|
|`false`
|
|
|Whether JMX clients connect using SSL/TLS. Must be either `true` or `false`. The value is used to specify the JVM parameter `-Dcom.sun.management.jmxremote.ssl=$JMXSSL`.
|
|
|=======================
|
|
|
|
|
|
The following example Docker command start a container using the `debezium/kafka` image with values for the `JMXPORT` and `HOST_NAME` environment variables, and maps the Docker host's port 9011 to the container's JMX port:
|
|
|
|
```
|
|
docker run -it --rm --name kafka -p 9092:9092 -p 9011:9011 -e JMXPORT=9011 -e JMXHOST=10.0.1.10 --link zookeeper:zookeeper debezium/kafka:latest
|
|
```
|
|
|
|
== Kafka Connect in Docker
|
|
|
|
The `debezium/connect` image recognizes the following JMX-related environment variables:
|
|
|
|
[cols="35%a,10%a,55%a",options="header,footer",role="table table-bordered table-striped"]
|
|
|=======================
|
|
|Environment Variable
|
|
|Default
|
|
|Description
|
|
|
|
|`JMXPORT`
|
|
|
|
|
|Required. The port number that will be used for JMX. The value is used to specify the JVM parameters `-Dcom.sun.management.jmxremote.port=$JMXPORT` and `-Dcom.sun.management.jmxremote.rmi.port=$JMXPORT`.
|
|
|
|
|`JMXHOST`
|
|
|
|
|
|Required. The IP address or resolvable hostname of the Docker host, which JMX uses to construct a URL sent to the JMX client. A value of `localhost` or `127.0.0.1` will not work. The value is used to specify the JVM parameter `-Djava.rmi.server.hostname=$JMXHOST`.
|
|
|
|
|`JMXAUTH`
|
|
|`false`
|
|
|Whether JMX clients must use password authentication when connecting. Must be either `true` or `false`. The value is used to specify the JVM parameter `-Dcom.sun.management.jmxremote.authenticate=$JMXAUTH`.
|
|
|
|
|`JMXSSL`
|
|
|`false`
|
|
|Whether JMX clients connect using SSL/TLS. Must be either `true` or `false`. The value is used to specify the JVM parameter `-Dcom.sun.management.jmxremote.ssl=$JMXSSL`.
|
|
|=======================
|
|
|
|
The following example Docker command start a container using the `debezium/connect` image with values for the `JMXPORT` and `HOST_NAME` environment variables, and maps the Docker host's port 9012 to the container's JMX port:
|
|
|
|
The Docker command to start a container using the `debezium/connect` image defines these variables using Docker's standard `-e` parameter, and maps the JMX port to a port on the Docker host. For example, the following command starts a container with JMX exposed on port 9011:
|
|
|
|
```
|
|
docker run -it --rm --name connect -p 8083:8083 -p 9012:9012 -e JMXPORT=9012 -e JMXHOST=10.0.1.10 -e GROUP_ID=1 -e CONFIG_STORAGE_TOPIC=my_connect_configs -e OFFSET_STORAGE_TOPIC=my_connect_offsets --link zookeeper:zookeeper --link kafka:kafka --link mysql:mysql debezium/connect:latest
|
|
```
|
|
|