208 lines
10 KiB
Plaintext
208 lines
10 KiB
Plaintext
[id="distributed-tracing"]
|
|
= Distributed Tracing
|
|
|
|
:toc:
|
|
:toc-placement: macro
|
|
:linkattrs:
|
|
:icons: font
|
|
:source-highlighter: highlight.js
|
|
|
|
toc::[]
|
|
|
|
== Overview
|
|
|
|
Observability is an important aspect of microservice-oriented applications.
|
|
One of the key ingredients for observability is https://microservices.io/patterns/observability/distributed-tracing.html[distributed tracing].
|
|
|
|
It is necessary to provide additional precautions when an application writes a record to a database that is later processed by Debezium.
|
|
The active trace is effectively demarcated by the write to the database.
|
|
If you want to include {prodname} data in your application tracer, you must pass the trace metadata to {prodname}.
|
|
|
|
You can add tracing support to {prodname} through the https://opentelemetry.io/docs/specs/otel/[OpenTelemetry specification].
|
|
|
|
== Installation
|
|
|
|
To enable traceability in {prodname}, you must install the required OpenTelemetry API packages on your Kafka Connect cluster, together with the OpenTelemetry SDK.
|
|
|
|
There are three main methods for installing OpenTelemetry on Kafka Connect, each having its advantages:
|
|
|
|
xref:open-telemetry-manual-installation[Manual installation]:: Offers more control and customization.
|
|
xref:open-telemetry-docker-installation[Docker installation]:: Provides easier integration.
|
|
xref:open-telemetry-strimzi-installation[Strimzi installation]:: Presents a ready-to-use solution on Kubernetes that includes all of the necessary OpenTelemetry components.
|
|
|
|
[id="open-telemetry-manual-installation"]
|
|
=== Manual installation of OpenTelemetry
|
|
|
|
A manual installation provides greater control and customization.
|
|
|
|
==== Installing the OpenTelemetry API
|
|
|
|
Install the following packages to the Kafka Connect classpath:
|
|
|
|
* https://mvnrepository.com/artifact/io.opentelemetry/opentelemetry-api/[opentelemetry-api]
|
|
* https://mvnrepository.com/artifact/io.opentelemetry/opentelemetry-context/[opentelemetry-context]
|
|
* https://mvnrepository.com/artifact/io.opentelemetry/opentelemetry-semconv/[opentelemetry-semconv]
|
|
* https://mvnrepository.com/artifact/io.opentelemetry.instrumentation/opentelemetry-instrumentation-api/[opentelemetry-instrumentation-api]
|
|
* https://mvnrepository.com/artifact/io.opentelemetry.instrumentation/opentelemetry-instrumentation-api-semconv/[opentelemetry-instrumentation-api-semconv]
|
|
* https://mvnrepository.com/artifact/io.opentelemetry.instrumentation/opentelemetry-kafka-clients-2.6/[opentelemetry-kafka-clients-2.6]
|
|
* https://mvnrepository.com/artifact/io.opentelemetry.instrumentation/opentelemetry-kafka-clients-common/[opentelemetry-kafka-clients-common]
|
|
|
|
|
|
You can download the OpenTelemetry packages directly from the Maven repository, or retrieve the packages by using the Maven dependency plugin to ensure version compatibility.
|
|
|
|
===== Using the Maven Dependency Plugin
|
|
|
|
1. Create a POM file similar to the one in the following example.
|
|
In your version of the file, specify the version of the OpenTelemetry package that you want to use.
|
|
+
|
|
[source,xml]
|
|
----
|
|
<?xml version="1.0"?>
|
|
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
|
<modelVersion>4.0.0</modelVersion>
|
|
<groupId>group.id</groupId>
|
|
<artifactId>artifact-id</artifactId>
|
|
<version>0.0.1</version>
|
|
<properties>
|
|
<version.opentelemetry.api>1.23.1</version.opentelemetry.api>
|
|
<version.opentelemetry.kafka.clients.2.6>1.23.0-alpha</version.opentelemetry.kafka.clients.2.6>
|
|
</properties>
|
|
<dependencies>
|
|
<dependency>
|
|
<groupId>io.opentelemetry</groupId>
|
|
<artifactId>opentelemetry-api</artifactId>
|
|
<version>${version.opentelemetry.api}</version>
|
|
</dependency>
|
|
<dependency>
|
|
<groupId>io.opentelemetry.instrumentation</groupId>
|
|
<artifactId>opentelemetry-kafka-clients-2.6</artifactId>
|
|
<version>${version.opentelemetry.kafka.clients.2.6}</version>
|
|
</dependency>
|
|
</dependencies>
|
|
</project>
|
|
----
|
|
+
|
|
2. In the directory that contains the `pom.xml` file, run the following command to download all dependencies (including transitives) to a specified directory (for example, `./lib`)
|
|
+
|
|
[source,bash]
|
|
----
|
|
mvn dependency:copy-dependencies -DoutputDirectory=./lib
|
|
----
|
|
|
|
|
|
==== Installing the OpenTelemetry SDK
|
|
After you install the API, install the OpenTelemetry SDK.
|
|
Install the OpenTelemetry SDK on your Kafka connect cluster by using the https://opentelemetry.io/docs/instrumentation/java/automatic/[Java agent] for automatic instrumentation.
|
|
|
|
[id="open-telemetry-docker-installation"]
|
|
=== Using Docker to install OpenTelemetry
|
|
Prefer this installation method if you want to use Docker for local development or testing.
|
|
|
|
The OpenTelemetry API dependencies are installed in the https://quay.io/repository/debezium/connect[{prodname} Kafka Connect] container image.
|
|
The JAR files are downloaded to a separate directory.
|
|
By default, the JAR files are not added to the class path.
|
|
To add the JAR files to the class path, set the environment variable `ENABLE_OTEL` to `true`.
|
|
You must also install the https://opentelemetry.io/docs/instrumentation/java/automatic/[OpenTelemetry Java agent].
|
|
|
|
[id="open-telemetry-strimzi-installation"]
|
|
=== Using Strimzi to install Open Telemetry
|
|
https://strimzi.io/[Strimzi] provides a way to deploy an Apache Kafka cluster on Kubernetes.
|
|
|
|
The Strimzi Kafka image includes all of the necessary OpenTelemetry components.
|
|
|
|
If you want to view the OpenTelemetry components that are available in the Strimzi image, examine the compose file that is provided in the https://github.com/debezium/debezium-examples/tree/main/outbox[{prodname} Outbox example].
|
|
|
|
=== Configuring OpenTelemetry
|
|
|
|
For information about how to configure OpenTelemetry, see the https://opentelemetry.io/docs/instrumentation/java/automatic/agent-config/[OpenTelemetry documentation].
|
|
|
|
[NOTE]
|
|
====
|
|
{prodname} was tested with a {prodname} Kafka Connect image, and a Strimzi distribution that included the following configuration settings:
|
|
|
|
* `otel.traces.exporter=otlp`
|
|
* `otel.propagators=tracecontext`
|
|
|
|
====
|
|
|
|
== ActivateTracingSpan SMT
|
|
|
|
The main implementation point of tracing in Debezium is `ActivateTracingSpan` SMT.
|
|
In this case, the application writing to a database is responsible for providing the tracing span context.
|
|
The writer must inject the span context into a `java.util.Properties` instance that is serialized and written to the database as a distinct field of the table.
|
|
|
|
If the span context is not provided then the SMT will create a new span.
|
|
In this case, Debezium operations together with metadata will be traced but will not be connected to business transaction traces to enable end-to-end tracing.
|
|
|
|
When this SMT is invoked with a message then it will:
|
|
|
|
* extract the parent span context if present in the message
|
|
* create the event `db-log-write` span context with the start timestamp set to the database log write timestamp
|
|
* insert fields from `source` block into the span as *tags*
|
|
* create the processing `debezium-read` span as a child of `db-log-write` span with the start timestamp set to the processing time of the even
|
|
* insert fields from envelope such as `op` into the processing span as *tags*
|
|
* injects the processing span context into message headers
|
|
|
|
=== Kafka Producer tracing
|
|
|
|
Optionally it is possible to enable tracing at the Kafka producer level.
|
|
If you enable tracing in the Kafka producer, when messages are written to the Kafka broker, the producer extracts the {prodname} processing span context from the Kafka message headers, creates a new child span, and then records information about the write operation to the broker.
|
|
Then it injects the new span into the message headers so a consumer of the message can restore the trace and resume end-to-end tracing.
|
|
|
|
The interceptor cannot propagate the traceability context if the Kafka instrumentation is enabled.
|
|
The interceptor simply propagates the traceability context before delegating the instrumentation to the OpenTelemetry SDK.
|
|
|
|
==== Enabling end-to-end traceability
|
|
|
|
1. Download and install the https://mvnrepository.com/artifact/io.debezium/debezium-interceptor/[debezium-interceptor] to the Kafka Connect classpath.
|
|
2. Disable the automatic OpenTelemetry instrumentation at the Kafka producer and consumer by setting the value of `otel.instrumentation.common.default-enabled` to `false`.
|
|
|
|
=== Configuration options
|
|
|
|
[cols="65%a,>15%a,>20%"]
|
|
|===
|
|
|Configuration property
|
|
|Type
|
|
|Default
|
|
|
|
|`tracing.span.context.field`::
|
|
The name of the field containing span context. +
|
|
+
|
|
_The sender must write the span context into the database column as a serialized instance of `java.util.Properties` with injected span context._
|
|
|string
|
|
|tracingspancontext
|
|
|
|
|`tracing.operation.name`::
|
|
The operation name representing the Debezium processing span. +
|
|
|string
|
|
|debezium-read
|
|
|
|
|`tracing.with.context.field.only`::
|
|
Only events that have serialized context field should be traced.
|
|
+
|
|
_If set to `true` then tracing span will be created only for events with associated tracing span context field.
|
|
If set to `false` then the tracing span is created for all incoming events regardless of having associated span context._
|
|
|boolean
|
|
|false
|
|
|
|
|===
|
|
|
|
== Outbox Extension
|
|
|
|
The {prodname} link:/documentation/reference/integrations/outbox[Quarkus extension] for implementing the outbox pattern provides the additional functionality necessary for tracing context propagation out-of-the-box.
|
|
Specifically, it provides the `tracingspancontext` field in the outbox table, which is used for passing the tracing span context from a service using the outbox extension to the {prodname} connector.
|
|
|
|
When an outbox event is emitted, the extension will:
|
|
|
|
* create a new `outbox-write` span as a child of current active span
|
|
* inject the span context into that `java.util.Properties` instance that is serialized into the `tracingspancontext` column
|
|
* write the record into the database
|
|
|
|
The tracing integration in the outbox extension is automatically enabled if the `quarkus-opentelemetry` extension is present.
|
|
If you want to disable tracing support for the outbox extension despite the presence of the `quarkus-opentelemetry` extension,
|
|
set the option `quarkus.debezium-outbox.tracing.enabled=false` in the Quarkus `application.properties` file.
|
|
|
|
== Event Router SMT
|
|
|
|
The link:/documentation/reference/configuration/outbox-event-router[Event Router SMT] acts as an Outbox extension counterpart, it executes the same steps as the `ActivateTracingSpan` SMT, and is used instead of it.
|