106 lines
5.1 KiB
Plaintext
106 lines
5.1 KiB
Plaintext
[id="distributed-tracing"]
|
|
= Distributed Tracing
|
|
|
|
:toc:
|
|
:toc-placement: macro
|
|
:linkattrs:
|
|
:icons: font
|
|
:source-highlighter: highlight.js
|
|
|
|
toc::[]
|
|
|
|
== Overview
|
|
|
|
Observability is an important aspect of microservice-oriented applications.
|
|
One of the key ingredients for observability is https://microservices.io/patterns/observability/distributed-tracing.html[distributed tracing].
|
|
|
|
It is necessary to provide additional precautions when an application writes a record to a database that is later processed by Debezium.
|
|
The active trace is effectively demarcated by the write to the database.
|
|
If we want Debezium to join the larger scope application tracer we need to pass the trace metadata to Debezium.
|
|
|
|
The tracing support is added to Debezium through the https://opentracing.io/[OpenTracing] specification.
|
|
It is also necessary to provide a client implementing the specification.
|
|
Debezium was tested with https://www.jaegertracing.io/[Jaeger] implementation.
|
|
|
|
[WARNING]
|
|
====
|
|
Neither the specification JAR files nor the Jaeger client are part of the Debezium Kafka Connect container image.
|
|
The user either needs to extend the image with them or can use the https://strimzi.io/[Strimzi] Kafka image.
|
|
In that case, the tracing of Kafka producer and consumer is also available.
|
|
|
|
See compose file of https://github.com/debezium/debezium-examples/tree/master/outbox[Outbox example].
|
|
====
|
|
|
|
== ActivateTracingSpan SMT
|
|
|
|
The main implementation point of tracing in Debezium is `ActivateTracingSpan` SMT.
|
|
In this case, the application writing to a database is responsible for providing the tracing span context.
|
|
The writer must inject the span context into a `java.util.Properties` instance that is serialized and written to the database as a distinct field of the table.
|
|
|
|
If the span context is not provided then the SMT will create a new span.
|
|
In this case, Debezium operations together with metadata will be traced but will not be connected to business transaction traces to enable end-to-end tracing.
|
|
|
|
When this SMT is invoked with a message then it will:
|
|
|
|
* extract the parent span context if present in the message
|
|
* create the event `db-log-write` span context with the start timestamp set to the database log write timestamp
|
|
* insert fields from `source` block into the span as *tags*
|
|
* create the processing `debezium-read` span as a child of `db-log-write` span with the start timestamp set to the processing time of the even
|
|
* insert fields from envelope such as `op` into the processing span as *tags*
|
|
* injects the processing span context into message headers
|
|
|
|
=== Kafka Producer tracing
|
|
|
|
Optionally it is possible to enable tracing at the Kafka producer level.
|
|
If enabled then when the message is being written to the Kafka broker the producer will extract Debezium's processing span context from the Kafka message headers, create a new child span and record information about the write to the broker.
|
|
Then it injects the new span into the message headers so a consumer of the message can restore the trace and resume end-to-end tracing.
|
|
|
|
=== Configuration options
|
|
|
|
[cols="65%a,>15%a,>20%"]
|
|
|===
|
|
|Configuration property
|
|
|Type
|
|
|Default
|
|
|
|
|`tracing.span.context.field`::
|
|
The name of the field containing span context. +
|
|
+
|
|
_The sender must write the span context into the database column as a serialized instance of `java.util.Properties` with injected span context._
|
|
|string
|
|
|tracingspancontext
|
|
|
|
|`tracing.operation.name`::
|
|
The operation name representing the Debezium processing span. +
|
|
|string
|
|
|debezium-read
|
|
|
|
|`tracing.with.context.field.only`::
|
|
Only events that have serialized context field should be traced.
|
|
+
|
|
_If set to `true` then tracing span will be created only for events with associated tracing span context field.
|
|
If set to `false` then the tracing span is created for all incoming events regardless of having associated span context._
|
|
|boolean
|
|
|false
|
|
|
|
|===
|
|
|
|
== Outbox Extension
|
|
|
|
The {prodname} link:/documentation/reference/integrations/outbox[Quarkus extension] for implementing the outbox pattern provides the additional functionality necessary for tracing context propagation out-of-the-box.
|
|
Specifically, it provides the `tracingspancontext` field in the outbox table, which is used for passing the tracing span context from a service using the outbox extension to the {prodname} connector.
|
|
|
|
When an outbox event is emitted, the extension will:
|
|
|
|
* create a new `outbox-write` span as a child of current active span
|
|
* inject the span context into that `java.util.Properties` instance that is serialized into the `tracingspancontext` column
|
|
* write the record into the database
|
|
|
|
The tracing integration in the outbox extension is automatically enabled if also the `smallrye-opentracing` Quarkus extension is present.
|
|
If you want to disable tracing support for the outbox extension despite the `smallrye-opentracing` Quarkus extension being is present,
|
|
you can disable it by setting an option `quarkus.debezium-outbox.tracing.enabled=false` in the Quarkus `application.properties` file.
|
|
|
|
== Event Router SMT
|
|
|
|
The link:/documentation/reference/configuration/outbox-event-router[Event Router SMT] acts as an Outbox extension counterpart, it executes the same steps as the `ActivateTracingSpan` SMT, and is used instead of it.
|