102 lines
4.5 KiB
Plaintext
102 lines
4.5 KiB
Plaintext
[id="distributed-tracing"]
|
|
= Distributed Tracing
|
|
|
|
:toc:
|
|
:toc-placement: macro
|
|
:linkattrs:
|
|
:icons: font
|
|
:source-highlighter: highlight.js
|
|
|
|
toc::[]
|
|
|
|
== Overview
|
|
|
|
Observability is an important aspect of microservice-oriented applications.
|
|
One of the key ingeredients for observability is https://microservices.io/patterns/observability/distributed-tracing.html[distributed tracing].
|
|
|
|
It is necessary to provide additional precautions when an application writes a record to a database that is later processed by Debezium.
|
|
The active trace is effectively demarcated by the write to the database.
|
|
If we want Debezium to join the larger scope application tracer we need to pass the trace metadata to Debezium.
|
|
|
|
The tracing support is added to Debezium through https://opentracing.io/[OpenTracing] specification.
|
|
It is also necessary to provide a client implementing the specification.
|
|
Debezium was tested with https://www.jaegertracing.io/[Jaeger] implementation.
|
|
|
|
[WARNING]
|
|
====
|
|
Neither specification JAR files nor Jaeger client are part of the Debezium Connect image.
|
|
The user either needs to extend the image with them or can use the https://strimzi.io/[Strimzi] Kafka image.
|
|
In that case, the tracing of Kafka producer and consumer is also available.
|
|
|
|
See compose file of https://github.com/debezium/debezium-examples/tree/master/outbox[Outbox example].
|
|
====
|
|
|
|
== ActivateTracingSpan SMT
|
|
|
|
The main implementation point of tracing in Debezium is `ActivateTracingSpan` SMT.
|
|
In this case, the application writing to a database is responsible for providing tracing span context.
|
|
The writer should have span context inject into `java.util.Properties` instance that is serialized then serialized and written to the database as a distinct field of the table.
|
|
|
|
If the span context is not provided then the SMT will create a new span.
|
|
In this case, Debezium operations together with metadata will be traced but will not be connected to business transaction traces to enable end-to-end testing.
|
|
|
|
When this SMT is invoked with a message then it will:
|
|
|
|
* extract parent span context if present in the message
|
|
* create event `db-log-write` span context with the start timestamp set to the database log write timestamp
|
|
* insert fields from `source` block into the span as *tags*
|
|
* create processing `debezium-read` span as a child of `db-log-write` span with the start timestamp set to the processing time of the even
|
|
* insert fields from envelope such as `op` into the processing span as *tags*
|
|
* injects the processing span context into message headers
|
|
|
|
=== Kafka Producer tracing
|
|
|
|
Optionally it is possible to enable tracing at the Kafka producer level.
|
|
If enabled then when the message is being written to the Kafka broker the producer will extract Debezium's processing span context from the Kafka message headers, create a new child span and record information about the write to the broker.
|
|
Then it injects the new span into the message headers so a consumer of the message can restore the trace and resume end-to-end tracing.
|
|
|
|
=== Configuration options
|
|
|
|
[cols="65%a,>15%a,>20%"]
|
|
|===
|
|
|Configuration property
|
|
|Type
|
|
|Default
|
|
|
|
|`tracing.span.context.field`::
|
|
The name of the field containing span context. +
|
|
+
|
|
_The sender must write the span context into the database column as a serialized instance of `java.util.Properties` with injected span context._
|
|
|string
|
|
|tracingspancontext
|
|
|
|
|`tracing.operation.name`::
|
|
The operation name representing the Debezium processing span. +
|
|
|string
|
|
|debezium-read
|
|
|
|
|`tracing.with.context.field.only`::
|
|
Only events that have serialized context field should be traced.
|
|
+
|
|
_If set to `true` then tracing span will be created only for events with associated tracing span context field.
|
|
If set to `false` then the tracing span is created for all incoming events regardless of having associated span context._
|
|
|boolean
|
|
|false
|
|
|
|
|===
|
|
|
|
== Outbox Extension
|
|
|
|
Outbox extension provides the additional operations necessary for tracing context propagation out-of-the-box.
|
|
It provides `tracingspancontext` field in an outbox table that is used as the vehicle to pass span context from service using the outbox extension to a Debezium connector.
|
|
|
|
When an outbox even is emitted the extension will:
|
|
|
|
* creates a new `outbox-write` span as a child of current active span
|
|
* injects the span context into that `java.util.Properties` instance that is serialized into the `tracingspancontext` column
|
|
* writes the record into the database
|
|
|
|
== Event Router SMT
|
|
|
|
Event Router acts as an Outbox Extension counterpart, executes the same steps as `ActivateTracingSpan` SMT, and is used instead of it.
|