242 lines
13 KiB
Plaintext
242 lines
13 KiB
Plaintext
// Category: debezium-using
|
|
// Type: assembly
|
|
// ModuleID: routing-change-event-records-to-topics-according-to-event-content
|
|
// Title: Routing change event records to topics according to event content
|
|
[id="content-based-routing"]
|
|
= Content-based routing
|
|
ifdef::community[]
|
|
:toc:
|
|
:toc-placement: macro
|
|
:linkattrs:
|
|
:icons: font
|
|
:source-highlighter: highlight.js
|
|
|
|
toc::[]
|
|
endif::community[]
|
|
By default, {prodname} streams all of the change events that it reads from a table to a single static topic.
|
|
However, there might be situations in which you might want to reroute selected events to other topics, based on the event content.
|
|
The process of routing messages based on their content is described in the https://www.enterpriseintegrationpatterns.com/patterns/messaging/ContentBasedRouter.html[Content-based routing] messaging pattern.
|
|
To apply this pattern in {prodname}, you use the content-based routing link:https://cwiki.apache.org/confluence/display/KAFKA/KIP-66%3A+Single+Message+Transforms+for+Kafka+Connect[single message transform] (SMT) to write expressions that are evaluated for each event.
|
|
Depending how an event is evaluated, the SMT either routes the event message to the original destination topic, or reroutes it to the topic that you specify in the expression.
|
|
|
|
ifdef::community[]
|
|
[NOTE]
|
|
====
|
|
The content-based routing SMT is under active development. The structure of the emitted message or other details might change as development progresses.
|
|
====
|
|
endif::community[]
|
|
|
|
ifdef::product[]
|
|
[IMPORTANT]
|
|
====
|
|
The {prodname} content-based routing SMT is a Technology Preview feature. Technology Preview features are not supported with Red Hat production service-level agreements (SLAs) and might not be functionally complete; therefore, Red Hat does not recommend implementing any Technology Preview features in production environments. This Technology Preview feature provides early access to upcoming product innovations, enabling you to test functionality and provide feedback during the development process. For more information about support scope, see link:https://access.redhat.com/support/offerings/techpreview/[Technology Preview Features Support Scope].
|
|
====
|
|
endif::product[]
|
|
|
|
While it is possible to use Java to create a custom SMT to encode routing logic, using a custom-coded SMT has its drawbacks.
|
|
For example:
|
|
|
|
* It is necessary to compile the transformation up front and deploy it to Kafka Connect.
|
|
* Every change needs code recompilation and redeployment, leading to inflexible operations.
|
|
|
|
The content-based routing SMT supports scripting languages that integrate with https://jcp.org/en/jsr/detail?id=223[JSR 223] (Scripting for the Java(TM) Platform).
|
|
|
|
{prodname} does not come with any implementations of the JSR 223 API.
|
|
To use an expression language with {prodname}, you must download the JSR 223 script engine implementation for the language, and add to your {prodname} connector plug-in directories, along any other JAR files used by the language implementation.
|
|
For example, for Groovy 3, you can download its JSR 223 implementation from https://groovy-lang.org/.
|
|
The JSR 223 implementation for GraalVM JavaScript is available at https://github.com/graalvm/graaljs.
|
|
|
|
// Type: procedure
|
|
// Title: Setting up the {prodname} content-based-routing SMT
|
|
// ModuleID: setting-up-the-debezium-content-based-routing-smt
|
|
[[set-up-content-based-routing]]
|
|
== Set up
|
|
|
|
For security reasons, the content-based routing SMT is not included with the {prodname} connector archives.
|
|
Instead, it is provided in a separate artifact, `debezium-scripting-{debezium-version}.tar.gz`.
|
|
To use the content-based routing SMT with a {prodname} connector plug-in, you must explicitly add the SMT artifact to your Kafka Connect environment.
|
|
|
|
IMPORTANT: After the routing SMT is present in a Kafka Connect instance, any user who is allowed to add a connector to the instance can run scripting expressions.
|
|
To ensure that scripting expressions can be run only by authorized users, be sure to secure the Kafka Connect instance and its configuration interface before you add the routing SMT.
|
|
|
|
ifdef::community[]
|
|
With https://zookeeper.apache.org[Zookeeper], http://kafka.apache.org/[Kafka], {link-kafka-docs}.html#connect[Kafka Connect], and one or more {prodname} connectors installed, the remaining tasks to install the filter SMT are:
|
|
|
|
. Download the link:https://repo1.maven.org/maven2/io/debezium/debezium-scripting/{debezium-version}/debezium-scripting-{debezium-version}.tar.gz[scripting SMT archive]
|
|
. Extract the contents of the archive into the {prodname} plug-in directories of your Kafka Connect environment.
|
|
. Obtain a JSR-223 script engine implementation and add its contents to the {prodname} plug-in directories of your Kafka Connect environment.
|
|
. Restart your Kafka Connect process to pick up the new JAR files.
|
|
endif::community[]
|
|
|
|
ifdef::product[]
|
|
.Procedure
|
|
. From a browser, open the link:{DebeziumDownload}, and download the {prodname} scripting SMT archive (`debezium-scripting-{debezium-version}.tar.gz`).
|
|
. Extract the contents of the archive into the {prodname} plug-in directories of your Kafka Connect environment.
|
|
. Obtain a JSR-223 script engine implementation and add its contents to the {prodname} plug-in directories of your Kafka Connect environment.
|
|
. Restart the Kafka Connect process to pick up the new JAR files.
|
|
endif::product[]
|
|
|
|
The Groovy language needs the following libraries on the classpath:
|
|
|
|
* `groovy`
|
|
* `groovy-json` (optional)
|
|
* `groovy-jsr223`
|
|
|
|
The JavaScript language needs the following libraries on the classpath:
|
|
|
|
* `graalvm.js`
|
|
* `graalvm.js.scriptengine`
|
|
|
|
// Type: concept
|
|
// ModuleID: example-debezium-basic-content-based-routing-configuration
|
|
// Title: Example: {prodname} basic content-based routing configuration
|
|
[[example-basic-content-based-routing-configuration]]
|
|
== Example: Basic configuration
|
|
|
|
To configure a {prodname} connector to route change event records based on the event content, you configure the `ContentBasedRouter` SMT in the Kafka Connect configuration for the connector.
|
|
|
|
Configuration of the content-based routing SMT requires you to specify a regular expression that defines the filtering criteria.
|
|
In the configuration, you create a regular expression that defines routing criteria.
|
|
The expression defines a pattern for evaluating event records.
|
|
It also specifies the name of a destination topic where events that match the pattern are routed.
|
|
The pattern that you specify might designate an event type, such as a table insert, update, or delete operation.
|
|
You might also define a pattern that matches a value in a specific column or row.
|
|
|
|
For example, to reroute all update (`u`) records to an `updates` topic, you might add the following configuration to your connector configuration:
|
|
|
|
[source]
|
|
----
|
|
...
|
|
transforms=route
|
|
transforms.route.type=io.debezium.transforms.ContentBasedRouter
|
|
transforms.route.language=jsr223.groovy
|
|
transforms.route.topic.expression=value.op == 'u' ? 'updates' : null
|
|
...
|
|
----
|
|
|
|
The preceding example specifies the use of the `Groovy` expression language.
|
|
|
|
Records that do not match the pattern are routed to the default topic.
|
|
|
|
// Type: concept
|
|
// ModuleID: variables-for-use-in-debezium-content-based-routing-expressions
|
|
//Title: Variables for use in {prodname} content-based routing expressions
|
|
== Variables for use in content-based routing expressions
|
|
|
|
{prodname} binds certain variables into the evaluation context for the SMT.
|
|
When you create expressions to specify conditions to control the routing destination,
|
|
the SMT can look up and interpret the values of these variables to evaluate conditions in an expression.
|
|
|
|
The following table lists the variables that {prodname} binds into the evaluation context for the content-based routing SMT:
|
|
|
|
.Content-based routing expression variables
|
|
[cols="25%a,35%a,40%a",subs="+attributes",options="header"]
|
|
|===
|
|
|Name |Description |Type
|
|
|`key` |A key of the message. |`org.apache.kafka.connect{zwsp}.data{zwsp}.Struct`
|
|
|`value` |A value of the message. |`org.apache.kafka.connect{zwsp}.data{zwsp}.Struct`
|
|
|`keySchema` |Schema of the message key.|`org.apache.kafka.connect{zwsp}.data{zwsp}.Schema`
|
|
|`valueSchema`|Schema of the message value.| `org.apache.kafka.connect{zwsp}.data{zwsp}.Schema`
|
|
|`topic`|Name of the target topic.| String
|
|
|`headers`
|
|
a|A Java map of message headers. The key field is the header name.
|
|
The `headers` variable exposes the following properties:
|
|
|
|
* `value` (of type `Object`)
|
|
|
|
* `schema` (of type `org.apache.kafka{zwsp}.connect{zwsp}.data{zwsp}.Schema`)
|
|
|
|
| `java.util.Map{zwsp}<String,{zwsp} io.debezium{zwsp}.transforms{zwsp}.scripting{zwsp}.RecordHeader>`
|
|
|===
|
|
|
|
An expression can invoke arbitrary methods on its variables.
|
|
Expressions should resolve to a Boolean value that determines how the SMT dispositions the message.
|
|
When the routing condition in an expression evaluates to `true`, the message is retained.
|
|
When the routing condition evaluates to `false`, the message is removed.
|
|
|
|
Expressions should not result in any side-effects. That is, they should not modify any variables that they pass.
|
|
|
|
[NOTE]
|
|
====
|
|
{prodname} emits not only data change events, but also metadata messages about schema changes, transactions, or heartbeat messages.
|
|
Typically, routing expressions should not be applied to such messages, as their structure is different from what is usually expected by the expression.
|
|
It is recommended to use either the xref:content-based-router-topic-regex[topic.regex] configuration option or an link:https://cwiki.apache.org/confluence/display/KAFKA/KIP-585%3A+Filter+and+Conditional+SMTs[SMT predicate] to ensure that the expression is applied only to data change events.
|
|
====
|
|
|
|
// Type: reference
|
|
// ModuleID: configuration-of-content-based-routing-conditions-for-other-scripting-languages
|
|
// Title: Configuration of content-based routing conditions for other scripting languages
|
|
== Language specifics
|
|
|
|
The way that you express content-based routing conditions depends on the scripting language that you use.
|
|
For example, as shown in the xref:example-basic-content-based-routing-configuration[basic configuration example], when you use `Groovy` as the expression language,
|
|
the following expression reroutes all update (`u`) records to the `updates` topic, while routing other records to the default topic:
|
|
|
|
[source,groovy]
|
|
----
|
|
value.op == 'u' ? 'updates' : null
|
|
----
|
|
|
|
Other languages use different methods to express the same condition.
|
|
|
|
[TIP]
|
|
====
|
|
The {prodname} MongoDB connector emits the `after` and `patch` fields as serialized JSON documents rather than as structures.
|
|
To use the ContentBasedRouting SMT with the MongoDB connector, you must first unwind the fields by applying the {link-prefix}:{link-mongodb-event-flattening}[`ExtractNewDocumentState`] SMT.
|
|
|
|
You could also take the approach of using a JSON parser within the expression.
|
|
For example, if you use Groovy as the expression language, add the `groovy-json` artifact to the classpath, and then add an expression such as `(new groovy.json.JsonSlurper()).parseText(value.after).last_name == 'Kretchmar'`.
|
|
====
|
|
|
|
.Javascript
|
|
When you use JavaScript as the expression language, you can call the `Struct#get()` method to specify the content-based routing condition, as in the following example:
|
|
|
|
[source,javascript]
|
|
----
|
|
value.get('op') == 'u' ? 'updates' : null
|
|
----
|
|
|
|
.Javascript with Graal.js
|
|
When you create content-based routing conditions by using JavaScript with Graal.js, you use an approach that is similar to the one use with Groovy.
|
|
For example:
|
|
|
|
[source,javascript]
|
|
----
|
|
value.op == 'u' ? 'updates' : null
|
|
----
|
|
|
|
|
|
// Type: reference
|
|
// ModuleID: options-for-configuring-the-content-based-routing-transformation
|
|
// Title: Options for configuring the content-based routing transformation
|
|
[[content-based-router-configuration-options]]
|
|
== Configuration options
|
|
[cols="30%a,25%a,45%a"]
|
|
|===
|
|
|Property
|
|
|Default
|
|
|Description
|
|
|
|
|[[content-based-router-topic-regex]]<<content-based-router-topic-regex, `topic.regex`>>
|
|
|
|
|
|An optional regular expression that evaluates the name of the destination topic for an event to determine whether to apply the condition logic.
|
|
If the name of the destination topic matches the value in `topic.regex`, the transformation applies the condition logic before it passes the event to the topic.
|
|
If the name of the topic does not match the value in `topic.regex`, the SMT passes the event to the topic unmodified.
|
|
|
|
|[[content-based-router-language]]<<content-based-router-language, `language`>>
|
|
|
|
|
|The language in which the expression is written. Must begin with `jsr223.`, for example, `jsr223.groovy`, or `jsr223.graal.js`. {prodname} supports bootstrapping through the https://jcp.org/en/jsr/detail?id=223[JSR 223 API ("Scripting for the Java (TM) Platform")] only.
|
|
|
|
|[[content-based-router-topic-expression]]<<content-based-router-topic-expression, `topic.expression`>>
|
|
|
|
|
|The expression to be evaluated for every message. Must evaluate to a `String` value where a result of non-null reroutes the message to a new topic, and a `null` value routes the message to the default topic.
|
|
|
|
|[[content-based-router-null-handling-mode]]<<content-based-router-null-handling-mode, `null.handling.mode`>>
|
|
|`keep`
|
|
a|Specifies how the transformation handles `null` (tombstone) messages. You can specify one of the following options:
|
|
|
|
`keep`:: (Default) Pass the messages through.
|
|
`drop`:: Remove the messages completely.
|
|
`evaluate`:: Apply the condition logic to the messages.
|
|
|===
|