69 lines
4.2 KiB
Plaintext
69 lines
4.2 KiB
Plaintext
= Transformations
|
|
|
|
Connectors can be configured with transformations to make lightweight per message modifications. {prodname} provides several link:{link-kafka-docs}/#connect_transforms[single message transformations] (SMTs) that you can use to either modify records before they are sent to Apache Kafka (by applying them to the {prodname} connectors), or when they are read from Kafka by a sink connector. Also xref:{link-debezium-server}[{prodname} Server] supports the usage of SMTs.
|
|
|
|
The following SMTs are provided by {prodname}:
|
|
|
|
[cols="30%a,70%a",options="header"]
|
|
|===
|
|
|Transform
|
|
|Description
|
|
|
|
|xref:transformations/topic-routing.adoc[Topic Routing]
|
|
|Re-routes records to different topics based on a regular expression applied to the original topic name.
|
|
|
|
|xref:transformations/content-based-routing.adoc[Content-Based Routing]
|
|
|Reroute selected events to other topics, based on the event content.
|
|
|
|
|xref:transformations/event-flattening.adoc[New Record State Extraction]
|
|
|Extracts the flat structure of field names and values from Debezium change events, facilitating sink connectors which cannot process Debezium's complex event structure.
|
|
|
|
|xref:transformations/mongodb-event-flattening.adoc[MongoDB New Document State Extraction]
|
|
|The MongoDB-specific counter-part to the xref:transformations/event-flattening.adoc[New Record State Extraction] SMT.
|
|
|
|
|xref:transformations/outbox-event-router.adoc[Outbox Event Router]
|
|
|Provides a way to safely and reliably exchange data between multiple (micro) services.
|
|
|
|
|xref:transformations/mongodb-outbox-event-router.adoc[MongoDB Outbox Event Router]
|
|
|The MongoDB-specific counter-part to the xref:transformations/outbox-event-router.adoc[Outbox Event Router] SMT.
|
|
|
|
|xref:transformations/filtering.adoc[Message Filtering]
|
|
|Applies a filter to the change events emitted by the connectors, based on their content. This lets you propagate only those records that are relevant to you.
|
|
|
|
|xref:transformations/header-to-value.adoc[HeaderToValue]
|
|
|Moves or copies headers into the record value.
|
|
|
|
|xref:transformations/partition-routing.adoc[Partition Routing]
|
|
|Re-routes records to specific partitions based on configured payload fields.
|
|
|
|
|xref:transformations/timezone-converter.adoc[Timezone Converter]
|
|
|Converts {prodname} and Kafka Connect timestamp fields in event records to a specified timezone.
|
|
|
|
|xref:transformations/timescaledb.adoc[TimescaleDB Integration]
|
|
|Routes and enriches messages that the {prodname} PostgreSQL connector captures from a TimescaleDB.
|
|
|
|
|xref:transformations/convert-cloudevent-to-saveable-form.adoc[Convert CloudEvents to Saveable Form]
|
|
|Converts values of records deserialized by {prodname} CloudEvents converter to a structure suitable for {prodname} JDBC sink connector.
|
|
|
|
|xref:transformations/vitess-use-local-vgtid.adoc[Vitess Use Local VGTID]
|
|
|A transformation that reduces the size of VGTIDs that the Vitess connector emits.
|
|
To reduce the amount of data in the event message, the SMT writes only the VGTID for the shard in which a change occurs.
|
|
The VGTIDs for other shards are removed.
|
|
This transformation is designed for use only with the {prodname} connector for Vitess.
|
|
|
|
|xref:transformations/vitess-remove-field.adoc[Vitess Remove Field]
|
|
|The Vitess-specific transformation to remove fields to reduce the size of the messages written (e.g., redundant VGTID fields).
|
|
|
|
|xref:transformations/vitess-filter-transaction-topic-records.adoc[Vitess Filter Transaction Topic Records]
|
|
|A transformation that filters out all messages that the connector emits to the transaction topic (when transaction metadata is enabled).
|
|
This transformation is designed for use only with the {prodname} connector for Vitess.
|
|
|
|
|===
|
|
|
|
By means of xref:transformations/applying-transformations-selectively.adoc[SMT Predicates] you can apply any of the transformations selectively, so that it modifies only that subset of change event messages that share a common characteristic.
|
|
|
|
[NOTE]
|
|
====
|
|
Most of the above SMTs are available by default with the {prodname} container image, but you need to opt-in for the scripting-based ones (Message Filtering or Content-based Routing). See the link:https://github.com/debezium/container-images/tree/main/connect/1.7#enable_debezium_scripting[README] file of the container image for more details.
|
|
====
|