185 lines
5.9 KiB
Plaintext
185 lines
5.9 KiB
Plaintext
:page-aliases: configuration/header-to-value.adoc
|
|
// Category: debezium-using
|
|
// Type: assembly
|
|
// ModuleID: converting-message-headers-to-event-record-values
|
|
// Title: Converting message headers into event record values
|
|
[id="header-to-value"]
|
|
= HeaderToValue
|
|
|
|
:toc:
|
|
:toc-placement: macro
|
|
:linkattrs:
|
|
:icons: font
|
|
:source-highlighter: highlight.js
|
|
|
|
toc::[]
|
|
|
|
The `HeaderToValue` SMT extracts specified header fields from event records, and then copies or moves the header fields to values in the event record.
|
|
The `move` options removes the fields from the header entirely before adding them as values in the payload.
|
|
You can configure the SMT to manipulate multiple headers in the original message.
|
|
You can use dot notation to specify a node within the payload in which you want to nest the header field.
|
|
For more information about configuring the SMT, see the following xref:example-header-to-value[example].
|
|
|
|
// Type: concept
|
|
// Title: Example: Basic configuration of the {prodname} `HeaderToValue` SMT
|
|
// ModuleID: basic-configuration-of-the-debezium-header-to-value-smt
|
|
[[example-header-to-value]]
|
|
== Example
|
|
|
|
To extract message headers in an event record into the record value, configure the `HeaderToValue` SMT in the Kafka Connect configuration for a connector.
|
|
You can configure the transformation to either remove the original headers or to copy them.
|
|
To remove header fields from the record, configure the SMT to use the `move` operations.
|
|
To retain the header fields in the original record, configure the SMT to use the `copy` operation.
|
|
For example, to remove the headers `event_timestamp` and `key` from an event message, add the following lines to your connector configuration:
|
|
|
|
[source]
|
|
----
|
|
transforms=moveHeadersToValue
|
|
transforms.moveHeadersToValue.type=io.debezium.transforms.HeaderToValue
|
|
transforms.moveHeadersToValue.headers=event_timestamp,key
|
|
transforms.moveHeadersToValue.fields=timestamp,source.id
|
|
transforms.moveHeadersToValue.operation=move
|
|
----
|
|
|
|
The following example shows the headers and values of an event record before and after the transformation is applied.
|
|
|
|
.Effect of applying the `HeaderToValue` SMT
|
|
====
|
|
Event record before it is processed by the `HeaderToValue` transformation::
|
|
|
|
Header before the SMT processes the event record:::
|
|
+
|
|
[source]
|
|
----
|
|
{
|
|
"header_x": 0,
|
|
"event_timestamp": 1626102708861,
|
|
"key": 100,
|
|
}
|
|
----
|
|
|
|
Value before the SMT processes the event record:::
|
|
+
|
|
[source]
|
|
----
|
|
{
|
|
"before": null,
|
|
"after": {
|
|
"id": 1,
|
|
"first_name": "Anne",
|
|
"last_name": "Kretchmar",
|
|
"email": "annek@noanswer.org"
|
|
},
|
|
"source": {
|
|
"version": "2.1.3.Final",
|
|
"connector": "postgresql",
|
|
"name": "PostgreSQL_server",
|
|
"ts_ms": 1559033904863,
|
|
"ts_us": 1559033904863091,
|
|
"ts_ns": 1559033904863091000,
|
|
"snapshot": true,
|
|
"db": "postgres",
|
|
"sequence": "[\"24023119\",\"24023128\"]"
|
|
"schema": "public",
|
|
"table": "customers",
|
|
"txId": 555,
|
|
"lsn": 24023128,
|
|
"xmin": null
|
|
},
|
|
"op": "c",
|
|
"ts_ms": 1559033904863,
|
|
"ts_us": 1559033904863741,
|
|
"ts_ns": 1559033904863741697
|
|
}
|
|
----
|
|
|
|
Event record after it is processed by the `HeaderToValue` transformation::
|
|
|
|
Header after the SMT removes the specified field:::
|
|
+
|
|
[source, json]
|
|
----
|
|
{
|
|
"header_x": 0
|
|
}
|
|
----
|
|
Value after the SMT moves header fields into the value:::
|
|
+
|
|
[source, json]
|
|
----
|
|
{
|
|
"before": null,
|
|
"after": {
|
|
"id": 1,
|
|
"first_name": "Anne",
|
|
"last_name": "Kretchmar",
|
|
"email": "annek@noanswer.org"
|
|
},
|
|
"source": {
|
|
"version": "2.1.3.Final",
|
|
"connector": "postgresql",
|
|
"name": "PostgreSQL_server",
|
|
"ts_ms": 1559033904863,
|
|
"ts_us": 1559033904863697,
|
|
"ts_ns": 1559033904863697000,
|
|
"snapshot": true,
|
|
"db": "postgres",
|
|
"sequence": "[\"24023119\",\"24023128\"]"
|
|
"schema": "public",
|
|
"table": "customers",
|
|
"txId": 555,
|
|
"lsn": 24023128,
|
|
"xmin": null,
|
|
"id": 100
|
|
},
|
|
"op": "c",
|
|
"ts_ms": 1559033904863,
|
|
"ts_us": 1559033904863631,
|
|
"ts_ns": 1559033904863631584,
|
|
"event_timestamp": 1626102708861
|
|
}
|
|
----
|
|
====
|
|
|
|
// Type: reference
|
|
// ModuleID: options-for-configuring-the-header-to-value-transformation
|
|
// Title: Options for configuring the `HeaderToValue` transformation
|
|
[[header-to-value-configuration-options]]
|
|
== Configuration options
|
|
|
|
The following table lists the configuration options that you can use with the `HeaderToValue` SMT.
|
|
|
|
.HeaderToValue SMT configuration options
|
|
[cols="14%a,40%a,10%a, 16%a, 16%a, 10%a"]
|
|
|===
|
|
|Property
|
|
|Description
|
|
|Type
|
|
|Default
|
|
|Valid Values
|
|
|Importance
|
|
|
|
|[[header-to-value-headers]]<<header-to-value-headers, `headers`>>
|
|
|A comma-separated list of header names in the record whose values are to be copied or moved to the record value.
|
|
|list
|
|
|No default value
|
|
|non-empty list
|
|
|high
|
|
|[[header-to-value-fields]]<<header-to-value-fields, `fields`>>
|
|
|A comma-separated list of field names, in the same order as the header names listed in the `headers` configuration property.
|
|
Use dot notation to instruct the SMT to nest fields within specific nodes of the message payload.
|
|
For information about how to configure the SMT to use dot notation, see the xref:example-header-to-value[example] that appears earlier in this topic.
|
|
|list
|
|
|No default value
|
|
|non-empty list
|
|
|high
|
|
|[[header-to-value-operation]]<<header-to-value-operation, `operation`>>
|
|
|Specifies one of the following options:
|
|
`move`:: The SMT moves header fields to values in the event record, and removes the fields from the header.
|
|
`copy`:: The SMT copies header field to values in the event record, and retains the original header fields.
|
|
|string
|
|
|No default value
|
|
|move or copy
|
|
|high
|
|
|===
|