0d2c96b5cf
Adds xref to `specified configuration` reference
260 lines
13 KiB
Plaintext
260 lines
13 KiB
Plaintext
:page-aliases: configuration/timezone-converter.adoc
|
|
// Category: debezium-using
|
|
// Type: assembly
|
|
// ModuleID: converting-timezone-values-in-debezium-event-records
|
|
// Title: Converting timezone values in {prodname} event records
|
|
|
|
[id="timezone-converter"]
|
|
= Timezone Converter
|
|
ifdef::community[]
|
|
:toc:
|
|
:toc-placement: macro
|
|
:linkattrs:
|
|
:icons: font
|
|
:source-highlighter: highlight.js
|
|
|
|
toc::[]
|
|
endif::community[]
|
|
|
|
[[timezone-converter-introduction]]
|
|
|
|
When {prodname} emits event records, the timezone values for timestamp fields in the record can vary, depending on the type and configuration of the data source.
|
|
To maintain data consistency and precision within data processing pipelines and applications, you can use the `Timezone Converter` SMT to ensure that event records use a consistent timezone to represent timestamp data.
|
|
|
|
The SMT converts the value of the specified field to the target timezone by using the `converted.timezone` configuration option.
|
|
You can specify the target timezone as a geographic timezone, for example, `America/New_York`, or as a UTC offset, such as `+02:00`.
|
|
It is assumed that the fields of the record are in UTC.
|
|
Along with the specified timezone, the SMT also provides configuration options to include or exclude specific fields from timezone conversion using the `include.list` and `exclude.list` configuration options.
|
|
|
|
The SMT supports all {prodname} and Kafka Connect temporal and non-temporal types.
|
|
|
|
ifdef::product[]
|
|
The following topics provide details:
|
|
|
|
* xref:example-basic-debezium-timezone-converter-smt-configuration[]
|
|
* xref:effect-of-applying-the-timezone-converter-smt-to-a-debezium-event-record[]
|
|
* xref:example-advanced-debezium-timezone-converter-smt-configuration[]
|
|
* xref:options-for-configuring-the-debezium-timezone-converter-transformation[]
|
|
endif::product[]
|
|
|
|
|
|
[NOTE]
|
|
====
|
|
To comply with daylight savings time, you must specify a geographic timezone in the `converted.timezone` configuration option.
|
|
If you specify a UTC offset, the transform applies a fixed offset from UTC that isn't accurate for regions that observe daylight savings time.
|
|
Providing a fixed UTC offset is useful when converting timestamp fields to a specific timezone that does not observe daylight saving time.
|
|
====
|
|
|
|
[NOTE]
|
|
====
|
|
The `include.list` and `exclude.list` configuration options are mutually exclusive. You must specify only one of the options.
|
|
====
|
|
|
|
The SMT also allows conversion of event metadata fields in the source information block, such as `ts_ms` to the target timezone. In order to convert the metadata fields, you must include the `source` prefix in the `fieldname` of the `include.list` or `exclude.list` configuration option.
|
|
|
|
[NOTE]
|
|
====
|
|
If the schema for timestamp fields in the source information block, like `ts_ms`, is currently set to `INT64`, which is not a timestamp type, future releases aim to support the conversion of such fields by introducing compatibility for a timestamp schema.
|
|
====
|
|
|
|
// Type: concept
|
|
// Title: Example: Basic {prodname} timezone converter SMT configuration
|
|
// ModuleID: example-basic-debezium-timezone-converter-smt-configuration
|
|
|
|
[[basic-example-timezone-converter]]
|
|
== Example: Basic configuration
|
|
|
|
[[timezone-converter-usage]]
|
|
Configure the `TimezoneConverter` SMT in the Kafka Connect configuration for a connector to convert the time-based fields in an event record to a target timezone.
|
|
|
|
For example, to convert all timestamp fields in an event record from UTC to the `Pacific/Easter` timezone, add the following lines to your connector configuration:
|
|
|
|
[source]
|
|
----
|
|
transforms=convertTimezone
|
|
transforms.convertTimezone.type=io.debezium.transforms.TimezoneConverter
|
|
transforms.convertTimezone.converted.timezone=Pacific/Easter
|
|
----
|
|
|
|
// Type: concept
|
|
// Title: Effect of applying the `TimezoneConverter` SMT to a {prodname} event record
|
|
// ModuleID: effect-of-applying-the-timezone-converter-smt-to-a-debezium-event-record
|
|
|
|
=== Effect of applying the `TimezoneConverter` SMT
|
|
|
|
The following examples show how the `TimezoneConverter` transformation modifies the timestamp fields in an event record.
|
|
The first example shows a {prodname} event record that is not processed by the transformation; the record retains its original timestamp values.
|
|
The next example shows the same event record after the transformation is applied.
|
|
Per the configuration specified in xref:basic-example-timezone-converter[the basic configuration example], the SMT converts the original UTC values of timestamp fields in the source message to `Pacific/Easter` timezone values.
|
|
|
|
.Event record value before processing by the `TimezoneConverter` transformation
|
|
====
|
|
The value of the `created_at` field shows the UTC time.
|
|
|
|
[source, json,subs="+attributes"]
|
|
----
|
|
{
|
|
"before": null,
|
|
"after": {
|
|
"id": 1,
|
|
"first_name": "Anne",
|
|
"last_name": "Kretchmar",
|
|
"email": "annek@noanswer.org",
|
|
"created_at": "2011-01-11T16:40:30.123456789+00:00"
|
|
},
|
|
"source": {
|
|
"version": "{debezium-version}",
|
|
"connector": "postgresql",
|
|
"name": "PostgreSQL_server",
|
|
"ts_ms": 1559033904863,
|
|
"ts_us": 1559033904863000,
|
|
"ts_ns": 1559033904863000000,
|
|
"snapshot": true,
|
|
"db": "postgres",
|
|
"sequence": "[\"24023119\",\"24023128\"]",
|
|
"schema": "public",
|
|
"table": "customers",
|
|
"txId": 555,
|
|
"lsn": 24023128,
|
|
"xmin": null
|
|
},
|
|
"op": "c",
|
|
"ts_ms": 1559033904863,
|
|
"ts_us": 1559033904863875,
|
|
"ts_ns": 1559033904863875124
|
|
}
|
|
----
|
|
|
|
====
|
|
.Event record value after processing by the `TimezoneConverter` transformation
|
|
====
|
|
The SMT converts the original UTC value of the `created_at` field to the time in the target `Pacific/Easter` timezone that is specified in the xref:basic-example-timezone-converter[Basic configuration] example.
|
|
The SMT also adds an `event_timestamp` field.
|
|
|
|
[source, json,subs="+attributes"]
|
|
----
|
|
{
|
|
"before": null,
|
|
"after": {
|
|
"id": 1,
|
|
"first_name": "Anne",
|
|
"last_name": "Kretchmar",
|
|
"email": "annek@noanswer.org",
|
|
"created_at": "2011-01-11T11:40:30.123456789-05:00"
|
|
},
|
|
"source": {
|
|
"version": "{debezium-version}",
|
|
"connector": "postgresql",
|
|
"name": "PostgreSQL_server",
|
|
"ts_ms": 1559033904863,
|
|
"ts_us": 1559033904863752,
|
|
"ts_ns": 1559033904863752000,
|
|
"snapshot": true,
|
|
"db": "postgres",
|
|
"sequence": "[\"24023119\",\"24023128\"]",
|
|
"schema": "public",
|
|
"table": "customers",
|
|
"txId": 555,
|
|
"lsn": 24023128,
|
|
"xmin": null,
|
|
"id": 100
|
|
},
|
|
"op": "c",
|
|
"ts_ms": 1559033904863,
|
|
"ts_us": 1559033904863971,
|
|
"ts_ns": 1559033904863971541,
|
|
"event_timestamp": 1626102708861
|
|
}
|
|
----
|
|
====
|
|
|
|
// Type: concept
|
|
// Title: Example: Advanced {prodname} timezone converter SMT configuration
|
|
// ModuleID: example-advanced-debezium-timezone-converter-smt-configuration
|
|
[[advanced-example-timezone-converter]]
|
|
== Example: Advanced configuration
|
|
|
|
Instead of converting all timestamp fields in an event record, you can configure the SMT to convert specific fields only.
|
|
The following example shows how you might use the `include.list` option in the SMT configuration to convert only the `created_at`, and `updated_at` timestamp fields in an event record.
|
|
The following configuration uses a fixed offset, rather than a geographic timezone designator, to convert the time from UTC to `+05:30`.
|
|
|
|
[source]
|
|
----
|
|
transforms=convertTimezone
|
|
transforms.convertTimezone.type=io.debezium.transforms.TimezoneConverter
|
|
transforms.convertTimezone.converted.timezone=+05:30
|
|
transforms.convertTimezone.include.list=source:customers:created_at,customers:updated_at
|
|
----
|
|
|
|
In some cases, you might want to exclude specific timestamp fields from timezone conversion.
|
|
For example, to exclude the `updated_at` timestamp field in an event record from timezone conversion, use the `exclude.list` configuration option as in the following example:
|
|
|
|
[source]
|
|
----
|
|
transforms=convertTimezone
|
|
transforms.convertTimezone.type=io.debezium.transforms.TimezoneConverter
|
|
transforms.convertTimezone.converted.timezone=+05:30
|
|
transforms.convertTimezone.exclude.list=source:customers:updated_at
|
|
----
|
|
|
|
// Type: reference
|
|
// Title: Options for configuring the {prodname} timezone converter transformation
|
|
// ModuleID: options-for-configuring-the-debezium-timezone-converter-transformation
|
|
[[timezone-converter-configuration-options]]
|
|
== Configuration options
|
|
|
|
The following table lists the configuration options for the `TimezoneConverter` SMT.
|
|
|
|
.TimezoneConverter SMT configuration options
|
|
[cols="14%a,40%a,10%a,10%a"]
|
|
|===
|
|
|Property
|
|
|Description
|
|
|Type
|
|
|Importance
|
|
|
|
|[[timezone-converter-converted-timezone]]<<timezone-converter-converted-timezone, `converted.timezone`>>
|
|
|A string that specifies the target timezone to which the timestamp fields should be converted.
|
|
The target timezone can be specified as a geographic timezone, such as, `America/New_York`, or as a UTC offset, for example, `+02:00`.
|
|
|string
|
|
|high
|
|
|
|
|[[timezone-converter-include-list]]<<timezone-converter-include-list, `include.list`>>
|
|
|A comma-separated list of rules that specify the fields that the SMT includes for timezone conversion.
|
|
Specify rules by using one of the following formats:
|
|
|
|
`source:<tablename>` :: Matches {prodname} change events with source information blocks that have the specified table name.
|
|
The SMT converts all time-based fields in the matched table.
|
|
|
|
`source:<tablename>:<fieldname>` :: Matches {prodname} change events with source information blocks that have the specified table name.
|
|
The SMT converts only fields in the specified table that have the specified field name. `fieldname` can be prefixed with `before`, `after`, or `source` to include the appropriate field in the event record. If no prefix is specified, both `before` and `after` fields are converted.
|
|
|
|
`topic:<topicname>` :: Matches events from the specified topic name, converting all time-based fields in the event record.
|
|
|
|
`topic:<topicname>:<fieldname>` :: Matches events from the specified topic name, and converts values for the specified fields only. `fieldname` can be prefixed with `before`, `after`, or `source` to include the appropriate field in the event record. If no prefix is specified, both `before` and `after` fields are converted.
|
|
|
|
`<matchname>:<fieldname>` :: Applies a heuristic matching algorithm to match against the table name of the source information block, if present; otherwise, matches against the topic name.
|
|
The SMT converts values for the specified field name only. `fieldname` can be prefixed with `before`, `after`, or `source` to include the appropriate field in the event record. If no prefix is specified, both `before` and `after` fields are converted.
|
|
|list
|
|
|medium
|
|
|
|
|[[timezone-converter-exclude-list]]<<timezone-converter-exclude-list, `exclude.list`>>
|
|
|A comma-separated list of rules that specify the fields to exclude from timezone conversion.
|
|
Specify rules by using one of the following formats:
|
|
|
|
`source:<tablename>` :: Matches {prodname} change events with source information blocks that have the specified table name.
|
|
The SMT excludes all time-based fields in the matched table from conversion.
|
|
|
|
`source:<tablename>:<fieldname>` :: Matches {prodname} change events with source information blocks that have the specified table name.
|
|
The SMT excludes from conversion fields in the specified table that match the specified field name. `fieldname` can be prefixed with `before`, `after`, or `source` to exclude the appropriate field in the event record. If no prefix is specified, both `before` and `after` fields are excluded from conversion.
|
|
|
|
`topic:<topicname>` :: Matches events from the specified topic name, and excludes from conversion all time-based fields in the topic.
|
|
|
|
`topic:<topicname>:<fieldname>` :: Matches events from the specified topic name, and excludes from conversion any fields in the topic that have the specified name. `fieldname` can be prefixed with `before`, `after`, or `source` to exclude the appropriate field in the event record. If no prefix is specified, both `before` and `after` fields are excluded from conversion.
|
|
|
|
`<matchname>:<fieldname>` :: Applies a heuristic matching algorithm to match against the table name of the source information block, if present; otherwise, matches against the topic name.
|
|
The SMT excludes from conversion only fields that have the specified name. `fieldname` can be prefixed with `before`, `after`, or `source` to exclude the appropriate field in the event record. If no prefix is specified, both `before` and `after` fields are excluded from conversion.
|
|
|list
|
|
|medium
|
|
|===
|