:page-aliases: configuration/timezone-converter.adoc // Category: debezium-using // Type: assembly // ModuleID: converting-timezone-values-in-debezium-event-records // Title: Converting timezone values in {prodname} event records [id="timezone-converter"] = Timezone Converter ifdef::community[] :toc: :toc-placement: macro :linkattrs: :icons: font :source-highlighter: highlight.js toc::[] endif::community[] [[timezone-converter-introduction]] When {prodname} emits event records, the timezone values for timestamp fields in the record can vary, depending on the type and configuration of the data source. To maintain data consistency and precision within data processing pipelines and applications, you can use the `Timezone Converter` SMT to ensure that event records use a consistent timezone to represent timestamp data. The SMT converts the value of the specified field to the target timezone by using the `converted.timezone` configuration option. You can specify the target timezone as a geographic timezone, for example, `America/New_York`, or as a UTC offset, such as `+02:00`. It is assumed that the fields of the record are in UTC. Along with the specified timezone, the SMT also provides configuration options to include or exclude specific fields from timezone conversion using the `include.list` and `exclude.list` configuration options. The SMT supports all {prodname} and Kafka Connect temporal and non-temporal types. ifdef::product[] The following topics provide details: * xref:example-basic-debezium-timezone-converter-smt-configuration[] * xref:effect-of-applying-the-timezone-converter-smt-to-a-debezium-event-record[] * xref:example-advanced-debezium-timezone-converter-smt-configuration[] * xref:options-for-configuring-the-debezium-timezone-converter-transformation[] endif::product[] [NOTE] ==== To comply with daylight savings time, you must specify a geographic timezone in the `converted.timezone` configuration option. If you specify a UTC offset, the transform applies a fixed offset from UTC that isn't accurate for regions that observe daylight savings time. Providing a fixed UTC offset is useful when converting timestamp fields to a specific timezone that does not observe daylight saving time. ==== [NOTE] ==== The `include.list` and `exclude.list` configuration options are mutually exclusive. You must specify only one of the options. ==== The SMT also allows conversion of event metadata fields in the source information block, such as `ts_ms` to the target timezone. In order to convert the metadata fields, you must include the `source` prefix in the `fieldname` of the `include.list` or `exclude.list` configuration option. [NOTE] ==== If the schema for timestamp fields in the source information block, like `ts_ms`, is currently set to `INT64`, which is not a timestamp type, future releases aim to support the conversion of such fields by introducing compatibility for a timestamp schema. ==== // Type: concept // Title: Example: Basic {prodname} timezone converter SMT configuration // ModuleID: example-basic-debezium-timezone-converter-smt-configuration [[basic-example-timezone-converter]] == Example: Basic configuration [[timezone-converter-usage]] Configure the `TimezoneConverter` SMT in the Kafka Connect configuration for a connector to convert the time-based fields in an event record to a target timezone. For example, to convert all timestamp fields in an event record from UTC to the `Pacific/Easter` timezone, add the following lines to your connector configuration: [source] ---- transforms=convertTimezone transforms.convertTimezone.type=io.debezium.transforms.TimezoneConverter transforms.convertTimezone.converted.timezone=Pacific/Easter ---- // Type: concept // Title: Effect of applying the `TimezoneConverter` SMT to a {prodname} event record // ModuleID: effect-of-applying-the-timezone-converter-smt-to-a-debezium-event-record === Effect of applying the `TimezoneConverter` SMT The following examples show how the `TimezoneConverter` transformation modifies the timestamp fields in an event record. The first example shows a {prodname} event record that is not processed by the transformation; the record retains its original timestamp values. The next example shows the same event record after the transformation is applied. Per the configuration specified in xref:basic-example-timezone-converter[the basic configuration example], the SMT converts the original UTC values of timestamp fields in the source message to `Pacific/Easter` timezone values. .Event record value before processing by the `TimezoneConverter` transformation ==== The value of the `created_at` field shows the UTC time. [source, json,subs="+attributes"] ---- { "before": null, "after": { "id": 1, "first_name": "Anne", "last_name": "Kretchmar", "email": "annek@noanswer.org", "created_at": "2011-01-11T16:40:30.123456789+00:00" }, "source": { "version": "{debezium-version}", "connector": "postgresql", "name": "PostgreSQL_server", "ts_ms": 1559033904863, "ts_us": 1559033904863000, "ts_ns": 1559033904863000000, "snapshot": true, "db": "postgres", "sequence": "[\"24023119\",\"24023128\"]", "schema": "public", "table": "customers", "txId": 555, "lsn": 24023128, "xmin": null }, "op": "c", "ts_ms": 1559033904863, "ts_us": 1559033904863875, "ts_ns": 1559033904863875124 } ---- ==== .Event record value after processing by the `TimezoneConverter` transformation ==== The SMT converts the original UTC value of the `created_at` field to the time in the target `Pacific/Easter` timezone that is specified in the xref:basic-example-timezone-converter[Basic configuration] example. The SMT also adds an `event_timestamp` field. [source, json,subs="+attributes"] ---- { "before": null, "after": { "id": 1, "first_name": "Anne", "last_name": "Kretchmar", "email": "annek@noanswer.org", "created_at": "2011-01-11T11:40:30.123456789-05:00" }, "source": { "version": "{debezium-version}", "connector": "postgresql", "name": "PostgreSQL_server", "ts_ms": 1559033904863, "ts_us": 1559033904863752, "ts_ns": 1559033904863752000, "snapshot": true, "db": "postgres", "sequence": "[\"24023119\",\"24023128\"]", "schema": "public", "table": "customers", "txId": 555, "lsn": 24023128, "xmin": null, "id": 100 }, "op": "c", "ts_ms": 1559033904863, "ts_us": 1559033904863971, "ts_ns": 1559033904863971541, "event_timestamp": 1626102708861 } ---- ==== // Type: concept // Title: Example: Advanced {prodname} timezone converter SMT configuration // ModuleID: example-advanced-debezium-timezone-converter-smt-configuration [[advanced-example-timezone-converter]] == Example: Advanced configuration Instead of converting all timestamp fields in an event record, you can configure the SMT to convert specific fields only. The following example shows how you might use the `include.list` option in the SMT configuration to convert only the `created_at`, and `updated_at` timestamp fields in an event record. The following configuration uses a fixed offset, rather than a geographic timezone designator, to convert the time from UTC to `+05:30`. [source] ---- transforms=convertTimezone transforms.convertTimezone.type=io.debezium.transforms.TimezoneConverter transforms.convertTimezone.converted.timezone=+05:30 transforms.convertTimezone.include.list=source:customers:created_at,customers:updated_at ---- In some cases, you might want to exclude specific timestamp fields from timezone conversion. For example, to exclude the `updated_at` timestamp field in an event record from timezone conversion, use the `exclude.list` configuration option as in the following example: [source] ---- transforms=convertTimezone transforms.convertTimezone.type=io.debezium.transforms.TimezoneConverter transforms.convertTimezone.converted.timezone=+05:30 transforms.convertTimezone.exclude.list=source:customers:updated_at ---- // Type: reference // Title: Options for configuring the {prodname} timezone converter transformation // ModuleID: options-for-configuring-the-debezium-timezone-converter-transformation [[timezone-converter-configuration-options]] == Configuration options The following table lists the configuration options for the `TimezoneConverter` SMT. .TimezoneConverter SMT configuration options [cols="14%a,40%a,10%a,10%a"] |=== |Property |Description |Type |Importance |[[timezone-converter-converted-timezone]]<> |A string that specifies the target timezone to which the timestamp fields should be converted. The target timezone can be specified as a geographic timezone, such as, `America/New_York`, or as a UTC offset, for example, `+02:00`. |string |high |[[timezone-converter-include-list]]<> |A comma-separated list of rules that specify the fields that the SMT includes for timezone conversion. Specify rules by using one of the following formats: `source:` :: Matches {prodname} change events with source information blocks that have the specified table name. The SMT converts all time-based fields in the matched table. `source::` :: Matches {prodname} change events with source information blocks that have the specified table name. The SMT converts only fields in the specified table that have the specified field name. `fieldname` can be prefixed with `before`, `after`, or `source` to include the appropriate field in the event record. If no prefix is specified, both `before` and `after` fields are converted. `topic:` :: Matches events from the specified topic name, converting all time-based fields in the event record. `topic::` :: Matches events from the specified topic name, and converts values for the specified fields only. `fieldname` can be prefixed with `before`, `after`, or `source` to include the appropriate field in the event record. If no prefix is specified, both `before` and `after` fields are converted. `:` :: Applies a heuristic matching algorithm to match against the table name of the source information block, if present; otherwise, matches against the topic name. The SMT converts values for the specified field name only. `fieldname` can be prefixed with `before`, `after`, or `source` to include the appropriate field in the event record. If no prefix is specified, both `before` and `after` fields are converted. |list |medium |[[timezone-converter-exclude-list]]<> |A comma-separated list of rules that specify the fields to exclude from timezone conversion. Specify rules by using one of the following formats: `source:` :: Matches {prodname} change events with source information blocks that have the specified table name. The SMT excludes all time-based fields in the matched table from conversion. `source::` :: Matches {prodname} change events with source information blocks that have the specified table name. The SMT excludes from conversion fields in the specified table that match the specified field name. `fieldname` can be prefixed with `before`, `after`, or `source` to exclude the appropriate field in the event record. If no prefix is specified, both `before` and `after` fields are excluded from conversion. `topic:` :: Matches events from the specified topic name, and excludes from conversion all time-based fields in the topic. `topic::` :: Matches events from the specified topic name, and excludes from conversion any fields in the topic that have the specified name. `fieldname` can be prefixed with `before`, `after`, or `source` to exclude the appropriate field in the event record. If no prefix is specified, both `before` and `after` fields are excluded from conversion. `:` :: Applies a heuristic matching algorithm to match against the table name of the source information block, if present; otherwise, matches against the topic name. The SMT excludes from conversion only fields that have the specified name. `fieldname` can be prefixed with `before`, `after`, or `source` to exclude the appropriate field in the event record. If no prefix is specified, both `before` and `after` fields are excluded from conversion. |list |medium |===