alexey.rassokhin/tet123
1
0
Fork 0
Code Issues Pull Requests 1 Packages Projects Releases Wiki Activity

DBZ-7065 Docs for ConvertCloudEventToSaveableForm

Browse Source
This commit is contained in:
Roman Kudryashov 2023-11-11 07:27:37 +03:00 committed by Jiri Pechanec
parent d1c08f154d
commit f429536a75
3 changed files with 122 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
documentation/modules/ROOT/nav.adoc
View File

@ -38,6 +38,7 @@
** xref:transformations/header-to-value.adoc[HeaderToValue]
** xref:transformations/timezone-converter.adoc[Timezone Converter]
** xref:transformations/timescaledb.adoc[TimescaleDB Integration]
** xref:transformations/convert-cloudevent-to-saveable-form.adoc[Convert CloudEvents to Saveable Form]
* API and SPI
** xref:development/engine.adoc[Debezium Engine]
** xref:development/converters.adoc[Custom Converters]

118
documentation/modules/ROOT/pages/transformations/convert-cloudevent-to-saveable-form.adoc Normal file
View File

@ -0,0 +1,118 @@
:page-aliases: configuration/convert-cloudevent-to-saveable-form.adoc
// Category: debezium-using
// Type: assembly
// ModuleID: convert-cloudevent-to-saveable-form
// Title: ConvertCloudEventToSaveableForm
[id="convert-cloudevent-to-saveable-form"]
= ConvertCloudEventToSaveableForm
:toc:
:toc-placement: macro
:linkattrs:
:icons: font
:source-highlighter: highlight.js
toc::[]
The `ConvertCloudEventToSaveableForm` SMT is used in conjunction with `CloudEventsConverter` and `JdbcSinkConnector` that is it makes a
CloudEvent suitable for saving to a database.
It is done using the mapping between a CloudEvent and target database table columns; it specifies which CloudEvent field is
associated with which column. A new structure and its schema created by the transform contain fields with the names of target table
columns and values from an original CloudEvent. If a CloudEvent contains `data` field as a structure, it is flattened, that is,
converted to JSON string.
To perform the conversion, SMT needs to know serializer type that is used to serialize and deserialize CloudEvents
(JSON or Avro). `serializer.type` should have the same value as `value.converter.serializer.type`.
// Type: concept
// Title: Example: Basic configuration of the {prodname} `ConvertCloudEventToSaveableForm` SMT
// ModuleID: basic-configuration-of-the-debezium-convert-cloudevent-to-saveable-form-smt
[[example-convert-cloudevent-to-saveable-form]]
== Example
To convert a CloudEvent to a form suitable for `JdbcSinkConnector`, configure the `ConvertCloudEventToSaveableForm` SMT in the Kafka Connect configuration
for a connector. For example, to convert CloudEvents that were serialized and deserialized with JSON provided that the target database has
`id`, `source`, `type`, and `payload` columns, use the following configuration for the SMT:
[source]
----
"connector.class": "io.debezium.connector.jdbc.JdbcSinkConnector",
...
"transforms": "convertCloudEvent",
"transforms.convertCloudEvent.type": "io.debezium.connector.jdbc.transforms.ConvertCloudEventToSaveableForm",
"transforms.convertCloudEvent.fields.mapping": "id,source,type,data:payload",
"transforms.convertCloudEvent.serializer.type": "json",
"value.converter": "io.debezium.converters.CloudEventsConverter",
"value.converter.serializer.type": "json",
"value.converter.data.serializer.type": "json"
...
----
The following example shows the value of a record before and after the transformation is applied.
.Effect of applying the `ConvertCloudEventToSaveableForm` SMT
====
Value before the SMT processes the record::
+
[source]
----
{
"id": "624e6565-99ee-4fdb-9228-653138c3a7b3",
"source": "/debezium/postgresql/book",
"specversion": "1.0",
"type": "BookCreated",
"time": "2023-11-11T07:11:01.825Z",
"datacontenttype": "application/json",
"data": {
"id": 4,
"name": "1984",
"publicationYear": 1949
}
}
----
Value after the SMT processes the record::
+
[source, json]
----
{
"id": "624e6565-99ee-4fdb-9228-653138c3a7b3",
"source": "/debezium/postgresql/book",
"type": "BookCreated",
"payload": "{"id": 4, "name": "1984", "publicationYear": 1949}"
}
----
====
// Type: reference
// ModuleID: options-for-configuring-the-cloudevent-to-saveable-form-transformation
// Title: Options for configuring the `ConvertCloudEventToSaveableForm` transformation
[[cloudevent-to-saveable-form-configuration-options]]
== Configuration options
The following table lists the configuration options that you can use with the `ConvertCloudEventToSaveableForm` SMT.
.ConvertCloudEventToSaveableForm SMT configuration options
[cols="14%a,40%a,10%a, 16%a, 16%a, 10%a"]
|===
|Property
|Description
|Type
|Default
|Valid Values
|Importance
|[[cloudevent-to-saveable-form-fields-mapping]]<<cloudevent-to-saveable-form-fields-mapping, `fields.mapping`>>
|A comma-separated list of pairs each of which contains the name of a CloudEvents field followed by the name of a target table column
|list
|No default value
|non-empty list
|high
|[[cloudevent-to-saveable-form-serializer-type]]<<cloudevent-to-saveable-form-serializer-type, `serializer.type`>>
|Serializer type that is used to serialize and deserialize CloudEvents. Should have the same value as `value.converter.serializer.type`.
|string
|No default value
|`json` or `avro`
|high
|===

3
documentation/modules/ROOT/pages/transformations/index.adoc
View File

@ -42,6 +42,9 @@ The following SMTs are provided by {prodname}:
|xref:transformations/timescaledb.adoc[TimescaleDB Integration]
|Routes and enriches messages that the {prodname} PostgreSQL connector captures from a TimescaleDB.
|xref:transformations/convert-cloudevent-to-saveable-form.adoc[Convert CloudEvents to Saveable Form]
|Converts values of records deserialized by {prodname} CloudEvents converter to a structure suitable for {prodname} JDBC sink connector.
|===
By means of xref:transformations/applying-transformations-selectively.adoc[SMT Predicates] you can apply any of the transformations selectively, so that it modifies only that subset of change event messages that share a common characteristic.