22d00575c8
The `id` column holds UUID in string form (36 characters) and can be appended with suffix `-open` or `-close` (a maximum of 6 characters); so 42 characters would be the minimal length of the `id` column.
173 lines
4.4 KiB
Plaintext
173 lines
4.4 KiB
Plaintext
[id="sending-signals-to-a-debezium-connector"]
|
|
= Sending signals to a Debezium connector
|
|
|
|
:toc:
|
|
:toc-placement: macro
|
|
:linkattrs:
|
|
:icons: font
|
|
:source-highlighter: highlight.js
|
|
|
|
toc::[]
|
|
|
|
[NOTE]
|
|
====
|
|
This feature is currently in incubating state, i.e. exact semantics, configuration options etc. may change in future revisions, based on the feedback we receive. Please let us know if you encounter any problems while using this extension.
|
|
====
|
|
|
|
== Overview
|
|
Sometimes it is necessary for the user to modify a connector behaviour or trigger a one-time action (e.g. snapshot of a single table).
|
|
To fulfill such needs {prodname} provides a mechanism how to send a signal to a {prodname} connector to trigger an action.
|
|
As it might be necessary to synchronize such an action with the dataflow the signal is implemented as a write to a data collection.
|
|
|
|
Signalling is disabled by default.
|
|
The name of the signalling data collection must be set via connector configuration parameter to enable the function.
|
|
The signalling data collection must be *explictly* added among captured data collections, i.e., included in the `table.include.list` parameter.
|
|
|
|
.Connector configuration
|
|
[cols="3,9",options="header"]
|
|
|===
|
|
|Parameter | Description
|
|
|
|
|`signal.data.collection`
|
|
|Fully-qualified name of data collection. +
|
|
The name format is connector specific, e.g. `"some_schema.debezium_signals"` in case of the {prodname} Postgres connector. +
|
|
For SQL Server it should be formatted as `"some_database.some_schema.debezium_signals"`.
|
|
|
|
|===
|
|
|
|
|
|
=== Data collection structure
|
|
|
|
The signalling data collection must conform to a required format:
|
|
It must contain three fields;
|
|
the naming of the fields is arbitrary and only order of the field is important.
|
|
It is recommended but not mandatory to use the field names as defined in the table below:
|
|
|
|
.Signalling data collection structure
|
|
[cols="1,1,9",options="header"]
|
|
|===
|
|
|Column | Type | Description
|
|
|
|
|`id`
|
|
|`string` +
|
|
(required)
|
|
|A unique identifier of this signal instance. +
|
|
It can be used for logging, debugging or deduplication.
|
|
Usually an UUID string.
|
|
|
|
|`type`
|
|
|`string` +
|
|
(required)
|
|
|The type of the signal to be sent. +
|
|
There are signals common for all connectors or specific to a subset of connectors.
|
|
|
|
|`data`
|
|
|`string` +
|
|
(optional)
|
|
|JSON formatted parameters that are passed to a signal action. +
|
|
Each signal has its own expected set of data.
|
|
|
|
|===
|
|
|
|
Such table could be created with a DDL command similar to:
|
|
|
|
[source,sql]
|
|
----
|
|
CREATE TABLE debezium_signal (id VARCHAR(42) PRIMARY KEY, type VARCHAR(32) NOT NULL, data VARCHAR(2048) NULL);
|
|
----
|
|
|
|
.Example of a signal record
|
|
[cols="1,9",options="header"]
|
|
|===
|
|
|Column | Value
|
|
|
|
|id
|
|
|`924e3ff8-2245-43ca-ba77-2af9af02fa07`
|
|
|
|
|type
|
|
|`log`
|
|
|
|
|data
|
|
|`{"message": "Signal message at offset {}"}`
|
|
|
|
|===
|
|
|
|
|
|
== Signal Actions
|
|
|
|
These signals are common to all connectors
|
|
|
|
=== Logging
|
|
|
|
The type of the logging action is `log`.
|
|
It will print a provided message to the log optionally including streaming position.
|
|
|
|
.Action parameters
|
|
[cols="1,9",options="header"]
|
|
|===
|
|
|Name | Description
|
|
|
|
|message
|
|
|The string printed to the log. +
|
|
If a placeholder `{}` is added to the message it will be replaced with streaming coordinates.
|
|
|
|
|===
|
|
|
|
.Example of a logging record
|
|
[cols="1,9",options="header"]
|
|
|===
|
|
|Column | Value
|
|
|
|
|id
|
|
|`924e3ff8-2245-43ca-ba77-2af9af02fa07`
|
|
|
|
|type
|
|
|`log`
|
|
|
|
|data
|
|
|`{"message": "Signal message at offset {}"}`
|
|
|
|
|===
|
|
|
|
|
|
=== Triggering an Incremental Snapshot
|
|
*Available for MySQL, PostgresSQL, SQL Server and Db2 connectors*
|
|
|
|
The type of the action is `execute-snapshot`.
|
|
It will start an incremental snapshot operation. The operation first reads the first and last primary key values and use those as start and end point for each table.
|
|
|
|
==== Optional Connector Configuration
|
|
`incremental.snapshot.chunk.size`: Integer value. The number of rows collected at each fetch operation on the database.
|
|
|
|
|
|
.Action parameters
|
|
[cols="3,9",options="header", source, adoc]
|
|
|===
|
|
|Name | Description
|
|
|
|
|data-collections
|
|
a|The list of data collections to be snapshoted. +
|
|
Formatting should be: +
|
|
|
|
* MySQL: `"some_database.some_table"`
|
|
* PostgresSQL and Db2: `"some_schema.some_table"`
|
|
* SQL Server: `"some_database.some_schema.some_table"`
|
|
|
|
|===
|
|
|
|
.Example of executing an incremental snapshot
|
|
[cols="1,9",options="header"]
|
|
|===
|
|
|Column | Value
|
|
|
|
|id
|
|
|`d139b9b7-7777-4547-917d-e1775ea61d41`
|
|
|
|
|type
|
|
|`execute-snapshot`
|
|
|
|
|data
|
|
|`{"data-collections": ["public.MyFirstTable", "public.MySecondTable"]}`
|
|
|
|
|===
|