276 lines
11 KiB
Plaintext
276 lines
11 KiB
Plaintext
// Category: debezium-using
|
|
// Type: assembly
|
|
[id="sending-signals-to-a-debezium-connector"]
|
|
= Sending signals to a {prodname} connector
|
|
ifdef::community[]
|
|
:toc:
|
|
:toc-placement: macro
|
|
:linkattrs:
|
|
:icons: font
|
|
:source-highlighter: highlight.js
|
|
|
|
toc::[]
|
|
|
|
== Overview
|
|
endif::community[]
|
|
|
|
The {prodname} signaling mechanism provides a way to modify the behavior of a connector, or to trigger a one-time action, such as initiating an xref:debezium-signaling-ad-hoc-snapshots[ad hoc snapshot] of a table.
|
|
To trigger a connector to perform a specified action, you issue a SQL command to add a signal message to a specialized signaling table, also referred to as a signaling data collection.
|
|
The signaling table, which you create on the source database, is designated exclusively for communicating with {prodname}.
|
|
When {prodname} detects that a new xref:debezium-signaling-example-of-a-logging-record[logging record] or xref:debezium-signaling-example-of-an-ad-hoc-signal-record[ad hoc snapshot record] is added to the signaling table, it reads the signal, and initiates the requested operation.
|
|
|
|
Signaling is available for use with the following {prodname} connectors:
|
|
|
|
* Db2
|
|
* MongoDB (Technology Preview)
|
|
* MySQL
|
|
* Oracle
|
|
* PostgreSQL
|
|
* SQL Server
|
|
|
|
|
|
// Type: procedure
|
|
// Title: Enabling {prodname} signaling
|
|
[id="debezium-signaling-enabling-signaling"]
|
|
== Enabling signaling
|
|
|
|
By default, the {prodname} signaling mechanism is disabled.
|
|
You must explicitly enable signaling for each connector that you want to use it with.
|
|
|
|
.Procedure
|
|
|
|
. On the source database, create a signaling data collection table for sending signals to the connector.
|
|
For information about the required structure of the signaling data collection, see xref:debezium-signaling-data-collection-structure[Structure of a signaling data collection].
|
|
|
|
. For source databases such as Db2 or SQL Server that implement a native change data capture (CDC) mechanism, enable CDC for the signaling table.
|
|
|
|
. Add the name of the signaling data collection to the {prodname} connector configuration. +
|
|
In the connector configuration, add the property `signal.data.collection`, and set its value to the fully-qualified name of the signaling data collection that you created in Step 1. +
|
|
+
|
|
For example, `signal.data.collection = inventory.debezium_signals`. +
|
|
+
|
|
The format for the fully-qualified name of the signaling collection depends on the connector. +
|
|
The following example shows the naming formats to use for each connector:
|
|
|
|
Db2:: `_<schemaName>_._<tableName>_`
|
|
MongoDB:: `_<databaseName>_._<collectionName>_`
|
|
MySQL:: `_<databaseName>_._<tableName>_`
|
|
Oracle:: `_<databaseName>_._<schemaName>_._<tableName>_`
|
|
PostgreSQL:: `_<schemaName>_._<tableName>_`
|
|
SQL Server:: `_<databaseName>_._<schemaName>_._<tableName>_` +
|
|
+
|
|
For more information about setting the `signal.data.collection` property, see the table of configuration properties for your connector.
|
|
. Add the signaling table to the list of tables to monitor. +
|
|
In the configuration for the {prodname} connector, add the name of the data collection that you created in Step 1 to the `table.include.list` property. +
|
|
+
|
|
For more information about the `table.include.list` property, see the table of configuration properties for your connector.
|
|
|
|
// Type: reference
|
|
// ModuleID: debezium-signaling-required-structure-of-a-signaling-data-collection
|
|
// Title: Required structure of a {prodname} signaling data collection
|
|
[id="debezium-signaling-data-collection-structure"]
|
|
=== Structure of a signaling data collection
|
|
|
|
A signaling data collection, or signaling table, stores signals that you send to a connector to trigger a specified operation.
|
|
The structure of the signaling table must conform to the following standard format.
|
|
|
|
* Contains three fields (columns).
|
|
* Fields are arranged in a specific order, as shown in xref:debezium-signaling-description-of-required-structure-of-a-signaling-data-collection[Table 1].
|
|
|
|
.Structure of a signaling data collection
|
|
[id="debezium-signaling-description-of-required-structure-of-a-signaling-data-collection"]
|
|
.Required structure of a signaling data collection
|
|
[cols="1,1,9",options="header"]
|
|
|===
|
|
|Field | Type | Description
|
|
|
|
|`id` +
|
|
(required)
|
|
|`string`
|
|
|
|
|An arbitrary unique string that identifies a signal instance. +
|
|
You assign an `id` to each signal that you submit to the signaling table. +
|
|
Typically the ID is a UUID string. +
|
|
You can use signal instances for logging, debugging, or de-duplication. +
|
|
When a signal triggers {prodname} to perform an incremental snapshot, it generates a signal message with an arbitrary `id` string.
|
|
The `id` string that the generated message contains is unrelated to the `id` string in the submitted signal.
|
|
|
|
|`type` +
|
|
(required)
|
|
|`string`
|
|
|
|
|Specifies the type of signal to send. +
|
|
You can use some signal types with any connector for which signaling is available, while other signal types are available for specific connectors only.
|
|
|
|
|`data` +
|
|
(optional)
|
|
|`string`
|
|
|
|
|Specifies JSON-formatted parameters to pass to a signal action. +
|
|
Each signal type requires a specific set of data.
|
|
|
|
|===
|
|
|
|
NOTE: The field names in a data collection are arbitrary.
|
|
The preceding table provides suggested names.
|
|
If you use a different naming convention, ensure that the values in each field are consistent with the expected content.
|
|
|
|
// Type: procedure
|
|
// Title: Creating a {prodname} signaling data collection
|
|
[id="debezium-signaling-creating-a-signal-data-collection"]
|
|
=== Creating a signaling data collection
|
|
|
|
You create a signaling table by submitting a standard SQL DDL query to the source database.
|
|
|
|
.Prerequisites
|
|
|
|
* You have sufficient access privileges to create a table on the target database.
|
|
|
|
.Procedure
|
|
|
|
* Submit a SQL query to the source database to create a table that is consistent with the xref:debezium-signaling-required-structure-of-a-signaling-data-collection[required structure], as shown in the following example: +
|
|
+
|
|
`CREATE TABLE _<tableName>_ (id VARCHAR(_<varcharValue>_) PRIMARY KEY, type VARCHAR(__<varcharValue>__) NOT NULL, data VARCHAR(_<varcharValue>_) NULL);` +
|
|
|
|
[NOTE]
|
|
====
|
|
The amount of space that you allocate to the `VARCHAR` parameter of the `id` variable must be sufficient to accommodate the size of the ID strings of signals sent to the signaling table. +
|
|
If the size of an ID exceeds the available space, the connector cannot process the signal.
|
|
====
|
|
|
|
The following example shows a `CREATE TABLE` command that creates a three-column `debezium_signal` table:
|
|
|
|
[source,sql]
|
|
----
|
|
CREATE TABLE debezium_signal (id VARCHAR(42) PRIMARY KEY, type VARCHAR(32) NOT NULL, data VARCHAR(2048) NULL);
|
|
----
|
|
|
|
// Type: concept
|
|
// ModuleID: debezium-signaling-types-of-signal-actions
|
|
// Title: Types of {prodname} signal actions
|
|
== Signal actions
|
|
|
|
You can use signaling to initiate the following actions:
|
|
|
|
* xref:debezium-signaling-logging[Add messages to the log].
|
|
* xref:debezium-signaling-ad-hoc-snapshots[Trigger ad hoc snapshots].
|
|
|
|
Some signals are not compatible with all connectors.
|
|
|
|
// Type: concept
|
|
[id="debezium-signaling-logging"]
|
|
=== Logging signals
|
|
|
|
You can request a connector to add an entry to the log by creating a signaling table entry with the `log` signal type.
|
|
After processing the signal, the connector prints the specified message to the log.
|
|
Optionally, you can configure the signal so that the resulting message includes the streaming coordinates.
|
|
|
|
[id="debezium-signaling-example-of-a-logging-record"]
|
|
.Example of a signaling record for adding a log message
|
|
[cols="1,9,9",options="header"]
|
|
|===
|
|
|Column | Value | Description
|
|
|
|
|id
|
|
|`924e3ff8-2245-43ca-ba77-2af9af02fa07`
|
|
|
|
|
|
|
|type
|
|
|`log`
|
|
|The action type of the signal.
|
|
|
|
|data
|
|
|`{"message": "Signal message at offset {}"}`
|
|
| The `message` parameter specifies the string to print to the log. +
|
|
If you add a placeholder (`{}`) to the message, it is replaced with streaming coordinates.
|
|
|===
|
|
|
|
// Type: concept
|
|
[id="debezium-signaling-ad-hoc-snapshots"]
|
|
=== Ad hoc snapshot signals
|
|
|
|
You can request a connector to initiate an ad hoc snapshot by creating a signaling table entry with the `execute-snapshot` signal type.
|
|
After processing the signal, the connector runs the requested snapshot operation.
|
|
|
|
Unlike the initial snapshot that a connector runs after it first starts, an ad hoc snapshot occurs during runtime, after the connector has already begun to stream change events from a database.
|
|
You can initiate ad hoc snapshots at any time.
|
|
|
|
Ad hoc snapshots are available for the following {prodname} connectors:
|
|
|
|
* Db2
|
|
ifdef::community[]
|
|
* MongoDB
|
|
endif::community[]
|
|
* MySQL
|
|
ifdef::community[]
|
|
* Oracle
|
|
endif::community[]
|
|
* PostgreSQL
|
|
* SQL Server
|
|
|
|
[id="debezium-signaling-example-of-an-ad-hoc-signal-record"]
|
|
.Example of an ad hoc snapshot signal record
|
|
[cols="1,9",options="header"]
|
|
|===
|
|
|Column | Value
|
|
|
|
|id
|
|
|`d139b9b7-7777-4547-917d-e1775ea61d41`
|
|
|
|
|type
|
|
|`execute-snapshot`
|
|
|
|
|data
|
|
|`{"data-collections": ["public.MyFirstTable", "public.MySecondTable"]}`
|
|
|
|
|===
|
|
|
|
Currently, the `execute-snapshot` action triggers xref:debezium-signaling-incremental-snapshots[incremental snapshots] only.
|
|
|
|
For more information about ad hoc snapshots, see the _Snapshots_ topic in the documentation for your connector.
|
|
|
|
|
|
.Additional resources
|
|
|
|
* xref:{link-db2-connector}#db2-ad-hoc-snapshots[Db2 connector ad hoc snapshots]
|
|
ifdef::community[]
|
|
* xref:{link-mongodb-connector}#mongodb-ad-hoc-snapshot[MongoDB connector ad hoc snapshots]
|
|
endif::community[]
|
|
* xref:{link-mysql-connector}#mysql-ad-hoc-snapshots[MySQL connector ad hoc snapshots]
|
|
ifdef::community[]
|
|
* xref:{link-oracle-connector}#oracle-ad-hoc-snapshots[Oracle connector ad hoc snapshots]
|
|
endif::community[]
|
|
* xref:{link-postgresql-connector}#postgresql-ad-hoc-snapshots[PostgreSQL connector ad hoc snapshots]
|
|
* xref:{link-sqlserver-connector}#sqlserver-ad-hoc-snapshots[SQL Server connector ad hoc snapshots]
|
|
|
|
|
|
// Type: concept
|
|
[id="debezium-signaling-incremental-snapshots"]
|
|
=== Incremental snapshots
|
|
|
|
Incremental snapshots are a specific type of ad hoc snapshot.
|
|
In an incremental snapshot, the connector captures the baseline state of the tables that you specify, similar to an initial snapshot.
|
|
However, unlike an initial snapshot, an incremental snapshot captures tables in chunks, rather than all at once.
|
|
The connector uses a watermarking method to track the progress of the snapshot.
|
|
|
|
By capturing the initial state of the specified tables in chunks rather than in a single monolithic operation, incremental snapshots provide the following advantages over the initial snapshot process:
|
|
|
|
* While the connector captures the baseline state of the specified tables, streaming of near real-time events from the transaction log continues uninterrupted.
|
|
* If the incremental snapshot process is interrupted, it can be resumed from the point at which it stopped.
|
|
* You can initiate an incremental snapshot at any time.
|
|
|
|
For more information about incremental snapshots, see the _Snapshots_ topic in the documentation for your connector.
|
|
|
|
.Additional resources
|
|
|
|
* xref:{link-db2-connector}#db2-incremental-snapshots[Db2 connector incremental snapshots]
|
|
ifdef::community[]
|
|
* xref:{link-mongodb-connector}#mongodb-incremental-snapshots[MongoDB connector incremental snapshots]
|
|
endif::community[]
|
|
* xref:{link-mysql-connector}#mysql-incremental-snapshots[MySQL connector incremental snapshots]
|
|
ifdef::community[]
|
|
* xref:{link-oracle-connector}#oracle-incremental-snapshots[Oracle connector incremental snapshots]
|
|
endif::community[]
|
|
* xref:{link-postgresql-connector}#postgresql-incremental-snapshots[PostgreSQL connector incremental snapshots]
|
|
* xref:{link-sqlserver-connector}#sqlserver-incremental-snapshots[SQL Server connector incremental snapshots]
|