DBZ-6731 Document blocking snapshot signal configuration
This commit is contained in:
mfvitale
Jiri Pechanec
parent
a8179932fa
commit
be843dd635
@ -96,7 +96,7 @@ The structure of the signaling table must conform to the following standard form
|
||||
|
||||
|An arbitrary unique string that identifies a signal instance. +
|
||||
You assign an `id` to each signal that you submit to the signaling table. +
|
||||
Typically the ID is a UUID string. +
|
||||
Typically, the ID is a UUID string. +
|
||||
You can use signal instances for logging, debugging, or de-duplication. +
|
||||
When a signal triggers {prodname} to perform an incremental snapshot, it generates a signal message with an arbitrary `id` string.
|
||||
The `id` string that the generated message contains is unrelated to the `id` string in the submitted signal.
|
||||
@ -185,7 +185,7 @@ The value is a JSON object with `type` and `data` fields.
|
||||
When the signal type is set to `execute-snapshot`, the `data` field must include the fields that are listed in the following table:
|
||||
|
||||
.Execute snapshot data fields
|
||||
[cols="2,2,6",options="header"]
|
||||
[cols="2,2,6a",options="header"]
|
||||
|===
|
||||
|Field | Default | Value
|
||||
|
||||
@ -196,12 +196,25 @@ Currently {prodname} supports the `incremental` and `blocking` types.
|
||||
|
||||
|`data-collections`
|
||||
|_N/A_
|
||||
| An array of comma-separated regular expressions that match the fully-qualified names of the tables to include in the snapshot. +
|
||||
| An array of comma-separated regular expressions that match the fully-qualified names of the data collections to include in the snapshot. +
|
||||
Specify the names by using the same format as is required for the `signal.data.collection` configuration option.
|
||||
|
||||
|`additional-condition`
|
||||
|`[.line-through]#additional-condition#`
|
||||
|_N/A_
|
||||
| An optional string that specifies a condition that the connector evaluates to designate a subset of records to include in a snapshot.
|
||||
| An optional string that specifies a condition that the connector evaluates to designate a subset of records to include in a snapshot. +
|
||||
+
|
||||
[NOTE]
|
||||
====
|
||||
This property is deprecated and should be replaced by additional-conditions.
|
||||
====
|
||||
|
||||
|`additional-conditions`
|
||||
|_N/A_
|
||||
| An optional array of additional condition that specifies a condition that the connector evaluates to designate a subset of records to include in a snapshot. +
|
||||
Each additional condition is an object with `data-collection` and `filter` properties. You can now specify different filters for different data collection.
|
||||
|
||||
* The `data-collection` property is the fully-qualified name of the data collection for which the filter will be applied.
|
||||
* The `filter` property will have the same value used in the old property for the `incremental snapshot` and the value used in the `snapshot.select.statement.overrides` for the `blocking snapshot`.
|
||||
|===
|
||||
|
||||
The following example shows a typical `execute-snapshot` Kafka message:
|
||||
@ -235,7 +248,7 @@ p2:: A JSON data field that contains additional information about the specified
|
||||
In the JSON data field, include the information that is listed in the following table:
|
||||
+
|
||||
.Execute snapshot data fields
|
||||
[cols="2,2,6",options="header"]
|
||||
[cols="2,2,6a",options="header"]
|
||||
|===
|
||||
|Field | Default | Value
|
||||
|
||||
@ -249,9 +262,22 @@ Currently {prodname} supports the `incremental` and `blocking` types.
|
||||
| An array of comma-separated regular expressions that match the fully-qualified names of the tables to include in the snapshot. +
|
||||
Specify the names by using the same format as is required for the xref:{context}-property-signal-data-collection[signal.data.collection] configuration option.
|
||||
|
||||
|`additional-condition`
|
||||
|`[.line-through]#additional-condition#`
|
||||
|_N/A_
|
||||
| An optional string that specifies a condition that the connector evaluates to designate a subset of records to include in a snapshot.
|
||||
| An optional string that specifies a condition that the connector evaluates to designate a subset of records to include in a snapshot. +
|
||||
|
||||
[NOTE]
|
||||
====
|
||||
This property is deprecated and should be replaced by additional-conditions.
|
||||
====
|
||||
|
||||
|`additional-conditions`
|
||||
|_N/A_
|
||||
| An optional array of additional condition that specifies a condition that the connector evaluates to designate a subset of records to include in a snapshot. +
|
||||
Each additional condition is an object with `data-collection` and `filter` properties. You can now specify different filters for different data collection.
|
||||
|
||||
* The `data-collection` property is the fully-qualified name of the data collection for which the filter will be applied.
|
||||
* The `filter` property will have the same value used in the old property for the `incremental snapshot` and the value used in the `snapshot.select.statement.overrides` for the `blocking snapshot`.
|
||||
|===
|
||||
+
|
||||
The following image shows an example of how to use JConsole to send a signal:
|
||||
@ -379,7 +405,11 @@ Optionally, you can configure the signal so that the resulting message includes
|
||||
|The action type of the signal.
|
||||
|
||||
|data
|
||||
|`{"message": "Signal message at offset {}"}`
|
||||
a|
|
||||
[source,json]
|
||||
----
|
||||
{"message": "Signal message at offset {}"}
|
||||
----
|
||||
| The `message` parameter specifies the string to print to the log. +
|
||||
If you add a placeholder (`{}`) to the message, it is replaced with streaming coordinates.
|
||||
|===
|
||||
@ -418,7 +448,11 @@ endif::community[]
|
||||
|`execute-snapshot`
|
||||
|
||||
|data
|
||||
|`{"data-collections": ["public.MyFirstTable", "public.MySecondTable"]}`
|
||||
a|
|
||||
[source,json]
|
||||
----
|
||||
{"data-collections": ["public.MyFirstTable", "public.MySecondTable"]}
|
||||
----
|
||||
|
||||
|===
|
||||
|
||||
@ -429,7 +463,11 @@ endif::community[]
|
||||
|Key | Value
|
||||
|
||||
|test_connector
|
||||
|`{"type":"execute-snapshot","data": {"data-collections": ["public.MyFirstTable"], "type": "INCREMENTAL", "additional-condition":"color='blue' AND brand='MyBrand'"}}`
|
||||
a|
|
||||
[source,json]
|
||||
----
|
||||
{"type":"execute-snapshot","data": {"data-collections": ["public.MyFirstTable"], "type": "INCREMENTAL", "additional-conditions":[{"data-collection": "public.MyFirstTable", "filter":"color='blue' AND brand='MyBrand'"]}}
|
||||
----
|
||||
|
||||
|===
|
||||
|
||||
@ -478,8 +516,11 @@ endif::community[]
|
||||
|`stop-snapshot`
|
||||
|
||||
|data
|
||||
|`{"type":"INCREMENTAL", "data-collections": ["public.MyFirstTable"]}`
|
||||
|
||||
a|
|
||||
[source,json]
|
||||
----
|
||||
{"type":"INCREMENTAL", "data-collections": ["public.MyFirstTable"]}
|
||||
----
|
||||
|===
|
||||
|
||||
You must specify the `type` of the signal.
|
||||
@ -620,7 +661,11 @@ endif::community[]
|
||||
|`execute-snapshot`
|
||||
|
||||
|data
|
||||
|`{"type": "blocking"}`
|
||||
a|
|
||||
[source,json]
|
||||
----
|
||||
{"type": "blocking", "data-collections": ["schema1.table1", "schema1.table2"], "additional-conditions": [{"data-collection": "schema1.table1", "filter": "SELECT * FROM [schema1].[table1] WHERE column1 = 0 ORDER BY column2 DESC"}, {"data-collection": "schema1.table2", "filter": "SELECT * FROM [schema1].[table2] WHERE column2 > 0"}]}
|
||||
----
|
||||
|
||||
|===
|
||||
|
||||
@ -631,8 +676,11 @@ endif::community[]
|
||||
|Key | Value
|
||||
|
||||
|test_connector
|
||||
|`{"type":"execute-snapshot","data": {"type": "blocking"}`
|
||||
|
||||
a|
|
||||
[source,json]
|
||||
----
|
||||
{"type":"execute-snapshot","data": {"type": "blocking"}
|
||||
----
|
||||
|===
|
||||
|
||||
|
||||
|
||||
@ -26,14 +26,9 @@ endif::[]
|
||||
You specify the {data-collection}s to capture by sending an `execute-snapshot` message to the signaling {data-collection}.
|
||||
Set the type of the `execute-snapshot` signal to `incremental` or `blocking`, and provide the names of the {data-collection}s to include in the snapshot, as described in the following table:
|
||||
|
||||
[NOTE]
|
||||
====
|
||||
Unlike incremental snapshot type, a blocking snapshot does not support the `data-collections`, `additional-condition`, or `surrogate-key' parameters.
|
||||
For details about the parameters supported for blocking snapshots, see the `snapshot.*` connectors properties.
|
||||
====
|
||||
|
||||
.Example of an ad hoc `execute-snapshot` signal record
|
||||
[cols="2,2,6",options="header"]
|
||||
[cols="2,2,6a",options="header"]
|
||||
|===
|
||||
|Field | Default | Value
|
||||
|
||||
@ -49,10 +44,23 @@ Currently, you can request `incremental` or `blocking` snapshots.
|
||||
The format of the names is the same as for the `signal.data.collection` configuration option.
|
||||
|
||||
ifeval::['{context}' != 'mongodb']
|
||||
|`additional-condition`
|
||||
|`[.line-through]#additional-condition#`
|
||||
|_N/A_
|
||||
| An optional string, which specifies a condition based on the column(s) of the {data-collection}(s), to capture a
|
||||
subset of the contents of the {data-collection}(s).
|
||||
subset of the contents of the {data-collection}(s). +
|
||||
|
||||
[NOTE]
|
||||
====
|
||||
This property is deprecated and should be replaced by additional-conditions.
|
||||
====
|
||||
|
||||
|`additional-conditions`
|
||||
|_N/A_
|
||||
| An optional array of additional condition that specifies a condition that the connector evaluates to designate a subset of records to include in a snapshot. +
|
||||
Each additional condition is an object with `data-collection` and `filter` properties. You can now specify different filters for different data collection.
|
||||
|
||||
* The `data-collection` property is the fully-qualified name of the data collection for which the filter will be applied.
|
||||
* The `filter` property will have the same value used in the old property for the `incremental snapshot` and the value used in the `snapshot.select.statement.overrides` for the `blocking snapshot`.
|
||||
endif::[]
|
||||
|
||||
ifeval::['{context}' != 'mongodb']
|
||||
|
||||
@ -12,7 +12,19 @@ You might want to run a blocking snapshot rather than use the standard initial s
|
||||
When you run a blocking snapshot, {prodname} stops streaming, and then initiates a snapshot of the specified {data-collection}, following the same process that it uses during an initial snapshot.
|
||||
After the snapshot completes, the streaming is resumed.
|
||||
|
||||
Use the `snapshot.*` properties in the connector configuration to control the behavior of the snapshot.
|
||||
.Configure snapshot
|
||||
|
||||
In the `data` section of the signal you can use the following properties:
|
||||
* data-collections: to specify which {data-collection}s must be snapshot
|
||||
* additional-conditions: You can specify different filters for different {data-collection}. +
|
||||
* The `data-collection` property is the fully-qualified name of the {data-collection} for which the filter will be applied.
|
||||
* The `filter` property will have the same value used in the `snapshot.select.statement.overrides`
|
||||
|
||||
Fo example
|
||||
[source,json]
|
||||
----
|
||||
{"type": "blocking", "data-collections": ["schema1.table1", "schema1.table2"], "additional-conditions": [{"data-collection": "schema1.table1", "filter": "SELECT * FROM [schema1].[table1] WHERE column1 = 0 ORDER BY column2 DESC"}, {"data-collection": "schema1.table2", "filter": "SELECT * FROM [schema1].[table2] WHERE column2 > 0"}]}
|
||||
----
|
||||
|
||||
.Possible duplicates
|
||||
A delay might exist between the time that you send the signal to trigger the snapshot, and the time when streaming stops and the snapshot starts.
|
||||
|
||||
@ -9,7 +9,7 @@ The value of the message is a JSON object with `type` and `data` fields.
|
||||
The signal type is `execute-snapshot`, and the `data` field must have the following fields:
|
||||
|
||||
.Execute snapshot data fields
|
||||
[cols="2,2,6",options="header"]
|
||||
[cols="2,2,6a",options="header"]
|
||||
|===
|
||||
|Field | Default | Value
|
||||
|
||||
@ -24,9 +24,22 @@ See the next section for more details.
|
||||
| An array of comma-separated regular expressions that match the fully-qualified names of tables to include in the snapshot. +
|
||||
Specify the names by using the same format as is required for the xref:{context}-property-signal-data-collection[signal.data.collection] configuration option.
|
||||
|
||||
|`additional-condition`
|
||||
|`[.line-through]#additional-condition#`
|
||||
|_N/A_
|
||||
| An optional string that specifies a condition that the connector evaluates to designate a subset of columns to include in a snapshot.
|
||||
| An optional string that specifies a condition that the connector evaluates to designate a subset of records to include in a snapshot. +
|
||||
|
||||
[NOTE]
|
||||
====
|
||||
This property is deprecated and should be replaced by additional-conditions.
|
||||
====
|
||||
|
||||
|`additional-conditions`
|
||||
|_N/A_
|
||||
| An optional array of additional condition that specifies a condition that the connector evaluates to designate a subset of records to include in a snapshot. +
|
||||
Each additional condition is an object with `data-collection` and `filter` properties. You can now specify different filters for different data collection.
|
||||
|
||||
* The `data-collection` property is the fully-qualified name of the data collection for which the filter will be applied.
|
||||
* The `filter` property will have the same value used in the old property for the `incremental snapshot` and the value used in the `snapshot.select.statement.overrides` for the `blocking snapshot`.
|
||||
|===
|
||||
|
||||
An example of the execute-snapshot Kafka message:
|
||||
@ -37,30 +50,30 @@ Key = `test_connector`
|
||||
Value = `{"type":"execute-snapshot","data": {"data-collections": ["schema1.table1", "schema1.table2"], "type": "INCREMENTAL"}}`
|
||||
----
|
||||
|
||||
==== Ad hoc incremental snapshots with additional-condition
|
||||
==== Ad hoc incremental snapshots with additional-conditions
|
||||
|
||||
{prodname} uses the `additional-condition` field to select a subset of a {data-collection}'s content.
|
||||
{prodname} uses the `additional-conditions` field to select a subset of a {data-collection}'s content.
|
||||
|
||||
Typically, when {prodname} runs a snapshot, it runs a SQL query such as:
|
||||
|
||||
`SELECT * FROM _<tableName>_ ....`
|
||||
|
||||
When the snapshot request includes an `additional-condition`, the `additional-condition` is appended to the SQL query, for example:
|
||||
When the snapshot request includes an `additional-conditions`, the `data-collection` and `filter` property of the `additional-conditions` is appended to the SQL query, for example:
|
||||
|
||||
`SELECT * FROM _<tableName>_ WHERE _<additional-condition>_ ....`
|
||||
`SELECT * FROM _<data-collection>_ WHERE _<filter>_ ....`
|
||||
|
||||
For example, given a `products` {data-collection} with the columns `id` (primary key), `color`, and `brand`, if you want a snapshot to include only content for which `color='blue'`, when you request the snapshot, you could append an `additional-condition` statement to filter the content:
|
||||
For example, given a `products` {data-collection} with the columns `id` (primary key), `color`, and `brand`, if you want a snapshot to include only content for which `color='blue'`, when you request the snapshot, you could add the `additional-conditions` property to filter the content:
|
||||
----
|
||||
Key = `test_connector`
|
||||
|
||||
Value = `{"type":"execute-snapshot","data": {"data-collections": ["schema1.products"], "type": "INCREMENTAL", "additional-condition":"color='blue'"}}`
|
||||
Value = `{"type":"execute-snapshot","data": {"data-collections": ["schema1.products"], "type": "INCREMENTAL", "additional-conditions": [{"data-collection": "schema1.products" ,"filter":"color='blue'"}]}}`
|
||||
----
|
||||
|
||||
You can use the `additional-condition` statement to pass conditions based on multiple columns.
|
||||
You can use the `additional-conditions` property to pass conditions based on multiple columns.
|
||||
For example, using the same `products` {data-collection} as in the previous example, if you want a snapshot to include only the content from the `products` {data-collection} for which `color='blue'`, and `brand='MyBrand'`, you could send the following request:
|
||||
|
||||
----
|
||||
Key = `test_connector`
|
||||
|
||||
Value = `{"type":"execute-snapshot","data": {"data-collections": ["schema1.products"], "type": "INCREMENTAL", "additional-condition":"color='blue' AND brand='MyBrand'"}}`
|
||||
Value = `{"type":"execute-snapshot","data": {"data-collections": ["schema1.products"], "type": "INCREMENTAL", "additional-conditions": [{"data-collection": "schema1.products" ,"filter":"color='blue' AND brand='MyBrand'"}]}}`
|
||||
----
|
||||
|
||||
@ -33,7 +33,7 @@ For example, to include a table that exists in the `*public*` schema and that ha
|
||||
+
|
||||
[source,sql,indent=0,subs="+attributes,+quotes"]
|
||||
----
|
||||
INSERT INTO _<signalTable>_ (id, type, data) VALUES (_'<id>'_, _'<snapshotType>'_, '{"data-collections": ["_<tableName>_","_<tableName>_"],"type":"_<snapshotType>_","additional-condition":"_<additional-condition>_"}');
|
||||
INSERT INTO _<signalTable>_ (id, type, data) VALUES (_'<id>'_, _'<snapshotType>'_, '{"data-collections": ["_<tableName>_","_<tableName>_"],"type":"_<snapshotType>_","additional-conditions":[{"data-collection": "_<tableName>_", "filter": "_<additional-condition>_"}]}');
|
||||
----
|
||||
+
|
||||
For example,
|
||||
@ -45,7 +45,7 @@ values ('ad-hoc-1', // <2>
|
||||
'execute-snapshot', // <3>
|
||||
'{"data-collections": ["schema1.table1", "schema2.table2"], // <4>
|
||||
"type":"incremental"}, // <5>
|
||||
"additional-condition":"color=blue"}'); // <6>
|
||||
"additional-conditions":[{"data-collection": "schema1.table1" ,"filter":"color='blue'"}]}'); // <6>
|
||||
----
|
||||
+
|
||||
The values of the `id`,`type`, and `data` parameters in the command correspond to the {link-prefix}:{link-signalling}#debezium-signaling-description-of-required-structure-of-a-signaling-data-collection[fields of the signaling {data-collection}].
|
||||
@ -84,16 +84,17 @@ Currently, the only valid option is the default value, `incremental`. +
|
||||
If you do not specify a value, the connector runs an incremental snapshot.
|
||||
|
||||
|6
|
||||
|`additional-condition`
|
||||
|An optional string, which specifies a condition based on the column(s) of the {data-collection}(s), to capture a
|
||||
subset of the contents of the {data-collection}s.
|
||||
For more information about the `additional-condition` parameter, see xref:{context}-incremental-snapshots-additional-condition[].
|
||||
|`additional-conditions`
|
||||
| An optional array of additional condition that specifies a condition that the connector evaluates to designate a subset of records to include in a snapshot. +
|
||||
Each additional condition is an object with `data-collection` and `filter` properties. You can now specify different filters for different data collection. +
|
||||
* The `data-collection` property is the fully-qualified name of the data collection for which the filter will be applied.
|
||||
For more information about the `additional-conditions` parameter, see xref:{context}-incremental-snapshots-additional-conditions[].
|
||||
|===
|
||||
|
||||
[id="{context}-incremental-snapshots-additional-condition"]
|
||||
.Ad hoc incremental snapshots with `additional-condition`
|
||||
[id="{context}-incremental-snapshots-additional-conditions"]
|
||||
.Ad hoc incremental snapshots with `additional-conditions`
|
||||
|
||||
If you want a snapshot to include only a subset of the content in a {data-collection}, you can modify the signal request by appending an `additional-condition` parameter to the snapshot signal.
|
||||
If you want a snapshot to include only a subset of the content in a {data-collection}, you can modify the signal request by appending an `additional-conditions` parameter to the snapshot signal.
|
||||
|
||||
The SQL query for a typical snapshot takes the following form:
|
||||
|
||||
@ -102,17 +103,17 @@ The SQL query for a typical snapshot takes the following form:
|
||||
SELECT * FROM _<tableName>_ ....
|
||||
----
|
||||
|
||||
By adding an `additional-condition` parameter, you append a `WHERE` condition to the SQL query, as in the following example:
|
||||
By adding an `additional-conditions` parameter, you append a `WHERE` condition to the SQL query, as in the following example:
|
||||
|
||||
[source,sql,subs="+attributes,+quotes"]
|
||||
----
|
||||
SELECT * FROM _<tableName>_ WHERE _<additional-condition>_ ....
|
||||
SELECT * FROM _<data-collection>_ WHERE _<filter>_ ....
|
||||
----
|
||||
|
||||
The following example shows a SQL query to send an ad hoc incremental snapshot request with an additional condition to the signaling {data-collection}:
|
||||
[source,sql,indent=0,subs="+attributes,+quotes"]
|
||||
----
|
||||
INSERT INTO _<signalTable>_ (id, type, data) VALUES (_'<id>'_, _'<snapshotType>'_, '{"data-collections": ["_<tableName>_","_<tableName>_"],"type":"_<snapshotType>_","additional-condition":"_<additional-condition>_"}');
|
||||
INSERT INTO _<signalTable>_ (id, type, data) VALUES (_'<id>'_, _'<snapshotType>'_, '{"data-collections": ["_<tableName>_","_<tableName>_"],"type":"_<snapshotType>_","additional-conditions":[{"data-collection": "_<tableName>_", "filter": "_<additional-condition>_"}]}');
|
||||
----
|
||||
|
||||
For example, suppose you have a `products` {data-collection} that contains the following columns:
|
||||
@ -125,15 +126,15 @@ If you want an incremental snapshot of the `products` {data-collection} to inclu
|
||||
|
||||
[source,sql,indent=0,subs="+attributes"]
|
||||
----
|
||||
INSERT INTO myschema.debezium_signal (id, type, data) VALUES('ad-hoc-1', 'execute-snapshot', '{"data-collections": ["schema1.products"],"type":"incremental", "additional-condition":"color=blue"}');
|
||||
INSERT INTO myschema.debezium_signal (id, type, data) VALUES('ad-hoc-1', 'execute-snapshot', '{"data-collections": ["schema1.products"],"type":"incremental", "additional-conditions":[{"data-collection": "schema1.products", "filter": "color=blue"}]}');
|
||||
----
|
||||
|
||||
The `additional-condition` parameter also enables you to pass conditions that are based on more than one column.
|
||||
The `additional-conditions` parameter also enables you to pass conditions that are based on more than one column.
|
||||
For example, using the `products` {data-collection} from the previous example, you can submit a query that triggers an incremental snapshot that includes the data of only those items for which `color=blue` and `quantity>10`:
|
||||
|
||||
[source,sql,indent=0,subs="+attributes"]
|
||||
----
|
||||
INSERT INTO myschema.debezium_signal (id, type, data) VALUES('ad-hoc-1', 'execute-snapshot', '{"data-collections": ["schema1.products"],"type":"incremental", "additional-condition":"color=blue AND quantity>10"}');
|
||||
INSERT INTO myschema.debezium_signal (id, type, data) VALUES('ad-hoc-1', 'execute-snapshot', '{"data-collections": ["schema1.products"],"type":"incremental", "additional-conditions":[{"data-collection": "schema1.products", "filter": "color=blue AND quantity>10"}]}');
|
||||
----
|
||||
|
||||
The following example, shows the JSON for an incremental snapshot event that is captured by a connector.
|
||||
|
||||
@ -48,7 +48,7 @@ tag::sql-based-snapshot[]
|
||||
+
|
||||
[source,sql,indent=0,subs="+attributes,+quotes"]
|
||||
----
|
||||
INSERT INTO _<signalTable>_ (id, type, data) VALUES (_'<id>'_, _'<snapshotType>'_, '{"data-collections": ["_<tableName>_","_<tableName>_"],"type":"_<snapshotType>_","additional-condition":"_<additional-condition>_","surrogate-key":"_<surrogate-key>_"}');
|
||||
INSERT INTO _<signalTable>_ (id, type, data) VALUES (_'<id>'_, _'<snapshotType>'_, '{"data-collections": ["_<tableName>_","_<tableName>_"],"type":"_<snapshotType>_","additional-conditions":[{"data-collection": "_<tableName>_", "filter": "_<additional-condition>_"}],"surrogate-key":"_<surrogate-key>_"}');
|
||||
----
|
||||
+
|
||||
For example,
|
||||
@ -60,7 +60,7 @@ values ('ad-hoc-1', // <2>
|
||||
'execute-snapshot', // <3>
|
||||
'{"data-collections": ["schema1.table1", "schema2.table2"], // <4>
|
||||
"type":"incremental", // <5>
|
||||
"additional-condition":"color='blue'", // <6>
|
||||
"additional-conditions":[{"data-collection": "schema1.products", "filter": "color='blue'"}], // <6>
|
||||
"surrogate-key":"tenant-id"}'); // <7>
|
||||
----
|
||||
end::sql-based-snapshot[]
|
||||
@ -125,19 +125,20 @@ Currently, the only valid option is the default value, `incremental`. +
|
||||
If you do not specify a value, the connector runs an incremental snapshot.
|
||||
|
||||
|6
|
||||
|`additional-condition`
|
||||
| An optional string, which specifies a condition based on the column(s) of the {data-collection}(s), to capture a
|
||||
subset of the contents of the {data-collection}s.
|
||||
For more information about the `additional-condition` parameter, see xref:{context}-incremental-snapshots-additional-condition[].
|
||||
|`additional-conditions`
|
||||
| An optional array of additional condition that specifies a condition that the connector evaluates to designate a subset of records to include in a snapshot. +
|
||||
Each additional condition is an object with `data-collection` and `filter` properties. You can now specify different filters for different data collection. +
|
||||
* The `data-collection` property is the fully-qualified name of the data collection for which the filter will be applied.
|
||||
For more information about the `additional-conditions` parameter, see xref:{context}-incremental-snapshots-additional-conditions[].
|
||||
|7
|
||||
|`surrogate-key`
|
||||
|An optional string that specifies the column name that the connector uses as the primary key for snapshots.
|
||||
|===
|
||||
|
||||
[id="{context}-incremental-snapshots-additional-condition"]
|
||||
.Ad hoc incremental snapshots with `additional-condition`
|
||||
[id="{context}-incremental-snapshots-additional-conditions"]
|
||||
.Ad hoc incremental snapshots with `additional-conditions`
|
||||
|
||||
If you want a snapshot to include only a subset of the content in a {data-collection}, you can modify the signal request by appending an `additional-condition` parameter to the snapshot signal.
|
||||
If you want a snapshot to include only a subset of the content in a {data-collection}, you can modify the signal request by appending an `additional-conditions` parameter to the snapshot signal.
|
||||
|
||||
The SQL query for a typical snapshot takes the following form:
|
||||
|
||||
@ -146,17 +147,17 @@ The SQL query for a typical snapshot takes the following form:
|
||||
SELECT * FROM _<tableName>_ ....
|
||||
----
|
||||
|
||||
By adding an `additional-condition` parameter, you append a `WHERE` condition to the SQL query, as in the following example:
|
||||
By adding an `additional-conditions` parameter, you append a `WHERE` condition to the SQL query, as in the following example:
|
||||
|
||||
[source,sql,subs="+attributes,+quotes"]
|
||||
----
|
||||
SELECT * FROM _<tableName>_ WHERE _<additional-condition>_ ....
|
||||
SELECT * FROM _<data-collection>_ WHERE _<filter>_ ....
|
||||
----
|
||||
|
||||
The following example shows a SQL query to send an ad hoc incremental snapshot request with an additional condition to the signaling {data-collection}:
|
||||
[source,sql,indent=0,subs="+attributes,+quotes"]
|
||||
----
|
||||
INSERT INTO _<signalTable>_ (id, type, data) VALUES (_'<id>'_, _'<snapshotType>'_, '{"data-collections": ["_<tableName>_","_<tableName>_"],"type":"_<snapshotType>_","additional-condition":"_<additional-condition>_"}');
|
||||
INSERT INTO _<signalTable>_ (id, type, data) VALUES (_'<id>'_, _'<snapshotType>'_, '{"data-collections": ["_<tableName>_","_<tableName>_"],"type":"_<snapshotType>_","additional-conditions":[{"data-collection": "_<tableName>_", "filter": "_<additional-condition>_"}]}');
|
||||
----
|
||||
|
||||
For example, suppose you have a `products` {data-collection} that contains the following columns:
|
||||
@ -169,15 +170,15 @@ If you want an incremental snapshot of the `products` {data-collection} to inclu
|
||||
|
||||
[source,sql,indent=0,subs="+attributes"]
|
||||
----
|
||||
INSERT INTO myschema.debezium_signal (id, type, data) VALUES('ad-hoc-1', 'execute-snapshot', '{"data-collections": ["schema1.products"],"type":"incremental", "additional-condition":"color='blue'"}');
|
||||
INSERT INTO myschema.debezium_signal (id, type, data) VALUES('ad-hoc-1', 'execute-snapshot', '{"data-collections": ["schema1.products"],"type":"incremental", [{"data-collection": "schema1.products", "filter": "color='blue'"}]}');
|
||||
----
|
||||
|
||||
The `additional-condition` parameter also enables you to pass conditions that are based on more than on column.
|
||||
The `additional-conditions` parameter also enables you to pass conditions that are based on more than on column.
|
||||
For example, using the `products` {data-collection} from the previous example, you can submit a query that triggers an incremental snapshot that includes the data of only those items for which `color='blue'` and `quantity>10`:
|
||||
|
||||
[source,sql,indent=0,subs="+attributes"]
|
||||
----
|
||||
INSERT INTO myschema.debezium_signal (id, type, data) VALUES('ad-hoc-1', 'execute-snapshot', '{"data-collections": ["schema1.products"],"type":"incremental", "additional-condition":"color='blue' AND quantity>10"}');
|
||||
INSERT INTO myschema.debezium_signal (id, type, data) VALUES('ad-hoc-1', 'execute-snapshot', '{"data-collections": ["schema1.products"],"type":"incremental", [{"data-collection": "schema1.products", "filter": "color='blue' AND quantity>10"}]}');
|
||||
----
|
||||
|
||||
end::sql-based-snapshot[]
|
||||
|
||||
Loading…
Reference in New Issue
Block a user