alexey.rassokhin/tet123
1
0
Fork 0
Code Issues Pull Requests 1 Packages Projects Releases Wiki Activity

DBZ-1933 Add anchors to connector properties

Browse Source
This commit is contained in:
Chris Cranford 2020-04-28 11:13:57 -04:00 committed by Gunnar Morling
parent 8093582487
commit 5eb29fb228
7 changed files with 288 additions and 285 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

54
documentation/modules/ROOT/pages/connectors/cassandra.adoc
View File

@ -895,113 +895,113 @@ Cassandra connector has built-in support for JMX metrics. The Cassandra driver a
|Default
|Description
|`snapshot.mode`
|[[connector-property-snapshot-mode]]<<connector-property-snapshot-mode, `snapshot.mode`>>
|`INITIAL`
|Specifies the criteria for running a snapshot (eg. initial sync) upon startup of the cassandra connector agent. Must be one of 'INITIAL', 'ALWAYS', or 'NEVER'. The default snapshot mode is 'INITIAL'.
|`snapshot.consistency`
|[[connector-property-snapshot-consistency]]<<connector-property-snapshot-consistency, `snapshot.consistency`>>
|`ALL`
|Specify the {@link ConsistencyLevel} used for the snapshot query.
|`http.port`
|[[connector-property-http-port]]<<connector-property-http-port, `http.port`>>
|`8000`
|The port used by the HTTP server for ping, health check, and build info
|`cassandra.config`
|[[connector-property-cassandra-config]]<<connector-property-cassandra-config, `cassandra.config`>>
|
|The absolute path of the YAML config file used by a Cassandra node.
|`cassandra.hosts`
|[[connector-property-cassandra-hosts]]<<connector-property-cassandra-hosts, `cassandra.hosts`>>
|`localhost`
|One or more addresses of Cassandra nodes that driver uses to discover topology, separated by ","
|`cassandra.port`
|[[connector-property-cassandra-port]]<<connector-property-cassandra-port, `cassandra.port`>>
|`9042`
|The port used to connect to Cassandra host(s).
|`cassandra.username`
|[[connector-property-cassandra-username]]<<connector-property-cassandra-username, `cassandra.username`>>
|
|The username used when connecting to Cassandra hosts.
|`cassandra.password`
|[[connector-property-cassandra-password]]<<connector-property-cassandra-password, `cassandra.password`>>
|
|The password used when connecting to Cassandra hosts.
|`cassandra.ssl.enabled`
|[[connector-property-cassandra-ssl-enabled]]<<connector-property-cassandra-ssl-enabled, `cassandra.ssl.enabled`>>
|`false`
|If set to true, Cassandra connector agent will use SSL to connect to Cassandra node.
|`cassandra.ssl.config.path`
|[[connector-property-cassandra-ssl-config-path]]<<connector-property-cassandra-ssl-config-path, `cassandra.ssl.config.path`>>
|
|The SSL config file path required for storage node.
|`commit.log.relocation.dir`
|[[connector-property-commit-log-relocation-dir]]<<connector-property-commit-log-relocation-dir, `commit.log.relocation.dir`>>
|
|The local directory which commit logs get relocated to once processed.
|`commit.log.post.processing.enabled`
|[[connector-property-commit-log-post-processing-enabled]]<<connector-property-commit-log-post-processing-enabled, `commit.log.post.processing.enabled`>>
|`true`
|Determines whether or not the CommitLogPostProcessor should run. If disabled, commit logs would not be deleted post-process.
|`commit.log.transfer.class`
|[[connector-property-commit-log-transfer-class]]<<connector-property-commit-log-transfer-class, `commit.log.transfer.class`>>
|`io.debezium.connector.cassandra.BlackHoleCommitLogTransfer`
|The fully qualified {@link CommitLogTransfer} class used to transfer commit logs. Built-in transfers are `BlackHoleCommitLogTransfer` and `GCSCommitLogTransfer`.
|`remote.commit.log.relocation.dir`
|[[connector-property-remote-commit-log-relocation-dir]]<<connector-property-remote-commit-log-relocation-dir, `remote.commit.log.relocation.dir`>>
|
|The remote base directory to transfer commit log to. Required to be specified if GCSCommitLogTransfer is used as COMMIT_LOG_TRANSFER_CLASS. For example: "gcs://bucket-name/prefix-name".
|`storage.credential.key.file`
|[[connector-property-storage-credential-key-file]]<<connector-property-storage-credential-key-file, `storage.credential.key.file`>>
|
|The path of the storage key file used for setting up credential to access remote storage service. If {@link GCSCommitLogTransfer} is used and this config is not provided, then Application Default Credential would be used.
|`offset.backing.store.dir`
|[[connector-property-offset-backing-store-dir]]<<connector-property-offset-backing-store-dir, `offset.backing.store.dir`>>
|
|The directory to store offset tracking files.
|`offset.flush.interval.ms`
|[[connector-property-offset-flush-interval-ms]]<<connector-property-offset-flush-interval-ms, `offset.flush.interval.ms`>>
|`0`
|The minimum amount of time to wait before committing the offset. The default value of 0 implies
the offset will be flushed every time.
|`max.offset.flush.size`
|[[connector-property-max-offset-flush-size]]<<connector-property-max-offset-flush-size, `max.offset.flush.size`>>
|`100`
|The maximum records that are allowed to be processed until it is required to flush offset to disk. This config is effective only if offset_flush_interval_ms != 0.
|`max.queue.size`
|[[connector-property-max-queue-size]]<<connector-property-max-queue-size, `max.queue.size`>>
|`8192`
|Positive integer value that specifies the maximum size of the blocking queue into which change events read from the commit log are placed before they are written to Kafka. This queue can provide back pressure to the commit log reader when, for example, writes to Kafka are slower or if Kafka is not available. Events that appear in the queue are not included in the offsets periodically recorded by this connector. Defaults to 8192, and should always be larger than the maximum batch size specified in the max.batch.size property. The capacity of the queue to hold deserialized records before they are converted to Kafka Connect structs and emitted to Kafka.
|`max.batch.size`
|[[connector-property-max-batch-size]]<<connector-property-max-batch-size, `max.batch.size`>>
|`2048`
|The maximum number of change events to dequeue each time.
|`poll.interval.ms`
|[[connector-property-poll-interval-ms]]<<connector-property-poll-interval-ms, `poll.interval.ms`>>
|`1000`
|Positive integer value that specifies the number of milliseconds the commit log processor should wait during each iteration for new change events to appear in the queue. Defaults to 1000 milliseconds, or 1 second.
|`schema.refresh.interval.ms`
|[[connector-property-schema-refresh-interval-ms]]<<connector-property-schema-refresh-interval-ms, `schema.refresh.interval.ms`>>
|`10000`
|Positive integer value that specifies the number of milliseconds the schema processor should wait before
refreshing the cached Cassandra table schemas.
|`cdc.dir.poll.interval.ms`
|[[connector-property-cdc-dir-poll-interval-ms]]<<connector-property-cdc-dir-poll-interval-ms, `cdc.dir.poll.interval.ms`>>
|`10000`
|The maximum amount of time to wait on each poll before reattempt.
|`snapshot.scan.interval.ms`
|[[connector-property-snapshot-scan-interval-ms]]<<connector-property-snapshot-scan-interval-ms, `snapshot.scan.interval.ms`>>
|`10000`
|Positive integer value that specifies the number of milliseconds the snapshot processor should wait before re-scanning tables to look for new cdc-enabled tables. Defaults to 10000 milliseconds, or 10 seconds.
|`tombstones.on.delete`
|[[connector-property-tombstones-on-delete]]<<connector-property-tombstones-on-delete, `tombstones.on.delete`>>
|`false`
|Whether deletion events should have a subsequent tombstone event (true) or not (false). It's important to note that in Cassandra, two events with the same key may be updating different columns of a given table. So this could potentially result in records being lost during compaction if they haven't been consumed by the consumer yet. In other words, do NOT set this to true if you have kafka compaction turned on.
|`field.blacklist`
|[[connector-property-field-blacklist]]<<connector-property-field-blacklist, `field.blacklist`>>
|
|A comma-separated list of fully-qualified names of fields that should be excluded from change event message values. Fully-qualified names for fields are in the form keyspace_name>.<field_name>.<nested_field_name>.
|`sanitize.field.names`
|[[connector-property-sanitize-field-names]]<<connector-property-sanitize-field-names, `sanitize.field.names`>>
|`true` when connector configuration explicitly specifies the `key.converter` or `value.converter` parameters to use Avro, otherwise defaults to `false`.
|Whether field names will be sanitized to adhere to Avro naming requirements.
See xref:configuration/avro.adoc#names[Avro naming] for more details.

72
documentation/modules/ROOT/pages/connectors/db2.adoc
View File

@ -1198,66 +1198,66 @@ The following configuration properties are _required_ unless a default value is
|=======================
|Property |Default |Description
|`name`
|[[connector-property-name]]<<connector-property-name, `name`>>
|
|Unique name for the connector. Attempting to register again with the same name will fail. (This property is required by all Kafka Connect connectors.)
|`connector.class`
|[[connector-property-connector-class]]<<connector-property-connector-class, `connector.class`>>
|
|The name of the Java class for the connector. Always use a value of `io.debezium.connector.db2.Db2Connector` for the Db2 connector.
|`tasks.max`
|[[connector-property-tasks-max]]<<connector-property-tasks-max, `tasks.max`>>
|`1`
|The maximum number of tasks that should be created for this connector. The Db2 connector always uses a single task and therefore does not use this value, so the default is always acceptable.
|`database.hostname`
|[[connector-property-database-hostname]]<<connector-property-database-hostname, `database.hostname`>>
|
|IP address or hostname of the Db2 database server.
|`database.port`
|[[connector-property-database-port]]<<connector-property-database-port, `database.port`>>
|`50000`
|Integer port number of the Db2 database server.
|`database.user`
|[[connector-property-database-user]]<<connector-property-database-user, `database.user`>>
|
|Username to use when connecting to the Db2 database server.
|`database.password`
|[[connector-property-database-password]]<<connector-property-database-password, `database.password`>>
|
|Password to use when connecting to the Db2 database server.
|`database.dbname`
|[[connector-property-database-dbname]]<<connector-property-database-dbname, `database.dbname`>>
|
|The name of the Db2 database from which to stream the changes
|`database.server.name`
|[[connector-property-database-server-name]]<<connector-property-database-server-name, `database.server.name`>>
|
|Logical name that identifies and provides a namespace for the particular Db2 database server being monitored. The logical name should be unique across all other connectors, since it is used as a prefix for all Kafka topic names emanating from this connector.
Only alphanumeric characters and underscores should be used.
|`database.history.kafka.topic`
|[[connector-property-database-history-kafka-topic]]<<connector-property-database-history-kafka-topic, `database.history.kafka.topic`>>
|
|The full name of the Kafka topic where the connector will store the database schema history.
|`database.history{zwsp}.kafka.bootstrap.servers`
|[[connector-property-database-history-kafka-bootstrap-servers]]<<connector-property-database-history-kafka-bootstrap-servers, `database.history{zwsp}.kafka.bootstrap.servers`>>
|
|A list of host/port pairs that the connector will use for establishing an initial connection to the Kafka cluster. This connection will be used for retrieving database schema history previously stored by the connector, and for writing each DDL statement read from the source database. This should point to the same Kafka cluster used by the Kafka Connect process.
|`table.whitelist`
|[[connector-property-table-whitelist]]<<connector-property-table-whitelist, `table.whitelist`>>
|
|An optional comma-separated list of regular expressions that match fully-qualified table identifiers for tables to be monitored; any table not included in the whitelist will be excluded from monitoring. Each identifier is of the form _schemaName_._tableName_. By default the connector will monitor every non-system table in each monitored schema. May not be used with `table.blacklist`.
|`table.blacklist`
|[[connector-property-table-blacklist]]<<connector-property-table-blacklist, `table.blacklist`>>
|
|An optional comma-separated list of regular expressions that match fully-qualified table identifiers for tables to be excluded from monitoring; any table not included in the blacklist will be monitored. Each identifier is of the form _schemaName_._tableName_. May not be used with `table.whitelist`.
|`column.blacklist`
|[[connector-property-column-blacklist]]<<connector-property-column-blacklist, `column.blacklist`>>
|_empty string_
|An optional comma-separated list of regular expressions that match the fully-qualified names of columns that should be excluded from change event message values.
Fully-qualified names for columns are of the form _schemaName_._tableName_._columnName_.
Note that primary key columns are always included in the event's key, also if blacklisted from the value.
|`column.mask.hash._hashAlgorithm_.with.salt._salt_`
|[[connector-property-column-mask-hash]]<<connector-property-column-mask-hash, `column.mask.hash._hashAlgorithm_.with.salt._salt_`>>
|_n/a_
|An optional comma-separated list of regular expressions that match the fully-qualified names of character-based columns whose values should be pseudonyms in the change event message values with a field value consisting of the hashed value using the algorithm `_hashAlgorithm_` and salt `_salt_`.
Based on the used hash function referential integrity is kept while data is pseudonymized. Supported hash functions are described in the {link-java7-standard-names}[MessageDigest section] of the Java Cryptography Architecture Standard Algorithm Name Documentation.
@ -1273,32 +1273,32 @@ where `CzQMA0cB5K` is a randomly selected salt.
Note: Depending on the `_hashAlgorithm_` used, the `_salt_` selected and the actual data set, the resulting masked data set may not be completely anonymized.
|`time.precision.mode`
|[[connector-property-time-precision-mode]]<<connector-property-time-precision-mode, `time.precision.mode`>>
|`adaptive`
| Time, date, and timestamps can be represented with different kinds of precision, including: `adaptive` (the default) captures the time and timestamp values exactly as in the database using either millisecond, microsecond, or nanosecond precision values based on the database column's type; or `connect` always represents time and timestamp values using Kafka Connect's built-in representations for Time, Date, and Timestamp, which uses millisecond precision regardless of the database columns' precision. See link:#temporal-values[temporal values].
|`tombstones.on.delete`
|[[connector-property-tombstones-on-delete]]<<connector-property-tombstones-on-delete, `tombstones.on.delete`>>
|`true`
| Controls whether a tombstone event should be generated after a delete event. +
When `true` the delete operations are represented by a delete event and a subsequent tombstone event. When `false` only a delete event is sent. +
Emitting the tombstone event (the default behavior) allows Kafka to completely delete all events pertaining to the given key once the source record got deleted.
|`column.truncate.to._length_.chars`
|[[connector-property-column-truncate-to-length-chars]]<<connector-property-column-truncate-to-length-chars, `column.truncate.to._length_.chars`>>
|_n/a_
|An optional comma-separated list of regular expressions that match the fully-qualified names of character-based columns whose values should be truncated in the change event message values if the field values are longer than the specified number of characters. Multiple properties with different lengths can be used in a single configuration, although in each the length must be a positive integer. Fully-qualified names for columns are of the form _databaseName_._tableName_._columnName_, or _databaseName_._schemaName_._tableName_._columnName_.
|`column.mask.with._length_.chars`
|[[connector-property-column-mask-with-length-chars]]<<connector-property-column-mask-with-length-chars, `column.mask.with._length_.chars`>>
|_n/a_
|An optional comma-separated list of regular expressions that match the fully-qualified names of character-based columns whose values should be replaced in the change event message values with a field value consisting of the specified number of asterisk (`*`) characters. Multiple properties with different lengths can be used in a single configuration, although in each the length must be a positive integer or zero. Fully-qualified names for columns are of the form _databaseName_._tableName_._columnName_, or _databaseName_._schemaName_._tableName_._columnName_.
|`column.propagate.source.type`
|[[connector-property-column-propagate-source-type]]<<connector-property-column-propagate-source-type, `column.propagate.source.type`>>
|_n/a_
|An optional comma-separated list of regular expressions that match the fully-qualified names of columns whose original type and length should be added as a parameter to the corresponding field schemas in the emitted change messages.
The schema parameters `pass:[_]pass:[_]debezium.source.column.type`, `pass:[_]pass:[_]debezium.source.column.length` and `pass:[_]pass:[_]debezium.source.column.scale` will be used to propagate the original type name and length (for variable-width types), respectively.
Useful to properly size corresponding columns in sink databases.
Fully-qualified names for columns are of the form _schemaName_._tableName_._columnName_.
|`datatype.propagate.source.type`
|[[connector-property-datatype-propagate-source-type]]<<connector-property-datatype-propagate-source-type, `datatype.propagate.source.type`>>
|_n/a_
|An optional comma-separated list of regular expressions that match the database-specific data type name of columns whose original type and length should be added as a parameter to the corresponding field schemas in the emitted change messages.
The schema parameters `pass:[_]pass:[_]debezium.source.column.type`, `pass:[_]pass:[_]debezium.source.column.length` and `pass:[_]pass:[_]debezium.source.column.scale` will be used to propagate the original type name and length (for variable-width types), respectively.
@ -1306,7 +1306,7 @@ Useful to properly size corresponding columns in sink databases.
Fully-qualified data type names are of the form _schemaName_._tableName_._typeName_.
See xref:data-types[] for the list of Db2-specific data type names.
|`message.key.columns`
|[[connector-property-message-key-columns]]<<connector-property-message-key-columns, `message.key.columns`>>
|_empty string_
| A semi-colon list of regular expressions that match fully-qualified tables and columns to map a primary key. +
Each item (regular expression) must match the fully-qualified `<fully-qualified table>:<a comma-separated list of columns>` representing the custom key. +
@ -1319,11 +1319,11 @@ The following _advanced_ configuration properties have good defaults that will w
|=======================
|Property |Default |Description
|`snapshot.mode`
|[[connector-property-snapshot-mode]]<<connector-property-snapshot-mode, `snapshot.mode`>>
|_initial_
|A mode for taking an initial snapshot of the structure and optionally data of captured tables. Supported values are _initial_ (will take a snapshot of structure and data of captured tables; useful if topics should be populated with a complete representation of the data from the captured tables) and _schema_only_ (will take a snapshot of the structure of captured tables only; useful if only changes happening from now onwards should be propagated to topics). Once the snapshot is complete, the connector will continue reading change events from the database's redo logs.
|`snapshot.isolation.mode`
|[[connector-property-snapshot-isolation-mode]]<<connector-property-snapshot-isolation-mode, `snapshot.isolation.mode`>>
|_repeatable_read_
|Mode to control which transaction isolation level is used and how long the connector locks the monitored tables.
There are four possible values: `read_uncommitted`, `read_committed`, `repeatable_read`, and `exclusive` (
@ -1340,26 +1340,26 @@ twice - once in initial snapshot and once in streaming phase. Nonetheless, that
data mirroring.
For `read_uncommitted` there are no data consistency guarantees at all (some data might be lost or corrupted).
|`event.processing{zwsp}.failure.handling.mode`
|[[connector-property-event-processing-failure-handling-mode]]<<connector-property-event-processing-failure-handling-mode, `event.processing{zwsp}.failure.handling.mode`>>
|`fail`
| Specifies how the connector should react to exceptions during processing of events.
`fail` will propagate the exception (indicating the offset of the problematic event), causing the connector to stop. +
`warn` will cause the problematic event to be skipped and the offset of the problematic event to be logged. +
`skip` will cause the problematic event to be skipped.
|`poll.interval.ms`
|[[connector-property-poll-interval-ms]]<<connector-property-poll-interval-ms, `poll.interval.ms`>>
|`1000`
|Positive integer value that specifies the number of milliseconds the connector should wait during each iteration for new change events to appear. Defaults to 1000 milliseconds, or 1 second.
|`max.queue.size`
|[[connector-property-max-queue-size]]<<connector-property-max-queue-size, `max.queue.size`>>
|`8192`
|Positive integer value that specifies the maximum size of the blocking queue into which change events read from the database log are placed before they are written to Kafka. This queue can provide backpressure to the CDC table reader when, for example, writes to Kafka are slower or if Kafka is not available. Events that appear in the queue are not included in the offsets periodically recorded by this connector. Defaults to 8192, and should always be larger than the maximum batch size specified in the `max.batch.size` property.
|`max.batch.size`
|[[connector-property-max-batch-size]]<<connector-property-max-batch-size, `max.batch.size`>>
|`2048`
|Positive integer value that specifies the maximum size of each batch of events that should be processed during each iteration of this connector. Defaults to 2048.
|`heartbeat.interval.ms`
|[[connector-property-heartbeat-interval-ms]]<<connector-property-heartbeat-interval-ms, `heartbeat.interval.ms`>>
|`0`
|Controls how frequently heartbeat messages are sent. +
This property contains an interval in milli-seconds that defines how frequently the connector sends messages into a heartbeat topic.
@ -1371,38 +1371,38 @@ This may result in more change events to be re-sent after a connector restart.
Set this parameter to `0` to not send heartbeat messages at all. +
Disabled by default.
|`heartbeat.topics.prefix`
|[[connector-property-heartbeat-topics-prefix]]<<connector-property-heartbeat-topics-prefix, `heartbeat.topics.prefix`>>
|`__debezium-heartbeat`
|Controls the naming of the topic to which heartbeat messages are sent. +
The topic is named according to the pattern `<heartbeat.topics.prefix>.<server.name>`.
|`snapshot.delay.ms`
|[[connector-property-snapshot-delay-ms]]<<connector-property-snapshot-delay-ms, `snapshot.delay.ms`>>
|
|An interval in milli-seconds that the connector should wait before taking a snapshot after starting up; +
Can be used to avoid snapshot interruptions when starting multiple connectors in a cluster, which may cause re-balancing of connectors.
|`snapshot.fetch.size`
|[[connector-property-snapshot-fetch-size]]<<connector-property-snapshot-fetch-size, `snapshot.fetch.size`>>
|`2000`
|Specifies the maximum number of rows that should be read in one go from each table while taking a snapshot.
The connector will read the table contents in multiple batches of this size. Defaults to 2000.
|`snapshot.lock.timeout.ms`
|[[connector-property-snapshot-lock-timeout-ms]]<<connector-property-snapshot-lock-timeout-ms, `snapshot.lock.timeout.ms`>>
|`10000`
|An integer value that specifies the maximum amount of time (in milliseconds) to wait to obtain table locks when performing a snapshot. If table locks cannot be acquired in this time interval, the snapshot will fail (also see link:#snapshots[snapshots]). +
When set to `0` the connector will fail immediately when it cannot obtain the lock. Value `-1` indicates infinite waiting.
|`snapshot.select.statement.overrides`
|[[connector-property-snapshot-select-statement-overrides]]<<connector-property-snapshot-select-statement-overrides, `snapshot.select.statement.overrides`>>
|
|Controls which rows from tables will be included in snapshot. +
This property contains a comma-separated list of fully-qualified tables _(SCHEMA_NAME.TABLE_NAME)_. Select statements for the individual tables are specified in further configuration properties, one for each table, identified by the id `snapshot.select.statement.overrides.[SCHEMA_NAME].[TABLE_NAME]`. The value of those properties is the SELECT statement to use when retrieving data from the specific table during snapshotting. _A possible use case for large append-only tables is setting a specific point where to start (resume) snapshotting, in case a previous snapshotting was interrupted._ +
*Note*: This setting has impact on snapshots only. Events captured during log reading are not affected by it.
|`sanitize.field.names`
|[[connector-property-sanitize-field-names]]<<connector-property-sanitize-field-names, `sanitize.field.names`>>
|`true` when connector configuration explicitly specifies the `key.converter` or `value.converter` parameters to use Avro, otherwise defaults to `false`.
|Whether field names will be sanitized to adhere to Avro naming requirements.
See xref:configuration/avro.adoc#names[Avro naming] for more details.
|`provide.transaction.metadata` (Incubating)
|[[connector-property-provide-transaction-metadata]]<<connector-property-provide-transaction-metadata, `provide.transaction.metadata` (Incubating)>>
|`false`
|When set to `true` Debezium generates events with transaction boundaries and enriches data events envelope with transaction metadata.

66
documentation/modules/ROOT/pages/connectors/mongodb.adoc
View File

@ -709,92 +709,92 @@ The following configuration properties are _required_ unless a default value is
|=======================
|Property |Default |Description
|`name`
|[[connector-property-name]]<<connector-property-name, `name`>>
|
|Unique name for the connector. Attempting to register again with the same name will fail. (This property is required by all Kafka Connect connectors.)
|`connector.class`
|[[connector-property-connector-class]]<<connector-property-connector-class, `connector.class`>>
|
|The name of the Java class for the connector. Always use a value of `io.debezium.connector.mongodb.MongoDbConnector` for the MongoDB connector.
|`mongodb.hosts`
|[[connector-property-mongodb-hosts]]<<connector-property-mongodb-hosts, `mongodb.hosts`>>
|
|The comma-separated list of hostname and port pairs (in the form 'host' or 'host:port') of the MongoDB servers in the replica set. The list can contain a single hostname and port pair. If `mongodb.members.auto.discover` is set to `false`, then the host and port pair should be prefixed with the replica set name (e.g., `rs0/localhost:27017`).
|`mongodb.name`
|[[connector-property-mongodb-name]]<<connector-property-mongodb-name, `mongodb.name`>>
|
|A unique name that identifies the connector and/or MongoDB replica set or sharded cluster that this connector monitors. Each server should be monitored by at most one {prodname} connector, since this server name prefixes all persisted Kafka topics emanating from the MongoDB replica set or cluster.
Only alphanumeric characters and underscores should be used.
|`mongodb.user`
|[[connector-property-mongodb-user]]<<connector-property-mongodb-user, `mongodb.user`>>
|
|Name of the database user to be used when connecting to MongoDB. This is required only when MongoDB is configured to use authentication.
|`mongodb.password`
|[[connector-property-mongodb-password]]<<connector-property-mongodb-password, `mongodb.password`>>
|
|Password to be used when connecting to MongoDB. This is required only when MongoDB is configured to use authentication.
|`mongodb.authsource`
|[[connector-property-mongodb-authsource]]<<connector-property-mongodb-authsource, `mongodb.authsource`>>
|`admin`
|Database (authentication source) containing MongoDB credentials. This is required only when MongoDB is configured to use authentication with another authentication database than `admin`.
|`mongodb.ssl.enabled`
|[[connector-property-mongodb-ssl-enabled]]<<connector-property-mongodb-ssl-enabled, `mongodb.ssl.enabled`>>
|`false`
|Connector will use SSL to connect to MongoDB instances.
|`mongodb.ssl.invalid.hostname.allowed`
|[[connector-property-mongodb-ssl-invalid-hostname-allowed]]<<connector-property-mongodb-ssl-invalid-hostname-allowed, `mongodb.ssl.invalid.hostname.allowed`>>
|`false`
|When SSL is enabled this setting controls whether strict hostname checking is disabled during connection phase. If `true` the connection will not prevent man-in-the-middle attacks.
|`database.whitelist`
|[[connector-property-database-whitelist]]<<connector-property-database-whitelist, `database.whitelist`>>
|_empty string_
|An optional comma-separated list of regular expressions that match database names to be monitored; any database name not included in the whitelist is excluded from monitoring. By default all databases is monitored.
May not be used with `database.blacklist`.
|`database.blacklist`
|[[connector-property-database-blacklist]]<<connector-property-database-blacklist, `database.blacklist`>>
|_empty string_
|An optional comma-separated list of regular expressions that match database names to be excluded from monitoring; any database name not included in the blacklist is monitored. May not be used with `database.whitelist`.
|`collection.whitelist`
|[[connector-property-collection-whitelist]]<<connector-property-collection-whitelist, `collection.whitelist`>>
|_empty string_
|An optional comma-separated list of regular expressions that match fully-qualified namespaces for MongoDB collections to be monitored; any collection not included in the whitelist is excluded from monitoring. Each identifier is of the form _databaseName_._collectionName_. By default the connector will monitor all collections except those in the `local` and `admin` databases. May not be used with `collection.blacklist`.
|`collection.blacklist`
|[[connector-property-collection-blacklist]]<<connector-property-collection-blacklist, `collection.blacklist`>>
|_empty string_
|An optional comma-separated list of regular expressions that match fully-qualified namespaces for MongoDB collections to be excluded from monitoring; any collection not included in the blacklist is monitored. Each identifier is of the form _databaseName_._collectionName_. May not be used with `collection.whitelist`.
|`snapshot.mode`
|[[connector-property-snapshot-mode]]<<connector-property-snapshot-mode, `snapshot.mode`>>
|`initial`
|Specifies the criteria for running a snapshot upon startup of the connector. The default is *initial*, and specifies the connector reads a snapshot when either no offset is found or if the oplog no longer contains the previous offset. The *never* option specifies that the connector should never use snapshots, instead the connector should proceed to tail the log.
|`field.blacklist`
|[[connector-property-field-blacklist]]<<connector-property-field-blacklist, `field.blacklist`>>
|_empty string_
|An optional comma-separated list of the fully-qualified names of fields that should be excluded from change event message values. Fully-qualified names for fields are of the form _databaseName_._collectionName_._fieldName_._nestedFieldName_, where _databaseName_ and _collectionName_ may contain the wildcard (*) which matches any characters.
|`field.renames`
|[[connector-property-field-renames]]<<connector-property-field-renames, `field.renames`>>
|_empty string_
|An optional comma-separated list of the fully-qualified replacements of fields that should be used to rename fields in change event message values. Fully-qualified replacements for fields are of the form _databaseName_._collectionName_._fieldName_._nestedFieldName_:__newNestedFieldName__, where _databaseName_ and _collectionName_ may contain the wildcard (*) which matches any characters, the colon character (:) is used to determine rename mapping of field. The next field replacement is applied to the result of the previous field replacement in the list, so keep this in mind when renaming multiple fields that are in the same path.
|`tasks.max`
|[[connector-property-tasks-max]]<<connector-property-tasks-max, `tasks.max`>>
|`1`
|The maximum number of tasks that should be created for this connector. The MongoDB connector will attempt to use a separate task for each replica set, so the default is acceptable when using the connector with a single MongoDB replica set. When using the connector with a MongoDB sharded cluster, we recommend specifying a value that is equal to or more than the number of shards in the cluster, so that the work for each replica set can be distributed by Kafka Connect.
|`initial.sync.max.threads`
|[[connector-property-initial-sync-max-threads]]<<connector-property-initial-sync-max-threads, `initial.sync.max.threads`>>
|`1`
|Positive integer value that specifies the maximum number of threads used to perform an intial sync of the collections in a replica set. Defaults to 1.
|`tombstones.on.delete`
|[[connector-property-tombstones-on-delete]]<<connector-property-tombstones-on-delete, `tombstones.on.delete`>>
|`true`
| Controls whether a tombstone event should be generated after a delete event. +
When `true` the delete operations are represented by a delete event and a subsequent tombstone event. When `false` only a delete event is sent. +
Emitting the tombstone event (the default behavior) allows Kafka to completely delete all events pertaining to the given key once the source record got deleted.
|`snapshot.delay.ms`
|[[connector-property-snapshot-delay-ms]]<<connector-property-snapshot-delay-ms, `snapshot.delay.ms`>>
|
|An interval in milli-seconds that the connector should wait before taking a snapshot after starting up; +
Can be used to avoid snapshot interruptions when starting multiple connectors in a cluster, which may cause re-balancing of connectors.
|`snapshot.fetch.size`
|[[connector-property-snapshot-fetch-size]]<<connector-property-snapshot-fetch-size, `snapshot.fetch.size`>>
|`0`
|Specifies the maximum number of documents that should be read in one go from each collection while taking a snapshot.
The connector will read the collection contents in multiple batches of this size. +
@ -811,36 +811,36 @@ The following _advanced_ configuration properties have good defaults that will w
|Default
|Description
|`max.queue.size`
|[[connector-property-max-queue-size]]<<connector-property-max-queue-size, `max.queue.size`>>
|`8192`
|Positive integer value that specifies the maximum size of the blocking queue into which change events read from the database log are placed before they are written to Kafka. This queue can provide backpressure to the oplog reader when, for example, writes to Kafka are slower or if Kafka is not available. Events that appear in the queue are not included in the offsets periodically recorded by this connector. Defaults to 8192, and should always be larger than the maximum batch size specified in the `max.batch.size` property.
|`max.batch.size`
|[[connector-property-max-batch-size]]<<connector-property-max-batch-size, `max.batch.size`>>
|`2048`
|Positive integer value that specifies the maximum size of each batch of events that should be processed during each iteration of this connector. Defaults to 2048.
|`poll.interval.ms`
|[[connector-property-poll-interval-ms]]<<connector-property-poll-interval-ms, `poll.interval.ms`>>
|`1000`
|Positive integer value that specifies the number of milliseconds the connector should wait during each iteration for new change events to appear. Defaults to 1000 milliseconds, or 1 second.
|`connect.backoff.initial.delay.ms`
|[[connector-property-connect-backoff-initial-delay-ms]]<<connector-property-connect-backoff-initial-delay-ms, `connect.backoff.initial.delay.ms`>>
|`1000`
|Positive integer value that specifies the initial delay when trying to reconnect to a primary after the first failed connection attempt or when no primary is available. Defaults to 1 second (1000 ms).
|`connect.backoff.max.delay.ms`
|[[connector-property-connect-backoff-max-delay-ms]]<<connector-property-connect-backoff-max-delay-ms, `connect.backoff.max.delay.ms`>>
|`1000`
|Positive integer value that specifies the maximum delay when trying to reconnect to a primary after repeated failed connection attempts or when no primary is available. Defaults to 120 seconds (120,000 ms).
|`connect.max.attempts`
|[[connector-property-connect-max-attempts]]<<connector-property-connect-max-attempts, `connect.max.attempts`>>
|`16`
|Positive integer value that specifies the maximum number of failed connection attempts to a replica set primary before an exception occurs and task is aborted. Defaults to 16, which with the defaults for `connect.backoff.initial.delay.ms` and `connect.backoff.max.delay.ms` results in just over 20 minutes of attempts before failing.
|`mongodb.members.auto.discover`
|[[connector-property-mongodb-members-auto-discover]]<<connector-property-mongodb-members-auto-discover, `mongodb.members.auto.discover`>>
|`true`
|Boolean value that specifies whether the addresses in 'mongodb.hosts' are seeds that should be used to discover all members of the cluster or replica set (`true`), or whether the address(es) in `mongodb.hosts` should be used as is (`false`). The default is `true` and should be used in all cases except where MongoDB is link:#mongodb-replicaset[fronted by a proxy].
ifndef::cdc-product[]
|`source.struct.version`
|[[connector-property-source-struct-version]]<<connector-property-source-struct-version, `source.struct.version`>>
|v2
|Schema version for the `source` block in CDC events. Debezium 0.10 introduced a few breaking +
changes to the structure of the `source` block in order to unify the exposed structure across
@ -849,7 +849,7 @@ By setting this option to `v1` the structure used in earlier versions can be pro
Note that this setting is not recommended and is planned for removal in a future {prodname} version.
endif::cdc-product[]
|`heartbeat.interval.ms`
|[[connector-property-heartbeat-interval-ms]]<<connector-property-heartbeat-interval-ms, `heartbeat.interval.ms`>>
|`0`
|Controls how frequently heartbeat messages are sent. +
This property contains an interval in milli-seconds that defines how frequently the connector sends messages into a heartbeat topic.
@ -862,19 +862,19 @@ This will cause the oplog files to be rotated out but connector will not notice
Set this parameter to `0` to not send heartbeat messages at all. +
Disabled by default.
|`heartbeat.topics.prefix`
|[[connector-property-heartbeat-topics-prefix]]<<connector-property-heartbeat-topics-prefix, `heartbeat.topics.prefix`>>
|`__debezium-heartbeat`
|Controls the naming of the topic to which heartbeat messages are sent. +
The topic is named according to the pattern `<heartbeat.topics.prefix>.<server.name>`.
|`sanitize.field.names`
|[[connector-property-sanitize-field-names]]<<connector-property-sanitize-field-names, `sanitize.field.names`>>
|`true` when connector configuration explicitly specifies the `key.converter` or `value.converter` parameters to use Avro, otherwise defaults to `false`.
|Whether field names are sanitized to adhere to Avro naming requirements.
ifndef::cdc-product[]
See xref:configuration/avro.adoc#names[Avro naming] for more details.
endif::cdc-product[]
|`skipped.operations`
|[[connector-property-skipped-operations]]<<connector-property-skipped-operations, `skipped.operations`>>
|
| comma-separated list of oplog operations that will be skipped during streaming.
The operations include: `i` for inserts, `u` for updates, and `d` for deletes.

68
documentation/modules/ROOT/pages/connectors/oracle.adoc
View File

@ -952,72 +952,72 @@ The following configuration properties are _required_ unless a default value is
|Default
|Description
|`name`
|[[connector-property-name]]<<connector-property-name, `name`>>
|
|Unique name for the connector. Attempting to register again with the same name will fail. (This property is required by all Kafka Connect connectors.)
|`connector.class`
|[[connector-property-connector-class]]<<connector-property-connector-class, `connector.class`>>
|
|The name of the Java class for the connector. Always use a value of `io.debezium{zwsp}.connector.oracle.OracleConnector` for the Oracle connector.
|`tasks.max`
|[[connector-property-tasks-max]]<<connector-property-tasks-max, `tasks.max`>>
|`1`
|The maximum number of tasks that should be created for this connector. The Oracle connector always uses a single task and therefore does not use this value, so the default is always acceptable.
|`database.hostname`
|[[connector-property-database-hostname]]<<connector-property-database-hostname, `database.hostname`>>
|
|IP address or hostname of the Oracle database server.
|`database.port`
|[[connector-property-database-port]]<<connector-property-database-port, `database.port`>>
|
|Integer port number of the Oracle database server.
|`database.user`
|[[connector-property-database-user]]<<connector-property-database-user, `database.user`>>
|
|Name of the user to use when connecting to the Oracle database server.
|`database.password`
|[[connector-property-database-password]]<<connector-property-database-password, `database.password`>>
|
|Password to use when connecting to the Oracle database server.
|`database.dbname`
|[[connector-property-database-dbname]]<<connector-property-database-dbname, `database.dbname`>>
|
|Name of the database to connect to. Must be the CDB name when working with the CDB + PDB model.
|`database.pdb.name`
|[[connector-property-database-pdb-name]]<<connector-property-database-pdb-name, `database.pdb.name`>>
|
|Name of the PDB to connect to, when working with the CDB + PDB model.
|`database.out.server.name`
|[[connector-property-database-out-server-name]]<<connector-property-database-out-server-name, `database.out.server.name`>>
|
|Name of the XStream outbound server configured in the database.
|`database.server.name`
|[[connector-property-database-server-name]]<<connector-property-database-server-name, `database.server.name`>>
|
|Logical name that identifies and provides a namespace for the particular Oracle database server being monitored. The logical name should be unique across all other connectors, since it is used as a prefix for all Kafka topic names emanating from this connector.
Only alphanumeric characters and underscores should be used.
|`database.history.kafka.topic`
|[[connector-property-database-history-kafka-topic]]<<connector-property-database-history-kafka-topic, `database.history.kafka.topic`>>
|
|The full name of the Kafka topic where the connector will store the database schema history.
|`database.history{zwsp}.kafka.bootstrap.servers`
|[[connector-property-database-history-kafka-bootstrap-servers]]<<connector-property-database-history-kafka-bootstrap-servers, `database.history{zwsp}.kafka.bootstrap.servers`>>
|
|A list of host/port pairs that the connector will use for establishing an initial connection to the Kafka cluster. This connection will be used for retrieving database schema history previously stored by the connector, and for writing each DDL statement read from the source database. This should point to the same Kafka cluster used by the Kafka Connect process.
|`snapshot.mode`
|[[connector-property-snapshot-mode]]<<connector-property-snapshot-mode, `snapshot.mode`>>
|_initial_
|A mode for taking an initial snapshot of the structure and optionally data of captured tables. Supported values are _initial_ (will take a snapshot of structure and data of captured tables; useful if topics should be populated with a complete representation of the data from the captured tables) and _schema_only_ (will take a snapshot of the structure of captured tables only; useful if only changes happening from now onwards should be propagated to topics). Once the snapshot is complete, the connector will continue reading change events from the database's redo logs.
|`table.whitelist`
|[[connector-property-table-whitelist]]<<connector-property-table-whitelist, `table.whitelist`>>
|_empty string_
|An optional comma-separated list of regular expressions that match fully-qualified table identifiers for tables to be monitored; any table not included in the whitelist will be excluded from monitoring. Each identifier is of the form _databaseName_._tableName_. By default the connector will monitor every non-system table in each monitored database. May not be used with `table.blacklist`.
|`table.blacklist`
|[[connector-property-table-blacklist]]<<connector-property-table-blacklist, `table.blacklist`>>
|_empty string_
|An optional comma-separated list of regular expressions that match fully-qualified table identifiers for tables to be excluded from monitoring; any table not included in the blacklist will be monitored. Each identifier is of the form _databaseName_._tableName_. May not be used with `table.whitelist`.
|`column.mask.hash._hashAlgorithm_.with.salt._salt_`
|[[connector-property-column-mask-hash]]<<connector-property-column-mask-hash, `column.mask.hash._hashAlgorithm_.with.salt._salt_`>>
|_n/a_
|An optional comma-separated list of regular expressions that match the fully-qualified names of character-based columns whose values should be pseudonyms in the change event message values with a field value consisting of the hashed value using the algorithm `_hashAlgorithm_` and salt `_salt_`.
Based on the used hash function referential integrity is kept while data is pseudonymized. Supported hash functions are described in the {link-java7-standard-names}[MessageDigest section] of the Java Cryptography Architecture Standard Algorithm Name Documentation.
@ -1033,57 +1033,57 @@ where `CzQMA0cB5K` is a randomly selected salt.
Note: Depending on the `_hashAlgorithm_` used, the `_salt_` selected and the actual data set, the resulting masked data set may not be completely anonymized.
|`decimal.handling.mode`
|[[connector-property-decimal-handling-mode]]<<connector-property-decimal-handling-mode, `decimal.handling.mode`>>
|`precise`
| Specifies how the connector should handle floating point values for `NUMBER`, `DECIMAL` and `NUMERIC` columns: `precise` (the default) represents them precisely using `java.math.BigDecimal` values represented in change events in a binary form; or `double` represents them using `double` values, which may result in a loss of precision but will be far easier to use. `string` option encodes values as formatted string which is easy to consume but a semantic information about the real type is lost. See <<decimal-values>>.
|`event.processing{zwsp}.failure.handling.mode`
|[[connector-property-event-processing-failure-handling-mode]]<<connector-property-event-processing-failure-handling-mode, `event.processing{zwsp}.failure.handling.mode`>>
|`fail`
| Specifies how the connector should react to exceptions during processing of events.
`fail` will propagate the exception (indicating the offset of the problematic event), causing the connector to stop. +
`warn` will cause the problematic event to be skipped and the offset of the problematic event to be logged. +
`skip` will cause the problematic event to be skipped.
|`max.queue.size`
|[[connector-property-max-queue-size]]<<connector-property-max-queue-size, `max.queue.size`>>
|`8192`
|Positive integer value that specifies the maximum size of the blocking queue into which change events read from the database log are placed before they are written to Kafka. This queue can provide backpressure to the binlog reader when, for example, writes to Kafka are slower or if Kafka is not available. Events that appear in the queue are not included in the offsets periodically recorded by this connector. Defaults to 8192, and should always be larger than the maximum batch size specified in the `max.batch.size` property.
|`max.batch.size`
|[[connector-property-max-batch-size]]<<connector-property-max-batch-size, `max.batch.size`>>
|`2048`
|Positive integer value that specifies the maximum size of each batch of events that should be processed during each iteration of this connector. Defaults to 2048.
|`poll.interval.ms`
|[[connector-property-poll-interval-ms]]<<connector-property-poll-interval-ms, `poll.interval.ms`>>
|`1000`
|Positive integer value that specifies the number of milliseconds the connector should wait during each iteration for new change events to appear. Defaults to 1000 milliseconds, or 1 second.
|`tombstones.on.delete`
|[[connector-property-tombstones-on-delete]]<<connector-property-tombstones-on-delete, `tombstones.on.delete`>>
|`true`
| Controls whether a tombstone event should be generated after a delete event. +
When `true` the delete operations are represented by a delete event and a subsequent tombstone event. When `false` only a delete event is sent. +
Emitting the tombstone event (the default behavior) allows Kafka to completely delete all events pertaining to the given key once the source record got deleted.
|`message.key.columns`
|[[connector-property-message-key-columns]]<<connector-property-message-key-columns, `message.key.columns`>>
|_empty string_
| A semi-colon list of regular expressions that match fully-qualified tables and columns to map a primary key. +
Each item (regular expression) must match the `<fully-qualified table>:<a comma-separated list of columns>` representing the custom key. +
Fully-qualified tables could be defined as `DB_NAME.TABLE_NAME` or `SCHEMA_NAME.TABLE_NAME`, depending on the specific connector.
|`column.truncate.to._length_.chars`
|[[connector-property-column-truncate-to-length-chars]]<<connector-property-column-truncate-to-length-chars, `column.truncate.to._length_.chars`>>
|_n/a_
|An optional comma-separated list of regular expressions that match the fully-qualified names of character-based columns whose values should be truncated in the change event message values if the field values are longer than the specified number of characters. Multiple properties with different lengths can be used in a single configuration, although in each the length must be a positive integer. Fully-qualified names for columns are of the form _databaseName_._tableName_._columnName_, or _databaseName_._schemaName_._tableName_._columnName_.
|`column.mask.with._length_.chars`
|[[connector-property-column-mask-with-length-chars]]<<connector-property-column-mask-with-length-chars, `column.mask.with._length_.chars`>>
|_n/a_
|An optional comma-separated list of regular expressions that match the fully-qualified names of character-based columns whose values should be replaced in the change event message values with a field value consisting of the specified number of asterisk (`*`) characters. Multiple properties with different lengths can be used in a single configuration, although in each the length must be a positive integer or zero. Fully-qualified names for columns are of the form _databaseName_._tableName_._columnName_, or _databaseName_._schemaName_._tableName_._columnName_.
|`column.propagate.source.type`
|[[connector-property-column-propagate-source-type]]<<connector-property-column-propagate-source-type, `column.propagate.source.type`>>
|_n/a_
|An optional comma-separated list of regular expressions that match the fully-qualified names of columns whose original type and length should be added as a parameter to the corresponding field schemas in the emitted change messages.
The schema parameters `pass:[_]pass:[_]debezium.source.column.type`, `pass:[_]pass:[_]debezium.source.column.length` and `pass:[_]pass:[_]debezium.source.column.scale` will be used to propagate the original type name and length (for variable-width types), respectively.
Useful to properly size corresponding columns in sink databases.
Fully-qualified names for columns are of the form _databaseName_._tableName_._columnName_, or _databaseName_._schemaName_._tableName_._columnName_.
|`datatype.propagate.source.type`
|[[connector-property-datatype-propagate-source-type]]<<connector-property-datatype-propagate-source-type, `datatype.propagate.source.type`>>
|_n/a_
|An optional comma-separated list of regular expressions that match the database-specific data type name of columns whose original type and length should be added as a parameter to the corresponding field schemas in the emitted change messages.
The schema parameters `pass:[_]pass:[_]debezium.source.column.type`, `pass:[_]pass:[_]debezium.source.column.length` and `pass:[_]pass:[_]debezium.source.column.scale` will be used to propagate the original type name and length (for variable-width types), respectively.
@ -1091,7 +1091,7 @@ Useful to properly size corresponding columns in sink databases.
Fully-qualified data type names are of the form _databaseName_._tableName_._typeName_, or _databaseName_._schemaName_._tableName_._typeName_.
See xref:data-types[] for the list of Oracle-specific data type names.
|`heartbeat.interval.ms`
|[[connector-property-heartbeat-interval-ms]]<<connector-property-heartbeat-interval-ms, `heartbeat.interval.ms`>>
|`0`
|Controls how frequently heartbeat messages are sent. +
This property contains an interval in milli-seconds that defines how frequently the connector sends messages into a heartbeat topic.
@ -1105,27 +1105,27 @@ and also may result in more change events to be re-sent after a connector restar
Set this parameter to `0` to not send heartbeat messages at all. +
Disabled by default.
|`heartbeat.topics.prefix`
|[[connector-property-heartbeat-topics-prefix]]<<connector-property-heartbeat-topics-prefix, `heartbeat.topics.prefix`>>
|`__debezium-heartbeat`
|Controls the naming of the topic to which heartbeat messages are sent. +
The topic is named according to the pattern `<heartbeat.topics.prefix>.<server.name>`.
|`snapshot.delay.ms`
|[[connector-property-snapshot-delay-ms]]<<connector-property-snapshot-delay-ms, `snapshot.delay.ms`>>
|
|An interval in milli-seconds that the connector should wait before taking a snapshot after starting up; +
Can be used to avoid snapshot interruptions when starting multiple connectors in a cluster, which may cause re-balancing of connectors.
|`snapshot.fetch.size`
|[[connector-property-snapshot-fetch-size]]<<connector-property-snapshot-fetch-size, `snapshot.fetch.size`>>
|`2000`
|Specifies the maximum number of rows that should be read in one go from each table while taking a snapshot.
The connector will read the table contents in multiple batches of this size. Defaults to 2000.
|`sanitize.field.names`
|[[connector-property-sanitize-field-names]]<<connector-property-sanitize-field-names, `sanitize.field.names`>>
|`true` when connector configuration explicitly specifies the `key.converter` or `value.converter` parameters to use Avro, otherwise defaults to `false`.
|Whether field names will be sanitized to adhere to Avro naming requirements.
See xref:configuration/avro.adoc#names[Avro naming] for more details.
|`provide.transaction.metadata` (Incubating)
|[[connector-property-provide-transaction-metadata]]<<connector-property-provide-transaction-metadata, `provide.transaction.metadata` (Incubating)>>
|`false`
|When set to `true` Debezium generates events with transaction boundaries and enriches data events envelope with transaction metadata.

119
documentation/modules/ROOT/pages/connectors/postgresql.adoc
View File

@ -1716,19 +1716,19 @@ The following configuration properties are _required_ unless a default value is
|Default
|Description
|`name`
|[[connector-property-name]]<<connector-property-name, `name`>>
|
|Unique name for the connector. Attempting to register again with the same name will fail. (This property is required by all Kafka Connect connectors.)
|`connector.class`
|[[connector-property-connector-class]]<<connector-property-connector-class, `connector.class`>>
|
|The name of the Java class for the connector. Always use a value of `io.debezium.connector.postgresql.PostgresConnector` for the PostgreSQL connector.
|`tasks.max`
|[[connector-property-tasks-max]]<<connector-property-tasks-max, `tasks.max`>>
|`1`
|The maximum number of tasks that should be created for this connector. The PostgreSQL connector always uses a single task and therefore does not use this value, so the default is always acceptable.
|`plugin.name`
|[[connector-property-plugin-name]]<<connector-property-plugin-name, `plugin.name`>>
|`decoderbufs`
|The name of the Postgres link:#output-plugin[logical decoding plugin] installed on the server.
ifdef::cdc-product[]
@ -1741,15 +1741,15 @@ endif::cdc-product[]
When the processed transactions are very large it is possible that the `JSON` batch event with all changes in the transaction will not fit into the hard-coded memory buffer of size 1 GB.
In such cases it is possible to switch to so-called *streaming* mode when every change in transactions is sent as a separate message from PostgreSQL into {prodname}.
|`slot.name`
|[[connector-property-slot-name]]<<connector-property-slot-name, `slot.name`>>
|`debezium`
|The name of the Postgres logical decoding slot created for streaming changes from a plugin and database instance. Values must conform to link:https://www.postgresql.org/docs/current/static/warm-standby.html#STREAMING-REPLICATION-SLOTS-MANIPULATION[Postgres replication slot naming rules] which state: _"Each replication slot has a name, which can contain lower-case letters, numbers, and the underscore character."_
|`slot.drop.on.stop`
|[[connector-property-slot-drop-on-stop]]<<connector-property-slot-drop-on-stop, `slot.drop.on.stop`>>
|`false`
|Whether or not to drop the logical replication slot when the connector finishes orderly. Should only be set to `true` in testing or development environments. Dropping the slot allows WAL segments to be discarded by the database, so it may happen that after a restart the connector cannot resume from the WAL position where it left off before.
|`publication.name`
|[[connector-property-publication-name]]<<connector-property-publication-name, `publication.name`>>
|`dbz_publication`
|The name of the PostgreSQL publication created created for streaming changes when using `pgoutput`.
@ -1760,111 +1760,112 @@ so it is usually preferable to create the publication upfront.
If the publication already exists (either for all tables or configured with a subset of tables),
{prodname} will instead use the publication as defined.
|`database.hostname`
|[[connector-property-database-hostname]]<<connector-property-database-hostname, `database.hostname`>>
|
|IP address or hostname of the PostgreSQL database server.
|`database.port`
|[[connector-property-database-port]]<<connector-property-database-port, `database.port`>>
|`5432`
|Integer port number of the PostgreSQL database server.
|`database.user`
|[[connector-property-database-user]]<<connector-property-database-user, `database.user`>>
|
|Name of the PostgreSQL database to use when connecting to the PostgreSQL database server.
|`database.password`
|[[connector-property-database-password]]<<connector-property-database-password, `database.password`>>
|
|Password to use when connecting to the PostgreSQL database server.
|`database.dbname`
|[[connector-property-database-dbname]]<<connector-property-database-dbname, `database.dbname`>>
|
|The name of the PostgreSQL database from which to stream the changes
|`database.server.name`
|[[connector-property-database-server-name]]<<connector-property-database-server-name, `database.server.name`>>
|
|Logical name that identifies and provides a namespace for the particular PostgreSQL database server/cluster being monitored. The logical name should be unique across all other connectors, since it is used as a prefix for all Kafka topic names coming from this connector.
Only alphanumeric characters and underscores should be used.
|`schema.whitelist`
|[[connector-property-schema-whitelist]]<<connector-property-schema-whitelist, `schema.whitelist`>>
|
|An optional comma-separated list of regular expressions that match schema names to be monitored; any schema name not included in the whitelist will be excluded from monitoring. By default all non-system schemas will be monitored. May not be used with `schema.blacklist`.
|`schema.blacklist`
|[[connector-property-schema-blacklist]]<<connector-property-schema-blacklist, `schema.blacklist`>>
|
|An optional comma-separated list of regular expressions that match schema names to be excluded from monitoring; any schema name not included in the blacklist will be monitored, with the exception of system schemas. May not be used with `schema.whitelist`.
|`table.whitelist`
|[[connector-property-table-whitelist]]<<connector-property-table-whitelist, `table.whitelist`>>
|
|An optional comma-separated list of regular expressions that match fully-qualified table identifiers for tables to be monitored; any table not included in the whitelist will be excluded from monitoring. Each identifier is of the form _schemaName_._tableName_. By default the connector will monitor every non-system table in each monitored schema. May not be used with `table.blacklist`.
|`table.blacklist`
|[[connector-property-table-blacklist]]<<connector-property-table-blacklist, `table.blacklist`>>
|
|An optional comma-separated list of regular expressions that match fully-qualified table identifiers for tables to be excluded from monitoring; any table not included in the blacklist will be monitored. Each identifier is of the form _schemaName_._tableName_. May not be used with `table.whitelist`.
|`column.whitelist`
|[[connector-property-column-whitelist]]<<connector-property-column-whitelist, `column.whitelist`>>
|
|An optional comma-separated list of regular expressions that match the fully-qualified names of columns that should be included in change event message values. Fully-qualified names for columns are of the form _schemaName_._tableName_._columnName_. May not be used with column.blacklist.
|`column.blacklist`
|[[connector-property-column-blacklist]]<<connector-property-column-blacklist, `column.blacklist`>>
|
|An optional comma-separated list of regular expressions that match the fully-qualified names of columns that should be excluded from change event message values. Fully-qualified names for columns are of the form _schemaName_._tableName_._columnName_. May not be used with column.whitelist.
|`time.precision.mode`
|[[connector-property-time-precision-mode]]<<connector-property-time-precision-mode, `time.precision.mode`>>
|`adaptive`
| Time, date, and timestamps can be represented with different kinds of precision, including: `adaptive` (the default) captures the time and timestamp values exactly as in the database using either millisecond, microsecond, or nanosecond precision values based on the database column's type; `adaptive_time_microseconds` captures the date, datetime and timestamp values exactly as in the database using either millisecond, microsecond, or nanosecond precision values based on the database column's type, with the exception of TIME type fields, which are always captured as microseconds; or `connect` always represents time and timestamp values using Kafka Connect's built-in representations for Time, Date, and Timestamp, which uses millisecond precision regardless of the database columns' precision. See link:#temporal-values[temporal values].
|`decimal.handling.mode`
|[[connector-property-decimal-handling-mode]]<<connector-property-decimal-handling-mode, `decimal.handling.mode`>>
|`precise`
| Specifies how the connector should handle values for `DECIMAL` and `NUMERIC` columns: `precise` (the default) represents them precisely using `java.math.BigDecimal` values represented in change events in a binary form; or `double` represents them using `double` values, which may result in a loss of precision but will be far easier to use. `string` option encodes values as formatted string which is easy to consume but a semantic information about the real type is lost. See <<decimal-values>>.
|`hstore.handling.mode`
|[[connector-property-hstore-handling-mode]]<<connector-property-hstore-handling-mode, `hstore.handling.mode`>>
|`map`
| Specifies how the connector should handle values for `hstore` columns: `map` (the default) represents using `MAP`; or `json` represents them using `json string`.`json` option encodes values as formatted string such as `{"key" : "val"}`. See <<hstore-values>>.
|`interval.handling.mode`
|[[connector-property-interval-handling-mode]]<<connector-property-interval-handling-mode, `interval.handling.mode`>>
|`numeric`
| Specifies how the connector should handle values for `interval` columns: `numeric` (the default) represents interval using approximate number of microseconds; `string` represents them exactly, using the string pattern representation `P<years>Y<months>M<days>DT<hours>H<minutes>M<seconds>S`, e.g. `P1Y2M3DT4H5M6.78S`. See <<data-types>>.
|`database.sslmode`
|[[connector-property-database-sslmode]]<<connector-property-database-sslmode, `database.sslmode`>>
|`disable`
|Whether to use an encrypted connection to the PostgreSQL server. Options include: *disable* (the default) to use an unencrypted connection ; *require* to use a secure (encrypted) connection, and fail if one cannot be established; *verify-ca* like `require` but additionally verify the server TLS certificate against the configured Certificate Authority (CA) certificates, or fail if no valid matching CA certificates are found; *verify-full* like `verify-ca` but additionally verify that the server certificate matches the host to which the connection is attempted. See https://www.postgresql.org/docs/current/static/libpq-connect.html[the PostgreSQL documentation] for more information.
|`database.sslcert`
|[[connector-property-database-sslcert]]<<connector-property-database-sslcert, `database.sslcert`>>
|
|The path to the file containing the SSL Certificate for the client. See https://www.postgresql.org/docs/current/static/libpq-connect.html[the PostgreSQL documentation] for more information.
|`database.sslkey`
|[[connector-property-database-sslkey]]<<connector-property-database-sslkey, `database.sslkey`>>
|
|The path to the file containing the SSL private key of the client. See https://www.postgresql.org/docs/current/static/libpq-connect.html[the PostgreSQL documentation] for more information.
|`database.sslpassword`
|[[connector-property-database-sslpassword]]<<connector-property-database-sslpassword, `database.sslpassword`>>
|
|The password to access the client private key from the file specified by `database.sslkey`. See https://www.postgresql.org/docs/current/static/libpq-connect.html[the PostgreSQL documentation] for more information.
|`database.sslrootcert`
|[[connector-property-database-sslrootcert]]<<connector-property-database-sslrootcert, `database.sslrootcert`>>
|
|The path to the file containing the root certificate(s) against which the server is validated. See https://www.postgresql.org/docs/current/static/libpq-connect.html[the PostgreSQL documentation] for more information.
|`database.tcpKeepAlive`
|[[connector-property-database-tcpkeepalive]]<<connector-property-database-tcpkeepalive, `database.tcpKeepAlive`>>
|
|Enable TCP keep-alive probe to verify that database connection is still alive. (enabled by default). See https://www.postgresql.org/docs/current/static/libpq-connect.html[the PostgreSQL documentation] for more information.
|`tombstones.on.delete`
|[[connector-property-tombstones-on-delete]]<<connector-property-tombstones-on-delete, `tombstones.on.delete`>>
|`true`
| Controls whether a tombstone event should be generated after a delete event. +
When `true` the delete operations are represented by a delete event and a subsequent tombstone event. When `false` only a delete event is sent. +
Emitting the tombstone event (the default behavior) allows Kafka to completely delete all events pertaining to the given key once the source record got deleted.
|`column.truncate.to._length_.chars`
|[[connector-property-column-truncate-to-length-chars]]<<connector-property-column-truncate-to-length-chars, `column.truncate.to._length_.chars`>>
|_n/a_
|An optional comma-separated list of regular expressions that match the fully-qualified names of character-based columns whose values should be truncated in the change event message values if the field values are longer than the specified number of characters. Multiple properties with different lengths can be used in a single configuration, although in each the length must be a positive integer. Fully-qualified names for columns are of the form _databaseName_._tableName_._columnName_, or _databaseName_._schemaName_._tableName_._columnName_.
|`column.mask.with._length_.chars`
|[[connector-property-column-mask-with-length-chars]]<<connector-property-column-mask-with-length-chars, `column.mask.with._length_.chars`>>
|_n/a_
|An optional comma-separated list of regular expressions that match the fully-qualified names of character-based columns whose values should be replaced in the change event message values with a field value consisting of the specified number of asterisk (`*`) characters. Multiple properties with different lengths can be used in a single configuration, although in each the length must be a positive integer or zero. Fully-qualified names for columns are of the form _databaseName_._tableName_._columnName_, or _databaseName_._schemaName_._tableName_._columnName_.
|`column.mask.hash._hashAlgorithm_.with.salt._salt_`
|[[connector-property-column-mask-hash]]<<connector-property-column-mask-hash, `column.mask.hash._hashAlgorithm_.with.salt._salt_`>>
|_n/a_
|An optional comma-separated list of regular expressions that match the fully-qualified names of character-based columns whose values should be pseudonyms in the change event message values with a field value consisting of the hashed value using the algorithm `_hashAlgorithm_` and salt `_salt_`.
Based on the used hash function referential integrity is kept while data is pseudonymized. Supported hash functions are described in the {link-java7-standard-names}[MessageDigest section] of the Java Cryptography Architecture Standard Algorithm Name Documentation.
@ -1880,14 +1881,14 @@ where `CzQMA0cB5K` is a randomly selected salt.
Note: Depending on the `_hashAlgorithm_` used, the `_salt_` selected and the actual data set, the resulting masked data set may not be completely anonymized.
|`column.propagate.source.type`
|[[connector-property-column-propagate-source-type]]<<connector-property-column-propagate-source-type, `column.propagate.source.type`>>
|_n/a_
|An optional comma-separated list of regular expressions that match the fully-qualified names of columns whose original type and length should be added as a parameter to the corresponding field schemas in the emitted change messages.
The schema parameters `pass:[_]pass:[_]debezium.source.column.type`, `pass:[_]pass:[_]debezium.source.column.length` and `pass:[_]pass:[_]debezium.source.column.scale` will be used to propagate the original type name and length (for variable-width types), respectively.
Useful to properly size corresponding columns in sink databases.
Fully-qualified names for columns are of the form _databaseName_._tableName_._columnName_, or _databaseName_._schemaName_._tableName_._columnName_.
|`datatype.propagate.source.type`
|[[connector-property-datatype-propagate-source-type]]<<connector-property-datatype-propagate-source-type, `datatype.propagate.source.type`>>
|_n/a_
|An optional comma-separated list of regular expressions that match the database-specific data type name of columns whose original type and length should be added as a parameter to the corresponding field schemas in the emitted change messages.
The schema parameters `pass:[_]pass:[_]debezium.source.column.type`, `pass:[_]pass:[_]debezium.source.column.length` and `pass:[_]pass:[_]debezium.source.column.scale` will be used to propagate the original type name and length (for variable-width types), respectively.
@ -1895,7 +1896,7 @@ Useful to properly size corresponding columns in sink databases.
Fully-qualified data type names are of the form _databaseName_._tableName_._typeName_, or _databaseName_._schemaName_._tableName_._typeName_.
See xref:data-types[] for the list of PostgreSQL-specific data type names.
|`message.key.columns`
|[[connector-property-message-key-columns]]<<connector-property-message-key-columns, `message.key.columns`>>
|_empty string_
| A semi-colon list of regular expressions that match fully-qualified tables and columns to map a primary key. +
Each item (regular expression) must match the fully-qualified `<fully-qualified table>:<a comma-separated list of columns>` representing the custom key. +
@ -1912,45 +1913,47 @@ The following _advanced_ configuration properties have good defaults that will w
|Default
|Description
|`snapshot.mode`
|[[connector-property-snapshot-mode]]<<connector-property-snapshot-mode, `snapshot.mode`>>
|`initial`
|Specifies the criteria for running a snapshot upon startup of the connector. The default is *initial*, and specifies the connector can run a snapshot only when no offsets have been recorded for the logical server name. The *always* option specifies that the connector run a snapshot each time on startup. The *never* option specifies that the connect should never use snapshots and that upon first startup with a logical server name the connector should read from either from where it last left off (last LSN position) or start from the beginning from the point of the view of the logical replication slot. The *initial_only* option specifies that the connector should only take an initial snapshot and then stop, without processing any subsequent changes. The *exported* option specifies that the database snapshot will be based on the point in time when the replication slot was created and is an excellent way to perform the snapshot in a lock-free way.
ifndef::cdc-product[]
Finally, if set to *custom* then the user must also set `snapshot.custom.class` which is a custom implementation of the `io.debezium.connector.postgresql.spi.Snapshotter` interface. See link:#snapshots[snapshots].
|`snapshot.custom.class`
|[[connector-property-snapshot-custom-class]]<<connector-property-snapshot-custom-class, `snapshot.custom.class`>>
|
| A full java class name that must be an implementation of the `io.debezium.connector.postgresql.spi.Snapshotter` interface. Only used when `snapshot.mode` is *custom
endif::cdc-product[]
|`snapshot.lock.timeout.ms`
|[[connector-property-snapshot-lock-timeout-ms]]<<connector-property-snapshot-lock-timeout-ms, `snapshot.lock.timeout.ms`>>
|`10000`
|Positive integer value that specifies the maximum amount of time (in milliseconds) to wait to obtain table locks when performing a snapshot. If table locks cannot be acquired in this time interval, the snapshot will fail. See link:#snapshots[snapshots]
|`snapshot.select.statement.overrides`
|[[connector-property-snapshot-select-statement-overrides]]<<connector-property-snapshot-select-statement-overrides, `snapshot.select.statement.overrides`>>
|
|Controls which rows from tables will be included in snapshot. +
This property contains a comma-separated list of fully-qualified tables _(DB_NAME.TABLE_NAME)_. Select statements for the individual tables are specified in further configuration properties, one for each table, identified by the id `snapshot.select.statement.overrides.[DB_NAME].[TABLE_NAME]`. The value of those properties is the SELECT statement to use when retrieving data from the specific table during snapshotting. _A possible use case for large append-only tables is setting a specific point where to start (resume) snapshotting, in case a previous snapshotting was interrupted._ +
NOTE: This setting has impact on snapshots only. Events generated by logical decoder are not affected by it at all.
|`event.processing{zwsp}.failure.handling.mode`
|[[connector-property-event-processing-failure-handling-mode]]<<connector-property-event-processing-failure-handling-mode, `event.processing{zwsp}.failure.handling.mode`>>
|`fail`
| Specifies how the connector should react to exceptions during processing of events.
`fail` will propagate the exception (indicating the offset of the problematic event), causing the connector to stop. +
`warn` will cause the problematic event to be skipped and the offset of the problematic event to be logged. +
`skip` will cause the problematic event to be skipped.
|`max.queue.size`
|[[connector-property-max-queue-size]]<<connector-property-max-queue-size, `max.queue.size`>>
|`20240`
|Positive integer value that specifies the maximum size of the blocking queue into which change events received via streaming replication are placed before they are written to Kafka. This queue can provide backpressure when, for example, writes to Kafka are slower or if Kafka is not available.
|`max.batch.size`
|[[connector-property-max-batch-size]]<<connector-property-max-batch-size, `max.batch.size`>>
|`10240`
|Positive integer value that specifies the maximum size of each batch of events that should be processed during each iteration of this connector.
|`poll.interval.ms`
|[[connector-property-poll-interval-ms]]<<connector-property-poll-interval-ms, `poll.interval.ms`>>
|`1000`
|Positive integer value that specifies the number of milliseconds the connector should wait during each iteration for new change events to appear. Defaults to 1000 milliseconds, or 1 second.
|`include.unknown.datatypes`
|[[connector-property-include-unknown-datatypes]]<<connector-property-include-unknown-datatypes, `include.unknown.datatypes`>>
|`false`
|When {prodname} meets a field whose data type is unknown, then by default the field is omitted from the change event and a warning is logged.
In some cases it may be preferable though to include the field and send it downstream to clients in the opaque binary representation so the clients will decode it themselves.
@ -1958,14 +1961,14 @@ Set to `false` to filter unknown data out of events and `true` to keep them in b
NOTE: The clients risk backward compatibility issues. Not only may the database specific binary representation change between releases, but also when the datatype is supported by {prodname} eventually, it will be sent downstream in a logical type, requiring adjustments by consumers. In general, when encountering unsupported data types, please file a feature request so that support can be added.
|`database.initial.statements`
|[[connector-property-database-initial-statements]]<<connector-property-database-initial-statements, `database.initial.statements`>>
|
|A semicolon separated list of SQL statements to be executed when a JDBC connection (not the transaction log reading connection) to the database is established.
Use doubled semicolon (';;') to use a semicolon as a character and not as a delimiter.
NOTE: The connector may establish JDBC connections at its own discretion, so this should typically be used for configuration of session parameters only, but not for executing DML statements.
|`heartbeat.interval.ms`
|[[connector-property-heartbeat-interval-ms]]<<connector-property-heartbeat-interval-ms, `heartbeat.interval.ms`>>
|`0`
|Controls how frequently heartbeat messages are sent. +
This property contains an interval in milli-seconds that defines how frequently the connector sends messages into a heartbeat topic.
@ -1979,12 +1982,12 @@ and also may result in more change events to be re-sent after a connector restar
Set this parameter to `0` to not send heartbeat messages at all. +
Disabled by default.
|`heartbeat.topics.prefix`
|[[connector-property-heartbeat-topics-prefix]]<<connector-property-heartbeat-topics-prefix, `heartbeat.topics.prefix`>>
|`__debezium-heartbeat`
|Controls the naming of the topic to which heartbeat messages are sent. +
The topic is named according to the pattern `<heartbeat.topics.prefix>.<server.name>`.
|`heartbeat.action.query`
|[[connector-property-heartbeat-action-query]]<<connector-property-heartbeat-action-query, `heartbeat.action.query`>>
|
|If specified, this query will be executed upon every heartbeat against the source database.
@ -1996,7 +1999,7 @@ preventing an unbounded WAL growth on the database host.
Example: `INSERT INTO test_heartbeat_table (text) VALUES ('test_heartbeat')`
|`schema.refresh.mode`
|[[connector-property-schema-refresh-mode]]<<connector-property-schema-refresh-mode, `schema.refresh.mode`>>
|`columns_diff`
|Specify the conditions that trigger a refresh of the in-memory schema for a table.
@ -2009,17 +2012,17 @@ This setting can improve connector performance significantly if there are freque
have TOASTed data that are rarely part of these updates. However, it is possible for the in-memory schema to
become outdated if TOASTable columns are dropped from the table.
|`snapshot.delay.ms`
|[[connector-property-snapshot-delay-ms]]<<connector-property-snapshot-delay-ms, `snapshot.delay.ms`>>
|
|An interval in milli-seconds that the connector should wait before taking a snapshot after starting up; +
Can be used to avoid snapshot interruptions when starting multiple connectors in a cluster, which may cause re-balancing of connectors.
|`snapshot.fetch.size`
|[[connector-property-snapshot-fetch-size]]<<connector-property-snapshot-fetch-size, `snapshot.fetch.size`>>
|`10240`
|Specifies the maximum number of rows that should be read in one go from each table while taking a snapshot.
The connector will read the table contents in multiple batches of this size. Defaults to 10240.
|`slot.stream.params`
|[[connector-property-slot-stream-params]]<<connector-property-slot-stream-params, `slot.stream.params`>>
|
|Optional list of parameters to be passed to the configured logical decoding plug-in.
ifndef::cdc-product[]
@ -2028,26 +2031,26 @@ Allowed values depend on the chosen plug-in are separated by semicolon.
endif::cdc-product[]
For example, `add-tables=public.table,public.table2;include-lsn=true`.
|`sanitize.field.names`
|[[connector-property-sanitize-field-names]]<<connector-property-sanitize-field-names, `sanitize.field.names`>>
|`true` when connector configuration explicitly specifies the `key.converter` or `value.converter` parameters to use Avro, otherwise defaults to `false`.
|Whether field names will be sanitized to adhere to Avro naming requirements.
See xref:configuration/avro.adoc#names[Avro naming] for more details.
|`slot.max.retries`
|[[connector-property-slot-max-retries]]<<connector-property-slot-max-retries, `slot.max.retries`>>
|6
|How many times to retry connecting to a replication slot when an attempt fails.
|`slot.retry.delay.ms` +
|[[connector-property-slot-retry-delay-ms]]<<connector-property-slot-retry-delay-ms, `slot.retry.delay.ms`>> +
|10000 (10 seconds)
|The number of milli-seconds to wait between retry attempts when the connector fails to connect to a replication slot.
|`toasted.value.placeholder`
|[[connector-property-toasted-value-placeholder]]<<connector-property-toasted-value-placeholder, `toasted.value.placeholder`>>
|`__debezium_unavailable_value`
|Specify the constant that will be provided by {prodname} to indicate that the original value is a toasted value not provided by the database.
If starts with `hex:` prefix it is expected that the rest of the string repesents hexadecimally encoded octets.
See link:#toasted-values[Toasted Values] for additional details.
|`provide.transaction.metadata` (Incubating)
|[[connector-property-provide-transaction-metadata]]<<connector-property-provide-transaction-metadata, `provide.transaction.metadata` (Incubating)>>
|`false`
|When set to `true` Debezium generates events with transaction boundaries and enriches data events envelope with transaction metadata.

76
documentation/modules/ROOT/pages/connectors/sqlserver.adoc
View File

@ -1219,68 +1219,68 @@ The following configuration properties are _required_ unless a default value is
|Default
|Description
|`name`
|[[connector-property-name]]<<connector-property-name, `name`>>
|
|Unique name for the connector. Attempting to register again with the same name will fail. (This property is required by all Kafka Connect connectors.)
|`connector.class`
|[[connector-property-connector-class]]<<connector-property-connector-class, `connector.class`>>
|
|The name of the Java class for the connector. Always use a value of `io.debezium.connector.sqlserver.SqlServerConnector` for the SQL Server connector.
|`tasks.max`
|[[connector-property-tasks-max]]<<connector-property-tasks-max, `tasks.max`>>
|`1`
|The maximum number of tasks that should be created for this connector. The SQL Server connector always uses a single task and therefore does not use this value, so the default is always acceptable.
|`database.hostname`
|[[connector-property-database-hostname]]<<connector-property-database-hostname, `database.hostname`>>
|
|IP address or hostname of the SQL Server database server.
|`database.port`
|[[connector-property-database-port]]<<connector-property-database-port, `database.port`>>
|`1433`
|Integer port number of the SQL Server database server.
|`database.user`
|[[connector-property-database-user]]<<connector-property-database-user, `database.user`>>
|
|Username to use when connecting to the SQL Server database server.
|`database.password`
|[[connector-property-database-password]]<<connector-property-database-password, `database.password`>>
|
|Password to use when connecting to the SQL Server database server.
|`database.dbname`
|[[connector-property-database-dbname]]<<connector-property-database-dbname, `database.dbname`>>
|
|The name of the SQL Server database from which to stream the changes
|`database.server.name`
|[[connector-property-database-server-name]]<<connector-property-database-server-name, `database.server.name`>>
|
|Logical name that identifies and provides a namespace for the particular SQL Server database server being monitored. The logical name should be unique across all other connectors, since it is used as a prefix for all Kafka topic names emanating from this connector.
Only alphanumeric characters and underscores should be used.
|`database.history.kafka.topic`
|[[connector-property-database-history-kafka-topic]]<<connector-property-database-history-kafka-topic, `database.history.kafka.topic`>>
|
|The full name of the Kafka topic where the connector will store the database schema history.
|`database.history{zwsp}.kafka.bootstrap.servers`
|[[connector-property-database-history-kafka-bootstrap-servers]]<<connector-property-database-history-kafka-bootstrap-servers, `database.history{zwsp}.kafka.bootstrap.servers`>>
|
|A list of host/port pairs that the connector will use for establishing an initial connection to the Kafka cluster.
This connection is used for retrieving database schema history previously stored by the connector, and for writing each DDL statement read from the source database. This should point to the same Kafka cluster used by the Kafka Connect process.
|`table.whitelist`
|[[connector-property-table-whitelist]]<<connector-property-table-whitelist, `table.whitelist`>>
|
|An optional comma-separated list of regular expressions that match fully-qualified table identifiers for tables to be monitored; any table not included in the whitelist is excluded from monitoring. Each identifier is of the form _schemaName_._tableName_. By default the connector will monitor every non-system table in each monitored schema. May not be used with `table.blacklist`.
|`table.blacklist`
|[[connector-property-table-blacklist]]<<connector-property-table-blacklist, `table.blacklist`>>
|
|An optional comma-separated list of regular expressions that match fully-qualified table identifiers for tables to be excluded from monitoring; any table not included in the blacklist is monitored.
Each identifier is of the form _schemaName_._tableName_. May not be used with `table.whitelist`.
|`column.blacklist`
|[[connector-property-column-blacklist]]<<connector-property-column-blacklist, `column.blacklist`>>
|_empty string_
|An optional comma-separated list of regular expressions that match the fully-qualified names of columns that should be excluded from change event message values.
Fully-qualified names for columns are of the form _schemaName_._tableName_._columnName_.
Note that primary key columns are always included in the event's key, also if blacklisted from the value.
|`column.mask.hash._hashAlgorithm_.with.salt._salt_`
|[[connector-property-column-mask-hash]]<<connector-property-column-mask-hash, `column.mask.hash._hashAlgorithm_.with.salt._salt_`>>
|_n/a_
|An optional comma-separated list of regular expressions that match the fully-qualified names of character-based columns whose values should be pseudonyms in the change event message values with a field value consisting of the hashed value using the algorithm `_hashAlgorithm_` and salt `_salt_`.
Based on the used hash function referential integrity is kept while data is pseudonymized. Supported hash functions are described in the {link-java7-standard-names}[MessageDigest section] of the Java Cryptography Architecture Standard Algorithm Name Documentation.
@ -1296,32 +1296,32 @@ where `CzQMA0cB5K` is a randomly selected salt.
Note: Depending on the `_hashAlgorithm_` used, the `_salt_` selected and the actual data set, the resulting masked data set may not be completely anonymized.
|`time.precision.mode`
|[[connector-property-time-precision-mode]]<<connector-property-time-precision-mode, `time.precision.mode`>>
|`adaptive`
| Time, date, and timestamps can be represented with different kinds of precision, including: `adaptive` (the default) captures the time and timestamp values exactly as in the database using either millisecond, microsecond, or nanosecond precision values based on the database column's type; or `connect` always represents time and timestamp values using Kafka Connect's built-in representations for Time, Date, and Timestamp, which uses millisecond precision regardless of the database columns' precision. See link:#sqlserver-temporal-values[temporal values].
|`tombstones.on.delete`
|[[connector-property-tombstones-on-delete]]<<connector-property-tombstones-on-delete, `tombstones.on.delete`>>
|`true`
| Controls whether a tombstone event should be generated after a delete event. +
When `true` the delete operations are represented by a delete event and a subsequent tombstone event. When `false` only a delete event is sent. +
Emitting the tombstone event (the default behavior) allows Kafka to completely delete all events pertaining to the given key once the source record got deleted.
|`column.truncate.to._length_.chars`
|[[connector-property-column-truncate-to-length-chars]]<<connector-property-column-truncate-to-length-chars, `column.truncate.to._length_.chars`>>
|_n/a_
|An optional comma-separated list of regular expressions that match the fully-qualified names of character-based columns whose values should be truncated in the change event message values if the field values are longer than the specified number of characters. Multiple properties with different lengths can be used in a single configuration, although in each the length must be a positive integer. Fully-qualified names for columns are of the form _databaseName_._tableName_._columnName_, or _databaseName_._schemaName_._tableName_._columnName_.
|`column.mask.with._length_.chars`
|[[connector-property-column-mask-with-length-chars]]<<connector-property-column-mask-with-length-chars, `column.mask.with._length_.chars`>>
|_n/a_
|An optional comma-separated list of regular expressions that match the fully-qualified names of character-based columns whose values should be replaced in the change event message values with a field value consisting of the specified number of asterisk (`*`) characters. Multiple properties with different lengths can be used in a single configuration, although in each the length must be a positive integer or zero. Fully-qualified names for columns are of the form _databaseName_._tableName_._columnName_, or _databaseName_._schemaName_._tableName_._columnName_.
|`column.propagate.source.type`
|[[connector-property-column-propagate-source-type]]<<connector-property-column-propagate-source-type, `column.propagate.source.type`>>
|_n/a_
|An optional comma-separated list of regular expressions that match the fully-qualified names of columns whose original type and length should be added as a parameter to the corresponding field schemas in the emitted change messages.
The schema parameters `pass:[_]pass:[_]debezium.source.column.type`, `pass:[_]pass:[_]debezium.source.column.length` and `pass:[_]pass:[_]debezium.source.column.scale` is used to propagate the original type name and length (for variable-width types), respectively.
Useful to properly size corresponding columns in sink databases.
Fully-qualified names for columns are of the form _schemaName_._tableName_._columnName_.
|`datatype.propagate.source.type`
|[[connector-property-datatype-propagate-source-type]]<<connector-property-datatype-propagate-source-type, `datatype.propagate.source.type`>>
|_n/a_
|An optional comma-separated list of regular expressions that match the database-specific data type name of columns whose original type and length should be added as a parameter to the corresponding field schemas in the emitted change messages.
The schema parameters `pass:[_]pass:[_]debezium.source.column.type`, `pass:[_]pass:[_]debezium.source.column.length` and `pass:[_]pass:[_]debezium.source.column.scale` will be used to propagate the original type name and length (for variable-width types), respectively.
@ -1329,7 +1329,7 @@ Useful to properly size corresponding columns in sink databases.
Fully-qualified data type names are of the form _schemaName_._tableName_._typeName_.
See xref:sqlserver-data-types[] for the list of SQL Server-specific data type names.
|`message.key.columns`
|[[connector-property-message-key-columns]]<<connector-property-message-key-columns, `message.key.columns`>>
|_empty string_
| A semi-colon list of regular expressions that match fully-qualified tables and columns to map a primary key. +
Each item (regular expression) must match the fully-qualified `<fully-qualified table>:<a comma-separated list of columns>` representing the custom key. +
@ -1344,7 +1344,7 @@ The following _advanced_ configuration properties have good defaults that will w
|Default
|Description
|`snapshot.mode`
|[[connector-property-snapshot-mode]]<<connector-property-snapshot-mode, `snapshot.mode`>>
|_initial_
|A mode for taking an initial snapshot of the structure and optionally data of captured tables.
Once the snapshot is complete, the connector will continue reading change events from the database's redo logs. +
@ -1353,7 +1353,7 @@ Supported values are: +
`initial`: Takes a snapshot of structure and data of captured tables; useful if topics should be populated with a complete representation of the data from the captured tables. +
`schema_only`: Takes a snapshot of the structure of captured tables only; useful if only changes happening from now onwards should be propagated to topics.
|`snapshot.isolation.mode`
|[[connector-property-snapshot-isolation-mode]]<<connector-property-snapshot-isolation-mode, `snapshot.isolation.mode`>>
|_repeatable_read_
|Mode to control which transaction isolation level is used and how long the connector locks the monitored tables.
There are five possible values: `read_uncommitted`, `read_committed`, `repeatable_read`, `snapshot`, and `exclusive` (
@ -1370,26 +1370,26 @@ twice - once in initial snapshot and once in streaming phase. Nonetheless, that
data mirroring.
For `read_uncommitted` there are no data consistency guarantees at all (some data might be lost or corrupted).
|`event.processing{zwsp}.failure.handling.mode`
|[[connector-property-event-processing-failure-handling-mode]]<<connector-property-event-processing-failure-handling-mode, `event.processing{zwsp}.failure.handling.mode`>>
|`fail`
| Specifies how the connector should react to exceptions during processing of events.
`fail` will propagate the exception (indicating the offset of the problematic event), causing the connector to stop. +
`warn` will cause the problematic event to be skipped and the offset of the problematic event to be logged. +
`skip` will cause the problematic event to be skipped.
|`poll.interval.ms`
|[[connector-property-poll-interval-ms]]<<connector-property-poll-interval-ms, `poll.interval.ms`>>
|`1000`
|Positive integer value that specifies the number of milliseconds the connector should wait during each iteration for new change events to appear. Defaults to 1000 milliseconds, or 1 second.
|`max.queue.size`
|[[connector-property-max-queue-size]]<<connector-property-max-queue-size, `max.queue.size`>>
|`8192`
|Positive integer value that specifies the maximum size of the blocking queue into which change events read from the database log are placed before they are written to Kafka. This queue can provide backpressure to the CDC table reader when, for example, writes to Kafka are slower or if Kafka is not available. Events that appear in the queue are not included in the offsets periodically recorded by this connector. Defaults to 8192, and should always be larger than the maximum batch size specified in the `max.batch.size` property.
|`max.batch.size`
|[[connector-property-max-batch-size]]<<connector-property-max-batch-size, `max.batch.size`>>
|`2048`
|Positive integer value that specifies the maximum size of each batch of events that should be processed during each iteration of this connector. Defaults to 2048.
|`heartbeat.interval.ms`
|[[connector-property-heartbeat-interval-ms]]<<connector-property-heartbeat-interval-ms, `heartbeat.interval.ms`>>
|`0`
|Controls how frequently heartbeat messages are sent. +
This property contains an interval in milli-seconds that defines how frequently the connector sends messages into a heartbeat topic.
@ -1401,34 +1401,34 @@ This may result in more change events to be re-sent after a connector restart.
Set this parameter to `0` to not send heartbeat messages at all. +
Disabled by default.
|`heartbeat.topics.prefix`
|[[connector-property-heartbeat-topics-prefix]]<<connector-property-heartbeat-topics-prefix, `heartbeat.topics.prefix`>>
|`__debezium-heartbeat`
|Controls the naming of the topic to which heartbeat messages are sent. +
The topic is named according to the pattern `<heartbeat.topics.prefix>.<server.name>`.
|`snapshot.delay.ms`
|[[connector-property-snapshot-delay-ms]]<<connector-property-snapshot-delay-ms, `snapshot.delay.ms`>>
|
|An interval in milli-seconds that the connector should wait before taking a snapshot after starting up; +
Can be used to avoid snapshot interruptions when starting multiple connectors in a cluster, which may cause re-balancing of connectors.
|`snapshot.fetch.size`
|[[connector-property-snapshot-fetch-size]]<<connector-property-snapshot-fetch-size, `snapshot.fetch.size`>>
|`2000`
|Specifies the maximum number of rows that should be read in one go from each table while taking a snapshot.
The connector will read the table contents in multiple batches of this size. Defaults to 2000.
|`snapshot.lock.timeout.ms`
|[[connector-property-snapshot-lock-timeout-ms]]<<connector-property-snapshot-lock-timeout-ms, `snapshot.lock.timeout.ms`>>
|`10000`
|An integer value that specifies the maximum amount of time (in milliseconds) to wait to obtain table locks when performing a snapshot. If table locks cannot be acquired in this time interval, the snapshot will fail (also see link:#snapshots[snapshots]). +
When set to `0` the connector will fail immediately when it cannot obtain the lock. Value `-1` indicates infinite waiting.
|`snapshot.select.statement.overrides`
|[[connector-property-snapshot-select-statement-overrides]]<<connector-property-snapshot-select-statement-overrides, `snapshot.select.statement.overrides`>>
|
|Controls which rows from tables are included in snapshot. +
This property contains a comma-separated list of fully-qualified tables _(SCHEMA_NAME.TABLE_NAME)_. Select statements for the individual tables are specified in further configuration properties, one for each table, identified by the id `snapshot.select.statement.overrides.[SCHEMA_NAME].[TABLE_NAME]`. The value of those properties is the SELECT statement to use when retrieving data from the specific table during snapshotting. _A possible use case for large append-only tables is setting a specific point where to start (resume) snapshotting, in case a previous snapshotting was interrupted._ +
*Note*: This setting has impact on snapshots only. Events captured during log reading are not affected by it.
ifndef::cdc-product[]
|`source.struct.version`
|[[connector-property-source-struct-version]]<<connector-property-source-struct-version, `source.struct.version`>>
|v2
|Schema version for the `source` block in CDC events; Debezium 0.10 introduced a few breaking +
changes to the structure of the `source` block in order to unify the exposed structure across
@ -1437,14 +1437,14 @@ By setting this option to `v1` the structure used in earlier versions can be pro
Note that this setting is not recommended and is planned for removal in a future {prodname} version.
endif::cdc-product[]
|`sanitize.field.names`
|[[connector-property-sanitize-field-names]]<<connector-property-sanitize-field-names, `sanitize.field.names`>>
|`true` when connector configuration explicitly specifies the `key.converter` or `value.converter` parameters to use Avro, otherwise defaults to `false`.
|Whether field names are sanitized to adhere to Avro naming requirements.
ifndef::cdc-product[]
See xref:configuration/avro.adoc#names[Avro naming] for more details.
endif::cdc-product[]
|`database.server.timezone`
|[[connector-property-database-server-timezone]]<<connector-property-database-server-timezone, `database.server.timezone`>>
|
| Timezone of the server.
@ -1452,7 +1452,7 @@ This is used to define the timezone of the transaction timestamp (ts_ms) retriev
When unset, default behavior is to use the timezone of the VM running the Debezium connector. In this case, when running on on SQL Server 2014 or older and using different timezones on server and the connector, incorrect ts_ms values may be produced. +
Possible values include "Z", "UTC", offset values like "+02:00", short zone ids like "CET", and long zone ids like "Europe/Paris".
|`provide.transaction.metadata` (Incubating)
|[[connector-property-provide-transaction-metadata]]<<connector-property-provide-transaction-metadata, `provide.transaction.metadata` (Incubating)>>
|`false`
|When set to `true` Debezium generates events with transaction boundaries and enriches data events envelope with transaction metadata.

118
documentation/modules/ROOT/partials/modules/cdc-mysql-connector/r_mysql-connector-configuration-properties.adoc
View File

@ -12,80 +12,80 @@ TIP: The {prodname} MySQL connector supports _pass-through_ configuration when c
|===
|Property |Default |Description
|`name`
|[[connector-property-name]]<<connector-property-name, `name`>>
|
|Unique name for the connector. Attempting to register again with the same name will fail. (This property is required by all Kafka Connect connectors.)
|`connector.class`
|[[connector-property-connector-class]]<<connector-property-connector-class, `connector.class`>>
|
|The name of the Java class for the connector. Always use a value of `io.debezium{zwsp}.connector.mysql.MySqlConnector` for the MySQL connector.
|`tasks.max`
|[[connector-property-tasks-max]]<<connector-property-tasks-max, `tasks.max`>>
|`1`
|The maximum number of tasks that should be created for this connector. The MySQL connector always uses a single task and therefore does not use this value, so the default is always acceptable.
|`database.hostname`
|[[connector-property-database-hostname]]<<connector-property-database-hostname, `database.hostname`>>
|
|IP address or hostname of the MySQL database server.
|`database.port`
|[[connector-property-database-port]]<<connector-property-database-port, `database.port`>>
|`3306`
|Integer port number of the MySQL database server.
|`database.user`
|[[connector-property-database-user]]<<connector-property-database-user, `database.user`>>
|
|Name of the MySQL database to use when connecting to the MySQL database server.
|`database.password`
|[[connector-property-database-password]]<<connector-property-database-password, `database.password`>>
|
|Password to use when connecting to the MySQL database server.
|`database.server.name`
|[[connector-property-database-server-name]]<<connector-property-database-server-name, `database.server.name`>>
|
|Logical name that identifies and provides a namespace for the particular MySQL database server/cluster being monitored. The logical name should be unique across all other connectors, since it is used as a prefix for all Kafka topic names emanating from this connector.
Only alphanumeric characters and underscores should be used.
|`database.server.id`
|[[connector-property-database-server-id]]<<connector-property-database-server-id, `database.server.id`>>
|_random_
|A numeric ID of this database client, which must be unique across all currently-running database processes in the MySQL cluster. This connector joins the MySQL database cluster as another server (with this unique ID) so it can read the binlog. By default, a random number is generated between 5400 and 6400, though we recommend setting an explicit value.
|`database.history.kafka.topic`
|[[connector-property-database-history-kafka-topic]]<<connector-property-database-history-kafka-topic, `database.history.kafka.topic`>>
|
|The full name of the Kafka topic where the connector will store the database schema history.
|`database.history{zwsp}.kafka.bootstrap.servers`
|[[connector-property-database-history-kafka-bootstrap-servers]]<<connector-property-database-history-kafka-bootstrap-servers, `database.history{zwsp}.kafka.bootstrap.servers`>>
|
|A list of host/port pairs that the connector will use for establishing an initial connection to the Kafka cluster. This connection will be used for retrieving database schema history previously stored by the connector, and for writing each DDL statement read from the source database. This should point to the same Kafka cluster used by the Kafka Connect process.
|`database.whitelist`
|[[connector-property-database-whitelist]]<<connector-property-database-whitelist, `database.whitelist`>>
|_empty string_
|An optional comma-separated list of regular expressions that match database names to be monitored; any database name not included in the whitelist will be excluded from monitoring. By default all databases will be monitored. May not be used with `database.blacklist`.
|`database.blacklist`
|[[connector-property-database-blacklist]]<<connector-property-database-blacklist, `database.blacklist`>>
|_empty string_
|An optional comma-separated list of regular expressions that match database names to be excluded from monitoring; any database name not included in the blacklist will be monitored. May not be used with `database.whitelist`.
|`table.whitelist`
|[[connector-property-table-whitelist]]<<connector-property-table-whitelist, `table.whitelist`>>
|_empty string_
|An optional comma-separated list of regular expressions that match fully-qualified table identifiers for tables to be monitored; any table not included in the whitelist will be excluded from monitoring. Each identifier is of the form _databaseName_._tableName_. By default the connector will monitor every non-system table in each monitored database. May not be used with `table.blacklist`.
|`table.blacklist`
|[[connector-property-table-blacklist]]<<connector-property-table-blacklist, `table.blacklist`>>
|_empty string_
|An optional comma-separated list of regular expressions that match fully-qualified table identifiers for tables to be excluded from monitoring; any table not included in the blacklist will be monitored. Each identifier is of the form _databaseName_._tableName_. May not be used with `table.whitelist`.
|`column.blacklist`
|[[connector-property-column-blacklist]]<<connector-property-column-blacklist, `column.blacklist`>>
|_empty string_
|An optional comma-separated list of regular expressions that match the fully-qualified names of columns that should be excluded from change event message values. Fully-qualified names for columns are of the form _databaseName_._tableName_._columnName_, or _databaseName_._schemaName_._tableName_._columnName_.
|`column.truncate.to._length_.chars`
|[[connector-property-column-truncate-to-length-chars]]<<connector-property-column-truncate-to-length-chars, `column.truncate.to._length_.chars`>>
|_n/a_
|An optional comma-separated list of regular expressions that match the fully-qualified names of character-based columns whose values should be truncated in the change event message values if the field values are longer than the specified number of characters. Multiple properties with different lengths can be used in a single configuration, although in each the length must be a positive integer. Fully-qualified names for columns are of the form _databaseName_._tableName_._columnName_, or _databaseName_._schemaName_._tableName_._columnName_.
|`column.mask.with._length_.chars`
|[[connector-property-column-mask-with-length-chars]]<<connector-property-column-mask-with-length-chars, `column.mask.with._length_.chars`>>
|_n/a_
|An optional comma-separated list of regular expressions that match the fully-qualified names of character-based columns whose values should be replaced in the change event message values with a field value consisting of the specified number of asterisk (`*`) characters. Multiple properties with different lengths can be used in a single configuration, although in each the length must be a positive integer or zero. Fully-qualified names for columns are of the form _databaseName_._tableName_._columnName_, or _databaseName_._schemaName_._tableName_._columnName_.
|`column.mask.hash._hashAlgorithm_.with.salt._salt_`
|[[connector-property-column-mask-hash]]<<connector-property-column-mask-hash, `column.mask.hash._hashAlgorithm_.with.salt._salt_`>>
|_n/a_
|An optional comma-separated list of regular expressions that match the fully-qualified names of character-based columns whose values should be pseudonyms in the change event message values with a field value consisting of the hashed value using the algorithm `_hashAlgorithm_` and salt `_salt_`.
Based on the used hash function referential integrity is kept while data is pseudonymized. Supported hash functions are described in the {link-java7-standard-names}[MessageDigest section] of the Java Cryptography Architecture Standard Algorithm Name Documentation.
@ -101,14 +101,14 @@ where `CzQMA0cB5K` is a randomly selected salt.
Note: Depending on the `_hashAlgorithm_` used, the `_salt_` selected and the actual data set, the resulting masked data set may not be completely anonymized.
|`column.propagate.source.type`
|[[connector-property-column-propagate-source-type]]<<connector-property-column-propagate-source-type, `column.propagate.source.type`>>
|_n/a_
|An optional comma-separated list of regular expressions that match the fully-qualified names of columns whose original type and length should be added as a parameter to the corresponding field schemas in the emitted change messages.
The schema parameters `pass:[_]pass:[_]{prodname}.source.column.type`, `pass:[_]pass:[_]{prodname}.source.column.length` and `pass:[_]{prodname}.source.column.scale` will be used to propagate the original type name and length (for variable-width types), respectively.
Useful to properly size corresponding columns in sink databases.
Fully-qualified names for columns are of the form _databaseName_._tableName_._columnName_, or _databaseName_._schemaName_._tableName_._columnName_.
|`datatype.propagate.source.type`
|[[connector-property-datatype-propagate-source-type]]<<connector-property-datatype-propagate-source-type, `datatype.propagate.source.type`>>
|_n/a_
|An optional comma-separated list of regular expressions that match the database-specific data type name of columns whose original type and length should be added as a parameter to the corresponding field schemas in the emitted change messages.
The schema parameters `pass:[_]pass:[_]debezium.source.column.type`, `pass:[_]pass:[_]debezium.source.column.length` and `pass:[_]pass:[_]debezium.source.column.scale` will be used to propagate the original type name and length (for variable-width types), respectively.
@ -116,7 +116,7 @@ Useful to properly size corresponding columns in sink databases.
Fully-qualified data type names are of the form _databaseName_._tableName_._typeName_, or _databaseName_._schemaName_._tableName_._typeName_.
See xref:how-the-mysql-connector-maps-data-types_{context}[] for the list of MySQL-specific data type names.
|`time.precision.mode`
|[[connector-property-time-precision-mode]]<<connector-property-time-precision-mode, `time.precision.mode`>>
|`adaptive_time{zwsp}_microseconds`
| Time, date, and timestamps can be represented with different kinds of precision, including: `adaptive_time_microseconds` (the default) captures the date, datetime and timestamp values exactly as in the database using either millisecond, microsecond, or nanosecond precision values based on the database column's type, with the exception of TIME type fields, which are always captured as microseconds;
ifndef::cdc-product[]
@ -125,77 +125,77 @@ ifndef::cdc-product[]
endif::cdc-product[]
or `connect` always represents time and timestamp values using Kafka Connect's built-in representations for Time, Date, and Timestamp, which uses millisecond precision regardless of the database columns' precision.
|`decimal.handling.mode`
|[[connector-property-decimal-handling-mode]]<<connector-property-decimal-handling-mode, `decimal.handling.mode`>>
|`precise`
| Specifies how the connector should handle values for `DECIMAL` and `NUMERIC` columns: `precise` (the default) represents them precisely using `java.math.BigDecimal` values represented in change events in a binary form; or `double` represents them using `double` values, which may result in a loss of precision but will be far easier to use. `string` option encodes values as formatted string which is easy to consume but a semantic information about the real type is lost.
|`bigint.unsigned.handling.mode`
|[[connector-property-bigint-unsigned-handling-mode]]<<connector-property-bigint-unsigned-handling-mode, `bigint.unsigned.handling.mode`>>
|`long`
| Specifies how BIGINT UNSIGNED columns should be represented in change events, including: `precise` uses `java.math.BigDecimal` to represent values, which are encoded in the change events using a binary representation and Kafka Connect's `org.apache.kafka.connect.data.Decimal` type; `long` (the default) represents values using Java's `long`, which may not offer the precision but will be far easier to use in consumers. `long` is usually the preferable setting. Only when working with values larger than 2^63, the `precise` setting should be used as those values can't be conveyed using `long`.
|`include.schema.changes`
|[[connector-property-include-schema-changes]]<<connector-property-include-schema-changes, `include.schema.changes`>>
|`true`
|Boolean value that specifies whether the connector should publish changes in the database schema to a Kafka topic with the same name as the database server ID. Each schema change will be recorded using a key that contains the database name and whose value includes the DDL statement(s). This is independent of how the connector internally records database history. The default is `true`.
|`include.query`
|[[connector-property-include-query]]<<connector-property-include-query, `include.query`>>
|`false`
|Boolean value that specifies whether the connector should include the original SQL query that generated the change event. +
Note: This option requires MySQL be configured with the binlog_rows_query_log_events option set to ON. Query will not be present for events generated from the snapshot process. +
WARNING: Enabling this option may expose tables or fields explicitly blacklisted or masked by including the original SQL statement in the change event. For this reason this option is defaulted to 'false'.
|`event.processing{zwsp}.failure.handling.mode`
|[[connector-property-event-processing-failure-handling-mode]]<<connector-property-event-processing-failure-handling-mode, `event.processing{zwsp}.failure.handling.mode`>>
|`fail`
| Specifies how the connector should react to exceptions during deserialization of binlog events.
`fail` will propagate the exception (indicating the problematic event and its binlog offset), causing the connector to stop. +
`warn` will cause the problematic event to be skipped and the problematic event and its binlog offset to be logged. +
`skip` will cause problematic event will be skipped.
|`inconsistent.schema.handling.mode`
|[[connector-property-inconsistent-schema-handling-mode]]<<connector-property-inconsistent-schema-handling-mode, `inconsistent.schema.handling.mode`>>
|`fail`
| Specifies how the connector should react to binlog events that relate to tables that are not present in internal schema representation (i.e. internal representation is not consistent with database)
`fail` will throw an exception (indicating the problematic event and its binlog offset), causing the connector to stop. +
`warn` will cause the problematic event to be skipped and the problematic event and its binlog offset to be logged. +
`skip` will cause the problematic event to be skipped.
|`max.queue.size`
|[[connector-property-max-queue-size]]<<connector-property-max-queue-size, `max.queue.size`>>
|`8192`
|Positive integer value that specifies the maximum size of the blocking queue into which change events read from the database log are placed before they are written to Kafka. This queue can provide backpressure to the binlog reader when, for example, writes to Kafka are slower or if Kafka is not available. Events that appear in the queue are not included in the offsets periodically recorded by this connector. Defaults to 8192, and should always be larger than the maximum batch size specified in the `max.batch.size` property.
|`max.batch.size`
|[[connector-property-max-batch-size]]<<connector-property-max-batch-size, `max.batch.size`>>
|`2048`
|Positive integer value that specifies the maximum size of each batch of events that should be processed during each iteration of this connector. Defaults to 2048.
|`poll.interval.ms`
|[[connector-property-poll-interval-ms]]<<connector-property-poll-interval-ms, `poll.interval.ms`>>
|`1000`
|Positive integer value that specifies the number of milliseconds the connector should wait during each iteration for new change events to appear. Defaults to 1000 milliseconds, or 1 second.
|`connect.timeout.ms`
|[[connector-property-connect-timeout-ms]]<<connector-property-connect-timeout-ms, `connect.timeout.ms`>>
|`30000`
|A positive integer value that specifies the maximum time in milliseconds this connector should wait after trying to connect to the MySQL database server before timing out. Defaults to 30 seconds.
|`gtid.source.includes`
|[[connector-property-gtid-source-includes]]<<connector-property-gtid-source-includes, `gtid.source.includes`>>
|
|A comma-separated list of regular expressions that match source UUIDs in the GTID set used to find the binlog position in the MySQL server. Only the GTID ranges that have sources matching one of these include patterns will be used. May not be used with `gtid.source.excludes`.
|`gtid.source.excludes`
|[[connector-property-gtid-source-excludes]]<<connector-property-gtid-source-excludes, `gtid.source.excludes`>>
|
|A comma-separated list of regular expressions that match source UUIDs in the GTID set used to find the binlog position in the MySQL server. Only the GTID ranges that have sources matching none of these exclude patterns will be used. May not be used with `gtid.source.includes`.
ifndef::cdc-product[]
// Do not include deprecated content in downstream doc
|`gtid.new.channel.position` +
|[[connector-property-gtid-new-channel-position]]<<connector-property-gtid-new-channel-position, `gtid.new.channel.position`>> +
_deprecated and scheduled for removal_
|`earliest`
| When set to `latest`, when the connector sees a new GTID channel, it will start consuming from the last executed transaction in that GTID channel. If set to `earliest` (default), the connector starts reading that channel from the first available (not purged) GTID position. `earliest` is useful when you have a active-passive MySQL setup where {prodname} is connected to master, in this case during failover the slave with new UUID (and GTID channel) starts receiving writes before {prodname} is connected. These writes would be lost when using `latest`.
endif::cdc-product[]
|`tombstones.on.delete`
|[[connector-property-tombstones-on-delete]]<<connector-property-tombstones-on-delete, `tombstones.on.delete`>>
|`true`
| Controls whether a tombstone event should be generated after a delete event. +
When `true` the delete operations are represented by a delete event and a subsequent tombstone event. When `false` only a delete event is sent. +
Emitting the tombstone event (the default behavior) allows Kafka to completely delete all events pertaining to the given key once the source record got deleted.
|`message.key.columns`
|[[connector-property-message-key-columns]]<<connector-property-message-key-columns, `message.key.columns`>>
|_empty string_
| A semi-colon list of regular expressions that match fully-qualified tables and columns to map a primary key. +
Each item (regular expression) must match the `<fully-qualified table>:<a comma-separated list of columns>` representing the custom key. +
@ -210,35 +210,35 @@ Fully-qualified tables could be defined as `DB_NAME.TABLE_NAME` or `SCHEMA_NAME.
|===
|Property |Default |Description
|`connect.keep.alive`
|[[connector-property-connect-keep-alive]]<<connector-property-connect-keep-alive, `connect.keep.alive`>>
|`true`
|A boolean value that specifies whether a separate thread should be used to ensure the connection to the MySQL server/cluster is kept alive.
|`table.ignore.builtin`
|[[connector-property-table-ignore-builtin]]<<connector-property-table-ignore-builtin, `table.ignore.builtin`>>
|`true`
|Boolean value that specifies whether built-in system tables should be ignored. This applies regardless of the table whitelist or blacklists. By default system tables are excluded from monitoring, and no events are generated when changes are made to any of the system tables.
|`database.history.kafka.recovery.poll.interval.ms`
|[[connector-property-database-history-kafka-recovery-poll-interval-ms]]<<connector-property-database-history-kafka-recovery-poll-interval-ms, `database.history.kafka.recovery.poll.interval.ms`>>
|`100`
|An integer value that specifies the maximum number of milliseconds the connector should wait during startup/recovery while polling for persisted data. The default is 100ms.
|`database.history.kafka.recovery.attempts`
|[[connector-property-database-history-kafka-recovery-attempts]]<<connector-property-database-history-kafka-recovery-attempts, `database.history.kafka.recovery.attempts`>>
|`4`
|The maximum number of times that the connector should attempt to read persisted history data before the connector recovery fails with an error. The maximum amount of time to wait after receiving no data is `recovery.attempts` x `recovery.poll.interval.ms`.
|`database.history.skip.unparseable.ddl`
|[[connector-property-database-history-skip-unparseable-ddl]]<<connector-property-database-history-skip-unparseable-ddl, `database.history.skip.unparseable.ddl`>>
|`false`
|Boolean value that specifies if connector should ignore malformed or unknown database statements or stop processing and let operator to fix the issue.
The safe default is `false`.
Skipping should be used only with care as it can lead to data loss or mangling when binlog is processed.
|`database.history.store.only.monitored.tables.ddl`
|[[connector-property-database-history-store-only-monitored-tables-ddl]]<<connector-property-database-history-store-only-monitored-tables-ddl, `database.history.store.only.monitored.tables.ddl`>>
|`false`
|Boolean value that specifies if connector should should record all DDL statements or (when `true`) only those that are relevant to tables that are monitored by {prodname} (via filter configuration).
The safe default is `false`.
This feature should be used only with care as the missing data might be necessary when the filters are changed.
|`database.ssl.mode`
|[[connector-property-database-ssl-mode]]<<connector-property-database-ssl-mode, `database.ssl.mode`>>
|`disabled`
|Specifies whether to use an encrypted connection. The default is `disabled`, and specifies to use an unencrypted connection.
@ -250,7 +250,7 @@ The `verify_ca` option behaves like `required` but additionally it verifies the
The `verify_identity` option behaves like `verify_ca` but additionally verifies that the server certificate matches the host of the remote connection.
|`binlog.buffer.size`
|[[connector-property-binlog-buffer-size]]<<connector-property-binlog-buffer-size, `binlog.buffer.size`>>
|0
|The size of a look-ahead buffer used by the binlog reader. +
Under specific conditions it is possible that MySQL binlog contains uncommitted data finished by a `ROLLBACK` statement.
@ -261,13 +261,13 @@ If the size of transaction is larger than the buffer then {prodname} needs to re
Disabled by default. +
_Note:_ This feature should be considered an incubating one. We need a feedback from customers but it is expected that it is not completely polished.
|`snapshot.mode`
|[[connector-property-snapshot-mode]]<<connector-property-snapshot-mode, `snapshot.mode`>>
|`initial`
|Specifies the criteria for running a snapshot upon startup of the connector. The default is `initial`, and specifies the connector can run a snapshot only when no offsets have been recorded for the logical server name. The `when_needed` option specifies that the connector run a snapshot upon startup whenever it deems it necessary (when no offsets are available, or when a previously recorded offset specifies a binlog location or GTID that is not available in the server). The `never` option specifies that the connect should never use snapshots and that upon first startup with a logical server name the connector should read from the beginning of the binlog; this should be used with care, as it is only valid when the binlog is guaranteed to contain the entire history of the database. If you don't need the topics to contain a consistent snapshot of the data but only need them to have the changes since the connector was started, you can use the `schema_only` option, where the connector only snapshots the schemas (not the data).
`schema_only_recovery` is a recovery option for an existing connector to recover a corrupted or lost database history topic, or to periodically "clean up" a database history topic (which requires infinite retention) that may be growing unexpectedly.
|`snapshot.locking.mode`
|[[connector-property-snapshot-locking-mode]]<<connector-property-snapshot-locking-mode, `snapshot.locking.mode`>>
|`minimal`
|Controls if and how long the connector holds onto the global MySQL read lock (preventing any updates to the database) while it is performing a snapshot. There are three possible values `minimal`, `extended`, and `none`. +
@ -277,50 +277,50 @@ _Note:_ This feature should be considered an incubating one. We need a feedback
`none` Will prevent the connector from acquiring any table locks during the snapshot process. This value can be used with all snapshot modes but it is safe to use if and _only_ if no schema changes are happening while the snapshot is taken. Note that for tables defined with MyISAM engine, the tables would still be locked despite this property being set as MyISAM acquires a table lock. This behaviour is unlike InnoDB engine which acquires row level locks.
|`snapshot.select.statement.overrides`
|[[connector-property-snapshot-select-statement-overrides]]<<connector-property-snapshot-select-statement-overrides, `snapshot.select.statement.overrides`>>
|
|Controls which rows from tables will be included in snapshot. +
This property contains a comma-separated list of fully-qualified tables _(DB_NAME.TABLE_NAME)_. Select statements for the individual tables are specified in further configuration properties, one for each table, identified by the id `snapshot.select.statement.overrides.[DB_NAME].[TABLE_NAME]`. The value of those properties is the SELECT statement to use when retrieving data from the specific table during snapshotting. _A possible use case for large append-only tables is setting a specific point where to start (resume) snapshotting, in case a previous snapshotting was interrupted._ +
*Note*: This setting has impact on snapshots only. Events captured from binlog are not affected by it at all.
|`min.row.count.to.stream.results`
|[[connector-property-min-row-count-to-stream-results]]<<connector-property-min-row-count-to-stream-results, `min.row.count.to.stream.results`>>
|`1000`
|During a snapshot operation, the connector will query each included table to produce a read event for all rows in that table. This parameter determines whether the MySQL connection will pull all results for a table into memory (which is fast but requires large amounts of memory), or whether the results will instead be streamed (can be slower, but will work for very large tables). The value specifies the minimum number of rows a table must contain before the connector will stream results, and defaults to 1,000. Set this parameter to '0' to skip all table size checks and always stream all results during a snapshot.
|`heartbeat.interval.ms`
|[[connector-property-heartbeat-interval-ms]]<<connector-property-heartbeat-interval-ms, `heartbeat.interval.ms`>>
|`0`
|Controls how frequently the heartbeat messages are sent. +
This property contains an interval in milli-seconds that defines how frequently the connector sends heartbeat messages into a heartbeat topic.
Set this parameter to `0` to not send heartbeat messages at all. +
Disabled by default.
|`heartbeat.topics.prefix`
|[[connector-property-heartbeat-topics-prefix]]<<connector-property-heartbeat-topics-prefix, `heartbeat.topics.prefix`>>
|`__debezium-heartbeat`
|Controls the naming of the topic to which heartbeat messages are sent. +
The topic is named according to the pattern `<heartbeat.topics.prefix>.<server.name>`.
|`database.initial.statements`
|[[connector-property-database-initial-statements]]<<connector-property-database-initial-statements, `database.initial.statements`>>
|
|A semicolon separated list of SQL statements to be executed when a JDBC connection (not the transaction log reading connection) to the database is established.
Use doubled semicolon (';;') to use a semicolon as a character and not as a delimiter. +
_Note: The connector may establish JDBC connections at its own discretion, so this should typically be used for configuration of session parameters only, but not for executing DML statements._
|`snapshot.delay.ms`
|[[connector-property-snapshot-delay-ms]]<<connector-property-snapshot-delay-ms, `snapshot.delay.ms`>>
|
|An interval in milli-seconds that the connector should wait before taking a snapshot after starting up; +
Can be used to avoid snapshot interruptions when starting multiple connectors in a cluster, which may cause re-balancing of connectors.
|`snapshot.fetch.size`
|[[connector-property-snapshot-fetch-size]]<<connector-property-snapshot-fetch-size, `snapshot.fetch.size`>>
|
|Specifies the maximum number of rows that should be read in one go from each table while taking a snapshot.
The connector will read the table contents in multiple batches of this size.
|`snapshot.lock.timeout.ms`
|[[connector-property-snapshot-lock-timeout-ms]]<<connector-property-snapshot-lock-timeout-ms, `snapshot.lock.timeout.ms`>>
|`10000`
|Positive integer value that specifies the maximum amount of time (in milliseconds) to wait to obtain table locks when performing a snapshot.
If table locks cannot be acquired in this time interval, the snapshot will fail. See xref:how-the-mysql-connector-performs-database-snapshots_{context}[How the MySQL connector performs database snapshots].
|`enable.time.adjuster`
|[[connector-property-enable-time-adjuster]]<<connector-property-enable-time-adjuster, `enable.time.adjuster`>>
|
|MySQL allows user to insert year value as either 2-digit or 4-digit.
In case of two digits the value is automatically mapped to 1970 - 2069 range.
@ -329,7 +329,7 @@ Set to `true` (the default) when {prodname} should do the conversion. +
Set to `false` when conversion is fully delegated to the database.
ifndef::cdc-product[]
|`source.struct.version`
|[[connector-property-source-struct-version]]<<connector-property-source-struct-version, `source.struct.version`>>
|v2
|Schema version for the `source` block in {prodname} events; {prodname} 0.10 introduced a few breaking +
changes to the structure of the `source` block in order to unify the exposed structure across
@ -338,11 +338,11 @@ By setting this option to `v1` the structure used in earlier versions can be pro
Note that this setting is not recommended and is planned for removal in a future {prodname} version.
endif::cdc-product[]
|`sanitize.field.names`
|[[connector-property-sanitize-field-names]]<<connector-property-sanitize-field-names, `sanitize.field.names`>>
|`true` when connector configuration explicitly specifies the `key.converter` or `value.converter` parameters to use Avro, otherwise defaults to `false`.
|Whether field names will be sanitized to adhere to Avro naming requirements.
|`skipped.operations`
|[[connector-property-skipped-operations]]<<connector-property-skipped-operations, `skipped.operations`>>
|
| comma-separated list of oplog operations that will be skipped during streaming.
The operations include: `c` for inserts, `u` for updates, and `d` for deletes.