34 lines
2.2 KiB
Plaintext
34 lines
2.2 KiB
Plaintext
[id="debezium-features"]
|
|
= {prodname} Features
|
|
|
|
:linkattrs:
|
|
:icons: font
|
|
|
|
{prodname} is a set of source connectors for Apache Kafka Connect,
|
|
ingesting changes from different databases using change data capture (CDC).
|
|
Unlike other approaches such as polling or dual writes,
|
|
log-based CDC as implemented by {prodname}:
|
|
|
|
* makes sure that *all data changes are captured*
|
|
* produces change events with a *very low delay* (e.g. ms range for MySQL or Postgres) while avoiding increased CPU usage of frequent polling
|
|
* requires *no changes to your data model* (such as "Last Updated" column)
|
|
* can *capture deletes*
|
|
* can *capture old record state and further metadata* such as transaction id and causing query (depending on the database's capabilities and configuration)
|
|
|
|
To learn more about the advantages of log-based CDC,
|
|
refer to link:/blog/2018/07/19/advantages-of-log-based-change-data-capture/[this blog post].
|
|
|
|
The actual change data capture feature of {prodname} is amended with a range of related capabilities and options:
|
|
|
|
* *Snapshots:* optionally, an initial snapshot of a database's current state can be taken if a connector gets started up and not all logs still exist (typically the case when the database has been running for some time and has discarded any transaction logs not needed any longer for transaction recovery or replication);
|
|
different modes exist for snapshotting, refer to the docs of the specific connector you're using to learn more
|
|
* *Filters:* the set of captured schemas, tables and columns can be configured via whitelist/blacklist filters
|
|
* *Masking:* the values from specific columns can be masked, e.g. for sensitive data
|
|
* *Monitoring:* most connectors can be monitored using JMX
|
|
* Different ready-to-use *message transformations*:
|
|
e.g. for {link-prefix}:{link-topic-routing}[message routing],
|
|
extraction of new record state ({link-prefix}:{link-event-flattening}[relational connectors], {link-prefix}:{link-mongodb-event-flattening}[MongoDB])
|
|
and {link-prefix}:{link-outbox-event-router}[routing of events] from a transactional outbox table
|
|
|
|
Refer to the {link-prefix}:{link-connectors}[connector documentation] for a list of all supported databases and detailed information about the features and configuration options of each connector.
|