// Category: debezium-using // Type: assembly [id="debezium-connector-for-mariadb"] = {prodname} connector for MariaDB :context: mariadb :data-collection: table :database-port: 3306 :mbean-name: {context} :connector-file: {context} :connector-class: MariaDb :connector-name: MariaDB :include-list-example: inventory.* ifdef::community[] :toc: :toc-placement: macro :linkattrs: :icons: font :source-highlighter: highlight.js :MARIADB: toc::[] endif::community[] include::{partialsdir}/modules/all-connectors/shared-mariadb-mysql.adoc[leveloffset=+1,tags=mariadb-mysql-intro] // Type: assembly // ModuleID: how-debezium-mariadb-connectors-work // Title: How {prodname} MariaDB connectors work [[how-the-mariadb-connector-works]] == How the connector works include::{partialsdir}/modules/all-connectors/shared-mariadb-mysql.adoc[leveloffset=+1,tags=how-connector-works-intro] // Type: concept // ModuleID: mariadb-topologies-supported-by-debezium-connectors // Title: MariaDB topologies supported by {prodname} connectors [id="supported-mariadb-topologies"] === Supported MariaDB topologies The {prodname} MariaDB connector supports the following MariaDB topologies: Standalone:: When a single MariaDB server is used, the server must have the binlog enabled so the {prodname} MariaDB connector can monitor the server. This is often acceptable, since the binary log can also be used as an incremental link:https://mariadb.com/kb/en/backup-and-restore-overview/[backup]. In this case, the MariaDB connector always connects to and follows this standalone MariaDB server instance. Primary and replica:: The {prodname} MariaDB connector can follow one of the primary servers or one of the replicas (_if that replica has its binlog enabled_), but the connector sees changes in only the cluster that is visible to that server. Generally, this is not a problem except for the multi-primary topologies. + The connector records its position in the server’s binlog, which is different on each server in the cluster. Therefore, the connector must follow just one MariaDB server instance. If that server fails, that server must be restarted or recovered before the connector can continue. High available clusters:: A variety of link:https://mariadb.com/docs/server/architecture/use-cases/high-availability/[high availability] solutions exist for MariaDB, and they make it significantly easier to tolerate and almost immediately recover from problems and failures. HA MariaDB clusters use GTIDs so that replicas are able to keep track of all changes on any of the primary servers. Multi-primary:: link:https://mariadb.com/kb/en/galera-cluster/[Galera cluster replication] uses one or more MariaDB replica nodes that each replicate from multiple primary servers. This is a powerful way to aggregate the replication of multiple MariaDB clusters. + A {prodname} MariaDB connector can use these multi-primary MariaDB replicas as sources, and can fail over to different multi-primary MariaDB replicas as long as the new replica is caught up to the old replica. That is, the new replica has all transactions that were seen on the first replica. This works even if the connector is using only a subset of databases and/or tables, as the connector can be configured to include or exclude specific GTID sources when attempting to reconnect to a new multi-primary MariaDB replica and find the correct position in the binlog. Hosted:: There is support for the {prodname} MariaDB connector to use hosted options such as Amazon RDS and Amazon Aurora. + Because these hosted options do not allow a global read lock, table-level locks are used to create the _consistent snapshot_. // Type: concept // Title: How {prodname} MariaDB connectors handle database schema changes [[mariadb-schema-history-topic]] === Schema history topic include::{partialsdir}/modules/all-connectors/shared-mariadb-mysql.adoc[leveloffset=+1,tags=schema-history] // Type: concept // Title: How {prodname} MariaDB connectors expose database schema changes [id="mariadb-schema-change-topic"] === Schema change topic include::{partialsdir}/modules/all-connectors/shared-mariadb-mysql.adoc[leveloffset=+1,tags=schema-change] // Type: concept // Title: How {prodname} MariaDB connectors perform database snapshots [[mariadb-snapshots]] === Snapshots include::{partialsdir}/modules/all-connectors/shared-mariadb-mysql.adoc[leveloffset=+1,tags=snapshots] // Type: concept // ModuleID: debezium-mariadb-description-of-the-workflow-for-initial-snapshots-that-use-a-global-read-lock [id="initial-snapshot-workflow-with-global-read-lock"] ==== Initial snapshots that use a global read lock include::{partialsdir}/modules/all-connectors/shared-mariadb-mysql.adoc[leveloffset=+1,tags=initial-snap-global] // Type: concept // ModuleID: workflow-of-mariadb-connector-initial-snapshots-that-use-table-level-locks [id="initial-snapshot-workflow-with-table-level-locks"] ==== Initial snapshots that use table-level locks include::{partialsdir}/modules/all-connectors/shared-mariadb-mysql.adoc[leveloffset=+1,tags=initial-snap-table-lock] // ModuleID: mariadb-description-of-why-initial-snapshots-capture-the-schema-history-for-all-tables // Title: Description of why initial snapshots capture the schema history for all tables // Type: concept [id="understanding-why-initial-snapshots-capture-the-schema-history-for-all-tables"] ==== Understanding why initial snapshots capture the schema history for all tables include::{partialsdir}/modules/all-connectors/shared-mariadb-mysql.adoc[leveloffset=+1,tags=why-snapshot-captures-history-all] // Type: procedure [id="mariadb-capturing-data-from-tables-not-captured-by-the-initial-snapshot-no-schema-change"] ==== Capturing data from tables not captured by the initial snapshot (no schema change) include::{partialsdir}/modules/all-connectors/shared-mariadb-mysql.adoc[leveloffset=+1,tags=cap-tbls-not-in-initial-no-schema-chg] // Type: procedure [id="mariadb-capturing-data-from-new-tables-with-schema-changes"] ==== Capturing data from tables not captured by the initial snapshot (schema change) include::{partialsdir}/modules/all-connectors/shared-mariadb-mysql.adoc[leveloffset=+1,tags=cap-tbls-not-in-initial-schema-chg] // Type: concept // ModuleID: debezium-mariadb-ad-hoc-snapshots [id="mariadb-ad-hoc-snapshots"] === Ad hoc snapshots include::{partialsdir}/modules/all-connectors/con-connector-ad-hoc-snapshots.adoc[leveloffset=+1] // Type: assembly // ModuleID: debezium-mariadb-incremental-snapshots [id="debezium-mariadb-incremental-snapshots"] === Incremental snapshots include::{partialsdir}/modules/all-connectors/con-connector-incremental-snapshot.adoc[leveloffset=+1] // Type: procedure // ModuleID: debezium-mariadb-triggering-an-incremental-snapshot [id="mariadb-triggering-an-incremental-snapshot"] ==== Triggering an incremental snapshot include::{partialsdir}/modules/all-connectors/proc-triggering-an-incremental-snapshot-sql.adoc[leveloffset=+1] // Type: procedure // ModuleID: debezium-mariadb-using-the-kafka-signaling-channel-to-trigger-an-incremental-snapshot [id="mariadb-triggering-an-incremental-snapshot-kafka"] ==== Using the Kafka signaling channel to trigger an incremental snapshot include::{partialsdir}/modules/all-connectors/proc-triggering-an-incremental-snapshot-kafka.adoc[leveloffset=+1] // Type: procedure // ModuleID: debezium-mariadb-stopping-an-incremental-snapshot [id="mariadb-stopping-an-incremental-snapshot"] ==== Stopping an incremental snapshot include::{partialsdir}/modules/all-connectors/proc-stopping-an-incremental-snapshot-sql.adoc[leveloffset=+1] // Type: procedure // ModuleID: debezium-mariadb-using-the-kafka-signaling-channel-to-stop-an-incremental-snapshot [id="mariadb-stopping-an-incremental-snapshot-kafka"] ==== Using the Kafka signaling channel to stop an incremental snapshot include::{partialsdir}/modules/all-connectors/proc-stopping-an-incremental-snapshot-kafka.adoc[leveloffset=+1] ifdef::community[] [id="mariadb-read-only-incremental-snapshots"] ==== Read-only incremental snapshots include::{partialsdir}/modules/all-connectors/shared-mariadb-mysql.adoc[leveloffset=+1,tags=readonly-incremental] ==== Ad hoc read-only incremental snapshots include::{partialsdir}/modules/all-connectors/shared-mariadb-mysql.adoc[leveloffset=+1,tags=adhoc-ro-incremental] endif::community[] ifdef::community[] [[connector-custom-snapshot]] === Custom snapshotter SPI include::{partialsdir}/modules/all-connectors/custom-snapshotter-spi.adoc[leveloffset=+3] endif::community[] // Type: concept [id="mariadb-blocking-snapshots"] === Blocking snapshots include::{partialsdir}/modules/all-connectors/con-connector-blocking-snapshot.adoc[leveloffset=+3] // Type: concept // Title: Default names of Kafka topics that receive {prodname} MariaDB change event records [[mariadb-topic-names]] === Topic names include::{partialsdir}/modules/all-connectors/shared-mariadb-mysql.adoc[leveloffset=+1,tags=topic-names] [[mariadb-transaction-metadata]] === Transaction metadata include::{partialsdir}/modules/all-connectors/shared-mariadb-mysql.adoc[leveloffset=+1,tags=trans-meta] // Type: assembly // ModuleID: descriptions-of-debezium-mariadb-connector-data-change-events // Title: Descriptions of {prodname} MariaDB connector data change events [[mariadb-events]] == Data change events include::{partialsdir}/modules/all-connectors/shared-mariadb-mysql.adoc[leveloffset=+1,tags=data-chg-events] // Type: concept // ModuleID: about-keys-in-debezium-mariadb-change-events // Title: About keys in {prodname} MariaDB change events [[mariadb-change-event-keys]] === Change event keys include::{partialsdir}/modules/all-connectors/shared-mariadb-mysql.adoc[leveloffset=+1,tags=event-keys] // Type: concept // ModuleID: about-values-in-debezium-mariadb-change-events // Title: About values in {prodname} MariaDB change events [[mariadb-change-event-values]] === Change event values include::{partialsdir}/modules/all-connectors/shared-mariadb-mysql.adoc[leveloffset=+1,tags=event-values] // Type: continue [id="mariadb-create-events"] === _create_ events include::{partialsdir}/modules/all-connectors/shared-mariadb-mysql.adoc[leveloffset=+1,tags=create-events] // Type: continue [id="mariadb-update-events"] === _update_ events include::{partialsdir}/modules/all-connectors/shared-mariadb-mysql.adoc[leveloffset=+1,tags=update-events] // Type: continue [id="mariadb-primary-key-updates"] === Primary key updates include::{partialsdir}/modules/all-connectors/shared-mariadb-mysql.adoc[leveloffset=+1,tags=pkey-updates] // Type: continue [id="mariadb-delete-events"] === _delete_ events include::{partialsdir}/modules/all-connectors/shared-mariadb-mysql.adoc[leveloffset=+1,tags=del-events] // Type: continue [id="mariadb-tombstone-events"] === Tombstone events include::{partialsdir}/modules/all-connectors/shared-mariadb-mysql.adoc[leveloffset=+1,tags=tombstone-events] // Type: continue [id="mariadb-truncate-events"] === _truncate_ events include::{partialsdir}/modules/all-connectors/shared-mariadb-mysql.adoc[leveloffset=+1,tags=truncate-events] // Type: reference // ModuleID: how-debezium-mariadb-connectors-map-data-types // Title: How {prodname} MariaDB connectors map data types [[mariadb-data-types]] == Data type mappings include::{partialsdir}/modules/all-connectors/shared-mariadb-mysql.adoc[leveloffset=+1,tags=data-type-mappings] [id="mariadb-basic-types"] === Basic types include::{partialsdir}/modules/all-connectors/shared-mariadb-mysql.adoc[leveloffset=+1,tags=basic-data-types] [id="mariadb-temporal-types"] === Temporal types include::{partialsdir}/modules/all-connectors/shared-mariadb-mysql.adoc[leveloffset=+1,tags=temporal-data-types] [id="mariadb-decimal-types"] === Decimal types include::{partialsdir}/modules/all-connectors/shared-mariadb-mysql.adoc[leveloffset=+1,tags=decimal-data-types] [id="mariadb-boolean-values"] === Boolean values include::{partialsdir}/modules/all-connectors/shared-mariadb-mysql.adoc[leveloffset=+1,tags=boolean-values] [id="mariadb-spatial-types"] === Spatial types include::{partialsdir}/modules/all-connectors/shared-mariadb-mysql.adoc[leveloffset=+1,tags=spatial-data-types] // Type: concept // Title: Custom converters for mapping {connector-name} data to alternative data types [id="debezium-mariadb-connector-converters"] == Custom converters include::{partialsdir}/modules/all-connectors/shared-mariadb-mysql.adoc[leveloffset=+1,tags=custom-converters] === `TINYINT(1)` to Boolean include::{partialsdir}/modules/all-connectors/shared-mariadb-mysql.adoc[leveloffset=+1,tags=tinyint-boolean-conv] === JDBC sink data types include::{partialsdir}/modules/all-connectors/shared-mariadb-mysql.adoc[leveloffset=+1,tags=jdbc-sink-data-types] // Type: assembly // ModuleID: setting-up-mariadb-to-run-a-debezium-connector // Title: Setting up MariaDB to run a {prodname} connector [[setting-up-mariadb]] == Setting up MariaDB include::{partialsdir}/modules/all-connectors/shared-mariadb-mysql.adoc[leveloffset=+1,tags=setting-up-db] // Type: procedure // ModuleID: creating-a-mariadb-user-for-a-debezium-connector // Title: Creating a MariaDB user for a {prodname} connector [[mariadb-creating-user]] === Creating a user include::{partialsdir}/modules/all-connectors/shared-mariadb-mysql.adoc[leveloffset=+1,tags=creating-a-db-user] // Type: procedure // ModuleID: enabling-the-mariadb-binlog-for-debezium // Title: Enabling the MariaDB binlog for {prodname} [[enable-mariadb-binlog]] === Enabling the binlog include::{partialsdir}/modules/all-connectors/shared-mariadb-mysql.adoc[leveloffset=+1,tags=enabling-binlog] // Type: procedure // ModuleID: enabling-mariadb-gtids-for-debezium // Title: Enabling MariaDB Global Transaction Identifiers for {prodname} [[enable-mariadb-gtids]] === Enabling GTIDs Global transaction identifiers (GTIDs) uniquely identify transactions that occur on a server within a cluster. Although not required for a {prodname} MariaDB connector, using GTIDs simplifies replication and enables you to more easily confirm if primary and replica servers are consistent. For MariaDB, GTIDs are enabled by default and no additional configuration is necessary. // Type: procedure // ModuleID: configuring-mariadb-session-timeouts-for-debezium // Title: Configuring MariaDB session timeouts for {prodname} [[mariadb-session-timeouts]] === Configuring session timeouts include::{partialsdir}/modules/all-connectors/shared-mariadb-mysql.adoc[leveloffset=+1,tags=cfg-session-timeouts] // Type: procedure // ModuleID: enabling-query-log-events-for-debezium-mariadb-connectors // Title: Enabling query log events for {prodname} MariaDB connectors [[enable-query-log-events]] === Enabling query log events You might want to see the original `SQL` statement for each binlog event. Enabling the `binlog_annotate_row_events` option in the MariaDB configuration allows you to do this. .Prerequisites * A MariaDB server. * Basic knowledge of SQL commands. * Access to the MariaDB configuration file. .Procedure * Enable `binlog_annotate_row_events` in MariaDB: + [source,SQL] ---- mariadb> binlog_annotate_row_events=ON ---- + `binlog_annotate_row_events` is set to a value that enables/disables support for including the original `SQL` statement in the binlog entry. + ** `ON` = enabled ** `OFF` = disabled // Type: procedure // ModuleID: validate-binlog-row-value-options-for-debezium-mariadb-connectors // Title: validate binlog row value options for {prodname} MariaDB connectors [[validate-binlog-row-value-options]] === Validating binlog row value options include::{partialsdir}/modules/all-connectors/shared-mariadb-mysql.adoc[leveloffset=+1,tags=validate-binlog-row-options] // Type: assembly // ModuleID: deployment-of-debezium-mariadb-connectors // Title: Deployment of {prodname} MariaDB connectors [[mariadb-deploying-a-connector]] == Deployment include::{partialsdir}/modules/all-connectors/shared-mariadb-mysql.adoc[leveloffset=+1,tags=deployment] ifdef::product[] // Type: concept [id="openshift-streams-mariadb-connector-deployment"] === MariaDB connector deployment using {StreamsName} include::{partialsdir}/modules/all-connectors/con-connector-streams-deployment.adoc[leveloffset=+1] //Type: procedure [id="using-streams-to-deploy-a-debezium-mariadb-connector"] === Using {StreamsName} to deploy a {prodname} MariaDB connector include::{partialsdir}/modules/all-connectors/proc-using-streams-to-deploy-a-debezium-mysql-sqlserver-connector.adoc[leveloffset=+1] // Type: procedure // ModuleID: deploying-debezium-mariadb-connectors === Deploying {prodname} MariaDB connectors by building a custom Kafka Connect container image from a Dockerfile include::{partialsdir}/modules/all-connectors/shared-mariadb-mysql.adoc[leveloffset=+1,tags=deploy-via-dockerfile] endif::product[] ifdef::community[] [[mariadb-example-configuration]] === MariaDB connector configuration example endif::community[] include::{partialsdir}/modules/all-connectors/shared-mariadb-mysql.adoc[leveloffset=+1,tags=connector-config-example] ifdef::community[] [[mariadb-adding-configuration]] === Adding connector configuration endif::community[] include::{partialsdir}/modules/all-connectors/shared-mariadb-mysql.adoc[leveloffset=+1,tags=add-connector-cfg] ifdef::product[] // Type: procedure [id="verifying-that-the-debezium-mariadb-connector-is-running"] === Verifying that the {prodname} MariaDB connector is running include::{partialsdir}/modules/all-connectors/proc-verifying-the-connector-deployment.adoc[leveloffset=+1] endif::product[] // Type: reference // Title: Descriptions of {prodname} MariaDB connector configuration properties [[mariadb-connector-properties]] === Connector properties include::{partialsdir}/modules/all-connectors/shared-mariadb-mysql.adoc[leveloffset=+1,tags=connector-props-intro] [id="mariadb-required-connector-configuration-properties"] ==== Required {prodname} MariaDB connector configuration properties include::{partialsdir}/modules/all-connectors/ref-mariadb-mysql-rqd-connector-cfg-props.adoc[leveloffset=+1] [id="mariadb-advanced-connector-configuration-properties"] ==== Advanced {prodname} MariaDB connector configuration properties include::{partialsdir}/modules/all-connectors/ref-mariadb-mysql-adv-connector-cfg-props.adoc[leveloffset=+1] [id="debezium-mariadb-connector-database-history-configuration-properties"] ==== {prodname} connector database schema history configuration properties include::{partialsdir}/modules/all-connectors/ref-connector-configuration-database-history-properties.adoc[leveloffset=+1] [id="debezium-mariadb-connector-kafka-signals-configuration-properties"] ==== {prodname} connector Kafka signals configuration properties include::{partialsdir}/modules/all-connectors/ref-connector-pass-through-kafka-signals-configuration-properties.adoc[leveloffset=+1] [id="debezium-mariadb-connector-kafka-notifications-configuration-properties"] ==== {prodname} connector sink notifications configuration properties include::{partialsdir}/modules/all-connectors/ref-connector-pass-through-kafka-notification-configuration-properties.adoc[leveloffset=+1] [id="debezium-mariadb-connector-pass-through-database-driver-configuration-properties"] ==== {prodname} connector pass-through database driver configuration properties include::{partialsdir}/modules/all-connectors/ref-connector-pass-through-database-driver-configuration-properties.adoc[leveloffset=+1] // Type: assembly // ModuleID: monitoring-debezium-mariadb-connector-performance // Title: Monitoring {prodname} MariaDB connector performance [[mariadb-monitoring]] == Monitoring include::{partialsdir}/modules/all-connectors/shared-mariadb-mysql.adoc[leveloffset=+1,tags=monitoring-intro] // Type: concept // ModuleID: monitoring-debezium-mariadb-connectors-customized-mbean-names // Title: Customized names for MariaDB connector snapshot and streaming MBean objects === Customized MBean names include::{partialsdir}/modules/all-connectors/frag-common-mbean-name.adoc[leveloffset=+1,tags=mbeans-shared] // Type: reference // ModuleID: monitoring-debezium-during-snapshots-of-mariadb-databases // Title: Monitoring {prodname} during snapshots of MariaDB databases [[mariadb-snapshot-metrics]] === Snapshot metrics include::{partialsdir}/modules/all-connectors/frag-common-mbean-name.adoc[leveloffset=+1,tags=common-snapshot] include::{partialsdir}/modules/all-connectors/ref-connector-monitoring-snapshot-metrics.adoc[leveloffset=+1] include::{partialsdir}/modules/all-connectors/ref-connector-monitoring-incremental-snapshot-metrics.adoc[leveloffset=+1] // Type: reference // ModuleID: monitoring-debezium-mariadb-connector-record-streaming // Title: Monitoring {prodname} MariaDB connector record streaming [[mariadb-streaming-metrics]] === Streaming metrics include::{partialsdir}/modules/all-connectors/shared-mariadb-mysql.adoc[leveloffset=+1,tags=monitoring-intro] include::{partialsdir}/modules/all-connectors/frag-common-mbean-name.adoc[leveloffset=+1,tags=common-streaming] include::{partialsdir}/modules/all-connectors/ref-connector-monitoring-streaming-metrics.adoc[leveloffset=+1] // Moved table of connector-specific metrics to conditonalized section of the preceding shared file. // Type: reference // ModuleID: monitoring-debezium-mariadb-connector-schema-history // Title: Monitoring {prodname} MariaDB connector schema history [[mariadb-schema-history-metrics]] === Schema history metrics include::{partialsdir}/modules/all-connectors/ref-connector-monitoring-schema-history-metrics.adoc[leveloffset=+1] // Type: concept // ModuleID: how-debezium-mariadb-connectors-handle-faults-and-problems // Title: How {prodname} MariaDB connectors handle faults and problems [[mariadb-when-things-go-wrong]] == Behavior when things go wrong include::{partialsdir}/modules/all-connectors/shared-mariadb-mysql.adoc[leveloffset=+1,tags=wrong-things]