mfvitale
c7a28021a6
DBZ-7308 Remove connector specific SnapshotLockProvider and SnapshotterServiceProvider
2024-03-25 05:46:46 +01:00
mfvitale
60a0b1b5c3
DBZ-7308 tableLockingStatement from SnapshotLock interface now takes just one table in input
2024-03-25 05:46:46 +01:00
mfvitale
ae9009303c
DBZ-7308 Move getSnapshottingTask to RelationalSnapshotChangeEventSource
2024-03-25 05:46:46 +01:00
mfvitale
265ba0f6e6
DBZ-7308 Check if configured snapshot mode permits streaming before starting it.
2024-03-25 05:46:46 +01:00
Chris Cranford
4356db3292
DBZ-7690 Fix history comparisons for SERVER_ID_KEY and TIMESTAMP_KEY
2024-03-21 08:41:47 +01:00
Chris Cranford
1822005d7c
DBZ-7665 Exclude jcl-over-slf4j dependency
2024-03-20 05:57:50 +01:00
harveyyue
48c67e40fb
DBZ-7618 Implement Versioned interfaces in Transformation and Converter plugins
2024-03-14 06:35:42 +01:00
Sean Wu
a71368cf62
DBZ-7594 Correct the negative value of data type TIME replicated from MySQL.
2024-03-12 08:01:04 +01:00
mfvitale
b8dcb958cd
DBZ-7608 Maintain order for schema events
2024-03-11 08:42:52 +01:00
mfvitale
5a45d8dcf4
DBZ-7608 Resolve performance issue during addSchemaEvent
2024-03-11 08:42:52 +01:00
Debezium Builder
2fb8fc3004
[maven-release-plugin] prepare for next development iteration
2024-03-06 07:47:33 +00:00
Debezium Builder
cd46b2b998
[maven-release-plugin] prepare release v2.6.0.Beta1
2024-03-06 07:47:33 +00:00
mfvitale
8a47b2333d
DBZ-7588 Disable debug prints on tests
2024-03-02 11:33:48 -05:00
mfvitale
e174eae235
DBZ-7461 Remove redundant log
2024-03-01 14:12:31 +01:00
mfvitale
4ade54351f
DBZ-7461 Rename SCHEMA_ONLY_RECOVERY to RECOVERY and SCHEMA_ONLY to NO_DATA
2024-03-01 14:12:31 +01:00
mfvitale
211675a355
DBZ-7461 Rename shouldSnapshot to shouldSnapshotData
2024-03-01 14:12:31 +01:00
mfvitale
562c999da4
DBZ-7461 Move out-of-the-box snapshotters to core
2024-03-01 14:12:31 +01:00
mfvitale
371905f135
DBZ-7461 Move snapshot mode validation logic to BaseSourceTask
2024-03-01 14:12:31 +01:00
mfvitale
99eee98e9d
DBZ-7461 Align common code for Snapshotters
2024-03-01 14:12:31 +01:00
mfvitale
ce1b5d7231
DBZ-7461 Support WhenNeededSnapshotter for PostgreSQL
2024-03-01 14:12:31 +01:00
mfvitale
b1a907757d
DBZ-7461 Support WhenNeededSnapshotter for Oracle
2024-03-01 14:12:31 +01:00
mfvitale
1f52b81b6d
DBZ-7461 Support AlwaysSnapshotter for MySQL
2024-03-01 14:12:31 +01:00
James Johnston
f632fa081e
DBZ-5071 Correctly handle NULL values in incremental snapshots
...
It turns out that the existing code for chunking a table when taking
an incremental snapshot was buggy and did not correctly handle NULL
values when building the chunk query. An example of such a situation
would be when the user has specified "message.key.columns" to reference
a column that is part of a PostgreSQL UNIQUE INDEX that was created with
the NULLS NOT DISTINCT option.
This commit updates the new AbstractChunkQueryBuilder so that it checks
whether a key column is optional. If it is, then additional will
appropriately consider NULL values when generating a chunk query using
"IS [NOT] NULL" clauses.
One complication is that different database engines have different
sorting behavior of ORDER BY. It is apparently not well-defined by the
SQL standard. Some databases consider NULL values to be higher than any
non-NULL values, and others consider them to be lower.
To handle this situation, a new nullsSortLast() function is added to the
JdbcConnection class. By default, it returns an empty value, indicating
that the behavior of the database engine is unknown. When an optional
field is encountered by AbstractChunkQueryBuilder in this situation, we
throw an error because we don't actually know how to correctly chunk the
query: there's no safe assumption that can be made here.
Derived JdbcConnection classes can then override the nullsSortLast
function, and return a value indicating the actual behavior of that
database engine. When this is done, the AbstractChunkQueryBuilder then
knows how to correctly build a chunk query that can handle NULL values.
To help test this, new tests have been added to
AbstractIncrementalSnapshotTest. First, the existing insertsWithoutPks
test has been moved and deduplicated from MySQL and PostgreSQL so that
the test case can be reused on other engines. Second, a new
insertsWithoutPksAndNull test is run, which inserts data with NULL
values in the message key columns. To demonstrate that chunk queries
are being correctly generated for practically every case, the
INCREMENTAL_SNAPSHOT_CHUNK_SIZE is set to 1 so that NULL values are not
returned in the middle of a chunk, which can cause us to skip testing
the code we need to test.
2024-02-29 13:36:26 +01:00
Vojtech Juranek
83c339c2ae
[ci] Mark ReadOnlyIncrementalSnapshotIT#testStopSnapshotKafkaSignal as flaky
2024-02-27 16:38:15 -05:00
Bue-Von-Hun
94e24532f6
DBZ-2015 Remove redundant fields
2024-02-27 15:30:48 +01:00
Lourens Naude
bba11001b3
DBZ-7567 Fix null event timestamp possible from FORMAT_DESCRIPTION and PREVIOUS_GTIDS events in MySqlStreamingChangeEventSource::setEventTimestamp
2024-02-27 10:05:34 +01:00
rkerner
1e69e40ec2
DBZ-7416 Fix duplicate SMTs sometimes returned by Kafka Connect. Moved deduplication from Map to LinkedHashSet.
...
+ minor fixes added for cleanup and centralization of common code
closes https://issues.redhat.com/browse/DBZ-7416
2024-02-22 13:34:08 -05:00
Jiri Pechanec
68b6591142
DBZ-7416 Fix duplicate SMTs sometimes returned by Kafka Connect. Moved deduplication from Map to LinkedHashSet.
...
closes https://issues.redhat.com/browse/DBZ-7416
2024-02-22 13:34:08 -05:00
mfvitale
08e46815e4
DBZ-7508 Exit from readChunk after createDataEventsForTable if snapshot is not running anymore
2024-02-22 12:13:37 -05:00
Chris Cranford
6af8a7c70b
[ci] Add Flaky to shouldCreateSnapshotOfSingleDatabaseWithoutGlobalLock (MySQL)
2024-02-21 09:00:45 -05:00
mfvitale
7ed5649e07
DBZ-7302 Implement Snapshotter for Oracle
2024-02-20 14:45:59 +01:00
mfvitale
9fe60a698d
DBZ-7302 Move snapshot.locking.mode.custom.name, snapshot.query.mode and snapshot.query.mode.custom.name to CommonConnectorConfig
2024-02-20 14:45:59 +01:00
Lourens Naude
b93218e3ce
DBZ-7500 prefer a more concise and self documenting version of setEventTimestamp as proposed by Chris Cranford
2024-02-17 14:22:43 -05:00
Lourens Naude
ae9539ab93
DBZ-7500 Fallback to seconds resolution for MariaDB as high resolution timestamps on MARIADB_GTID events are not supported
2024-02-17 14:22:43 -05:00
Lourens Naude
70aa83b371
DBZ-7500 Fix MySQL 8 event timestamp resolution logic error where fallback to seconds occurs erroneously for non-GTID events
2024-02-17 14:22:43 -05:00
Chris Cranford
928aa26cdb
DBZ-7107 Fix source info test failures
2024-02-16 12:52:20 +01:00
Debezium Builder
10e327602c
[maven-release-plugin] prepare for next development iteration
2024-02-13 09:20:04 +00:00
Debezium Builder
0c5b05738c
[maven-release-plugin] prepare release v2.6.0.Alpha2
2024-02-13 09:20:04 +00:00
Vojtech Juranek
eef8ee4cea
DBZ-7024 Move async engine into separate package
2024-02-12 13:43:21 +01:00
Vojtech Juranek
78f7c40410
DBZ-7024 Don't stop engine before tasks are fully running
2024-02-12 13:43:21 +01:00
Vojtech Juranek
834b522739
DBZ-7024 Fix ReadOnlyIncrementalSnapshotIT after switch to async engine
2024-02-12 13:43:21 +01:00
Vojtech Juranek
6c71cb7f88
DBZ-7024 Fix StreamingSourceIT after switch to async engine
2024-02-12 13:43:21 +01:00
Vojtech Juranek
ec58dd6a1c
DBZ-7024 Fix MySqlRestartIT after switch to async engine
2024-02-12 13:43:21 +01:00
Vojtech Juranek
3a902ff1d6
DBZ-7024 Fix MySqlConnectorSchemaValidateIT after switch to async engine
2024-02-12 13:43:21 +01:00
Vojtech Juranek
0137d5f15e
DBZ-7024 Switch MySQL tests inheriting from AbstractConnectorTest to async engine
2024-02-12 13:43:21 +01:00
Vojtech Juranek
4452e3d095
DBZ-7024 Move EmbeddedConfig into separate class
...
`EmbeddedConfig` needs to be shared with other implmentations of
`DebeziumEngine` as long as Debezium embedded depends on the Kafka
model.
2024-02-12 13:43:21 +01:00
mfvitale
3ff775ea6c
DBZ-7301 Add test for a custom Snapshotter and SnapshotQuery
2024-02-06 07:12:42 +01:00
mfvitale
a8a07e35f1
DBZ-7301 Implement SnapshotLock for MySQL connector
2024-02-06 07:12:42 +01:00
mfvitale
6cb01f67d1
DBZ-7301 Implement SnapshotQuery for MySQL connector
2024-02-06 07:12:42 +01:00
mfvitale
0e68bc642d
DBZ-7301 Support Snapshotter for MariaDB
2024-02-06 07:12:42 +01:00
mfvitale
1cdf2836dd
DBZ-7301 Implement Snapshotter for MySQL connector
2024-02-06 07:12:42 +01:00
mfvitale
f78d0cbd19
DBZ-7301 Add SnapshotterService to ChangeEventSourceFactory and SnapshotChangeEventSource
2024-02-06 07:12:42 +01:00
mfvitale
d0e4ad7e14
DBZ-7441 Postpone SignalProcessor start after streaming is initialized
...
This will avoid that channels that not depends on the event streaming like the 'source' channel, will start processing signals before the IncrementalSnapshotChangeEventSource is initialized.
2024-02-05 14:06:40 +01:00
nicholas-fwang
4b62ee44f7
DBZ-7143 Fix test code for database replica environment
2024-01-29 13:48:19 +01:00
nicholas-fwang
1f8c957d18
DBZ-7143 Use log with trace record
2024-01-29 13:48:19 +01:00
nicholas-fwang
be35033d6b
DBZ-7143 fix format
2024-01-29 13:48:19 +01:00
nicholas-fwang
fcee5e7bdc
DBZ-7143 Add constructor for testing in MysqlValueConverters
2024-01-29 13:48:19 +01:00
nicholas-fwang
d710ee6b9f
DBZ-7143 Rollback ValueConverter and handle when parse default value
2024-01-29 13:48:19 +01:00
nicholas-fwang
5aae5f51f4
DBZ-7143 Move event converting failure handler to ValueConverter interface.
2024-01-29 13:48:19 +01:00
nicholas-fwang
a36caea1de
DBZ-7143 Add table DATE_TIME_TABLE_CONNECT_MODE to avoid consume error when connect mode
2024-01-29 13:48:19 +01:00
nicholas-fwang
7d99605886
DBZ-7143 Add case when EventConvertingFailureHandlingMode is null
2024-01-29 13:48:19 +01:00
nicholas-fwang
4f2bbb023a
DBZ-7143 fix checkstyle format
2024-01-29 13:48:19 +01:00
nicholas-fwang
57a46943af
DBZ-7143 throw exception in JdbcValueConverters
2024-01-29 13:48:19 +01:00
nicholas-fwang
eeea0f1e70
DBZ-7143 Add event.converting.failure.handling.mode option
2024-01-29 13:48:19 +01:00
Animesh Kumar
986303babc
DBZ-7376 Fix logging for schema only recovery in mysql
2024-01-23 15:01:28 +01:00
Debezium Builder
65d63ed42d
[maven-release-plugin] prepare for next development iteration
2024-01-21 10:12:45 +00:00
Debezium Builder
485fa82a8f
[maven-release-plugin] prepare release v2.6.0.Alpha1
2024-01-21 10:12:44 +00:00
Vojtech Juranek
33c8430a84
DBZ-7356 Update comment to include relevant DBZ issue
2024-01-17 09:45:43 +01:00
Vojtech Juranek
5ca719d1d0
DBZ-7355 Skip Avro serialization validation
...
Test fails due to bug in Apicurio
https://github.com/Apicurio/apicurio-registry/issues/2980
2024-01-17 09:45:43 +01:00
Peter Hamer
e2c0a59ec3
DBZ-7347 Update notification tests.
2024-01-17 09:42:27 +01:00
Jakub Cechacek
5dbe073d3b
DBZ-7260 Offset value conversion extracted into common util class
2024-01-11 09:58:24 +01:00
mfvitale
20fbe73bc8
DBZ-7311 Use a more meaningful name for identify an on demand blocking snapshot
2024-01-08 12:52:03 +01:00
mfvitale
47cbdee526
DBZ-7311 Permits to execute a blocking snapshot even if snapshot.mode=never
2024-01-08 12:52:03 +01:00
Mickael Maison
70e7186868
DBZ-7298 Add service loader manifests for all Connect plugins
...
Add manifests for source/sink connectors, converters and transformations.
2024-01-04 09:22:29 +01:00
Debezium Builder
3853d20f44
[maven-release-plugin] prepare for next development iteration
2023-12-21 06:52:01 +00:00
Debezium Builder
5d35e9caaa
[maven-release-plugin] prepare release v2.5.0.Final
2023-12-21 06:52:01 +00:00
ani-sha
4b36ed9975
DBZ-7177 DBZ-6764 Use correct context name and add suggestions for code review
2023-12-20 08:56:56 +01:00
ani-sha
5a27d42257
DBZ-7177 Add isStreaming check for all tests, remove unused files
2023-12-20 08:56:56 +01:00
rkerner
e06f96f339
DBZ-6764 Fix waitForStreamingRunning/isStreamingRunning
2023-12-20 08:56:56 +01:00
ani-sha
a12b1a9872
DBZ-6764 Connector metrics implementation for REST extension
2023-12-20 08:56:56 +01:00
ani-sha
5841eacb79
DBZ-7177 Wait for streaming, adjust test cases
2023-12-20 08:56:56 +01:00
ani-sha
e535e5b78d
DBZ-7177 Add suggestions from code review, adjust code style changes
2023-12-20 08:56:56 +01:00
ani-sha
2555a1cee2
DBZ-6764 DBZ-7178 DBZ-7177 Use MBeanServer to query metrics for REST endpoint and create new metrics descriptor
2023-12-20 08:56:56 +01:00
ani-sha
2165d9bed1
DBZ-6764 Connector metrics implementation for REST extension
2023-12-20 08:56:56 +01:00
Roman Kudryashov
516aa87fad
DBZ-7235 Add option to customize CloudEvents schema name
2023-12-20 06:53:37 +01:00
Jiri Pechanec
7354f7aa38
DBZ-7251 Enable fitering of MySQL connector
2023-12-19 17:29:37 +01:00
“vsantonastaso”
9dac11e1cd
DBZ-7164 Support offset commit for the signal topic
2023-12-19 15:51:10 +01:00
harveyyue
225277c5c0
DBZ-7251 Return generic DataCollectionId instead of REST response DataCollection in getMatchingCollections method
2023-12-19 12:53:57 +01:00
harveyyue
4719396ffc
DBZ-7145 Introduce a common parent connect for all Debezium source connectors
2023-12-19 12:53:57 +01:00
harveyyue
5553059fb8
DBZ-7167 Provide a public API from the connector implementations to retrieve the list of matching collections or tables based on the different include-/exclude lists
2023-12-19 12:53:57 +01:00
Vojtech Juranek
37ebb8ecae
DBZ-7275 Close connection registerd with bean registry
...
New connection created for bean registry is never closed. Close it
during when stopping the task.
2023-12-19 10:22:02 +01:00
Debezium Builder
2c1def7241
[maven-release-plugin] prepare for next development iteration
2023-12-14 09:43:13 +00:00
Debezium Builder
ef8260f802
[maven-release-plugin] prepare release v2.5.0.CR1
2023-12-14 09:43:12 +00:00
Chris Cranford
2b02b3982e
DBZ-4321 Rework configuration options
2023-12-13 11:27:40 -05:00
Chris Cranford
22a22d65d1
DBZ-4321 Pull getQualifiedTableName up to AbstractConnectorConnection
2023-12-13 11:27:40 -05:00
Chris Cranford
ada157d117
DBZ-4321 Introduce BeanRegistry
and ServiceRegistry
...
As a part of this work to handle injection in a cleaner way, this commit
adds two new broad concepts called `BeanRegistry` and `ServiceRegistry`.
A BeanRegistry is a glorified registry of different objects that are not
necessarily services but may be desired by a service. This contract will
allow Debezium to integrate in the future with other CDI providers.
A ServiceRegistry is more of an internal concept, where various systems
can be started based on their dependency order and provides a universal
way to split larger parts of the code into smaller, focused modules that
can be accessed using the Service Locator pattern.
2023-12-13 11:27:40 -05:00
Chris Cranford
26b3881941
DBZ-4321 Add MongoDB hook & polish injection handler
2023-12-13 11:27:40 -05:00
Chris Cranford
59027ed5ed
DBZ-4321 New PostProcessor
contract and Column Reselection
2023-12-13 11:27:40 -05:00
harveyyue
c5ce1cf945
DBZ-7251 Length value is not removed when changing a column's type
2023-12-12 10:18:21 +01:00
gongzhongqiang
018ecfe9f6
[polish] remove redundant Geometry type checking code
2023-12-08 10:16:46 +01:00