Chris Cranford
4361d3a30a
DBZ-7593 Fix test failures
2024-03-07 11:01:43 +01:00
Chris Cranford
a76dfdebfa
DBZ-7593 Always capture all permitted schema structures by default
2024-03-07 11:01:43 +01:00
Debezium Builder
2fb8fc3004
[maven-release-plugin] prepare for next development iteration
2024-03-06 07:47:33 +00:00
Debezium Builder
cd46b2b998
[maven-release-plugin] prepare release v2.6.0.Beta1
2024-03-06 07:47:33 +00:00
mfvitale
2d06d847c3
DBZ-7303 Add tests for Always, When Needed and Custom snapshot mode for SQLServer connector
2024-03-04 11:17:06 +01:00
mfvitale
9ad4273791
DBZ-7303 Align snapshot modes for SqlServer connector
2024-03-04 11:17:06 +01:00
Vojtech Juranek
0c34c6ac68
[ci] Add missing ConditionalFail rule
2024-03-04 08:30:16 +01:00
mfvitale
8a47b2333d
DBZ-7588 Disable debug prints on tests
2024-03-02 11:33:48 -05:00
mfvitale
e174eae235
DBZ-7461 Remove redundant log
2024-03-01 14:12:31 +01:00
mfvitale
4ade54351f
DBZ-7461 Rename SCHEMA_ONLY_RECOVERY to RECOVERY and SCHEMA_ONLY to NO_DATA
2024-03-01 14:12:31 +01:00
James Johnston
f632fa081e
DBZ-5071 Correctly handle NULL values in incremental snapshots
...
It turns out that the existing code for chunking a table when taking
an incremental snapshot was buggy and did not correctly handle NULL
values when building the chunk query. An example of such a situation
would be when the user has specified "message.key.columns" to reference
a column that is part of a PostgreSQL UNIQUE INDEX that was created with
the NULLS NOT DISTINCT option.
This commit updates the new AbstractChunkQueryBuilder so that it checks
whether a key column is optional. If it is, then additional will
appropriately consider NULL values when generating a chunk query using
"IS [NOT] NULL" clauses.
One complication is that different database engines have different
sorting behavior of ORDER BY. It is apparently not well-defined by the
SQL standard. Some databases consider NULL values to be higher than any
non-NULL values, and others consider them to be lower.
To handle this situation, a new nullsSortLast() function is added to the
JdbcConnection class. By default, it returns an empty value, indicating
that the behavior of the database engine is unknown. When an optional
field is encountered by AbstractChunkQueryBuilder in this situation, we
throw an error because we don't actually know how to correctly chunk the
query: there's no safe assumption that can be made here.
Derived JdbcConnection classes can then override the nullsSortLast
function, and return a value indicating the actual behavior of that
database engine. When this is done, the AbstractChunkQueryBuilder then
knows how to correctly build a chunk query that can handle NULL values.
To help test this, new tests have been added to
AbstractIncrementalSnapshotTest. First, the existing insertsWithoutPks
test has been moved and deduplicated from MySQL and PostgreSQL so that
the test case can be reused on other engines. Second, a new
insertsWithoutPksAndNull test is run, which inserts data with NULL
values in the message key columns. To demonstrate that chunk queries
are being correctly generated for practically every case, the
INCREMENTAL_SNAPSHOT_CHUNK_SIZE is set to 1 so that NULL values are not
returned in the middle of a chunk, which can cause us to skip testing
the code we need to test.
2024-02-29 13:36:26 +01:00
rkerner
1e69e40ec2
DBZ-7416 Fix duplicate SMTs sometimes returned by Kafka Connect. Moved deduplication from Map to LinkedHashSet.
...
+ minor fixes added for cleanup and centralization of common code
closes https://issues.redhat.com/browse/DBZ-7416
2024-02-22 13:34:08 -05:00
Jiri Pechanec
68b6591142
DBZ-7416 Fix duplicate SMTs sometimes returned by Kafka Connect. Moved deduplication from Map to LinkedHashSet.
...
closes https://issues.redhat.com/browse/DBZ-7416
2024-02-22 13:34:08 -05:00
Chris Cranford
0516c0cdbb
DBZ-6236 Reset ErrorHandler retry counter on successful poll
2024-02-22 00:14:26 -05:00
mfvitale
e680840c4a
DBZ-7541 Add wait check to let database return online during SqlServerConnectorIT tests
2024-02-21 20:07:23 -05:00
Chris Cranford
93cf3b06bf
DBZ-7516 Correctly enable table CDC to avoid failures
2024-02-20 10:59:39 +01:00
Chris Cranford
928aa26cdb
DBZ-7107 Fix source info test failures
2024-02-16 12:52:20 +01:00
Debezium Builder
10e327602c
[maven-release-plugin] prepare for next development iteration
2024-02-13 09:20:04 +00:00
Debezium Builder
0c5b05738c
[maven-release-plugin] prepare release v2.6.0.Alpha2
2024-02-13 09:20:04 +00:00
Vojtech Juranek
4452e3d095
DBZ-7024 Move EmbeddedConfig into separate class
...
`EmbeddedConfig` needs to be shared with other implmentations of
`DebeziumEngine` as long as Debezium embedded depends on the Kafka
model.
2024-02-12 13:43:21 +01:00
Vadzim Ramanenka
237755331a
DBZ-7463: SQL Server queries with special characters fail after applying DBZ-7273
...
Enclose column names in square brackets to account for spaces and
special characters.
2024-02-09 14:46:38 -05:00
mfvitale
6dc3b578f8
DBZ-7467 Fix missing signaling actions
2024-02-09 11:45:56 -05:00
mfvitale
f78d0cbd19
DBZ-7301 Add SnapshotterService to ChangeEventSourceFactory and SnapshotChangeEventSource
2024-02-06 07:12:42 +01:00
Jan.Lieskovsky
1d8a56778d
DBZ-7374 Fix parsing NULL as the default value for SQL server (Unicode) character strings
2024-01-29 17:09:03 +01:00
nicholas-fwang
7d99605886
DBZ-7143 Add case when EventConvertingFailureHandlingMode is null
2024-01-29 13:48:19 +01:00
nicholas-fwang
4f2bbb023a
DBZ-7143 fix checkstyle format
2024-01-29 13:48:19 +01:00
nicholas-fwang
eeea0f1e70
DBZ-7143 Add event.converting.failure.handling.mode option
2024-01-29 13:48:19 +01:00
Vadzim Ramanenka
a52fc18a45
DBZ-7273: Introduce mode to query capture tables directly
2024-01-25 06:29:26 +01:00
Animesh Kumar
2f8114029e
DBZ-7380 Offset transaction id only when it's non null
2024-01-24 15:32:34 +01:00
Debezium Builder
65d63ed42d
[maven-release-plugin] prepare for next development iteration
2024-01-21 10:12:45 +00:00
Debezium Builder
485fa82a8f
[maven-release-plugin] prepare release v2.6.0.Alpha1
2024-01-21 10:12:44 +00:00
Peter Hamer
e2c0a59ec3
DBZ-7347 Update notification tests.
2024-01-17 09:42:27 +01:00
mfvitale
20fbe73bc8
DBZ-7311 Use a more meaningful name for identify an on demand blocking snapshot
2024-01-08 12:52:03 +01:00
mfvitale
47cbdee526
DBZ-7311 Permits to execute a blocking snapshot even if snapshot.mode=never
2024-01-08 12:52:03 +01:00
Mickael Maison
70e7186868
DBZ-7298 Add service loader manifests for all Connect plugins
...
Add manifests for source/sink connectors, converters and transformations.
2024-01-04 09:22:29 +01:00
Debezium Builder
3853d20f44
[maven-release-plugin] prepare for next development iteration
2023-12-21 06:52:01 +00:00
Debezium Builder
5d35e9caaa
[maven-release-plugin] prepare release v2.5.0.Final
2023-12-21 06:52:01 +00:00
ani-sha
8bb7869c7e
DBZ-7177 DBZ-6764 Remove unused loggers, sort imports
2023-12-20 08:56:56 +01:00
ani-sha
4b36ed9975
DBZ-7177 DBZ-6764 Use correct context name and add suggestions for code review
2023-12-20 08:56:56 +01:00
ani-sha
5a27d42257
DBZ-7177 Add isStreaming check for all tests, remove unused files
2023-12-20 08:56:56 +01:00
ani-sha
a12b1a9872
DBZ-6764 Connector metrics implementation for REST extension
2023-12-20 08:56:56 +01:00
ani-sha
e535e5b78d
DBZ-7177 Add suggestions from code review, adjust code style changes
2023-12-20 08:56:56 +01:00
ani-sha
2555a1cee2
DBZ-6764 DBZ-7178 DBZ-7177 Use MBeanServer to query metrics for REST endpoint and create new metrics descriptor
2023-12-20 08:56:56 +01:00
ani-sha
2165d9bed1
DBZ-6764 Connector metrics implementation for REST extension
2023-12-20 08:56:56 +01:00
Roman Kudryashov
516aa87fad
DBZ-7235 Add option to customize CloudEvents schema name
2023-12-20 06:53:37 +01:00
Jiri Pechanec
16ba4881e6
DBZ-7251 Enable fitering of SQL Server connector
2023-12-19 17:29:37 +01:00
harveyyue
225277c5c0
DBZ-7251 Return generic DataCollectionId instead of REST response DataCollection in getMatchingCollections method
2023-12-19 12:53:57 +01:00
harveyyue
4719396ffc
DBZ-7145 Introduce a common parent connect for all Debezium source connectors
2023-12-19 12:53:57 +01:00
harveyyue
5553059fb8
DBZ-7167 Provide a public API from the connector implementations to retrieve the list of matching collections or tables based on the different include-/exclude lists
2023-12-19 12:53:57 +01:00
Debezium Builder
2c1def7241
[maven-release-plugin] prepare for next development iteration
2023-12-14 09:43:13 +00:00
Debezium Builder
ef8260f802
[maven-release-plugin] prepare release v2.5.0.CR1
2023-12-14 09:43:12 +00:00
Chris Cranford
2b02b3982e
DBZ-4321 Rework configuration options
2023-12-13 11:27:40 -05:00
Chris Cranford
ada157d117
DBZ-4321 Introduce BeanRegistry
and ServiceRegistry
...
As a part of this work to handle injection in a cleaner way, this commit
adds two new broad concepts called `BeanRegistry` and `ServiceRegistry`.
A BeanRegistry is a glorified registry of different objects that are not
necessarily services but may be desired by a service. This contract will
allow Debezium to integrate in the future with other CDI providers.
A ServiceRegistry is more of an internal concept, where various systems
can be started based on their dependency order and provides a universal
way to split larger parts of the code into smaller, focused modules that
can be accessed using the Service Locator pattern.
2023-12-13 11:27:40 -05:00
Chris Cranford
26b3881941
DBZ-4321 Add MongoDB hook & polish injection handler
2023-12-13 11:27:40 -05:00
Chris Cranford
59027ed5ed
DBZ-4321 New PostProcessor
contract and Column Reselection
2023-12-13 11:27:40 -05:00
Chris Cranford
aea6cc003d
[ci] Use a relaxed property check for tests
2023-12-06 18:04:59 -05:00
Debezium Builder
0fd1c0dc9a
[maven-release-plugin] prepare for next development iteration
2023-12-04 13:55:35 +00:00
Debezium Builder
3e2d75f0da
[maven-release-plugin] prepare release v2.5.0.Beta1
2023-12-04 13:55:35 +00:00
“vsantonastaso”
8c1c369449
DBZ-6878 add table specific notification in initial snapshot
2023-11-29 08:32:21 +01:00
rkerner
aeaba3ae70
DBZ-6762 Add "validate filters" endpoint/s for connector-specific Connect REST Extensions
...
closes to https://issues.redhat.com/browse/DBZ-6762
2023-11-22 06:10:40 +01:00
Chris Cranford
529be44712
[ci] Mark stopCurrentIncrementalSnapshotWithAllCollectionsAndTakeNewNewIncrementalSnapshotAfterRestart as flaky
2023-11-18 09:25:15 -05:00
Debezium Builder
1521445908
[maven-release-plugin] prepare for next development iteration
2023-11-10 10:26:05 +00:00
Debezium Builder
6c6f6e9138
[maven-release-plugin] prepare release v2.5.0.Alpha2
2023-11-10 10:26:05 +00:00
Jiri Pechanec
f3eb643822
DBZ-7043 Use code idiom
2023-11-10 10:53:00 +01:00
Vadzim Ramanenka
a46069a8f7
DBZ-7043: Emit a notification when completed reading from a capture instance
...
Let the database know when the connector has finished reading
from a capture instance by calling a stored procedure.
2023-11-10 10:53:00 +01:00
Vojtech Juranek
11d2ff0b9b
DBZ-7007 Use better name for auxiliary variable detecing if engine is running
2023-11-06 10:50:21 +01:00
Vojtech Juranek
fdab2cd61b
DBZ-7007 Remove engine.isRunning() from testsuite
2023-11-06 10:50:21 +01:00
Debezium Builder
78c5204444
[maven-release-plugin] prepare for next development iteration
2023-10-26 15:39:35 +00:00
Debezium Builder
356e488e83
[maven-release-plugin] prepare release v2.5.0.Alpha1
2023-10-26 15:39:35 +00:00
Vojtech Juranek
d7e8943d4e
DBZ-6778 Initialize ElapsedTimeStrategy upon its creation
...
Currently, newly created `ElapsedTimeStrategy` is uninitialized and its
`hasElapsed()` has to be called once `ElapsedTimeStrategy` is created to
initialize the strategy. This is confusing and error prone.
Move initialization of `ElapsedTimeStrategy` into it's constructor, so
it's initialized once it's created.
2023-10-05 11:57:59 +02:00
Debezium Builder
40a8f4026d
[maven-release-plugin] prepare for next development iteration
2023-10-03 14:18:54 +00:00
Debezium Builder
412c6f7979
[maven-release-plugin] prepare release v2.4.0.Final
2023-10-03 14:18:54 +00:00
Roman Kudryashov
98c426dec4
DBZ-3642 CloudEventsConverter can retrieve metadata info from headers
2023-09-26 13:06:09 +02:00
Debezium Builder
cc1de0aa51
[maven-release-plugin] prepare for next development iteration
2023-09-22 09:00:45 +00:00
Debezium Builder
6809978083
[maven-release-plugin] prepare release v2.4.0.CR1
2023-09-22 09:00:45 +00:00
rkerner
6d3528d4fb
DBZ-4395 Add connector specific Debezium Connect REST Extension/s and move logic from UI backend to the Debezium Connect REST Extension (part 2 of the re-architecturing/re-factoring)
...
DBZ-6761 Add "validate connection" endpoints in connector specific Connect REST extensions
closes to https://issues.redhat.com/browse/DBZ-4395
closes to https://issues.redhat.com/browse/DBZ-6761
2023-09-21 13:29:59 +02:00
mfvitale
ae199d2053
DBZ-6828 Blocking snapshot will generate schema events only for signaled tables
2023-09-20 08:18:33 +02:00
Debezium Builder
65169a9315
[maven-release-plugin] prepare for next development iteration
2023-09-13 09:41:46 +00:00
Debezium Builder
845e3e9d09
[maven-release-plugin] prepare release v2.4.0.Beta2
2023-09-13 09:41:46 +00:00
Debezium Builder
d97a18e8dc
[maven-release-plugin] prepare for next development iteration
2023-08-29 11:41:08 +00:00
Debezium Builder
5ce25f2cfe
[maven-release-plugin] prepare release v2.4.0.Beta1
2023-08-29 11:41:08 +00:00
mfvitale
542b3619fa
DBZ-6731 Blocking snapshot takes configuration from signal payload
2023-08-15 12:43:43 +02:00
Vojtech Juranek
6aebee6934
DBZ-6811 Send heartbeats when there are no updates in the DB
...
Heartbeats should be sent when Debezium is not sending any events
obatined from the DB. Send heartbeat before finishing itteration
which doesn't contain any updates from the DB.
2023-08-14 20:17:59 +02:00
Debezium Builder
3c72f071b0
[maven-release-plugin] prepare for next development iteration
2023-08-09 10:33:06 +00:00
Debezium Builder
915c465440
[maven-release-plugin] prepare release v2.4.0.Alpha2
2023-08-09 10:33:06 +00:00
mfvitale
61651a94b5
DBZ-6566 Force blocking snapshot even if a snapshot has already executed
2023-08-03 09:22:24 +02:00
mfvitale
f883a5c9a3
DBZ-6566 Support Blocking snapshot for SQLServer
2023-08-03 09:22:24 +02:00
jcechace
699a1612ae
DBZ-6670 Replace error handler instead of recreate in order to keep retry counter
2023-07-25 13:46:56 +02:00
Debezium Builder
267dfda8a6
[maven-release-plugin] prepare for next development iteration
2023-07-17 10:40:52 +00:00
Debezium Builder
0d3cef6a29
[maven-release-plugin] prepare release v2.4.0.Alpha1
2023-07-17 10:40:52 +00:00
harveyyue
5bc7fd1f7c
DBZ-6603 Support for custom tags in the connector metrics
2023-07-13 09:52:33 +02:00
tjwornjs
2767f469aa
DBZ-6573 add max retries config to common config
2023-06-30 09:39:32 +02:00
mfvitale
d5febcc054
DBZ-6417 Provide extensible signal action mechanism through SPI
2023-06-23 15:03:27 +02:00
mfvitale
9ed928a1c9
DBZ-6416 Improve tests stability
2023-06-22 13:22:34 +02:00
mfvitale
34e28ac52d
DBZ-6416 Add initial snapshot notifications
2023-06-22 13:22:34 +02:00
Jiri Pechanec
090aaa47ec
DBZ-5676 Add missing prefix
2023-06-21 17:05:35 +02:00
ani-sha
f7734c226c
DBZ-5676 Modify SqlServerConnection definition
2023-06-21 17:05:35 +02:00
ani-sha
2b6c1a591a
DBZ-5676 Align QUERY_FETCH_SIZE config in SQLServer Connector to use a default value
2023-06-21 17:05:35 +02:00
Debezium Builder
60f730d383
[maven-release-plugin] prepare for next development iteration
2023-06-20 13:21:31 +00:00
Debezium Builder
150396e4c8
[maven-release-plugin] prepare release v2.3.0.Final
2023-06-20 13:21:31 +00:00