Commit Graph

1079 Commits

Author SHA1 Message Date
mfvitale
7429b4e6f1 DBZ-1973 Add incremental snapshot notifications 2023-05-10 12:22:02 +02:00
mfvitale
811b800f55 DBZ-1973 Add support for Notifications
DBZ-1973 Add more tests for other connectors

DBZ-1973 Add send method with offset parameter

DBZ-1973 Instantiate NotificationService in the task class

DBZ-1973 Instantiate NotificationService in the task class

DBZ-4027 Move specific sink channel configuration to SinkNotificationChannel

DBZ-4027 Remove not used SPI file

DBZ-1973 Moved SPI file definition to debezium-core

DBZ-1973 Rename KafkaNotificationChannel to more generic SinkNotificationChannel

DBZ-1973 Code refactor

DBZ-1973 Improve configuration property description

DBZ-1973 Improve test

DBZ-1973 Add close method to NotificationChannel

DBZ-1973 Implement KafkaNotificationChannel

DBZ-1973 Add NotificationService and LogNotificationChannel
2023-05-10 12:22:02 +02:00
Chris Cranford
fc4a31e3d7 DBZ-6436 Improve Oracle LogMiner failure logging
This change explicitly captures the state of two critical tables when the connector
fails to start a LogMiner session, namely V$LOGMNR_LOGS and V$LOGMNR_PARAMETERS.
While these were previously captured by the LogMinerDatabaseStateWriter, the old
code required DEBUG log level and that isn't always feasible, so this change will
capture these in ERROR log level regardless moving forward.
2023-05-09 12:09:36 +02:00
Chris Cranford
30eb53c764 DBZ-6254 Add LogMiner Query and Username Include Filtering
* Add `log.mining.query.filter.mode` configuration option
* Add `log.mining.username.include.list` configuration option
2023-05-09 11:31:08 +02:00
mfvitale
952340286e DBZ-4027 Make signalling channel configurable
DBZ-4027 Add an Incremental snapshot test with kafka signaling

DBZ-4027 Add an Incremental snapshot test with kafka signaling

DBZ-4027 Add an Incremental snapshot test with kafka signaling

DBZ-4027 Code style

DBZ-4027 Make SignalPayload more generic and extensible

DBZ-4027 Rename DatabaseSignalChannel to SourceSignalChannel

DBZ-4027 Improve logging

DBZ-4027 Moved SPI file definition to debezium-core

DBZ-4027 Move SignalProcessor synchronization point to be processed only when a signal cdc event arrives.

DBZ-4027 Add EventDispatcher constructor without signalProcessor for spanner connector

DBZ-4027 Fix NPE

DBZ-4027 Fix NPE

DBZ-4027 Formatting

DBZ-4027 Correctly manage signal on not supported connector

DBZ-4027 Use the correct MongoDbOffset

DBZ-4027 Correctly initialize offset for Oracle and SqlServer connectors

DBZ-4027 Register SPI implementations

DBZ-4027 Improve SignalProcessor instantiation

DBZ-4027 Pass source info in case of SchemaChanges action

DBZ-4027 Manage close event in a synchronous way

DBZ-4027 Correctly init offset context also in case of snapshot mode 'never'

DBZ-4027 Fix MySqlMetricsIT test

DBZ-4027 Move KafkaSignalChannel to core

DBZ-4027 Move KafkaSignalChannel to core

DBZ-4027 Set pass offset context after initial snapshot to SignalProcessor

DBZ-4027 Pass OffsetContext to signal processor

DBZ-4027 Pass CommonConnectorConfig to SignalChannelReader init method

DBZ-4027 Move Incremental snapshot window actions to dedicated package

DBZ-4027 Align SignalsIT test with new code

DBZ-4027 Fix SignalsIT test

DBZ-4027 Fix SignalProcessor scheduling

DBZ-4027 Moved DatabaseSignalChannel and SignalChannelReader to dedicated package

DBZ-4027 Start SignalProcessor from ChangeEventSourceCoordinator

DBZ-4027 Create SignalProcessor and renamed Signal to DatabaseSignalChannel

DBZ-4027 Initial refactoring of signal feature
2023-05-05 13:35:40 +02:00
harveyyue
3860a4faca DBZ-6388 __source_ts_ms r (read) operation date is set to future for SQL Server 2023-05-02 08:11:02 -04:00
ani-sha
a453c3ca19 DBZ-6259 Add missing description of various snapshot modes in the connector field definitions 2023-05-01 11:03:04 -04:00
Chris Cranford
3f554645b6 DBZ-6256 Add log.mining.flush.table.name Oracle config option
There is the potential when using multiple connector deployments on Oracle that
there could be some level of lock contention with all connectors using the same
table. This allows users to configure the flush table name themselves, reducing
the lock contention across multiple connector deployments.
2023-05-01 10:41:05 -04:00
Chris Cranford
4664dd6df6 DBZ-6355 Add log.mining.transaction.retention.ms; deprecate log.mining.transaction.retention.hours 2023-04-27 09:28:29 -04:00
Chris Cranford
036d7f831d DBZ-6355 Fix test logic 2023-04-27 09:28:29 -04:00
Chris Cranford
2ee3249d84 DBZ-6355 Fix memory-leak with heap-based abandoned transaction cache 2023-04-27 09:28:29 -04:00
Chris Cranford
8ef9a6f015 DBZ-6355 Reduce logging verbosity 2023-04-27 09:28:29 -04:00
Chris Cranford
e81af64adc DBZ-6355 Add ORA-01555 fallback based on change time 2023-04-27 09:28:29 -04:00
Chris Cranford
1fbb6f228d DBZ-6355 Fix test failures 2023-04-27 09:28:29 -04:00
Chris Cranford
3d7c87401f DBZ-6355 Fix format 2023-04-27 09:28:29 -04:00
Sergey Eizner
b954afd2cc DBZ-6355 Fix log message 2023-04-27 09:28:29 -04:00
Chris Cranford
9f28ef17e4 DBZ-6355 Don't abandon transactions if calculated SCN is before oldest buffered SCN 2023-04-27 09:28:29 -04:00
Chris Cranford
34f6b221c6 DBZ-6355 Remove unused code 2023-04-27 09:28:29 -04:00
Chris Cranford
a0fedf9439 DBZ-6355 Rework test 2023-04-27 09:28:29 -04:00
eizners
fa42a1b0ce DBZ-6355 fix unit tests errors 2023-04-27 09:28:29 -04:00
Sergey Eizner
cac5485176 DBZ-6355 Change to the last offset 2023-04-27 09:28:29 -04:00
Chris Cranford
e6e19fb578 DBZ-6329 Trim value of rs_id 2023-04-23 18:29:07 +02:00
Chris Cranford
f624e39bbc DBZ-6329 Make sure Oracle source rs_id field populated 2023-04-23 18:29:07 +02:00
Debezium Builder
bb1b79f3bd [maven-release-plugin] prepare for next development iteration 2023-04-20 09:09:43 +02:00
Debezium Builder
5483b4de99 [maven-release-plugin] prepare release v2.2.0.Final 2023-04-20 09:09:43 +02:00
Jiri Pechanec
ee5ba8e56f DBZ-2979 Fix Oracle Xstream tests 2023-04-19 15:57:49 +02:00
Chris Cranford
16385994d8 DBZ-5395 Improve Oracle offset management when LOB enabled 2023-04-19 11:24:10 +02:00
Chris Cranford
1805f6e75e DBZ-6276 Marked test ignored by default
The default Debezium Oracle images are pre-configured with archive log mode
enabled and this isn't something we can simply turn off and back on in the
test suite proper. This test must be invoked manually and separately, so it
is disabled by default in the overall test suite execution.
2023-04-19 10:52:18 +02:00
Mark Bereznitsky
4df3895d65 DBZ-6276 changed Oracle archive log check to fail if conditions require it 2023-04-19 10:52:18 +02:00
Jiri Pechanec
2181251a9f DBZ-2979 Fix Oracle tests 2023-04-19 10:50:41 +02:00
Ronak Jain
2c2b87b846 DBZ-2979 If column.include.list/column.exclude.list are used and the target table receives an update for the excluded (or not included) column - such events should be ignored
* [Unstable][Untested] Add config in Oracle Connector

Addresses/Closes DBZ-2979
2023-04-19 09:04:29 +02:00
Debezium Builder
976a47a3a3 [maven-release-plugin] prepare for next development iteration 2023-04-14 08:46:06 +00:00
Debezium Builder
1ca8399e8b [maven-release-plugin] prepare release v2.2.0.CR1 2023-04-14 08:46:06 +00:00
Chris Cranford
36bc3ce07a DBZ-6129 Use Infinispan Jakarta EE based artifacts 2023-04-13 17:18:23 -04:00
Chris Cranford
84d57628b7 DBZ-6155 Avoid batch size "ping-pong" in Oracle metrics
This avoids a scenario where we want to increment the batch size; however, the
current batch size is equal to the max and therefore triggers a decrement and
the next iteration increments. The new behavior will be that we don't trigger
this ping pong. With this change, we can track specifically when we reach the
max batch size and only log the warning once. If the batch size drops, and it
later increments, the warning will be logged again but this should be expected.
2023-04-10 16:51:11 -04:00
doupengwei
352d616173 DBZ-6155 Adjust LogMiner batch size based on the default batch size rather than current batch size.
Using current batch size for comparison is wrong in case that when currentScn is topScn as described in DBZ-6155.
In the other case (when topScn is behind currentScn) can eventually lead to this situation.
when topScn would fall behind currentScn more and more as we would compare currentScn - topScn with bigger and bigger number (current batch size).

Comapring with logMiningBatchSizeMin could result in very small window which would mean we will send many small queries instead of several bigger ones during mining.
Therefore reverting back to do the adjustments based on the defaultBatchSize.
2023-04-10 16:51:11 -04:00
harveyyue
c525f34cd0 DBZ-6070 DDL events not stored in schema history topic for excluded tables 2023-04-06 12:10:30 +02:00
Chris Cranford
2ef6bd66e4 DBZ-6273 Fix Infinispan serialization of table names with spaces 2023-04-01 17:04:06 -04:00
Chris Cranford
3bb82d6b73 DBZ-6275 Correctly sync Infinispan cache on event counter reset 2023-04-01 16:37:25 -04:00
Chris Cranford
6419c0e704 DBZ-6274 Fix ClusterLoader deprecation using embedded Infinispan
The embedded Infinipsan configuration used a deprecated index model, and
moved to a non-deprecated model called SIFS using segments.
2023-04-01 15:29:12 -04:00
Chris Cranford
42f375f570 DBZ-6274 Align test configurations for remote Infinispan caches
The "fetch-state" attribute was deprecated and is no longer a valid option
with Infinispan 14, which causes the tests to fail to execute.  Additionally,
the "segmented" attribute must be set to true going forward as the file
store implementation no longer supports non-segmented configurations.
2023-04-01 15:29:12 -04:00
Debezium Builder
5c650fd054 [maven-release-plugin] prepare for next development iteration 2023-03-31 10:00:16 +00:00
Debezium Builder
68ffc4833b [maven-release-plugin] prepare release v2.2.0.Beta1 2023-03-31 10:00:16 +00:00
Jiri Pechanec
04def67900 DBZ-6221 Remove print statements 2023-03-28 14:45:12 +02:00
Chris Cranford
f9bb5eb82f DBZ-6221 Correctly interpret CHARACTER/CHARACTER VARYING data types while streaming 2023-03-28 14:45:12 +02:00
Debezium Builder
55d65e9618 [maven-release-plugin] prepare for next development iteration 2023-03-08 11:38:17 +00:00
Debezium Builder
6d183c3dc1 [maven-release-plugin] prepare release v2.2.0.Alpha3 2023-03-08 11:38:16 +00:00
Jiri Pechanec
f62caf9c83 DBZ-6143 Make tests running with xstream 2023-03-08 09:15:48 +01:00
Jiri Pechanec
3da2434c6c DBZ-823 Restrict the type with generics 2023-03-03 12:47:15 +01:00
Jiri Pechanec
5e666f00ee DBZ-823 Add JavaDoc to new interfaces 2023-03-03 12:47:15 +01:00