Commit Graph

1221 Commits

Author SHA1 Message Date
Jiri Pechanec
f8e4e589d8 DBZ-6903 Use distinct dir for protobuf generated sources 2023-09-13 11:50:47 -04:00
Debezium Builder
65169a9315 [maven-release-plugin] prepare for next development iteration 2023-09-13 09:41:46 +00:00
Debezium Builder
845e3e9d09 [maven-release-plugin] prepare release v2.4.0.Beta2 2023-09-13 09:41:46 +00:00
Chris Cranford
6e10618c94 DBZ-2543 Checkstyle fixes 2023-09-12 18:20:07 -04:00
Chris Cranford
1d9b81b58c DBZ-2543 Confirm only checkpoints 2023-09-12 18:20:07 -04:00
Chris Cranford
e8e11cd40a DBZ-2543 Fix value converter failures 2023-09-12 18:20:07 -04:00
Chris Cranford
46fa3bef4d DBZ-2543 Refactor value conversion logic 2023-09-12 18:20:07 -04:00
Chris Cranford
ca6ccbcd35 DBZ-2543 Remove unnecessary test class 2023-09-12 18:20:07 -04:00
Chris Cranford
e841e6c74d DBZ-2543 Use protoc plugin setup like PostgreSQL 2023-09-12 18:20:07 -04:00
Chris Cranford
c076d9526b DBZ-2543 Remove vert.x dependency 2023-09-12 18:20:07 -04:00
Chris Cranford
2cae1b396d DBZ-2543 Add OpenLogReplicator configuration properties
Added source, host, and port configurable OpenLogReplicator connector properties.
2023-09-12 18:20:07 -04:00
Chris Cranford
0955d0e7d6 DBZ-2543 Disable XML tests for OpenLogReplicator 2023-09-12 18:20:07 -04:00
Chris Cranford
07c7a8236e DBZ-2543 Disable ROWID tests for OpenLogReplicator 2023-09-12 18:20:07 -04:00
Chris Cranford
a87533e4a8 DBZ-2543 Interval data types support 2023-09-12 18:20:07 -04:00
Chris Cranford
51b95e3bfb DBZ-2543 Add timestamp with local time zone support 2023-09-12 18:20:07 -04:00
Chris Cranford
c9f6f4596e DBZ-2543 Clean-up value resolution logic 2023-09-12 18:20:07 -04:00
Chris Cranford
73aa4a6b35 DBZ-2543 Properly decode non-variable RAW data 2023-09-12 18:20:07 -04:00
Chris Cranford
1e803edee8 DBZ-2543 Cleanup timestamptz parsing and DDL handling 2023-09-12 18:20:07 -04:00
Chris Cranford
d130fde297 DBZ-2543 Improve timestamp handling for precision modes 2023-09-12 18:20:07 -04:00
Chris Cranford
26db0b3961 DBZ-2543 Support timestamps with time zone data types 2023-09-12 18:20:07 -04:00
Chris Cranford
748420605e DBZ-2543 Add support for new db field in streaming events 2023-09-12 18:20:07 -04:00
Chris Cranford
2d4d34428d DBZ-2543 Add some streaming metrics 2023-09-12 18:20:07 -04:00
Chris Cranford
d977e9b0f1 DBZ-2543 Initial OpenLogReplicator adapter 2023-09-12 18:20:07 -04:00
Chris Cranford
dd5f1ac0f9 DBZ-6880 Include details about unsupported operations 2023-09-08 14:16:02 -04:00
Chris Cranford
b5700a62c9 DBZ-6880 Move some Oracle log messages to DEBUG 2023-09-08 14:16:02 -04:00
Chris Cranford
a0c67aa0e8 DBZ-6860 Fix NPE Oracle shouldContinueToUpdateOffsetsEvenWhenTableIsNotChanged 2023-08-31 19:13:55 -04:00
Chris Cranford
6aa6b58fdc DBZ-6859 Restore Oracle connector test suite logging 2023-08-31 17:09:25 -04:00
Debezium Builder
d97a18e8dc [maven-release-plugin] prepare for next development iteration 2023-08-29 11:41:08 +00:00
Debezium Builder
5ce25f2cfe [maven-release-plugin] prepare release v2.4.0.Beta1 2023-08-29 11:41:08 +00:00
Ondrej Babec
f03eda1cc4 DBZ-6827 Testing farm onboarding 2023-08-28 10:39:30 +02:00
Chris Cranford
e093a7bdd2
[ci] Fix some failing Oracle tests 2023-08-23 12:53:19 -04:00
Chris Cranford
e3bbc47367
DBZ-6808 Fix tests: Allow global configuration with embedded Infinispan 2023-08-23 10:00:40 -04:00
Chris Cranford
b44cae62e3
DBZ-6798 Fix tests: Expose SCN-based metrics as BigInteger 2023-08-23 10:00:33 -04:00
Chris Cranford
0b2e65c741 DBZ-6615 Fix test failures 2023-08-17 06:24:55 +02:00
Chris Cranford
d5e86e1380 DBZ-6615 Add OldestScnAgeInMilliseconds LogMiner based metric.
This returns the age in milliseconds from the poll time for the age of the oldest
transaction's starting system change number in the transaction buffer.
2023-08-17 06:24:55 +02:00
Jeremy Ford
829b969491 DBZ-6808 Allow the embedded infinispan global configuration to be configurable 2023-08-15 12:53:29 +02:00
mfvitale
542b3619fa DBZ-6731 Blocking snapshot takes configuration from signal payload 2023-08-15 12:43:43 +02:00
Chris Cranford
aad842522b DBZ-6798 Expose SCN-based metrics as BigInteger
The SCN data types were previously exposed as `String` types, which is not
consumable by Grafana and Prometheus. By using `BigInteger`, we can now
make these accessible on dashboards.
2023-08-15 06:06:49 +02:00
Debezium Builder
3c72f071b0 [maven-release-plugin] prepare for next development iteration 2023-08-09 10:33:06 +00:00
Debezium Builder
915c465440 [maven-release-plugin] prepare release v2.4.0.Alpha2 2023-08-09 10:33:06 +00:00
Chris Cranford
dc23f18456 DBZ-6782 Properly solve XMLTYPE data types during DDL changes 2023-08-09 06:46:31 +02:00
mfvitale
61651a94b5 DBZ-6566 Force blocking snapshot even if a snapshot has already executed 2023-08-03 09:22:24 +02:00
mfvitale
85d725fde8 DBZ-6566 Support Blocking snapshot for Oracle 2023-08-03 09:22:24 +02:00
Chris Cranford
fe2573cf86 DBZ-6751 Fix failing test OracleSchemaMigrationIT 2023-08-03 02:49:47 -04:00
Chris Cranford
c920479336 DBZ-6733 Skip mining distance calc. if upper bounds not within distance 2023-08-02 08:38:50 +02:00
Chris Cranford
dbeac568b0 DBZ-6729 Use default of 10000 for snapshot/query fetch sizes 2023-07-31 13:47:53 +02:00
jcechace
699a1612ae DBZ-6670 Replace error handler instead of recreate in order to keep retry counter 2023-07-25 13:46:56 +02:00
mfvitale
b8462e8fe0 DBZ-6599 Support single line statement without semicolon ending 2023-07-20 10:10:16 -04:00
Chris Cranford
afb2ecec63 DBZ-6677 Fix IndexOutOfBoundsException with ALTER TABLE listener
When a user supplies a column visibility clause in an ALTER TABLE statement,
there are no "modify_col_properties" clauses present, and this will lead to
the aforementioned exception. The listener should be tolerant of this case
and should not initialize any column editors.
2023-07-18 09:46:27 +02:00
Chris Cranford
72acea5628 DBZ-6673 Fix flaky Oracle test
The shouldCaptureChangesForTransactionsAcrossSnapshotBoundaryWithoutReemittingDDLChanges test
only expects the tables created by the entire test to exist but tables from other tests not,
and it would appear this commonly happens when another test fails to cleanup after itself.

This fix is to guarantee that the Oracle database state is set properly so that tests from
within this class are executed with the right number of tables expected to exist.
2023-07-18 09:40:41 +02:00
Debezium Builder
267dfda8a6 [maven-release-plugin] prepare for next development iteration 2023-07-17 10:40:52 +00:00
Debezium Builder
0d3cef6a29 [maven-release-plugin] prepare release v2.4.0.Alpha1 2023-07-17 10:40:52 +00:00
Chris Cranford
de8cf36b2c DBZ-6660 Introduce internal.log.mining.max.scn.deviation.ms
We hypothesize that there could be a situation where we may be mining precisely
around the CURRENT_SCN and this may lead to situations where LGWR may not have
flushed all records for the same SCN before being mined by the connector.
2023-07-17 10:22:46 +02:00
Chris Cranford
877a19e278 DBZ-6660 Add redo thread id to abandoned transaction warning 2023-07-17 10:22:46 +02:00
Chris Cranford
d660909fc1 DBZ-3605 Cleanup documentation, dependencies, and SAX requirements 2023-07-17 10:21:11 +02:00
Chris Cranford
34882f9468 DBZ-3605 Add support for RAW data type 2023-07-17 10:21:11 +02:00
Chris Cranford
94d363a004 DBZ-3605 Add support for Oracle XMLTYPE data types 2023-07-17 10:21:11 +02:00
Chris Cranford
b6f299b09c DBZ-6650 Align LogMiner included DML event behavior for truncates 2023-07-14 11:21:39 +02:00
Chris Cranford
56ddbe21d5 DBZ-6650 Avoid ConcurrentModificationException during dispatch 2023-07-14 11:21:39 +02:00
harveyyue
5bc7fd1f7c DBZ-6603 Support for custom tags in the connector metrics 2023-07-13 09:52:33 +02:00
mfvitale
b0b9975782 DBZ-6633 Throw a RetriableException when a SQLRecoverableException occurs during OracleConnection instantiation 2023-07-05 16:50:26 -04:00
Chris Cranford
0518999152 DBZ-6610 Fix retrying nested ORA-01327 exception 2023-06-28 08:45:33 +02:00
mfvitale
d5febcc054 DBZ-6417 Provide extensible signal action mechanism through SPI 2023-06-23 15:03:27 +02:00
mfvitale
9ed928a1c9 DBZ-6416 Improve tests stability 2023-06-22 13:22:34 +02:00
mfvitale
34e28ac52d DBZ-6416 Add initial snapshot notifications 2023-06-22 13:22:34 +02:00
Debezium Builder
60f730d383 [maven-release-plugin] prepare for next development iteration 2023-06-20 13:21:31 +00:00
Debezium Builder
150396e4c8 [maven-release-plugin] prepare release v2.3.0.Final 2023-06-20 13:21:31 +00:00
Jiri Pechanec
b8607db61c DBZ-6528 Use correct connection 2023-06-19 20:23:17 +02:00
Vojtech Juranek
8172434dda DBZ-6513 Fix the computation of Oracle's negative interval 2023-06-16 07:32:20 +02:00
Jiri Pechanec
067683e56a DBZ-6076 Consistent config for all connectors 2023-06-15 11:55:28 +02:00
Anil Dasari
0431a8b9fc DBZ-6076: Add support for custom SourceInfoStructMaker for adding new fields to source field 2023-06-15 11:55:28 +02:00
Chris Cranford
45d6f2c5ef DBZ-6193 Upgrade Infinispan to 14.0.11.Final 2023-06-15 06:25:50 +02:00
Chris Cranford
ef8059f3ff DBZ-6528 Fix Oracle NPE when storing only captured table schemas with signals 2023-06-14 10:48:19 +02:00
Chris Cranford
ab28e12d2e DBZ-6359 Allow specifying SIDs in Oracle rac.nodes parameter 2023-06-14 08:48:23 +02:00
Jiri Pechanec
c41e70a116 DBZ-6499 Atribute not needed to be passed as param 2023-06-13 07:09:01 +02:00
Chris Cranford
e025230d44 DBZ-6499 Support restarting JDBC connections
Introduces a new configuration option, `log.mining.restart.connection`,
which closes and re-opens the JDBC connection when an Oracle Log switch
occurs or when the optionally configured log mining session max lifetime
is reached.
2023-06-13 07:09:01 +02:00
Debezium Builder
5d1b00588a [maven-release-plugin] prepare for next development iteration 2023-06-09 09:23:38 +00:00
Debezium Builder
d1e6107517 [maven-release-plugin] prepare release v2.3.0.CR1 2023-06-09 09:23:37 +00:00
mfvitale
7338fde84e DBZ-6424 Emit JMX Notification when Debezium generate its own notification
DBZ-6424 Emit JMX Notification when Debezium generate its own notification

DBZ-6424 Emit JMX Notification when Debezium generate its own notification
2023-06-07 07:36:58 +02:00
Ronak Jain
3f3967c2bb DBZ-6366: Code Improvements for skip.messages.without.change
Addresses/Closes DBZ-6366
2023-06-05 14:50:25 +02:00
Debezium Builder
aa5b2131fc [maven-release-plugin] prepare for next development iteration 2023-05-26 10:35:20 +00:00
Debezium Builder
9cc9b3c29a [maven-release-plugin] prepare release v2.3.0.Beta1 2023-05-26 10:35:20 +00:00
Chris Cranford
9ed6a3ae97 DBZ-6502 Fix Oracle 23.x JDBC driver compatibility
Solves ORA-18176 error "{} not in any time zone.DATE" error.
2023-05-25 09:29:09 +02:00
Jiri Pechanec
65b46f204a DBZ-6496 Speed-up another incremental snapshot test 2023-05-24 18:51:49 +02:00
mfvitale
e0659f1a0e DBZ-6496 Fix performance issue on testNotification test with Oracle and MS SQL Server connectors 2023-05-24 18:51:49 +02:00
Jiri Pechanec
d5a3095717 DBZ-6457 Fix formatting 2023-05-19 14:04:14 +02:00
Chris Cranford
d40da4f810 DBZ-6457 Use PDB for parallel snapshot connections 2023-05-19 14:04:14 +02:00
Vojtech Juranek
a58f4b4d4f DBZ-6439 Filter tables only when user want to capture schema for included tables only 2023-05-16 14:48:48 -04:00
Vojtech Juranek
ee4ebaa957 DBZ-6439 Don't scan all the tables when loading schema
Currently we scan all the tables, which may result into a substantial
delay in initial snapshot when the database is very large. We need to
filter out tables which we are not interested in.

Add back table filter when loading schema of tables. As per comment of
this block of code, passing all tables and table filter should be faster
than passing only list of tables we are interested in.
2023-05-15 14:11:47 +02:00
Debezium Builder
a5b44767bc [maven-release-plugin] prepare for next development iteration 2023-05-11 11:05:24 +00:00
Debezium Builder
1b1a7531f2 [maven-release-plugin] prepare release v2.3.0.Alpha1 2023-05-11 11:05:24 +00:00
Jiri Pechanec
b57ac1ca5a DBZ-1973 Guard against uninited fields in Xstream 2023-05-11 11:13:38 +02:00
mfvitale
7429b4e6f1 DBZ-1973 Add incremental snapshot notifications 2023-05-10 12:22:02 +02:00
mfvitale
811b800f55 DBZ-1973 Add support for Notifications
DBZ-1973 Add more tests for other connectors

DBZ-1973 Add send method with offset parameter

DBZ-1973 Instantiate NotificationService in the task class

DBZ-1973 Instantiate NotificationService in the task class

DBZ-4027 Move specific sink channel configuration to SinkNotificationChannel

DBZ-4027 Remove not used SPI file

DBZ-1973 Moved SPI file definition to debezium-core

DBZ-1973 Rename KafkaNotificationChannel to more generic SinkNotificationChannel

DBZ-1973 Code refactor

DBZ-1973 Improve configuration property description

DBZ-1973 Improve test

DBZ-1973 Add close method to NotificationChannel

DBZ-1973 Implement KafkaNotificationChannel

DBZ-1973 Add NotificationService and LogNotificationChannel
2023-05-10 12:22:02 +02:00
Chris Cranford
fc4a31e3d7 DBZ-6436 Improve Oracle LogMiner failure logging
This change explicitly captures the state of two critical tables when the connector
fails to start a LogMiner session, namely V$LOGMNR_LOGS and V$LOGMNR_PARAMETERS.
While these were previously captured by the LogMinerDatabaseStateWriter, the old
code required DEBUG log level and that isn't always feasible, so this change will
capture these in ERROR log level regardless moving forward.
2023-05-09 12:09:36 +02:00
Chris Cranford
30eb53c764 DBZ-6254 Add LogMiner Query and Username Include Filtering
* Add `log.mining.query.filter.mode` configuration option
* Add `log.mining.username.include.list` configuration option
2023-05-09 11:31:08 +02:00
mfvitale
952340286e DBZ-4027 Make signalling channel configurable
DBZ-4027 Add an Incremental snapshot test with kafka signaling

DBZ-4027 Add an Incremental snapshot test with kafka signaling

DBZ-4027 Add an Incremental snapshot test with kafka signaling

DBZ-4027 Code style

DBZ-4027 Make SignalPayload more generic and extensible

DBZ-4027 Rename DatabaseSignalChannel to SourceSignalChannel

DBZ-4027 Improve logging

DBZ-4027 Moved SPI file definition to debezium-core

DBZ-4027 Move SignalProcessor synchronization point to be processed only when a signal cdc event arrives.

DBZ-4027 Add EventDispatcher constructor without signalProcessor for spanner connector

DBZ-4027 Fix NPE

DBZ-4027 Fix NPE

DBZ-4027 Formatting

DBZ-4027 Correctly manage signal on not supported connector

DBZ-4027 Use the correct MongoDbOffset

DBZ-4027 Correctly initialize offset for Oracle and SqlServer connectors

DBZ-4027 Register SPI implementations

DBZ-4027 Improve SignalProcessor instantiation

DBZ-4027 Pass source info in case of SchemaChanges action

DBZ-4027 Manage close event in a synchronous way

DBZ-4027 Correctly init offset context also in case of snapshot mode 'never'

DBZ-4027 Fix MySqlMetricsIT test

DBZ-4027 Move KafkaSignalChannel to core

DBZ-4027 Move KafkaSignalChannel to core

DBZ-4027 Set pass offset context after initial snapshot to SignalProcessor

DBZ-4027 Pass OffsetContext to signal processor

DBZ-4027 Pass CommonConnectorConfig to SignalChannelReader init method

DBZ-4027 Move Incremental snapshot window actions to dedicated package

DBZ-4027 Align SignalsIT test with new code

DBZ-4027 Fix SignalsIT test

DBZ-4027 Fix SignalProcessor scheduling

DBZ-4027 Moved DatabaseSignalChannel and SignalChannelReader to dedicated package

DBZ-4027 Start SignalProcessor from ChangeEventSourceCoordinator

DBZ-4027 Create SignalProcessor and renamed Signal to DatabaseSignalChannel

DBZ-4027 Initial refactoring of signal feature
2023-05-05 13:35:40 +02:00
harveyyue
3860a4faca DBZ-6388 __source_ts_ms r (read) operation date is set to future for SQL Server 2023-05-02 08:11:02 -04:00
ani-sha
a453c3ca19 DBZ-6259 Add missing description of various snapshot modes in the connector field definitions 2023-05-01 11:03:04 -04:00
Chris Cranford
3f554645b6 DBZ-6256 Add log.mining.flush.table.name Oracle config option
There is the potential when using multiple connector deployments on Oracle that
there could be some level of lock contention with all connectors using the same
table. This allows users to configure the flush table name themselves, reducing
the lock contention across multiple connector deployments.
2023-05-01 10:41:05 -04:00