Commit Graph

1714 Commits

Author SHA1 Message Date
mfvitale
811b800f55 DBZ-1973 Add support for Notifications
DBZ-1973 Add more tests for other connectors

DBZ-1973 Add send method with offset parameter

DBZ-1973 Instantiate NotificationService in the task class

DBZ-1973 Instantiate NotificationService in the task class

DBZ-4027 Move specific sink channel configuration to SinkNotificationChannel

DBZ-4027 Remove not used SPI file

DBZ-1973 Moved SPI file definition to debezium-core

DBZ-1973 Rename KafkaNotificationChannel to more generic SinkNotificationChannel

DBZ-1973 Code refactor

DBZ-1973 Improve configuration property description

DBZ-1973 Improve test

DBZ-1973 Add close method to NotificationChannel

DBZ-1973 Implement KafkaNotificationChannel

DBZ-1973 Add NotificationService and LogNotificationChannel
2023-05-10 12:22:02 +02:00
mfvitale
0ca87d8031 DBZ-4027 During event dispatch process only signal from source channel 2023-05-05 13:35:40 +02:00
mfvitale
952340286e DBZ-4027 Make signalling channel configurable
DBZ-4027 Add an Incremental snapshot test with kafka signaling

DBZ-4027 Add an Incremental snapshot test with kafka signaling

DBZ-4027 Add an Incremental snapshot test with kafka signaling

DBZ-4027 Code style

DBZ-4027 Make SignalPayload more generic and extensible

DBZ-4027 Rename DatabaseSignalChannel to SourceSignalChannel

DBZ-4027 Improve logging

DBZ-4027 Moved SPI file definition to debezium-core

DBZ-4027 Move SignalProcessor synchronization point to be processed only when a signal cdc event arrives.

DBZ-4027 Add EventDispatcher constructor without signalProcessor for spanner connector

DBZ-4027 Fix NPE

DBZ-4027 Fix NPE

DBZ-4027 Formatting

DBZ-4027 Correctly manage signal on not supported connector

DBZ-4027 Use the correct MongoDbOffset

DBZ-4027 Correctly initialize offset for Oracle and SqlServer connectors

DBZ-4027 Register SPI implementations

DBZ-4027 Improve SignalProcessor instantiation

DBZ-4027 Pass source info in case of SchemaChanges action

DBZ-4027 Manage close event in a synchronous way

DBZ-4027 Correctly init offset context also in case of snapshot mode 'never'

DBZ-4027 Fix MySqlMetricsIT test

DBZ-4027 Move KafkaSignalChannel to core

DBZ-4027 Move KafkaSignalChannel to core

DBZ-4027 Set pass offset context after initial snapshot to SignalProcessor

DBZ-4027 Pass OffsetContext to signal processor

DBZ-4027 Pass CommonConnectorConfig to SignalChannelReader init method

DBZ-4027 Move Incremental snapshot window actions to dedicated package

DBZ-4027 Align SignalsIT test with new code

DBZ-4027 Fix SignalsIT test

DBZ-4027 Fix SignalProcessor scheduling

DBZ-4027 Moved DatabaseSignalChannel and SignalChannelReader to dedicated package

DBZ-4027 Start SignalProcessor from ChangeEventSourceCoordinator

DBZ-4027 Create SignalProcessor and renamed Signal to DatabaseSignalChannel

DBZ-4027 Initial refactoring of signal feature
2023-05-05 13:35:40 +02:00
Jiri Pechanec
58ef4f0b98 DBZ-6157 Disable potentially dangerous MySQL JDBC props 2023-05-04 11:54:04 +02:00
Jiri Pechanec
6b1455666a DBZ-6411 Message content logged only at TRACE 2023-05-03 06:51:28 +02:00
mfvitale
e5d4332835 DBZ-6411 Fix HeaderToValue schema update cache.
When the connector starts for the first time with initial snapshot record with read operation will be created. In this phase the ExtractChangedRecordState will produce no headers and since HeaderToValue was not skipping record without headers the cache of the new schema will be created only with the old fields.
2023-05-03 06:51:28 +02:00
harveyyue
3860a4faca DBZ-6388 __source_ts_ms r (read) operation date is set to future for SQL Server 2023-05-02 08:11:02 -04:00
Chris Cranford
4664dd6df6 DBZ-6355 Add log.mining.transaction.retention.ms; deprecate log.mining.transaction.retention.hours 2023-04-27 09:28:29 -04:00
eizners
fa42a1b0ce DBZ-6355 fix unit tests errors 2023-04-27 09:28:29 -04:00
Sergey Eizner
cac5485176 DBZ-6355 Change to the last offset 2023-04-27 09:28:29 -04:00
Jiri Pechanec
22e19c0ff3 DBZ-6209 Extract additional common code 2023-04-27 08:13:58 +02:00
harveyyue
795f0dd977 DBZ-6209 Refactor storage implementations 2023-04-27 08:13:58 +02:00
Jiri Pechanec
8c52ccfd0c DBZ-6391 Use incdluded term for consistency 2023-04-26 12:31:15 +02:00
zalmane
2b8bf33876 DBZ-6391 change logging level 2023-04-26 12:31:15 +02:00
Debezium Builder
bb1b79f3bd [maven-release-plugin] prepare for next development iteration 2023-04-20 09:09:43 +02:00
Debezium Builder
5483b4de99 [maven-release-plugin] prepare release v2.2.0.Final 2023-04-20 09:09:43 +02:00
Ronak Jain
762122ecf5 DBZ-2979 If column.include.list/column.exclude.list are used and the target table receives an update for the excluded (or not included) column - such events should be ignored
* Add Mysql and SqlServer Connector Changes
* Change Docs
* Clean Test Case
* Formatting and Checkstyle

Addresses/Closes DBZ-2979
2023-04-19 09:04:29 +02:00
Jiri Pechanec
5eb5e3555d DBZ-6324 Rename flaky test config option 2023-04-18 08:22:43 +02:00
Jiri Pechanec
ab66f038e4 DBZ-6324 Add retry for flaking tests 2023-04-18 08:22:43 +02:00
Jiri Pechanec
f64c35de29 DBZ-6324 Provide @Flaky annotation for CI unstable tests 2023-04-18 08:22:43 +02:00
Debezium Builder
976a47a3a3 [maven-release-plugin] prepare for next development iteration 2023-04-14 08:46:06 +00:00
Debezium Builder
1ca8399e8b [maven-release-plugin] prepare release v2.2.0.CR1 2023-04-14 08:46:06 +00:00
Jiri Pechanec
8a648fbb4d DBZ-4488 Refactor code to keep error handler immutable 2023-04-13 08:22:45 +02:00
Jacob Gminder
5e615ff796 DBZ-4488: Reuse the ErrorHandler between retries 2023-04-13 08:22:45 +02:00
Jiri Pechanec
84d0d580bc DBZ-4488 Refactor the code 2023-04-13 08:22:45 +02:00
Jacob Gminder
9d5b6df0f7 DBZ-4488: Allow SQL Server connectors to set a maximum number of retriable exception restarts. 2023-04-13 08:22:45 +02:00
Jiri Pechanec
943b9485d7 DBZ-6216 Remove registry name for tag query 2023-04-12 14:07:35 +02:00
harveyyue
c525f34cd0 DBZ-6070 DDL events not stored in schema history topic for excluded tables 2023-04-06 12:10:30 +02:00
Jiri Pechanec
39ee498e64 DBZ-5902 Fix wording 2023-04-06 10:55:19 +02:00
mfvitale
bad6ad6bae DBZ-5902 Change fields configuration from 'partition.payload.field' to 'partition.payload.fields' 2023-04-06 10:55:19 +02:00
mfvitale
41416293ca DBZ-5902 Refactor and better field validation 2023-04-06 10:55:19 +02:00
mfvitale
fdf79b0379 DBZ-5902 Add warn log for deprecation 2023-04-06 10:55:19 +02:00
mfvitale
34f021988e DBZ-5902 Add new PartitionRouting SMT that will replace ComputePartition 2023-04-06 10:55:19 +02:00
mfvitale
6e9528a47d DBZ-5902 Deprecate ComputePartition SMT 2023-04-06 10:55:19 +02:00
mfvitale
80ac1b362f DBZ-5902 Moved list validation to CommonConnectorConfig 2023-04-06 10:55:19 +02:00
Debezium Builder
5c650fd054 [maven-release-plugin] prepare for next development iteration 2023-03-31 10:00:16 +00:00
Debezium Builder
68ffc4833b [maven-release-plugin] prepare release v2.2.0.Beta1 2023-03-31 10:00:16 +00:00
Jiri Pechanec
b6c984e4de DBZ-6072 Sort imports 2023-03-28 17:06:58 +02:00
Jiri Pechanec
0e42935b35 DBZ-6072 Use more descriptive name 2023-03-28 16:59:05 +02:00
harveyyue
5dd4b53ec3 DBZ-6072 NPE when setting schema.history.internal.store.only.captured.tables.ddl=true 2023-03-28 16:59:05 +02:00
harveyyue
2f46c83b2a DBZ-6104 java.lang.NullPointerException in MySQL connector with max.queue.size.in.bytes 2023-03-28 14:51:52 +02:00
harveyyue
c9c1ce8a2f DBZ-6185 Zerofill property failed for different int types 2023-03-28 09:34:58 +02:00
Jiri Pechanec
e3fdd873e1 DBZ-6201 Skip expensive logging 2023-03-22 14:02:44 +01:00
mfvitale
2d70f47c40 DBZ-6201: Add more fail test scenarios 2023-03-22 14:02:44 +01:00
mfvitale
16191a5a35 DBZ-6201: Add logs 2023-03-22 14:02:44 +01:00
mfvitale
039a3f04e8 DBZ-6201: Add support for move and nested fields for HeaderToValue SMT 2023-03-22 14:02:44 +01:00
mfvitale
c45bf39eb0 DBZ-6201: Add new HeaderToValue SMT to copy headers to value 2023-03-22 14:02:44 +01:00
mfvitale
4e544debdc DBZ-6201: Fix validation message that reports null as value 2023-03-22 14:02:44 +01:00
Chris Cranford
1c5f14f7fd DBZ-6225 DBZ-6226 DBZ-6231 Introduce CustomConverter for JDBC sink consistency
In DBZ-6225, it was identified that the schema type associated with TINYINT(1)
and BOOLEAN data types wasn't consistent. This converter implementation will
always emit BOOLEAN data types as INT16, similar to TINYINT(1).

In DBZ-6226, it was identified that the schema type associated with REAL data
types wasn't consistent, emitted as FLOAT64 during snapshot and FLOAT32 in
the streaming phase. This converter implementation will always emit such types
as FLOAT64 for consistency.

In DBZ-6231, during streaming the DDL parser will get the column type as the
literal type supplied by the user's DDL, i.e. NCHAR or NVARCHAR but when the
DDL is sourced from SHOW CREATE TABLE, MySQL returns these columns as types
of CHAR/VARCHAR but with a character set.  By passing the character_set in
the schema consistency, we can deduce nationalized vs non-nationalized.
2023-03-21 14:53:56 +01:00
harveyyue
1877906dd8 DBZ-6197 Remove duplicated createDdlFilter method from historized connector config 2023-03-13 10:56:43 +01:00
Nir Levy
16f460d2d7 DBZ-6190: Fix support for ddl filter 2023-03-09 08:48:43 +01:00
Debezium Builder
55d65e9618 [maven-release-plugin] prepare for next development iteration 2023-03-08 11:38:17 +00:00
Debezium Builder
6d183c3dc1 [maven-release-plugin] prepare release v2.2.0.Alpha3 2023-03-08 11:38:16 +00:00
subodh
b781960019 DBZ-6179 Fix swallowing of exception cause of finally block in RelationalSnapshotChangeEventSource 2023-03-06 10:28:57 +01:00
Jiri Pechanec
98dcdb8f55 DBZ-6023 Remove nnecessary code 2023-03-06 10:25:59 +01:00
PlugaruT
7d0492fee5 DBZ-6023 Unit tests for buildChunkQuery and buildMaxPrimaryKeyQuery methods 2023-03-06 10:25:59 +01:00
PlugaruT
472162e46e DBZ-6023 Add support for providing a surrogate key when triggering incremental snapshots
For tables that have a composite Primary Key, SQL queries for incremental snapshot perform bad. This change allows to provide a surrogate key that will be used for the queries. The key should be unique in order for the snapshot to guarantee consistency.
2023-03-06 10:25:59 +01:00
Jiri Pechanec
0585c6beb8 DBZ-823 Add NULL instance for the class 2023-03-03 12:47:15 +01:00
Jiri Pechanec
d492ec892c DBZ-823 Use enum for enumeration 2023-03-03 12:47:15 +01:00
Jiri Pechanec
3da2434c6c DBZ-823 Restrict the type with generics 2023-03-03 12:47:15 +01:00
Jiri Pechanec
5e666f00ee DBZ-823 Add JavaDoc to new interfaces 2023-03-03 12:47:15 +01:00
Jiri Pechanec
7d92dc0211 DBZ-823 Don't use getter based name 2023-03-03 12:47:15 +01:00
Jiri Pechanec
0f4adb9fe6 DBZ-823 Use better name for CF with main connection 2023-03-03 12:47:15 +01:00
ggaborg
e875e784a0 DBZ-823 CR changes 2023-03-03 12:47:15 +01:00
ggaborg
065dc997b9 DBZ-823 Optionally parallelize initial snapshots 2023-03-03 12:47:15 +01:00
Jiri Pechanec
0656de70c9 DBZ-6156 Add an extra test case 2023-03-03 11:20:21 +01:00
jcechace
7d3cca89bd DBZ-6032 MongoConnection allways rethrows InterruptedException 2023-03-03 05:46:33 +01:00
jcechace
922191cdd7 DBZ-6032 Connection management refactoring 2023-03-03 05:46:33 +01:00
harveyyue
440192851d DBZ-6156 Add hostname validator to connector configuration 2023-02-28 13:22:14 +01:00
Vojtech Juranek
0868bfd55e DBZ-3594 Remove unused deprecated method 2023-02-23 09:26:07 -05:00
Debezium Builder
2245ceda44 [maven-release-plugin] prepare for next development iteration 2023-02-16 11:28:25 +00:00
Debezium Builder
20f3b16e8a [maven-release-plugin] prepare release v2.2.0.Alpha2 2023-02-16 11:28:25 +00:00
Chris Cranford
a7d1bb0f86 DBZ-5283 Decoupled changed SMT vs new state SMT, updated docs 2023-02-10 13:53:05 +01:00
Chris Cranford
297225d141 DBZ-5283 Add unchanged field support to ExtractNewRecordState 2023-02-10 13:53:05 +01:00
Chris Cranford
f8eb32cf6f DBZ-5283 Move ExtractChangedRecordState tests to own class 2023-02-10 13:53:05 +01:00
harveyyue
4cb6e7c5b3 DBZ-5283 Introduce a new SMT ExtractChangedRecordState to capture the names of changed/unchanged field within headers 2023-02-10 13:53:05 +01:00
Jeremy Ford
517b43b870 DBZ-5827 ActivateTracingSpan wrong timestamps reported
Change txLogSpan to use processingTimestamp for span finish.
2023-02-07 13:25:44 +01:00
Ismail Simsek
3548daf8b9 DBZ-6075 Use Instantiator to load ClassLoader 2023-02-07 07:24:03 +01:00
Jiri Pechanec
3a646519de DBZ-5743 Modify Javadoc 2023-02-02 13:21:45 +01:00
harveyyue
57b902ff8c DBZ-5743 Introduce alternative AVRO_UNICODE mode for field name selector 2023-02-02 13:21:45 +01:00
harveyyue
f093d7a6c9 DBZ-5743 Introduce alternative AVRO_UNICODE mode for schema adjuster 2023-02-02 13:21:45 +01:00
harveyyue
003bef0736 DBZ-5743 Introduce a new unicode topic naming strategy 2023-02-02 13:21:45 +01:00
Jiri Pechanec
795014e040 DBZ-6037 Data no longer logged on error 2023-01-26 13:30:20 +01:00
Jiri Pechanec
ad3c6981bd DBZ-5879 Remove duplicate dependency 2023-01-24 14:29:00 +01:00
Jiri Pechanec
8f3ca36e23 DBZ-6026 Don't update last offsets in statistics 2023-01-24 14:26:05 +01:00
Vojtech Juranek
f00f2a8735 DBZ-5917 Fix querying column metadata for tables with back slash in the name
If the table contains back slash, which is ANSI SQL escape chracter,
in its name, querying column metadata would fail in some cases (*)
which would result into NPE. Fix table name before the query and
escape escape character if the table name contains it.

Default backslash works for all currently supported databases as it's
ANSI SQL standard,

(*) everything works when we collect column metadata for all tables,
i.e. when there are no excluded tables
2023-01-20 00:01:11 +01:00
Debezium Builder
04f5291a2d [maven-release-plugin] prepare for next development iteration 2023-01-19 11:32:10 +00:00
Debezium Builder
027527963e [maven-release-plugin] prepare release v2.2.0.Alpha1 2023-01-19 11:32:10 +00:00
Anil Dasari
e1f9d64fbb DBZ-5852:Snapshotter#snapshotCompleted is invoked regardless of snapshot result 2023-01-19 09:24:02 +01:00
Jiri Pechanec
1ec41fa120 DBZ-5879 Added logging; removed commented out code 2023-01-18 14:46:14 +01:00
Jeremy Ford
553cb9b1dd DBZ-5879 Add wait to tests in order to maintain expectations of existing integration tests 2023-01-18 14:46:14 +01:00
Jeremy Ford
f2a6f0b8db DBZ-5879 Support retrying database connection failures during connector start
Reworked when task requests start from subclasses
Add support for restarting
2023-01-18 14:46:14 +01:00
Jiri Pechanec
e8729bb0eb DBZ-5996 Properly handle zoned timestamp width in Oracle 2023-01-18 10:42:36 +01:00
Jiri Pechanec
bcd6787cdc DBZ-6008 Revert "DBZ-3653 Introduce incremental.snapshot.isolation.mode config"
This reverts commit 6d8bcd42fd.
2023-01-17 10:40:20 +01:00
Jiri Pechanec
8738e62459 DBZ-6008 Revert "DBZ-3653 Remove extra modifiers"
This reverts commit bdfc1e6861.
2023-01-17 10:40:20 +01:00
Jiri Pechanec
0c5e4f0051 DBZ-6002 Fix tests now requiring fractions 2023-01-13 12:00:00 +01:00
harveyyue
bea65acc0e DBZ-6002 Run PostgresConnectorIT.shouldReceiveChangesForChangeColumnDefault() failed 2023-01-13 12:00:00 +01:00
akshansh
444dab3ffa DBZ-5996 Handled case where fractionalWidth was -1 2023-01-11 12:09:26 +01:00
akshansh
b87d80f462 DBZ-5996 Implemented custom DateTimeFormatter to parse ZonedDateTime while preserving length of column. 2023-01-11 12:09:26 +01:00
harveyyue
5e56ad75f8 DBZ-5985 Disable to print invalid estimated row count when config "snapshot.select.statement.overrides" 2023-01-11 09:20:15 +01:00