jcechace
5cb579bf12
DBZ-6525 Removing runtime dependency on core for testcontainers module in order to prevent logback.xml downstream conflicts
2023-07-11 12:07:04 +02:00
mfvitale
0baad2f7b7
DBZ-6417 Add test for custom action
2023-06-23 15:03:27 +02:00
mfvitale
d5febcc054
DBZ-6417 Provide extensible signal action mechanism through SPI
2023-06-23 15:03:27 +02:00
mfvitale
9ed928a1c9
DBZ-6416 Improve tests stability
2023-06-22 13:22:34 +02:00
mfvitale
34e28ac52d
DBZ-6416 Add initial snapshot notifications
2023-06-22 13:22:34 +02:00
Anatolii Popov
b207f96320
DBZ-6577 Don't try to alter publication created for all tables when configuration changes to 'filtered'
2023-06-21 15:58:11 +02:00
Debezium Builder
60f730d383
[maven-release-plugin] prepare for next development iteration
2023-06-20 13:21:31 +00:00
Debezium Builder
150396e4c8
[maven-release-plugin] prepare release v2.3.0.Final
2023-06-20 13:21:31 +00:00
Jiri Pechanec
067683e56a
DBZ-6076 Consistent config for all connectors
2023-06-15 11:55:28 +02:00
Anil Dasari
0431a8b9fc
DBZ-6076: Add support for custom SourceInfoStructMaker for adding new fields to source field
2023-06-15 11:55:28 +02:00
Vojtech Juranek
78668df5c1
DBZ-6547 Declare exactly-once semantic support for Postgres connector
2023-06-12 12:44:12 +02:00
Debezium Builder
5d1b00588a
[maven-release-plugin] prepare for next development iteration
2023-06-09 09:23:38 +00:00
Debezium Builder
d1e6107517
[maven-release-plugin] prepare release v2.3.0.CR1
2023-06-09 09:23:37 +00:00
mfvitale
7338fde84e
DBZ-6424 Emit JMX Notification when Debezium generate its own notification
...
DBZ-6424 Emit JMX Notification when Debezium generate its own notification
DBZ-6424 Emit JMX Notification when Debezium generate its own notification
2023-06-07 07:36:58 +02:00
mfvitale
769d9361e7
DBZ-6424 Add JMX Notification channel
...
This channel permits to send Debezium notification to JMX bean
DBZ-6424 Add JMX Signal channel
This channel permits to send signal to Debezium through the JMX operation
2023-06-07 07:36:58 +02:00
mfvitale
537cc50d65
DBZ-6524 Add support for ltree
and isbn
PostgreSQL data-types in case INCLUDE_UNKNOWN_DATATYPES
is set to false
2023-06-06 06:31:00 +02:00
Ronak Jain
3f3967c2bb
DBZ-6366: Code Improvements for skip.messages.without.change
...
Addresses/Closes DBZ-6366
2023-06-05 14:50:25 +02:00
Debezium Builder
aa5b2131fc
[maven-release-plugin] prepare for next development iteration
2023-05-26 10:35:20 +00:00
Debezium Builder
9cc9b3c29a
[maven-release-plugin] prepare release v2.3.0.Beta1
2023-05-26 10:35:20 +00:00
Debezium Builder
a5b44767bc
[maven-release-plugin] prepare for next development iteration
2023-05-11 11:05:24 +00:00
Debezium Builder
1b1a7531f2
[maven-release-plugin] prepare release v2.3.0.Alpha1
2023-05-11 11:05:24 +00:00
mfvitale
7429b4e6f1
DBZ-1973 Add incremental snapshot notifications
2023-05-10 12:22:02 +02:00
mfvitale
811b800f55
DBZ-1973 Add support for Notifications
...
DBZ-1973 Add more tests for other connectors
DBZ-1973 Add send method with offset parameter
DBZ-1973 Instantiate NotificationService in the task class
DBZ-1973 Instantiate NotificationService in the task class
DBZ-4027 Move specific sink channel configuration to SinkNotificationChannel
DBZ-4027 Remove not used SPI file
DBZ-1973 Moved SPI file definition to debezium-core
DBZ-1973 Rename KafkaNotificationChannel to more generic SinkNotificationChannel
DBZ-1973 Code refactor
DBZ-1973 Improve configuration property description
DBZ-1973 Improve test
DBZ-1973 Add close method to NotificationChannel
DBZ-1973 Implement KafkaNotificationChannel
DBZ-1973 Add NotificationService and LogNotificationChannel
2023-05-10 12:22:02 +02:00
mfvitale
0ca87d8031
DBZ-4027 During event dispatch process only signal from source channel
2023-05-05 13:35:40 +02:00
mfvitale
952340286e
DBZ-4027 Make signalling channel configurable
...
DBZ-4027 Add an Incremental snapshot test with kafka signaling
DBZ-4027 Add an Incremental snapshot test with kafka signaling
DBZ-4027 Add an Incremental snapshot test with kafka signaling
DBZ-4027 Code style
DBZ-4027 Make SignalPayload more generic and extensible
DBZ-4027 Rename DatabaseSignalChannel to SourceSignalChannel
DBZ-4027 Improve logging
DBZ-4027 Moved SPI file definition to debezium-core
DBZ-4027 Move SignalProcessor synchronization point to be processed only when a signal cdc event arrives.
DBZ-4027 Add EventDispatcher constructor without signalProcessor for spanner connector
DBZ-4027 Fix NPE
DBZ-4027 Fix NPE
DBZ-4027 Formatting
DBZ-4027 Correctly manage signal on not supported connector
DBZ-4027 Use the correct MongoDbOffset
DBZ-4027 Correctly initialize offset for Oracle and SqlServer connectors
DBZ-4027 Register SPI implementations
DBZ-4027 Improve SignalProcessor instantiation
DBZ-4027 Pass source info in case of SchemaChanges action
DBZ-4027 Manage close event in a synchronous way
DBZ-4027 Correctly init offset context also in case of snapshot mode 'never'
DBZ-4027 Fix MySqlMetricsIT test
DBZ-4027 Move KafkaSignalChannel to core
DBZ-4027 Move KafkaSignalChannel to core
DBZ-4027 Set pass offset context after initial snapshot to SignalProcessor
DBZ-4027 Pass OffsetContext to signal processor
DBZ-4027 Pass CommonConnectorConfig to SignalChannelReader init method
DBZ-4027 Move Incremental snapshot window actions to dedicated package
DBZ-4027 Align SignalsIT test with new code
DBZ-4027 Fix SignalsIT test
DBZ-4027 Fix SignalProcessor scheduling
DBZ-4027 Moved DatabaseSignalChannel and SignalChannelReader to dedicated package
DBZ-4027 Start SignalProcessor from ChangeEventSourceCoordinator
DBZ-4027 Create SignalProcessor and renamed Signal to DatabaseSignalChannel
DBZ-4027 Initial refactoring of signal feature
2023-05-05 13:35:40 +02:00
Jiri Pechanec
ab5d643aa8
DBZ-6353 Use correct SQL codes for other cases
2023-05-03 12:15:18 +02:00
umachi
6ced91fe53
DBZ-6353 Add a conditions of 'OBJECT_IN_USE', 'NOT_IMPLEMENTED'.
2023-05-03 12:15:18 +02:00
miguelbirdie
63cc36e9a2
DBZ-6112 Set Replica Identity for PostgreSQL tables
...
Introduces a new PostgreSQL configuration parameter called `replica.autoset.type`. With this parameter, you can easily specify the Replica Identity value for each table that is captured by the connector
2023-05-02 14:33:46 +02:00
ani-sha
a453c3ca19
DBZ-6259 Add missing description of various snapshot modes in the connector field definitions
2023-05-01 11:03:04 -04:00
Nir Levy
34aa5ed8b9
DBZ-6396 Honor slot.max.retries when committing LSN before starting replication
2023-04-28 15:01:26 +02:00
Frederic Laurent
5139957f91
DBZ-6340 Add support to the Postgres connector for allow and prefer SSL modes (with prefer as the new default value)
2023-04-26 12:48:54 +02:00
Bertrand Paquet
82cf55e3ec
DBZ-6384: Fix snapshotting with hstore in Map mode
2023-04-26 12:43:21 +02:00
Jiri Pechanec
643a82e4ee
DBZ-6122 Refactor for more safety
2023-04-24 07:06:42 +02:00
Bertrand Paquet
979d4529b7
DBZ-6122 Toasted varying character array and date array are not correctly processed
2023-04-24 07:06:42 +02:00
Jiri Pechanec
28563b7244
DBZ-6353 Use SQL state to identify missing function
2023-04-24 06:36:58 +02:00
Jiri Pechanec
07947e1540
DBZ-6379 Fix imports
2023-04-24 06:03:56 +02:00
Bertrand Paquet
8a224e6588
DBZ-6379: Fix handling of toasted hstore
2023-04-24 06:03:56 +02:00
Debezium Builder
bb1b79f3bd
[maven-release-plugin] prepare for next development iteration
2023-04-20 09:09:43 +02:00
Debezium Builder
5483b4de99
[maven-release-plugin] prepare release v2.2.0.Final
2023-04-20 09:09:43 +02:00
Mark Bereznitsky
83629089a0
DBZ-6265 always check the WAL_LEVEL but fail if streaming is required
2023-04-19 10:54:39 +02:00
Jiri Pechanec
f4c200ac3e
DBZ-2979 Fix tests
2023-04-19 09:04:29 +02:00
Ronak Jain
465b355dd0
DBZ-2979 If column.include.list/column.exclude.list are used and the target table receives an update for the excluded (or not included) column - such events should be ignored
...
* Add More Test Cases
Addresses/Closes DBZ-2979
2023-04-19 09:04:29 +02:00
Ronak Jain
762122ecf5
DBZ-2979 If column.include.list/column.exclude.list are used and the target table receives an update for the excluded (or not included) column - such events should be ignored
...
* Add Mysql and SqlServer Connector Changes
* Change Docs
* Clean Test Case
* Formatting and Checkstyle
Addresses/Closes DBZ-2979
2023-04-19 09:04:29 +02:00
Debezium Builder
976a47a3a3
[maven-release-plugin] prepare for next development iteration
2023-04-14 08:46:06 +00:00
Debezium Builder
1ca8399e8b
[maven-release-plugin] prepare release v2.2.0.CR1
2023-04-14 08:46:06 +00:00
Jiri Pechanec
925de0184c
DBZ-6191 Do not fail if slot cannot be advanced
2023-04-14 07:44:16 +02:00
Vojtech Juranek
44f2f40c56
DBZ-6216 Update testcontainers to use quay.io images
...
When needed update also version.
Update also leftovers in readme files.
2023-04-12 14:07:35 +02:00
Jiri Pechanec
75e680cfb0
DBZ-6265 Correctly format error message
2023-04-04 07:05:55 +02:00
Mark Bereznitsky
6b82ee67e1
DBZ-6265 warn in case no wal support and no cdc
2023-04-04 07:05:55 +02:00
Mark Bereznitsky
4fea0cc895
DBZ-6265 skip the wal level check if no cdc is to occur
2023-04-04 07:05:55 +02:00
Jiri Pechanec
77c971880c
DBZ-6261 Allow different version for protoc
2023-04-04 05:40:27 +02:00
Debezium Builder
5c650fd054
[maven-release-plugin] prepare for next development iteration
2023-03-31 10:00:16 +00:00
Debezium Builder
68ffc4833b
[maven-release-plugin] prepare release v2.2.0.Beta1
2023-03-31 10:00:16 +00:00
mfvitale
673a42b541
DBZ-6232 Support 'binary.handling.mode' also on bytea array
2023-03-27 15:55:38 +02:00
mfvitale
ed67ff6eb9
DBZ-6232 Manage BYTEA_ARRAY in PostgresValueConverter
2023-03-27 15:55:38 +02:00
Anatolii Popov
930d9b0971
DBZ-6251 Ignore Debezium internal transactions during fallback to restart_lsn in Postgres connector if confirmed_flush_lsn is NULL
2023-03-27 15:55:13 +02:00
Nir Levy
c7e00d3881
DBZ-6249 Fix error thrown when snapshot.custom_class=custom and no snapshot.custom.class
2023-03-27 10:53:39 +02:00
Vojtech Juranek
cfca3e41a3
DBZ-6216 Update postgres notes to use quay.io images
...
Also update postgres version currently being used
2023-03-23 13:29:19 +01:00
Jiri Pechanec
d4c3e23767
DBZ-6092 Simplify the code; change logging
2023-03-16 12:58:03 +01:00
Anatolii Popov
cbb785d67a
DBZ-6092 Fallback to restart_lsn in Postgres connector if confirmed_flush_lsn is NULL and no long-running transactions were committed during timeout
2023-03-16 12:58:03 +01:00
Vojtěch Juránek
42b374613a
DBZ-6204 Fix typo
...
Co-authored-by: Jiri Pechanec <jpechane@redhat.com>
2023-03-16 11:32:52 +01:00
Vojtech Juranek
3b5740f1a8
DBZ-6204 Avoid skipping record when determining resume LSN
...
When the last operation before connector stop is tx BEGIN, subsequent
INSERT done after connector stop can have same LSN as BEGIN and thus it
was possible to skip this INSERT when we search for the resume LSN. This
issue was described and fixed in DBZ-5915. However, the same LSN can
have also previous tx COMMIT, so the WAL can look like this:
LSN | operation
------------------
123 | COMMIT
123 | BEGIN
123 | INSERT
124 | COMMIT
When the last stored LSN is 123 for COMMIT operation, we skip also BEGIN
and INSERT operations with LSN 123 and resume from COMMIT with LSN 124.
Adjust condition introduced in DBZ-5915 to consider also the case when
the last processed operation is COMMIT.
2023-03-16 11:32:52 +01:00
Debezium Builder
55d65e9618
[maven-release-plugin] prepare for next development iteration
2023-03-08 11:38:17 +00:00
Debezium Builder
6d183c3dc1
[maven-release-plugin] prepare release v2.2.0.Alpha3
2023-03-08 11:38:16 +00:00
Jiri Pechanec
3da2434c6c
DBZ-823 Restrict the type with generics
2023-03-03 12:47:15 +01:00
Jiri Pechanec
5e666f00ee
DBZ-823 Add JavaDoc to new interfaces
2023-03-03 12:47:15 +01:00
Jiri Pechanec
7d92dc0211
DBZ-823 Don't use getter based name
2023-03-03 12:47:15 +01:00
Jiri Pechanec
0f4adb9fe6
DBZ-823 Use better name for CF with main connection
2023-03-03 12:47:15 +01:00
ggaborg
e875e784a0
DBZ-823 CR changes
2023-03-03 12:47:15 +01:00
ggaborg
065dc997b9
DBZ-823 Optionally parallelize initial snapshots
2023-03-03 12:47:15 +01:00
Chris Cranford
a9d4663456
DBZ-6163 Properly handle zoned timestamp within PostgreSQL
2023-02-28 13:32:57 +01:00
Debezium Builder
2245ceda44
[maven-release-plugin] prepare for next development iteration
2023-02-16 11:28:25 +00:00
Debezium Builder
20f3b16e8a
[maven-release-plugin] prepare release v2.2.0.Alpha2
2023-02-16 11:28:25 +00:00
Yang Wu
d7ab7146f0
DBZ-6089 Expose sequence in postgres parser
2023-02-08 10:13:19 +01:00
harveyyue
57b902ff8c
DBZ-5743 Introduce alternative AVRO_UNICODE mode for field name selector
2023-02-02 13:21:45 +01:00
Jiri Pechanec
8f3ca36e23
DBZ-6026 Don't update last offsets in statistics
2023-01-24 14:26:05 +01:00
Vojtech Juranek
f00f2a8735
DBZ-5917 Fix querying column metadata for tables with back slash in the name
...
If the table contains back slash, which is ANSI SQL escape chracter,
in its name, querying column metadata would fail in some cases (*)
which would result into NPE. Fix table name before the query and
escape escape character if the table name contains it.
Default backslash works for all currently supported databases as it's
ANSI SQL standard,
(*) everything works when we collect column metadata for all tables,
i.e. when there are no excluded tables
2023-01-20 00:01:11 +01:00
Debezium Builder
04f5291a2d
[maven-release-plugin] prepare for next development iteration
2023-01-19 11:32:10 +00:00
Debezium Builder
027527963e
[maven-release-plugin] prepare release v2.2.0.Alpha1
2023-01-19 11:32:10 +00:00
Anil Dasari
e1f9d64fbb
DBZ-5852:Snapshotter#snapshotCompleted is invoked regardless of snapshot result
2023-01-19 09:24:02 +01:00
Jeremy Ford
8da86b78e8
DBZ-5879 Fix unit tests
2023-01-18 14:46:14 +01:00
Jeremy Ford
f2a6f0b8db
DBZ-5879 Support retrying database connection failures during connector start
...
Reworked when task requests start from subclasses
Add support for restarting
2023-01-18 14:46:14 +01:00
Jiri Pechanec
1ee4850273
DBZ-6012 Allow skipping of LSN check failure
2023-01-18 10:34:56 +01:00
Jiri Pechanec
107f09a2e7
DBZ-6001 DBZ-5991 Use strict class for money
2023-01-17 14:52:03 +01:00
harveyyue
93b82ab7b4
DBZ-6001 Postgresql connector parses the null of the money type into 0
2023-01-17 14:52:03 +01:00
harveyyue
bde7af1423
DBZ-5991 There is a problem with postgresql connector parsing the boundary value of money type
2023-01-17 14:52:03 +01:00
Jiri Pechanec
bcd6787cdc
DBZ-6008 Revert "DBZ-3653 Introduce incremental.snapshot.isolation.mode config"
...
This reverts commit 6d8bcd42fd
.
2023-01-17 10:40:20 +01:00
Animesh Kumar
90cd3fdb6d
DBZ-5988 Impose null check on schema and name values prior to applying lowercase operation
2023-01-11 09:04:31 +01:00
govi20
6d8bcd42fd
DBZ-3653 Introduce incremental.snapshot.isolation.mode config
2023-01-11 08:55:33 +01:00
Jiri Pechanec
8ca5e0a9b5
DBZ-4669 Reformat after plugin version consolidation
2023-01-11 06:44:53 +01:00
Mark Lambert
e5d16b14a6
DBZ-2439 Remove redundant modifiers
2023-01-11 06:44:53 +01:00
erdinc
5a063867df
DBZ-5976 Use as much as possible "replace" method rather than "replaceAll"
2023-01-05 07:30:49 +01:00
rajdangwal
b03b4cdd19
DBZ-5915: correctly determine the startStreamingLsn with messageType
2023-01-04 14:50:14 +01:00
rajdangwal
5dde1ab3b5
DBZ-5915: correctly determine the startStreamingLsn
2023-01-04 14:50:14 +01:00
Debezium Builder
cf1720e4cb
[maven-release-plugin] prepare for next development iteration
2022-12-22 10:11:54 +00:00
Debezium Builder
b4ea9b46cd
[maven-release-plugin] prepare release v2.1.0.Final
2022-12-22 10:11:53 +00:00
jchipmunk
189f3c023e
DBZ-5949 Support partitioned tables when updating PG publication
...
PostgresConnection.getAllTableIds(String) method was added to unify getting all tables in database catalog, including partitioned tables.
2022-12-20 09:28:05 +01:00
Praveen Burgu
31745a7dd5
DBZ-5936 Add test for json[] column
2022-12-20 09:18:34 +01:00
Praveen Burgu
48e1db8fc3
DBZ-5936 Add Javadoc
2022-12-20 09:18:34 +01:00
Jiri Pechanec
fff4719fb1
DBZ-5936 Remove separate config option
2022-12-20 09:18:34 +01:00
Praveen Burgu
4cada81661
DBZ-5936 Handle json and jsonb arrays
2022-12-20 09:18:34 +01:00
Praveen Burgu
28d12a836f
DBZ-5936 Move all placeholder logic into one class
2022-12-20 09:18:34 +01:00
Praveen Burgu
54c0cfc14a
DBZ-5936 Handle TOASTed int and bigint array columns
2022-12-20 09:18:34 +01:00
Debezium Builder
177a60a9d3
[maven-release-plugin] prepare for next development iteration
2022-12-16 09:33:38 +00:00
Debezium Builder
f805db8c3c
[maven-release-plugin] prepare release v2.1.0.Beta1
2022-12-16 09:33:38 +00:00
Praveen Burgu
0b4769c793
DBZ-5944 Toasted varchar array is not correctly processed
2022-12-16 08:50:32 +01:00
Praveen Burgu
b4be663168
DBZ-4941 add a comment
2022-12-14 14:26:17 +01:00
Praveen Burgu
e61a75ff4e
DBZ-4941 Create a separate table for test
2022-12-14 14:26:17 +01:00
Praveen Burgu
4bdcd66d48
DBZ-4941 Handle Toasted array columns
2022-12-14 14:26:17 +01:00
xiaowu
08119e5368
Clean up useless methods
2022-12-14 08:34:44 +01:00
Vojtech Juranek
9494552a3a
DBZ-5838 Add missing Apicurio converter dependency
2022-12-12 14:44:05 +01:00
Vojtech Juranek
68b624cf7b
DBZ-5811 DBZ-5914 Don't manipulate with LSN when LSN flushing LSN is disabled
...
Befre we start streamig, we adjust `confirmed_flush_lsn` in Postgres to
the position where we determined that the streaming should start. When
user disables LSN flushing, which is not desired as it's the user
reponsibility to flush LSN in Postgres.
Also adjust related tests, as when LSN flushing is not disabled, flushed
LSN in Postgres may change (as we e.g. determined start of streaming
from offset or from `xlogpos`) and can result in the test failure. When
flushing is enabled, only LSNs after start of streaming can be compared.
2022-12-09 21:03:19 +01:00
Jiri Pechanec
7f05ef44f9
DBZ-5895 Remove hard-coded plugin
2022-12-09 21:03:19 +01:00
Jiri Pechanec
67ff871569
DBZ-5895 Wait for REPLICA IDENTITY set completed
2022-12-09 21:03:19 +01:00
Jiri Pechanec
f437788325
DBZ-5895 Remove race condition between table reading and re-creating
2022-12-09 21:03:19 +01:00
Marcelo Avancini
7754163c5d
DBZ-2432: Fixing misuse of assertj
...
- Some usages of assertj was not callig the actual assert of the DSL, fixing it
- Changing the Assertions to be staticlly imported
2022-12-08 13:10:26 +01:00
Vojtech Juranek
e1b65cd6f7
DBZ-5889 Run integration tests in specified order
...
Run the tests alwyas in thr same order to make it more easy to debug
failures. If needed, the order can be changed (e.g. to `random`) by
overriding propeperty `runOrder`.
2022-12-08 13:00:41 +01:00
Jiri Pechanec
574c1b0015
DBZ-5811 Use validate method
2022-12-06 13:35:52 +01:00
subodh
66fb51b55a
DBZ-5811 Implement support for skipping LSN commit on postgres source based on config
2022-12-06 13:35:52 +01:00
Debezium Builder
81896ed334
[maven-release-plugin] prepare for next development iteration
2022-11-30 07:52:24 +00:00
Debezium Builder
1aac97b87c
[maven-release-plugin] prepare release v2.1.0.Alpha2
2022-11-30 07:52:23 +00:00
Praveen Burgu
2158ea22d4
DBZ-5739 Pass the exception as argument to the logger
2022-11-30 06:55:35 +01:00
Praveen Burgu
ad6ec2be6a
DBZ-5739 Handle permission error from RDS Postgres
2022-11-30 06:55:35 +01:00
Jiri Pechanec
5eb147129d
DBZ-5874 Fix syntax errors
2022-11-25 11:43:49 +01:00
Gunnar Morling
e7aba47a24
DBZ-5874 Making sure trace message gets logged
2022-11-25 11:43:49 +01:00
Praveen Burgu
2b73ad6216
DBZ-5739 Seek to LSN based on lastCommitStoredLsn
2022-11-22 14:50:11 +01:00
Praveen Burgu
00321ccb4d
DBZ-5739 Fix remaining failing tests
2022-11-22 14:50:11 +01:00
Praveen Burgu
cb535d2ce6
DBZ-5739 Fix another test with expected error message
2022-11-22 14:50:11 +01:00
Praveen Burgu
8e3867f23f
DBZ-5739 Fix other failing tests
2022-11-22 14:50:11 +01:00
Praveen Burgu
1a0576e340
DBZ-5739 auto close statement
2022-11-22 14:50:11 +01:00
Praveen Burgu
2aed6b55ba
DBZ-5739 Fix test PostgresConnectorIT#shouldRegularlyFlushLsnWithTxMonitoring by not dropping the slot
2022-11-22 14:50:11 +01:00
Praveen Burgu
895d651841
DBZ-5739 Fix test DebeziumEngineIT.testOffsetsCommitAfterStop() by not dropping the slot
2022-11-22 14:50:11 +01:00
Jiri Pechanec
b4c076a27c
DBZ-5739 Make validation mandatory; more robust tests
2022-11-22 14:50:11 +01:00
Praveen Burgu
822f87147c
DBZ-5739 Postgres connector results in silent data loss if replication slot is recreated
...
Detect replication slot recreation and fail the connector
2022-11-22 14:50:11 +01:00
Debezium Builder
e311dd1a67
[maven-release-plugin] prepare for next development iteration
2022-11-10 13:29:06 +00:00
Debezium Builder
e3bf53440a
[maven-release-plugin] prepare release v2.1.0.Alpha1
2022-11-10 13:29:05 +00:00
Vojtech Juranek
96e4326d35
DBZ-2131 Use system property to determine if Apicurio should be used
2022-11-09 08:44:45 +01:00
Vojtech Juranek
391c1b921d
DBZ-2131 Add Apicurion profile for Postgres
2022-11-09 08:44:45 +01:00
Vojtech Juranek
1e86ab7c7a
DBZ-2131 Unify intendation in Postgres pom.xml
2022-11-09 08:44:45 +01:00
rajdangwal
2350442a8c
DBZ-5792: minor nit
2022-11-09 08:28:17 +01:00
rajdangwal
7bf889006f
DBZ-5792: Handle TRUNCATE type
2022-11-09 08:28:17 +01:00
rajdangwal
ea2983c8d8
DBZ-5792: Fix for - Message with LSN XYZ not present among LSNs seen in the location phase
2022-11-09 08:28:17 +01:00
Jiri Pechanec
4ccf91d70d
DBZ-5800 Disable test failing on CI
2022-11-08 12:10:19 +01:00
harveyyue
506311b469
DBZ-5783 Suppress logging of undetermined optionality for explicitly excluded columns
2022-11-04 10:46:29 -04:00
jcechace
e887670a8e
DBZ-5779 Migrated debezium-connector-postgres to AssertJ
2022-11-02 08:50:26 +01:00
Jiri Pechanec
53731e5662
DBZ-5370 Support for PostgreSQL 15
2022-10-18 12:04:55 +02:00
Debezium Builder
4d35fb8011
[maven-release-plugin] prepare for next development iteration
2022-10-14 11:26:32 +00:00
Debezium Builder
7bd1551c99
[maven-release-plugin] prepare release v2.0.0.Final
2022-10-14 11:26:32 +00:00
Vojtech Juranek
820b9d10d5
DBZ-5329 Add LSN to Postgres transation ID
...
Add `EventDispatcher` constructor which accepts `TransactionMonior`
instance as a parameter and in case of Postgres pass into
`EventDispatcher` custom `PostgresTransactionMonitor` which adjusts
transactions IDs by adding LSN, i.e. Postgres transaction is now of
form `txId:LSN`.
2022-10-11 15:38:33 +02:00
Vojtech Juranek
44dfa893ae
DBZ-5329 Remove unsed PostgresEventDispatcher constructors
2022-10-11 15:38:33 +02:00
Debezium Builder
884254aba1
[maven-release-plugin] prepare for next development iteration
2022-10-07 09:27:52 +00:00
Debezium Builder
a0f3263fa6
[maven-release-plugin] prepare release v2.0.0.CR1
2022-10-07 09:27:51 +00:00
harveyyue
f6b72f060c
DBZ-5671 Remove logic name parameter from sub connector config
2022-10-03 11:40:17 -04:00
Vojtech Juranek
97532042b3
DBZ-5657 Allow to use Docker maven plugin external properties
...
Allow to override Docker maven plugin properties [1] from command line
to be able to change various Docker parameters more easily when starting
the container.
[1] https://dmp.fabric8.io/#combining-property-config
2022-09-29 05:41:09 +02:00
Jiri Pechanec
1d423d7f09
DBZ-5597 Record LSNs seen the location phase
2022-09-22 11:12:04 -04:00
harveyyue
d1153e8ad4
DBZ-5594 Clean up "logical name" config
2022-09-22 14:29:37 +02:00
harveyyue
288c93434f
DBZ-5585 Remove classloader parameter from JdbcConnection
2022-09-19 17:15:29 +02:00
Debezium Builder
218d414552
[maven-release-plugin] prepare for next development iteration
2022-09-16 06:36:40 +00:00
Debezium Builder
567f57c47b
[maven-release-plugin] prepare release v2.0.0.Beta2
2022-09-16 06:36:39 +00:00
Vojtech Juranek
acc6be6df3
DBZ-5043 Move TOPIC_PREFIX into CommonConnectorConfig
...
`TOPIC_PREFIX` is now mandatory to all connectors therefore it make
sense to have it in common config. Beside that, it also makes it more
easy to use it in Debezium UI without any workarounds - if the field is
not member of the given connector config, the field has to be
explicitely added into known fields otherwise is invisible for UI.
2022-09-13 09:25:54 +02:00
Anisha Mohanty
f41b6dd458
DBZ-5044 Get rid of schema factory instances
2022-09-08 10:35:20 +02:00
Anisha Mohanty
b0dc7e557a
DBZ-5044 Update schemaHistoryColumnSchema to add new fields
2022-09-08 10:35:20 +02:00
Anisha Mohanty
ec792527e9
DBZ-5044 Use SchemaFactory for centralizing Postgres schemas
2022-09-08 10:35:20 +02:00
Vojtech Juranek
0823cedf25
DBZ-5043 Replace database.server.name with topic.prefix
2022-09-07 12:57:31 +02:00
Vojtech Juranek
161a7229b6
DBZ-5043 Use topic prefix as the logical name
...
With this change the user cannot direcly set connector logical name and
thus in the future it can be remove without breaking user config.
If the turn out that the logical name is useful and user should be able
to configure it, dedicated config option can be added.
Originally it was prposed in the Jira to replace it with connector name,
but it turned out that logical name defaults to `database.server.name`
and is heavily used in the tests and JMX, so it would require another
big refactoring. Thus, use topic prefix for now. Once we know further
direction (remove logical name or add new option), do this refactoring.
2022-09-07 12:57:31 +02:00
Vadzim Ramanenka
917de37a01
DBZ-5557: Supply partition when comiting offsets with source database
...
Keep track of partitions offsets being committed belong to in
`BaseSourceTask`. Supply the partition along with the offset in calls to
`commitOffset`.
2022-09-05 14:04:23 +02:00
ggaborg
244de2bd96
DBZ-3293 Add signal table automatically to include list
2022-09-01 20:16:47 +02:00
Marek Winkler
1c9ea5166c
DBZ-3007: limit 'Streaming requested from LSN' logging
2022-08-31 11:45:47 -04:00
Jiri Pechanec
de53591dba
DBZ-5244 Retry all communication exceptions by default
2022-08-31 09:45:09 -04:00
pkgonan
b031770300
DBZ-5544 Support BASE64_URL_SAFE in BinaryHandlingMode
2022-08-24 09:50:23 +02:00
Sergei Morozov
1c99437890
DBZ-5470: Remove the dependency of JdbcConnection on DatabaseSchema
2022-08-19 11:06:59 +02:00
Chris Cranford
37b97970a6
DBZ-5497 Skip truncate operations by default
...
* Promote truncate skip default from PostgreSQL to core library
* Adjust behavior for Oracle with truncates skipped by default
2022-08-19 11:05:29 +02:00
Vojtech Juranek
db446f85c4
DBZ-4329 Don't send fileds with obsolete values
...
During incremental snapshot fields `xmin`, `lsn` and `txId` are not
updated and therefore we stream obsolete values from previous records
in `source` struct. Don't include these obsolete value in the `source`
struct during incremental snapshot.
2022-08-05 10:23:47 -04:00
Vojtech Juranek
c7d8353d7b
DBZ-4727 Add snapshot paused JXM metrics
...
Add two JXM objects:
* `snapshotPaused` - detemines if the incremental snapshot is paused
or not
* `snapshotPausedDurationInSeconds` - overall time when the incremental
snapshot was paused. The time adds up - if the snapshot was paused
e.g. two times, the `snapshotPausedDurationInSeconds` is the sum of
these two paused times.
2022-07-29 12:41:58 -04:00
harveyyue
5f430f1a8e
DBZ-5457 Deprecate legacy topic selector for all connectors
2022-07-28 12:58:59 -04:00
Debezium Builder
1ab51ae8d0
[maven-release-plugin] prepare for next development iteration
2022-07-26 18:13:10 +00:00
Debezium Builder
d247a48b79
[maven-release-plugin] prepare release v2.0.0.Beta1
2022-07-26 18:13:09 +00:00
harveyyue
187ccb25db
DBZ-5398 The column is referenced as PRIMARY KEY, but a matching column is not defined in table
2022-07-25 10:11:43 -04:00
Vojtech Juranek
3527479cd8
DBZ-5408 Make sure streaming is fully running before dropping connection
2022-07-14 11:10:22 -04:00
Chris Cranford
67dbd189a6
DBZ-5403 Convert timestamps with tz default values to GMT
2022-07-14 08:23:06 +02:00
harveyyue
6ea7abc8fc
DBZ-4180 Pluggable topic selector
2022-07-13 14:45:25 +02:00
harveyyue
7af6037078
DBZ-5384 org.postgresql.util.PSQLException: Bad value for type timestamp/date/time: CURRENT_TIMESTAMP
2022-07-12 17:29:15 +02:00
Vojtech Juranek
5573bb51ed
DBZ-5048 Rename GH repo with containes
...
Besides updating links to renamed GH repo, update also documentation and
replace 'Docker' with 'containers' when possible.
2022-07-12 13:16:53 +02:00
Chris Cranford
d2a76a7ffc
DBZ-5295 Reselect LOB columns on primary key update
2022-07-12 11:59:34 +02:00
Chris Cranford
bb0a640219
DBZ-5340 Add test case
2022-07-08 06:54:18 +02:00
Chris Cranford
c909dd78d2
DBZ-5340 Allow schema prefix on default value function calls
2022-07-08 06:54:18 +02:00
Debezium Builder
6fd1441af3
[maven-release-plugin] prepare for next development iteration
2022-07-01 09:21:49 +00:00
Debezium Builder
99402af634
[maven-release-plugin] prepare release v2.0.0.Alpha3
2022-07-01 09:21:48 +00:00
yangrong688
774d1e4141
[docs] Fix typo for javadoc newSlotInfo
2022-06-30 12:27:06 +02:00
Vojtech Juranek
499aece7eb
DBZ-5240 Add partitioned tables into supported table types
...
Postgres supports partitioned tables. Debezium requires tables to have
primary key to be able to snapshot them. Primary key support for
partitioned was added in Postgres 11, see [1]. Add partitioned table
type into supported table tables so that Debezium fetches the schema
and can do the snapshot for partitioned tables.
N.B.: On Postgres < 11 is still possible to define primary key
constraint on partitioned sub-tables. In such cases sub-tables are
snapshotted, but parent table is not.
Fix RecordsSnapshotProducerIT#shouldGenerateSnapshotsForPartitionedTables
test after adding support for partitioned tables. Test assumed that
the parent table is not snapshotted and filters out duplicate records
which results into lossing `LAST` snapshot record. Keepting original
test by filtering parent table.
[1] https://www.postgresql.org/docs/11/release-11.html
2022-06-30 10:46:54 +02:00
Mickael Maison
5ff71aa992
DBZ-5335 Fix formatting issues in the connectors configdefs
2022-06-30 09:35:54 +02:00
harveyyue
20dc353262
DBZ-5170 Mysql Commit Timestamp
2022-06-27 15:50:52 +02:00
kanha gupta
1b17372d2d
DBZ-4902 Replace long durations with Duration
2022-06-27 14:29:51 +02:00
Vojtech Juranek
e15f148deb
DBZ-5070 Don't reset lsn when lastCommitLsn is null
...
Currently we update first `lsn` and then update `lastCommitLsn` in
`PostgresOffsetContext` constructor. However, when `lastCommitLsn`
is for whatever reason `null`, previously updated `lsn` is reset to
`null` as `updateLastCommit()` updates also `lsn`. This can have
unwanted consequences like streaming again records which were already
streamed. To prevent this, update `lsn` in `updateLastCommit()` only
when `lastCommitLsn` is not `null`.
2022-06-22 13:55:17 +02:00
Jiri Pechanec
4fc4c8c374
DBZ-5031 Stop connector on failure to find resume LSN
2022-06-20 12:00:05 +02:00
Oskar Polak
3e288b8385
DBZ-5031 Additional logging when offset flushing is not allowed
2022-06-20 12:00:05 +02:00
Jiri Pechanec
38cd6907ae
DBZ-3942 Load correct last commit LSN from offsets
2022-06-17 14:49:44 +02:00
Jiri Pechanec
04c474bac7
DBZ-5031 Enable LSN flushing only after processed message
2022-06-17 12:39:56 +02:00
Jiri Pechanec
5721dd7ace
DBZ-5031 Emit warning if not exact LSN is found
2022-06-17 12:39:56 +02:00
Jun Zhao
29fbdc03bf
DBZ-5031 Stop skipping message if last received lsn is higher or equal to the startStreamingLsn in WalPositionLocation
2022-06-17 12:39:56 +02:00
Chris Cranford
3fb5bf1339
DBZ-4451 Introduce SchemaChangeEvent factory methods
2022-06-16 09:19:41 +02:00
Chris Cranford
863abcd081
DBZ-4451 Store previous table id in alter schema change events
...
When a table is renamed with the ALTER TABLE statement, the schema
history record not only will reference the current table's unique id
in the "id" metadata, but will also refer to the old table name in
the "previousId" metadata field.
2022-06-16 09:19:41 +02:00
Chris Cranford
20bb2adea5
DBZ-4834 Fix incremental snapshot of table added to include list
2022-06-16 08:39:16 +02:00
Debezium Builder
6fff3a2876
[maven-release-plugin] prepare for next development iteration
2022-06-09 11:49:26 +00:00
Debezium Builder
b2e2e351df
[maven-release-plugin] prepare release v2.0.0.Alpha2
2022-06-09 11:49:26 +00:00
Chris Cranford
660e391851
DBZ-5204 Do not cache connection details in TypeRegistry
.
2022-06-08 09:21:04 +02:00
Vojtech Juranek
4a48eb33df
DBZ-5045 Remove legacy V1 source info
2022-06-08 09:11:00 +02:00
Vojtech Juranek
7b8cb693ed
DBZ-5045 Remove deprecated Postgres TOASTED_VALUE_PLACEHOLDER
2022-06-08 09:11:00 +02:00
Vojtech Juranek
1eacbdc590
DBZ-5045 Remove deprecated Postgres EXPORTED snapshot option
2022-06-08 09:11:00 +02:00
Vojtech Juranek
83dfd9f638
DBZ-5045 Remove deprecated Postgres TRUNCATE_HANDLING_MODE
2022-06-08 09:11:00 +02:00
Vojtech Juranek
b0b3066ed5
DBZ-5045 Remove deprecated COLUMN_BLACKLIST
2022-06-08 09:11:00 +02:00
Vojtech Juranek
5031594b75
DBZ-5045 Remove deprecated COLUMN_WHITELIST
2022-06-08 09:11:00 +02:00
Vojtech Juranek
f84893a8b4
DBZ-5045 Remove deprecated SCHEMA_BLACKLIST
2022-06-08 09:11:00 +02:00
Vojtech Juranek
261188482f
DBZ-5045 Remove deprecated SCHEMA_WHITELIST
2022-06-08 09:11:00 +02:00
Vojtech Juranek
6abbcee6a3
DBZ-5045 Remove deprecated TABLE_BLACKLIST
2022-06-08 09:11:00 +02:00
Vojtech Juranek
db08e894fc
DBZ-5045 Remove deprecated TABLE_WHITELIST
2022-06-08 09:11:00 +02:00
harveyyue
acb743bd34
DBZ-5206 Remove the deprecated monitored tables metrics
2022-06-07 16:13:29 -04:00
Chris Cranford
9d71296b5c
DBZ-4251 Support stopping incremental snapshots
2022-06-02 07:30:07 +02:00
Jiri Pechanec
88b7ae2050
DBZ-5136 Move sourceInfo to common class
2022-06-01 16:56:29 +02:00
Mark Bereznitsky
c64b8af31a
DBZ-5136 reintroduce the CommonOffsetContext class and refactor common functionality into it
2022-06-01 16:56:29 +02:00
Mark Bereznitsky
2d72dae649
DBZ-5136 further refining of the algorithm for determining position in snapshot and IT test fixes
2022-06-01 16:56:29 +02:00
Jiri Pechanec
08e8c3d795
DBZ-5136 Move common class to common package
2022-06-01 16:56:29 +02:00
Andrew Walker
d0759461a4
DBZ-5136: Convert RelationalOffsetContext to CommonOffsetContext
2022-06-01 16:56:29 +02:00
Andrew Walker
b6222d42bf
DBZ-5136: Add first flag and refactor OffsetContext
2022-06-01 16:56:29 +02:00
Andrew Walker
593c2769e6
DBZ-5136: Add last in table flag
2022-06-01 16:56:29 +02:00
Vojtech Juranek
0757c966d0
DBZ-5042 Add DB name logging support into Postgres connector
2022-05-27 14:20:30 -04:00
Chris Cranford
7d8c9486d6
DBZ-5119 Move Heartbeat creation to ConnectorConfig & descendants
2022-05-25 09:59:59 +02:00
Chris Cranford
7ff7826743
DBZ-5119 Pull heartbeat.action.query
support into debezium-core
2022-05-24 12:54:40 +02:00
Chris Cranford
7eee39800f
DBZ-5038 Fix test failures
2022-05-23 11:46:15 +02:00
Chris Cranford
2a4c76cabe
DBZ-5038 Support default values for PostgreSQL enum types
2022-05-23 11:46:15 +02:00
Jiri Pechanec
a11767d05b
DBZ-3921 Remove incorrect comment
2022-05-17 14:56:05 +02:00
troeselereos
836632e5dd
DBZ-3921 fixed logging
...
DBZ-3921: update message
Co-authored-by: Chris Cranford <ccranfor@redhat.com>
2022-05-17 14:56:05 +02:00
roeselert
6b72d22be8
DBZ-3921: fix postgres existing publication is not updated with the new table
2022-05-17 14:56:05 +02:00
Jiri Pechanec
cc90b1cfeb
DBZ-5014 Fix delete tests
2022-05-09 08:53:59 +02:00
Cesar Martinez
61a77c2467
DBZ-5014 Remove unused import
2022-05-09 08:53:59 +02:00
Cesar Martinez
c0ac30593c
DBZ-5014 Uncomment validations
2022-05-09 08:53:59 +02:00
Cesar Martinez
72e51e02c7
DBZ-5014 Remove invalid line
2022-05-09 08:53:59 +02:00
Cesar Martinez
6994c1db56
DBZ-5014 Test case for delete register with infinity date
2022-05-09 08:53:59 +02:00
Cesar Martinez
840979a454
DBZ-5014 Add AbstractRecordsProducerTest.tableIdFromDeleteStmt(String)
2022-05-09 08:53:59 +02:00
Cesar Martinez
46f6db2d50
DBZ-5014 Add infinity date constants (Date & LocalDate
2022-05-09 08:53:59 +02:00
Cesar Martinez
24173cc548
DBZ-5014 Parse infinity date on delete
2022-05-09 08:53:59 +02:00
Cesar Martinez
b1f7fa1a47
DBZ-5014 Add test case for (+/-)infinity date
2022-05-09 08:53:59 +02:00
Jiri Pechanec
6e092d39f2
DBZ-5074 No separate connection for pgoutput decoder
2022-05-06 16:36:28 +02:00
Jiri Pechanec
88dab4d8d3
DBZ-5074 Track connections by name; close heartbeat connection by default
2022-05-06 16:36:28 +02:00
Mark Allanson
0edb86d861
DBZ-5097: Add tests to cover PostgresErrorHandler behaviour
...
The error handler had no unit tests at all, so I thought it might be a good idea
to add some tests to verify the behaviour given there's a bunch of different
ways to determine if something is retryable.
2022-05-06 09:55:06 -04:00
Mark Allanson
dd815f5226
DBZ-5097: Add two new retryable postgresql error messages
...
These two error messages can occur when running inside docker and the
postgres container is either killed (sigtermed) or brought down
using `docker stop`.
In both cases these are retryable as the postgres server can be
restarted shortly thereafter.
2022-05-06 09:55:06 -04:00
Debezium Builder
901825810a
[maven-release-plugin] prepare for next development iteration
2022-04-28 13:51:13 +00:00
Debezium Builder
05dcb45647
[maven-release-plugin] prepare release v2.0.0.Alpha1
2022-04-28 13:51:12 +00:00
Jiri Pechanec
d186ec800f
DBZ-703 Remove support for messages without metadata
2022-04-22 09:57:15 +02:00
Jiri Pechanec
d299c9e707
DBZ-4156 Remove wal2json support
2022-04-22 09:57:15 +02:00