alexey.rassokhin/tet123
1
0
Fork 0
Code Issues Pull Requests 1 Packages Projects Releases Wiki Activity
10,786 Commits 3 Branches 0 Tags 51 MiB
d4431e17e4
Commit Graph

1564 Commits

Author SHA1 Message Date
rkerner
6d3528d4fb DBZ-4395 Add connector specific Debezium Connect REST Extension/s and move logic from UI backend to the Debezium Connect REST Extension (part 2 of the re-architecturing/re-factoring) 
DBZ-6761 Add "validate connection" endpoints in connector specific Connect REST extensions

closes to https://issues.redhat.com/browse/DBZ-4395
closes to https://issues.redhat.com/browse/DBZ-6761
 2023-09-21 13:29:59 +02:00
mfvitale
ae199d2053 DBZ-6828 Blocking snapshot will generate schema events only for signaled tables 2023-09-20 08:18:33 +02:00
Jiri Pechanec
29e45b0ce0 DBZ-6911 Support for PostgreSQL 16 2023-09-18 15:33:11 +02:00
xiaowu
e50d0eb87a DBZ-6935 fix logger named 2023-09-18 12:34:47 +02:00
Debezium Builder
65169a9315 [maven-release-plugin] prepare for next development iteration 2023-09-13 09:41:46 +00:00
Debezium Builder
845e3e9d09 [maven-release-plugin] prepare release v2.4.0.Beta2 2023-09-13 09:41:46 +00:00
Jiri Pechanec
f1e7999431 DBZ-6481 Support Postgres enum as key in incremental snapshot 2023-09-11 06:08:57 +02:00
Debezium Builder
d97a18e8dc [maven-release-plugin] prepare for next development iteration 2023-08-29 11:41:08 +00:00
Debezium Builder
5ce25f2cfe [maven-release-plugin] prepare release v2.4.0.Beta1 2023-08-29 11:41:08 +00:00
Jiri Pechanec
dac07d53c3 DBZ-6482 TimescaleDB transformation 2023-08-29 11:45:27 +02:00
Vojtech Juranek
28b06e6795 DBZ-6839 Fix PostgresConnectorIT#shouldAddNewFieldToSourceInfo 
PostgresConnectorIT#shouldAddNewFieldToSourceInfo fails only when run
together with other tests and the failure is random. It seem there is
a caching issue in Apicuro, when `test_server.s1.a-value` artifact
references `io.debezium.connector.postgresql.Source` version 1, which
hasnt't `newField` field and this reference is used also in
`shouldAddNewFieldToSourceInfo` where artifact with version 2 should
be used. Using dedicated table and thus creating new artifact in
Apucurio should fix this issue.

Also remove unused variable from `CustomPostgresSourceInfoStructMaker`.
 2023-08-25 15:02:18 +02:00
mfvitale
542b3619fa DBZ-6731 Blocking snapshot takes configuration from signal payload 2023-08-15 12:43:43 +02:00
Vojtech Juranek
6ca842b772 DBZ-6635 Send heartbeats also before processing first event 
We should send heartbeats no matter if there is already any DB record
processed or not. This prevents situation when after the start there is
no new record in the Db and Debezium is not sending neither DB events
nor heartbeats.
 2023-08-14 18:45:43 +02:00
harveyyue
76738a6402 DBZ-6758 Add it test for (+/-) infinity values for numeric/decimal types 2023-08-14 09:04:33 +02:00
harveyyue
15e81c1a84 DBZ-6758 When using pgoutput in postgres connector, (+/-)Infinity is not supported in decimal values 2023-08-14 09:04:33 +02:00
rajdangwal
b3098de0fd DBZ-6720: minor nit 2023-08-14 08:59:54 +02:00
rajdangwal
153834c650 DBZ-6720: handle unchanged toasted uuid array 2023-08-14 08:59:54 +02:00
Debezium Builder
3c72f071b0 [maven-release-plugin] prepare for next development iteration 2023-08-09 10:33:06 +00:00
Debezium Builder
915c465440 [maven-release-plugin] prepare release v2.4.0.Alpha2 2023-08-09 10:33:06 +00:00
harveyyue
b16d991a25 DBZ-6684 Rename back fields with avro_unicode 2023-08-08 10:22:10 +02:00
Jiri Pechanec
a8664ae3e9 DBZ-6767 Create publication for pgoutput only 2023-08-08 08:18:17 +02:00
Jiri Pechanec
3d263d645e DBZ-6767 Provide bytea suport 2023-08-08 08:18:17 +02:00
Jiri Pechanec
3aa6759ebd DBZ-6767 Add missing type name 2023-08-08 08:18:17 +02:00
mfvitale
61651a94b5 DBZ-6566 Force blocking snapshot even if a snapshot has already executed 2023-08-03 09:22:24 +02:00
mfvitale
def7379a0d DBZ-6566 Support Blocking snapshot for MySQL 2023-08-03 09:22:24 +02:00
mfvitale
6e0b323e83 DBZ-6566 Set correct log context during blocking snapshot 2023-08-03 09:22:24 +02:00
mfvitale
70f16af7c4 DBZ-6566 Support Blocking snapshot for PostgreSQL 2023-08-03 09:22:24 +02:00
jcechace
699a1612ae DBZ-6670 Replace error handler instead of recreate in order to keep retry counter 2023-07-25 13:46:56 +02:00
mfvitale
8ea2b3c5b7 DBZ-6669 Snapshot will get data when only signal data collection table is specified 2023-07-24 10:02:13 -04:00
Debezium Builder
267dfda8a6 [maven-release-plugin] prepare for next development iteration 2023-07-17 10:40:52 +00:00
Debezium Builder
0d3cef6a29 [maven-release-plugin] prepare release v2.4.0.Alpha1 2023-07-17 10:40:52 +00:00
Jiri Pechanec
4ca2a1c5d0 DBZ-6517 Use correct constructor 2023-07-14 09:11:10 +02:00
jcechace
318c55cc74 DBZ-6659 Fixing junit dependencies 2023-07-14 09:11:10 +02:00
Vojtech Juranek
5cf61444c0 DBZ-6648 Use String fallback when interval handling mode is String 
The root cause of DBZ-6648 is using long number as a fallback value,
while String is expected and thus falling conversion and returning
`null`.

Use String fallback when interval handling mode is String.
 2023-07-14 06:03:59 +02:00
Jiri Pechanec
764427081b DBZ-6517 DBZ-6654 Make CE converter compatible with Kafka 3.5 2023-07-13 11:26:21 +02:00
harveyyue
58d3e150b9 DBZ-6603 Add integration test for custom metric tags 2023-07-13 09:52:33 +02:00
harveyyue
5bc7fd1f7c DBZ-6603 Support for custom tags in the connector metrics 2023-07-13 09:52:33 +02:00
jcechace
5cb579bf12 DBZ-6525 Removing runtime dependency on core for testcontainers module in order to prevent logback.xml downstream conflicts 2023-07-11 12:07:04 +02:00
mfvitale
0baad2f7b7 DBZ-6417 Add test for custom action 2023-06-23 15:03:27 +02:00
mfvitale
d5febcc054 DBZ-6417 Provide extensible signal action mechanism through SPI 2023-06-23 15:03:27 +02:00
mfvitale
9ed928a1c9 DBZ-6416 Improve tests stability 2023-06-22 13:22:34 +02:00
mfvitale
34e28ac52d DBZ-6416 Add initial snapshot notifications 2023-06-22 13:22:34 +02:00
Anatolii Popov
b207f96320 DBZ-6577 Don't try to alter publication created for all tables when configuration changes to 'filtered' 2023-06-21 15:58:11 +02:00
Debezium Builder
60f730d383 [maven-release-plugin] prepare for next development iteration 2023-06-20 13:21:31 +00:00
Debezium Builder
150396e4c8 [maven-release-plugin] prepare release v2.3.0.Final 2023-06-20 13:21:31 +00:00
Jiri Pechanec
067683e56a DBZ-6076 Consistent config for all connectors 2023-06-15 11:55:28 +02:00
Anil Dasari
0431a8b9fc DBZ-6076: Add support for custom SourceInfoStructMaker for adding new fields to source field 2023-06-15 11:55:28 +02:00
Vojtech Juranek
78668df5c1 DBZ-6547 Declare exactly-once semantic support for Postgres connector 2023-06-12 12:44:12 +02:00
Debezium Builder
5d1b00588a [maven-release-plugin] prepare for next development iteration 2023-06-09 09:23:38 +00:00
Debezium Builder
d1e6107517 [maven-release-plugin] prepare release v2.3.0.CR1 2023-06-09 09:23:37 +00:00
mfvitale
7338fde84e DBZ-6424 Emit JMX Notification when Debezium generate its own notification 
DBZ-6424 Emit JMX Notification when Debezium generate its own notification

DBZ-6424 Emit JMX Notification when Debezium generate its own notification
 2023-06-07 07:36:58 +02:00
mfvitale
769d9361e7 DBZ-6424 Add JMX Notification channel 
This channel permits to send Debezium notification to JMX bean

DBZ-6424 Add JMX Signal channel

This channel permits to send signal to Debezium through the JMX operation
 2023-06-07 07:36:58 +02:00
mfvitale
537cc50d65 DBZ-6524 Add support for ltree and isbn PostgreSQL data-types in case INCLUDE_UNKNOWN_DATATYPES is set to false 2023-06-06 06:31:00 +02:00
Ronak Jain
3f3967c2bb DBZ-6366: Code Improvements for skip.messages.without.change 
Addresses/Closes DBZ-6366
 2023-06-05 14:50:25 +02:00
Debezium Builder
aa5b2131fc [maven-release-plugin] prepare for next development iteration 2023-05-26 10:35:20 +00:00
Debezium Builder
9cc9b3c29a [maven-release-plugin] prepare release v2.3.0.Beta1 2023-05-26 10:35:20 +00:00
Debezium Builder
a5b44767bc [maven-release-plugin] prepare for next development iteration 2023-05-11 11:05:24 +00:00
Debezium Builder
1b1a7531f2 [maven-release-plugin] prepare release v2.3.0.Alpha1 2023-05-11 11:05:24 +00:00
mfvitale
7429b4e6f1 DBZ-1973 Add incremental snapshot notifications 2023-05-10 12:22:02 +02:00
mfvitale
811b800f55 DBZ-1973 Add support for Notifications 
DBZ-1973 Add more tests for other connectors

DBZ-1973 Add send method with offset parameter

DBZ-1973 Instantiate NotificationService in the task class

DBZ-1973 Instantiate NotificationService in the task class

DBZ-4027 Move specific sink channel configuration to SinkNotificationChannel

DBZ-4027 Remove not used SPI file

DBZ-1973 Moved SPI file definition to debezium-core

DBZ-1973 Rename KafkaNotificationChannel to more generic SinkNotificationChannel

DBZ-1973 Code refactor

DBZ-1973 Improve configuration property description

DBZ-1973 Improve test

DBZ-1973 Add close method to NotificationChannel

DBZ-1973 Implement KafkaNotificationChannel

DBZ-1973 Add NotificationService and LogNotificationChannel
 2023-05-10 12:22:02 +02:00
mfvitale
0ca87d8031 DBZ-4027 During event dispatch process only signal from source channel 2023-05-05 13:35:40 +02:00
mfvitale
952340286e DBZ-4027 Make signalling channel configurable 
DBZ-4027 Add an Incremental snapshot test with kafka signaling

DBZ-4027 Add an Incremental snapshot test with kafka signaling

DBZ-4027 Add an Incremental snapshot test with kafka signaling

DBZ-4027 Code style

DBZ-4027 Make SignalPayload more generic and extensible

DBZ-4027 Rename DatabaseSignalChannel to SourceSignalChannel

DBZ-4027 Improve logging

DBZ-4027 Moved SPI file definition to debezium-core

DBZ-4027 Move SignalProcessor synchronization point to be processed only when a signal cdc event arrives.

DBZ-4027 Add EventDispatcher constructor without signalProcessor for spanner connector

DBZ-4027 Fix NPE

DBZ-4027 Fix NPE

DBZ-4027 Formatting

DBZ-4027 Correctly manage signal on not supported connector

DBZ-4027 Use the correct MongoDbOffset

DBZ-4027 Correctly initialize offset for Oracle and SqlServer connectors

DBZ-4027 Register SPI implementations

DBZ-4027 Improve SignalProcessor instantiation

DBZ-4027 Pass source info in case of SchemaChanges action

DBZ-4027 Manage close event in a synchronous way

DBZ-4027 Correctly init offset context also in case of snapshot mode 'never'

DBZ-4027 Fix MySqlMetricsIT test

DBZ-4027 Move KafkaSignalChannel to core

DBZ-4027 Move KafkaSignalChannel to core

DBZ-4027 Set pass offset context after initial snapshot to SignalProcessor

DBZ-4027 Pass OffsetContext to signal processor

DBZ-4027 Pass CommonConnectorConfig to SignalChannelReader init method

DBZ-4027 Move Incremental snapshot window actions to dedicated package

DBZ-4027 Align SignalsIT test with new code

DBZ-4027 Fix SignalsIT test

DBZ-4027 Fix SignalProcessor scheduling

DBZ-4027 Moved DatabaseSignalChannel and SignalChannelReader to dedicated package

DBZ-4027 Start SignalProcessor from ChangeEventSourceCoordinator

DBZ-4027 Create SignalProcessor and renamed Signal to DatabaseSignalChannel

DBZ-4027 Initial refactoring of signal feature
 2023-05-05 13:35:40 +02:00
Jiri Pechanec
ab5d643aa8 DBZ-6353 Use correct SQL codes for other cases 2023-05-03 12:15:18 +02:00
umachi
6ced91fe53 DBZ-6353 Add a conditions of 'OBJECT_IN_USE', 'NOT_IMPLEMENTED'. 2023-05-03 12:15:18 +02:00
miguelbirdie
63cc36e9a2 DBZ-6112 Set Replica Identity for PostgreSQL tables 
Introduces a new PostgreSQL configuration parameter called `replica.autoset.type`. With this parameter, you can easily specify the Replica Identity value for each table that is captured by the connector
 2023-05-02 14:33:46 +02:00
ani-sha
a453c3ca19 DBZ-6259 Add missing description of various snapshot modes in the connector field definitions 2023-05-01 11:03:04 -04:00
Nir Levy
34aa5ed8b9 DBZ-6396 Honor slot.max.retries when committing LSN before starting replication 2023-04-28 15:01:26 +02:00
Frederic Laurent
5139957f91 DBZ-6340 Add support to the Postgres connector for allow and prefer SSL modes (with prefer as the new default value) 2023-04-26 12:48:54 +02:00
Bertrand Paquet
82cf55e3ec DBZ-6384: Fix snapshotting with hstore in Map mode 2023-04-26 12:43:21 +02:00
Jiri Pechanec
643a82e4ee DBZ-6122 Refactor for more safety 2023-04-24 07:06:42 +02:00
Bertrand Paquet
979d4529b7 DBZ-6122 Toasted varying character array and date array are not correctly processed 2023-04-24 07:06:42 +02:00
Jiri Pechanec
28563b7244 DBZ-6353 Use SQL state to identify missing function 2023-04-24 06:36:58 +02:00
Jiri Pechanec
07947e1540 DBZ-6379 Fix imports 2023-04-24 06:03:56 +02:00
Bertrand Paquet
8a224e6588 DBZ-6379: Fix handling of toasted hstore 2023-04-24 06:03:56 +02:00
Debezium Builder
bb1b79f3bd [maven-release-plugin] prepare for next development iteration 2023-04-20 09:09:43 +02:00
Debezium Builder
5483b4de99 [maven-release-plugin] prepare release v2.2.0.Final 2023-04-20 09:09:43 +02:00
Mark Bereznitsky
83629089a0 DBZ-6265 always check the WAL_LEVEL but fail if streaming is required 2023-04-19 10:54:39 +02:00
Jiri Pechanec
f4c200ac3e DBZ-2979 Fix tests 2023-04-19 09:04:29 +02:00
Ronak Jain
465b355dd0 DBZ-2979 If column.include.list/column.exclude.list are used and the target table receives an update for the excluded (or not included) column - such events should be ignored 
* Add More Test Cases

Addresses/Closes DBZ-2979
 2023-04-19 09:04:29 +02:00
Ronak Jain
762122ecf5 DBZ-2979 If column.include.list/column.exclude.list are used and the target table receives an update for the excluded (or not included) column - such events should be ignored 
* Add Mysql and SqlServer Connector Changes
* Change Docs
* Clean Test Case
* Formatting and Checkstyle

Addresses/Closes DBZ-2979
 2023-04-19 09:04:29 +02:00
Debezium Builder
976a47a3a3 [maven-release-plugin] prepare for next development iteration 2023-04-14 08:46:06 +00:00
Debezium Builder
1ca8399e8b [maven-release-plugin] prepare release v2.2.0.CR1 2023-04-14 08:46:06 +00:00
Jiri Pechanec
925de0184c DBZ-6191 Do not fail if slot cannot be advanced 2023-04-14 07:44:16 +02:00
Vojtech Juranek
44f2f40c56 DBZ-6216 Update testcontainers to use quay.io images 
When needed update also version.
Update also leftovers in readme files.
 2023-04-12 14:07:35 +02:00
Jiri Pechanec
75e680cfb0 DBZ-6265 Correctly format error message 2023-04-04 07:05:55 +02:00
Mark Bereznitsky
6b82ee67e1 DBZ-6265 warn in case no wal support and no cdc 2023-04-04 07:05:55 +02:00
Mark Bereznitsky
4fea0cc895 DBZ-6265 skip the wal level check if no cdc is to occur 2023-04-04 07:05:55 +02:00
Jiri Pechanec
77c971880c DBZ-6261 Allow different version for protoc 2023-04-04 05:40:27 +02:00
Debezium Builder
5c650fd054 [maven-release-plugin] prepare for next development iteration 2023-03-31 10:00:16 +00:00
Debezium Builder
68ffc4833b [maven-release-plugin] prepare release v2.2.0.Beta1 2023-03-31 10:00:16 +00:00
mfvitale
673a42b541 DBZ-6232 Support 'binary.handling.mode' also on bytea array 2023-03-27 15:55:38 +02:00
mfvitale
ed67ff6eb9 DBZ-6232 Manage BYTEA_ARRAY in PostgresValueConverter 2023-03-27 15:55:38 +02:00
Anatolii Popov
930d9b0971 DBZ-6251 Ignore Debezium internal transactions during fallback to restart_lsn in Postgres connector if confirmed_flush_lsn is NULL 2023-03-27 15:55:13 +02:00
Nir Levy
c7e00d3881 DBZ-6249 Fix error thrown when snapshot.custom_class=custom and no snapshot.custom.class 2023-03-27 10:53:39 +02:00
Vojtech Juranek
cfca3e41a3 DBZ-6216 Update postgres notes to use quay.io images 
Also update postgres version currently being used
 2023-03-23 13:29:19 +01:00
Jiri Pechanec
d4c3e23767 DBZ-6092 Simplify the code; change logging 2023-03-16 12:58:03 +01:00
Anatolii Popov
cbb785d67a DBZ-6092 Fallback to restart_lsn in Postgres connector if confirmed_flush_lsn is NULL and no long-running transactions were committed during timeout 2023-03-16 12:58:03 +01:00
Vojtěch Juránek
42b374613a DBZ-6204 Fix typo 
Co-authored-by: Jiri Pechanec <jpechane@redhat.com>
 2023-03-16 11:32:52 +01:00
Vojtech Juranek
3b5740f1a8 DBZ-6204 Avoid skipping record when determining resume LSN 
When the last operation before connector stop is tx BEGIN, subsequent
INSERT done after connector stop can have same LSN as BEGIN and thus it
was possible to skip this INSERT when we search for the resume LSN. This
issue was described and fixed in DBZ-5915. However, the same LSN can
have also previous tx COMMIT, so the WAL can look like this:

    LSN | operation
    ------------------
    123 | COMMIT
    123 | BEGIN
    123 | INSERT
    124 | COMMIT

When the last stored LSN is 123 for COMMIT operation, we skip also BEGIN
and INSERT operations with LSN 123 and resume from COMMIT with LSN 124.

Adjust condition introduced in DBZ-5915 to consider also the case when
the last processed operation is COMMIT.
 2023-03-16 11:32:52 +01:00
Debezium Builder
55d65e9618 [maven-release-plugin] prepare for next development iteration 2023-03-08 11:38:17 +00:00
Debezium Builder
6d183c3dc1 [maven-release-plugin] prepare release v2.2.0.Alpha3 2023-03-08 11:38:16 +00:00
Jiri Pechanec
3da2434c6c DBZ-823 Restrict the type with generics 2023-03-03 12:47:15 +01:00
Jiri Pechanec
5e666f00ee DBZ-823 Add JavaDoc to new interfaces 2023-03-03 12:47:15 +01:00
Jiri Pechanec
7d92dc0211 DBZ-823 Don't use getter based name 2023-03-03 12:47:15 +01:00
Jiri Pechanec
0f4adb9fe6 DBZ-823 Use better name for CF with main connection 2023-03-03 12:47:15 +01:00
ggaborg
e875e784a0 DBZ-823 CR changes 2023-03-03 12:47:15 +01:00
ggaborg
065dc997b9 DBZ-823 Optionally parallelize initial snapshots 2023-03-03 12:47:15 +01:00
Chris Cranford
a9d4663456 DBZ-6163 Properly handle zoned timestamp within PostgreSQL 2023-02-28 13:32:57 +01:00
Debezium Builder
2245ceda44 [maven-release-plugin] prepare for next development iteration 2023-02-16 11:28:25 +00:00
Debezium Builder
20f3b16e8a [maven-release-plugin] prepare release v2.2.0.Alpha2 2023-02-16 11:28:25 +00:00
Yang Wu
d7ab7146f0 DBZ-6089 Expose sequence in postgres parser 2023-02-08 10:13:19 +01:00
harveyyue
57b902ff8c DBZ-5743 Introduce alternative AVRO_UNICODE mode for field name selector 2023-02-02 13:21:45 +01:00
Jiri Pechanec
8f3ca36e23 DBZ-6026 Don't update last offsets in statistics 2023-01-24 14:26:05 +01:00
Vojtech Juranek
f00f2a8735 DBZ-5917 Fix querying column metadata for tables with back slash in the name 
If the table contains back slash, which is ANSI SQL escape chracter,
in its name, querying column metadata would fail in some cases (*)
which would result into NPE. Fix table name before the query and
escape escape character if the table name contains it.

Default backslash works for all currently supported databases as it's
ANSI SQL standard,

(*) everything works when we collect column metadata for all tables,
i.e. when there are no excluded tables
 2023-01-20 00:01:11 +01:00
Debezium Builder
04f5291a2d [maven-release-plugin] prepare for next development iteration 2023-01-19 11:32:10 +00:00
Debezium Builder
027527963e [maven-release-plugin] prepare release v2.2.0.Alpha1 2023-01-19 11:32:10 +00:00
Anil Dasari
e1f9d64fbb DBZ-5852:Snapshotter#snapshotCompleted is invoked regardless of snapshot result 2023-01-19 09:24:02 +01:00
Jeremy Ford
8da86b78e8 DBZ-5879 Fix unit tests 2023-01-18 14:46:14 +01:00
Jeremy Ford
f2a6f0b8db DBZ-5879 Support retrying database connection failures during connector start 
Reworked when task requests start from subclasses
Add support for restarting
 2023-01-18 14:46:14 +01:00
Jiri Pechanec
1ee4850273 DBZ-6012 Allow skipping of LSN check failure 2023-01-18 10:34:56 +01:00
Jiri Pechanec
107f09a2e7 DBZ-6001 DBZ-5991 Use strict class for money 2023-01-17 14:52:03 +01:00
harveyyue
93b82ab7b4 DBZ-6001 Postgresql connector parses the null of the money type into 0 2023-01-17 14:52:03 +01:00
harveyyue
bde7af1423 DBZ-5991 There is a problem with postgresql connector parsing the boundary value of money type 2023-01-17 14:52:03 +01:00
Jiri Pechanec
bcd6787cdc DBZ-6008 Revert "DBZ-3653 Introduce incremental.snapshot.isolation.mode config" 
This reverts commit 6d8bcd42fd.
 2023-01-17 10:40:20 +01:00
Animesh Kumar
90cd3fdb6d DBZ-5988 Impose null check on schema and name values prior to applying lowercase operation 2023-01-11 09:04:31 +01:00
govi20
6d8bcd42fd DBZ-3653 Introduce incremental.snapshot.isolation.mode config 2023-01-11 08:55:33 +01:00
Jiri Pechanec
8ca5e0a9b5 DBZ-4669 Reformat after plugin version consolidation 2023-01-11 06:44:53 +01:00
Mark Lambert
e5d16b14a6 DBZ-2439 Remove redundant modifiers 2023-01-11 06:44:53 +01:00
erdinc
5a063867df DBZ-5976 Use as much as possible "replace" method rather than "replaceAll" 2023-01-05 07:30:49 +01:00
rajdangwal
b03b4cdd19 DBZ-5915: correctly determine the startStreamingLsn with messageType 2023-01-04 14:50:14 +01:00
rajdangwal
5dde1ab3b5 DBZ-5915: correctly determine the startStreamingLsn 2023-01-04 14:50:14 +01:00
Debezium Builder
cf1720e4cb [maven-release-plugin] prepare for next development iteration 2022-12-22 10:11:54 +00:00
Debezium Builder
b4ea9b46cd [maven-release-plugin] prepare release v2.1.0.Final 2022-12-22 10:11:53 +00:00
jchipmunk
189f3c023e DBZ-5949 Support partitioned tables when updating PG publication 
PostgresConnection.getAllTableIds(String) method was added to unify getting all tables in database catalog, including partitioned tables.
 2022-12-20 09:28:05 +01:00
Praveen Burgu
31745a7dd5 DBZ-5936 Add test for json[] column 2022-12-20 09:18:34 +01:00
Praveen Burgu
48e1db8fc3 DBZ-5936 Add Javadoc 2022-12-20 09:18:34 +01:00
Jiri Pechanec
fff4719fb1 DBZ-5936 Remove separate config option 2022-12-20 09:18:34 +01:00
Praveen Burgu
4cada81661 DBZ-5936 Handle json and jsonb arrays 2022-12-20 09:18:34 +01:00
Praveen Burgu
28d12a836f DBZ-5936 Move all placeholder logic into one class 2022-12-20 09:18:34 +01:00
Praveen Burgu
54c0cfc14a DBZ-5936 Handle TOASTed int and bigint array columns 2022-12-20 09:18:34 +01:00
Debezium Builder
177a60a9d3 [maven-release-plugin] prepare for next development iteration 2022-12-16 09:33:38 +00:00
Debezium Builder
f805db8c3c [maven-release-plugin] prepare release v2.1.0.Beta1 2022-12-16 09:33:38 +00:00
Praveen Burgu
0b4769c793 DBZ-5944 Toasted varchar array is not correctly processed 2022-12-16 08:50:32 +01:00
Praveen Burgu
b4be663168 DBZ-4941 add a comment 2022-12-14 14:26:17 +01:00
Praveen Burgu
e61a75ff4e DBZ-4941 Create a separate table for test 2022-12-14 14:26:17 +01:00
Praveen Burgu
4bdcd66d48 DBZ-4941 Handle Toasted array columns 2022-12-14 14:26:17 +01:00
xiaowu
08119e5368 Clean up useless methods 2022-12-14 08:34:44 +01:00
Vojtech Juranek
9494552a3a DBZ-5838 Add missing Apicurio converter dependency 2022-12-12 14:44:05 +01:00
Vojtech Juranek
68b624cf7b DBZ-5811 DBZ-5914 Don't manipulate with LSN when LSN flushing LSN is disabled 
Befre we start streamig, we adjust `confirmed_flush_lsn` in Postgres to
the position where we determined that the streaming should start. When
user disables LSN flushing, which is not desired as it's the user
reponsibility to flush LSN in Postgres.

Also adjust related tests, as when LSN flushing is not disabled, flushed
LSN in Postgres may change (as we e.g. determined start of streaming
from offset or from `xlogpos`) and can result in the test failure. When
flushing is enabled, only LSNs after start of streaming can be compared.
 2022-12-09 21:03:19 +01:00
Jiri Pechanec
7f05ef44f9 DBZ-5895 Remove hard-coded plugin 2022-12-09 21:03:19 +01:00
Jiri Pechanec
67ff871569 DBZ-5895 Wait for REPLICA IDENTITY set completed 2022-12-09 21:03:19 +01:00
Jiri Pechanec
f437788325 DBZ-5895 Remove race condition between table reading and re-creating 2022-12-09 21:03:19 +01:00
Marcelo Avancini
7754163c5d DBZ-2432: Fixing misuse of assertj 
- Some usages of assertj was not callig the actual assert of the DSL, fixing it
- Changing the Assertions to be staticlly imported
 2022-12-08 13:10:26 +01:00
Vojtech Juranek
e1b65cd6f7 DBZ-5889 Run integration tests in specified order 
Run the tests alwyas in thr same order to make it more easy to debug
failures. If needed, the order can be changed (e.g. to `random`) by
overriding propeperty `runOrder`.
 2022-12-08 13:00:41 +01:00
Jiri Pechanec
574c1b0015 DBZ-5811 Use validate method 2022-12-06 13:35:52 +01:00
subodh
66fb51b55a DBZ-5811 Implement support for skipping LSN commit on postgres source based on config 2022-12-06 13:35:52 +01:00
Debezium Builder
81896ed334 [maven-release-plugin] prepare for next development iteration 2022-11-30 07:52:24 +00:00
Debezium Builder
1aac97b87c [maven-release-plugin] prepare release v2.1.0.Alpha2 2022-11-30 07:52:23 +00:00
Praveen Burgu
2158ea22d4 DBZ-5739 Pass the exception as argument to the logger 2022-11-30 06:55:35 +01:00
Praveen Burgu
ad6ec2be6a DBZ-5739 Handle permission error from RDS Postgres 2022-11-30 06:55:35 +01:00
Jiri Pechanec
5eb147129d DBZ-5874 Fix syntax errors 2022-11-25 11:43:49 +01:00
Gunnar Morling
e7aba47a24 DBZ-5874 Making sure trace message gets logged 2022-11-25 11:43:49 +01:00
Praveen Burgu
2b73ad6216 DBZ-5739 Seek to LSN based on lastCommitStoredLsn 2022-11-22 14:50:11 +01:00
Praveen Burgu
00321ccb4d DBZ-5739 Fix remaining failing tests 2022-11-22 14:50:11 +01:00
Praveen Burgu
cb535d2ce6 DBZ-5739 Fix another test with expected error message 2022-11-22 14:50:11 +01:00
Praveen Burgu
8e3867f23f DBZ-5739 Fix other failing tests 2022-11-22 14:50:11 +01:00
Praveen Burgu
1a0576e340 DBZ-5739 auto close statement 2022-11-22 14:50:11 +01:00
Praveen Burgu
2aed6b55ba DBZ-5739 Fix test PostgresConnectorIT#shouldRegularlyFlushLsnWithTxMonitoring by not dropping the slot 2022-11-22 14:50:11 +01:00
Praveen Burgu
895d651841 DBZ-5739 Fix test DebeziumEngineIT.testOffsetsCommitAfterStop() by not dropping the slot 2022-11-22 14:50:11 +01:00
Jiri Pechanec
b4c076a27c DBZ-5739 Make validation mandatory; more robust tests 2022-11-22 14:50:11 +01:00
Praveen Burgu
822f87147c DBZ-5739 Postgres connector results in silent data loss if replication slot is recreated 
Detect replication slot recreation and fail the connector
 2022-11-22 14:50:11 +01:00
Debezium Builder
e311dd1a67 [maven-release-plugin] prepare for next development iteration 2022-11-10 13:29:06 +00:00
Debezium Builder
e3bf53440a [maven-release-plugin] prepare release v2.1.0.Alpha1 2022-11-10 13:29:05 +00:00
Vojtech Juranek
96e4326d35 DBZ-2131 Use system property to determine if Apicurio should be used 2022-11-09 08:44:45 +01:00
Vojtech Juranek
391c1b921d DBZ-2131 Add Apicurion profile for Postgres 2022-11-09 08:44:45 +01:00
Vojtech Juranek
1e86ab7c7a DBZ-2131 Unify intendation in Postgres pom.xml 2022-11-09 08:44:45 +01:00
rajdangwal
2350442a8c DBZ-5792: minor nit 2022-11-09 08:28:17 +01:00
rajdangwal
7bf889006f DBZ-5792: Handle TRUNCATE type 2022-11-09 08:28:17 +01:00
rajdangwal
ea2983c8d8 DBZ-5792: Fix for - Message with LSN XYZ not present among LSNs seen in the location phase 2022-11-09 08:28:17 +01:00
Jiri Pechanec
4ccf91d70d DBZ-5800 Disable test failing on CI 2022-11-08 12:10:19 +01:00
harveyyue
506311b469 DBZ-5783 Suppress logging of undetermined optionality for explicitly excluded columns 2022-11-04 10:46:29 -04:00
jcechace
e887670a8e DBZ-5779 Migrated debezium-connector-postgres to AssertJ 2022-11-02 08:50:26 +01:00
Jiri Pechanec
53731e5662 DBZ-5370 Support for PostgreSQL 15 2022-10-18 12:04:55 +02:00
Debezium Builder
4d35fb8011 [maven-release-plugin] prepare for next development iteration 2022-10-14 11:26:32 +00:00
Debezium Builder
7bd1551c99 [maven-release-plugin] prepare release v2.0.0.Final 2022-10-14 11:26:32 +00:00
Vojtech Juranek
820b9d10d5 DBZ-5329 Add LSN to Postgres transation ID 
Add `EventDispatcher` constructor which accepts `TransactionMonior`
instance as a parameter and in case of Postgres pass into
`EventDispatcher` custom `PostgresTransactionMonitor` which adjusts
transactions IDs by adding LSN, i.e. Postgres transaction is now of
form `txId:LSN`.
 2022-10-11 15:38:33 +02:00
Vojtech Juranek
44dfa893ae DBZ-5329 Remove unsed PostgresEventDispatcher constructors 2022-10-11 15:38:33 +02:00
Debezium Builder
884254aba1 [maven-release-plugin] prepare for next development iteration 2022-10-07 09:27:52 +00:00
Debezium Builder
a0f3263fa6 [maven-release-plugin] prepare release v2.0.0.CR1 2022-10-07 09:27:51 +00:00
harveyyue
f6b72f060c DBZ-5671 Remove logic name parameter from sub connector config 2022-10-03 11:40:17 -04:00
Vojtech Juranek
97532042b3 DBZ-5657 Allow to use Docker maven plugin external properties 
Allow to override Docker maven plugin properties [1] from command line
to be able to change various Docker parameters more easily when starting
the container.

[1] https://dmp.fabric8.io/#combining-property-config
 2022-09-29 05:41:09 +02:00
Jiri Pechanec
1d423d7f09 DBZ-5597 Record LSNs seen the location phase 2022-09-22 11:12:04 -04:00
harveyyue
d1153e8ad4 DBZ-5594 Clean up "logical name" config 2022-09-22 14:29:37 +02:00
harveyyue
288c93434f DBZ-5585 Remove classloader parameter from JdbcConnection 2022-09-19 17:15:29 +02:00
Debezium Builder
218d414552 [maven-release-plugin] prepare for next development iteration 2022-09-16 06:36:40 +00:00
Debezium Builder
567f57c47b [maven-release-plugin] prepare release v2.0.0.Beta2 2022-09-16 06:36:39 +00:00
Vojtech Juranek
acc6be6df3 DBZ-5043 Move TOPIC_PREFIX into CommonConnectorConfig 
`TOPIC_PREFIX` is now mandatory to all connectors therefore it make
sense to have it in common config. Beside that, it also makes it more
easy to use it in Debezium UI without any workarounds - if the field is
not member of the given connector config, the field has to be
explicitely added into known fields otherwise is invisible for UI.
 2022-09-13 09:25:54 +02:00
Anisha Mohanty
f41b6dd458 DBZ-5044 Get rid of schema factory instances 2022-09-08 10:35:20 +02:00
Anisha Mohanty
b0dc7e557a DBZ-5044 Update schemaHistoryColumnSchema to add new fields 2022-09-08 10:35:20 +02:00
Anisha Mohanty
ec792527e9 DBZ-5044 Use SchemaFactory for centralizing Postgres schemas 2022-09-08 10:35:20 +02:00
Vojtech Juranek
0823cedf25 DBZ-5043 Replace database.server.name with topic.prefix 2022-09-07 12:57:31 +02:00
Vojtech Juranek
161a7229b6 DBZ-5043 Use topic prefix as the logical name 
With this change the user cannot direcly set connector logical name and
thus in the future it can be remove without breaking user config.
If the turn out that the logical name is useful and user should be able
to configure it, dedicated config option can be added.

Originally it was prposed in the Jira to replace it with connector name,
but it turned out that logical name defaults to `database.server.name`
and is heavily used in the tests and JMX, so it would require another
big refactoring. Thus, use topic prefix for now. Once we know further
direction (remove logical name or add new option), do this refactoring.
 2022-09-07 12:57:31 +02:00
Vadzim Ramanenka
917de37a01 DBZ-5557: Supply partition when comiting offsets with source database 
Keep track of partitions offsets being committed belong to in
`BaseSourceTask`. Supply the partition along with the offset in calls to
`commitOffset`.
 2022-09-05 14:04:23 +02:00
ggaborg
244de2bd96 DBZ-3293 Add signal table automatically to include list 2022-09-01 20:16:47 +02:00
Marek Winkler
1c9ea5166c DBZ-3007: limit 'Streaming requested from LSN' logging 2022-08-31 11:45:47 -04:00
Jiri Pechanec
de53591dba DBZ-5244 Retry all communication exceptions by default 2022-08-31 09:45:09 -04:00
pkgonan
b031770300 DBZ-5544 Support BASE64_URL_SAFE in BinaryHandlingMode 2022-08-24 09:50:23 +02:00
Sergei Morozov
1c99437890 DBZ-5470: Remove the dependency of JdbcConnection on DatabaseSchema 2022-08-19 11:06:59 +02:00
Chris Cranford
37b97970a6 DBZ-5497 Skip truncate operations by default 
* Promote truncate skip default from PostgreSQL to core library
* Adjust behavior for Oracle with truncates skipped by default
 2022-08-19 11:05:29 +02:00
Vojtech Juranek
db446f85c4 DBZ-4329 Don't send fileds with obsolete values 
During incremental snapshot fields `xmin`, `lsn` and `txId` are not
updated and therefore we stream obsolete values from previous records
in `source` struct. Don't include these obsolete value in the `source`
struct during incremental snapshot.
 2022-08-05 10:23:47 -04:00
Vojtech Juranek
c7d8353d7b DBZ-4727 Add snapshot paused JXM metrics 
Add two JXM objects:
* `snapshotPaused` - detemines if the incremental snapshot is paused
  or not
* `snapshotPausedDurationInSeconds` - overall time when the incremental
  snapshot was paused. The time adds up - if the snapshot was paused
  e.g. two times, the `snapshotPausedDurationInSeconds` is the sum of
  these two paused times.
 2022-07-29 12:41:58 -04:00
harveyyue
5f430f1a8e DBZ-5457 Deprecate legacy topic selector for all connectors 2022-07-28 12:58:59 -04:00
Debezium Builder
1ab51ae8d0 [maven-release-plugin] prepare for next development iteration 2022-07-26 18:13:10 +00:00
Debezium Builder
d247a48b79 [maven-release-plugin] prepare release v2.0.0.Beta1 2022-07-26 18:13:09 +00:00
harveyyue
187ccb25db DBZ-5398 The column is referenced as PRIMARY KEY, but a matching column is not defined in table 2022-07-25 10:11:43 -04:00
Vojtech Juranek
3527479cd8 DBZ-5408 Make sure streaming is fully running before dropping connection 2022-07-14 11:10:22 -04:00
Chris Cranford
67dbd189a6 DBZ-5403 Convert timestamps with tz default values to GMT 2022-07-14 08:23:06 +02:00
harveyyue
6ea7abc8fc DBZ-4180 Pluggable topic selector 2022-07-13 14:45:25 +02:00
harveyyue
7af6037078 DBZ-5384 org.postgresql.util.PSQLException: Bad value for type timestamp/date/time: CURRENT_TIMESTAMP 2022-07-12 17:29:15 +02:00
Vojtech Juranek
5573bb51ed DBZ-5048 Rename GH repo with containes 
Besides updating links to renamed GH repo, update also documentation and
replace 'Docker' with 'containers' when possible.
 2022-07-12 13:16:53 +02:00
Chris Cranford
d2a76a7ffc DBZ-5295 Reselect LOB columns on primary key update 2022-07-12 11:59:34 +02:00
Chris Cranford
bb0a640219 DBZ-5340 Add test case 2022-07-08 06:54:18 +02:00
Chris Cranford
c909dd78d2 DBZ-5340 Allow schema prefix on default value function calls 2022-07-08 06:54:18 +02:00
Debezium Builder
6fd1441af3 [maven-release-plugin] prepare for next development iteration 2022-07-01 09:21:49 +00:00
Debezium Builder
99402af634 [maven-release-plugin] prepare release v2.0.0.Alpha3 2022-07-01 09:21:48 +00:00
yangrong688
774d1e4141 [docs] Fix typo for javadoc newSlotInfo 2022-06-30 12:27:06 +02:00
Vojtech Juranek
499aece7eb DBZ-5240 Add partitioned tables into supported table types 
Postgres supports partitioned tables. Debezium requires tables to have
primary key to be able to snapshot them. Primary key support for
partitioned was added in Postgres 11, see [1]. Add partitioned table
type into supported table tables so that Debezium fetches the schema
and can do the snapshot for partitioned tables.

N.B.: On Postgres < 11 is still possible to define primary key
constraint on partitioned sub-tables. In such cases sub-tables are
snapshotted, but parent table is not.

Fix RecordsSnapshotProducerIT#shouldGenerateSnapshotsForPartitionedTables
test after adding support for partitioned tables. Test assumed that
the parent table is not snapshotted and filters out duplicate records
which results into lossing `LAST` snapshot record. Keepting original
test by filtering parent table.

[1] https://www.postgresql.org/docs/11/release-11.html
 2022-06-30 10:46:54 +02:00
Mickael Maison
5ff71aa992 DBZ-5335 Fix formatting issues in the connectors configdefs 2022-06-30 09:35:54 +02:00
harveyyue
20dc353262 DBZ-5170 Mysql Commit Timestamp 2022-06-27 15:50:52 +02:00
kanha gupta
1b17372d2d DBZ-4902 Replace long durations with Duration 2022-06-27 14:29:51 +02:00
Vojtech Juranek
e15f148deb DBZ-5070 Don't reset lsn when lastCommitLsn is null 
Currently we update first `lsn` and then update `lastCommitLsn` in
`PostgresOffsetContext` constructor. However, when `lastCommitLsn`
is for whatever reason `null`, previously updated `lsn` is reset to
`null` as `updateLastCommit()` updates also `lsn`. This can have
unwanted consequences like streaming again records which were already
streamed. To prevent this, update `lsn` in `updateLastCommit()` only
when `lastCommitLsn` is not `null`.
 2022-06-22 13:55:17 +02:00
Jiri Pechanec
4fc4c8c374 DBZ-5031 Stop connector on failure to find resume LSN 2022-06-20 12:00:05 +02:00
Oskar Polak
3e288b8385 DBZ-5031 Additional logging when offset flushing is not allowed 2022-06-20 12:00:05 +02:00
Jiri Pechanec
38cd6907ae DBZ-3942 Load correct last commit LSN from offsets 2022-06-17 14:49:44 +02:00
Jiri Pechanec
04c474bac7 DBZ-5031 Enable LSN flushing only after processed message 2022-06-17 12:39:56 +02:00
Jiri Pechanec
5721dd7ace DBZ-5031 Emit warning if not exact LSN is found 2022-06-17 12:39:56 +02:00
Jun Zhao
29fbdc03bf DBZ-5031 Stop skipping message if last received lsn is higher or equal to the startStreamingLsn in WalPositionLocation 2022-06-17 12:39:56 +02:00
Chris Cranford
3fb5bf1339 DBZ-4451 Introduce SchemaChangeEvent factory methods 2022-06-16 09:19:41 +02:00
Chris Cranford
863abcd081 DBZ-4451 Store previous table id in alter schema change events 
When a table is renamed with the ALTER TABLE statement, the schema
history record not only will reference the current table's unique id
in the "id" metadata, but will also refer to the old table name in
the "previousId" metadata field.
 2022-06-16 09:19:41 +02:00
Chris Cranford
20bb2adea5 DBZ-4834 Fix incremental snapshot of table added to include list 2022-06-16 08:39:16 +02:00
Debezium Builder
6fff3a2876 [maven-release-plugin] prepare for next development iteration 2022-06-09 11:49:26 +00:00
Debezium Builder
b2e2e351df [maven-release-plugin] prepare release v2.0.0.Alpha2 2022-06-09 11:49:26 +00:00
Chris Cranford
660e391851 DBZ-5204 Do not cache connection details in TypeRegistry. 2022-06-08 09:21:04 +02:00
Vojtech Juranek
4a48eb33df DBZ-5045 Remove legacy V1 source info 2022-06-08 09:11:00 +02:00
Vojtech Juranek
7b8cb693ed DBZ-5045 Remove deprecated Postgres TOASTED_VALUE_PLACEHOLDER 2022-06-08 09:11:00 +02:00
Vojtech Juranek
1eacbdc590 DBZ-5045 Remove deprecated Postgres EXPORTED snapshot option 2022-06-08 09:11:00 +02:00
Vojtech Juranek
83dfd9f638 DBZ-5045 Remove deprecated Postgres TRUNCATE_HANDLING_MODE 2022-06-08 09:11:00 +02:00
Vojtech Juranek
b0b3066ed5 DBZ-5045 Remove deprecated COLUMN_BLACKLIST 2022-06-08 09:11:00 +02:00
Vojtech Juranek
5031594b75 DBZ-5045 Remove deprecated COLUMN_WHITELIST 2022-06-08 09:11:00 +02:00
Vojtech Juranek
f84893a8b4 DBZ-5045 Remove deprecated SCHEMA_BLACKLIST 2022-06-08 09:11:00 +02:00