alexey.rassokhin/tet123
1
0
Fork 0
Code Issues Pull Requests 1 Packages Projects Releases Wiki Activity
10,789 Commits 3 Branches 0 Tags 51 MiB
6d3da2871b
Commit Graph

1564 Commits

Author SHA1 Message Date
mfvitale
8dc9605aac DBZ-7917 Flush incremental snapshot window with a heartbeat message 2024-06-13 08:41:49 +02:00
mfvitale
3bf9f4d9d5 DBZ-7917 Implement read-only incremental snapshot for PostgreSQL 2024-06-13 08:41:49 +02:00
Chris Cranford
992528fcb3 DBZ-7128 Move to PostgreSQL 12 as baseline 2024-06-13 06:29:31 +02:00
Debezium Builder
5273bb1601 [maven-release-plugin] prepare for next development iteration 2024-06-06 05:44:28 +00:00
Debezium Builder
a79c9ef799 [maven-release-plugin] prepare release v2.7.0.Beta1 2024-06-06 05:44:27 +00:00
mfvitale
5c35b7b061 DBZ-7878 Add test for GIS types during snapshot 2024-06-03 14:25:51 +02:00
Jiri Pechanec
b052d8a472 DBZ-7880 Make schema backward compatible 2024-05-22 14:20:02 +02:00
Jiri Pechanec
c09c2c6712 DBZ-7641 Update tests 2024-05-15 05:42:12 +02:00
Aristofanis Lekkos
f45264d28e DBZ-5182 Fetch Postgres types by ascending order 2024-05-14 13:48:25 +02:00
Debezium Builder
34a7c11de6 [maven-release-plugin] prepare for next development iteration 2024-05-13 06:42:03 +00:00
Debezium Builder
ebf74a8933 [maven-release-plugin] prepare release v2.7.0.Alpha2 2024-05-13 06:42:03 +00:00
Jiri Pechanec
57f2446f4b DBZ-7806 use dedicated method for check 2024-05-10 09:34:45 +02:00
mfvitale
70c526eb16 DBZ-7806 Skip search for WAl position when offset exist and is just after the initial snapshot. 2024-05-10 09:34:45 +02:00
mfvitale
c3a8ba4afb DBZ-7828 Correct filter query in snapshot task to handle SQLServer table quote 2024-05-10 07:42:09 +02:00
Anil Dasari
6d49102c9b DBZ-7816: Renamed stateLock variable 2024-05-07 13:30:18 +02:00
Anil Dasari
5b9d955db9 DBZ-7816: Added empty commitRecord 2024-05-07 13:30:18 +02:00
Anil Dasari
18447d33c1 DBZ-7816: Fixed failed tests by removing marjor db upgrade case changes 2024-05-07 13:30:18 +02:00
Chris Cranford
89dac2c287 DBZ-7816 Checkstyle fixes 2024-05-07 13:30:18 +02:00
Anil Dasari
f98545c125 DBZ-7816: Postgres: Potential data loss on connector restart 2024-05-07 13:30:18 +02:00
Debezium Builder
bcac0c5c7a [maven-release-plugin] prepare for next development iteration 2024-04-25 09:21:27 +00:00
Debezium Builder
0be6e6b8fd [maven-release-plugin] prepare release v2.7.0.Alpha1 2024-04-25 09:21:26 +00:00
mfvitale
8206d2d98d DBZ-7616 Add query timeout on JdbcConnection controlled by query.timeout.ms property 2024-04-22 09:17:44 +02:00
twthorn
261253d9f0 DBZ-7698 Refactor to generic transaction block customization 2024-04-19 10:23:30 +02:00
twthorn
24e4e4eb80 DBZ-7698 fix some failing tests 2024-04-19 10:23:30 +02:00
twthorn
ea8dddade3 DBZ-7698 Add ordered transaction metadata 2024-04-19 10:23:30 +02:00
mfvitale
717888f6f6 DBZ-7765 Use different query for validating log position for SQLServer 2024-04-11 07:35:49 +02:00
Roman Kudryashov
50c9f042a3 DBZ-7755 Refactor exporting to CloudEvents: remove RecordParser 2024-04-08 13:22:31 +02:00
mfvitale
ae7628a732 DBZ-7732 Add ConnectorSpecific annotation to manage multiple SPI connector specific implementation 2024-04-08 12:46:53 +02:00
mfvitale
ec3c68b0d8 DBZ-7717 Move LogPositionValidator outside JdbcConnection 2024-04-05 11:57:43 +02:00
mfvitale
1190389edc DBZ-7718 Support table name escaping in BlockingSnapshotIT 2024-04-04 14:58:24 +02:00
Debezium Builder
ea182d83f3 [maven-release-plugin] prepare for next development iteration 2024-04-02 07:38:53 +00:00
Debezium Builder
7dcd94d19e [maven-release-plugin] prepare release v2.6.0.Final 2024-04-02 07:38:53 +00:00
mfvitale
213456c5af DBZ-7497 Properties related to configuration based snapshot mode are now optional 2024-03-28 08:59:54 +01:00
mfvitale
ee5f25ccc9 DBZ-7497 Add a configuration based snapshot modes configurable via connector properties 2024-03-28 08:59:54 +01:00
Debezium Builder
4df18d9f43 [maven-release-plugin] prepare for next development iteration 2024-03-25 09:57:05 +00:00
Debezium Builder
9656da1fad [maven-release-plugin] prepare release v2.6.0.CR1 2024-03-25 09:57:04 +00:00
mfvitale
c7a28021a6 DBZ-7308 Remove connector specific SnapshotLockProvider and SnapshotterServiceProvider 2024-03-25 05:46:46 +01:00
mfvitale
60a0b1b5c3 DBZ-7308 tableLockingStatement from SnapshotLock interface now takes just one table in input 2024-03-25 05:46:46 +01:00
mfvitale
ae9009303c DBZ-7308 Move getSnapshottingTask to RelationalSnapshotChangeEventSource 2024-03-25 05:46:46 +01:00
mfvitale
265ba0f6e6 DBZ-7308 Check if configured snapshot mode permits streaming before starting it. 2024-03-25 05:46:46 +01:00
Vaibhav Kushwaha
a4190be7b4 DBZ-7649 added missing test annotation 2024-03-19 13:37:19 +01:00
Chris Cranford
3a66a45632 DBZ-7596 Support reselection of ARRAY schema types 2024-03-18 13:44:14 +01:00
Chris Cranford
0e267d8ef6 DBZ-7596 Support reselection of PostgreSQL hstore values 2024-03-18 13:44:14 +01:00
mfvitale
71256cf1bc DBZ-7616 Honor the message.key.columns order while building query for incremental snapshot 2024-03-14 08:08:51 +01:00
harveyyue
48c67e40fb DBZ-7618 Implement Versioned interfaces in Transformation and Converter plugins 2024-03-14 06:35:42 +01:00
Nick Golubev
69efa7e218 DBZ-7599 Write toast placeholder instead of null for binary types when "hex" mode configured 2024-03-06 15:06:09 +01:00
Debezium Builder
2fb8fc3004 [maven-release-plugin] prepare for next development iteration 2024-03-06 07:47:33 +00:00
Debezium Builder
cd46b2b998 [maven-release-plugin] prepare release v2.6.0.Beta1 2024-03-06 07:47:33 +00:00
mfvitale
9ad4273791 DBZ-7303 Align snapshot modes for SqlServer connector 2024-03-04 11:17:06 +01:00
mfvitale
8a47b2333d DBZ-7588 Disable debug prints on tests 2024-03-02 11:33:48 -05:00
mfvitale
c942bd26d1 DBZ-7461 Revisit snapshot modes behavior when snapshot already completed 2024-03-01 14:12:31 +01:00
mfvitale
e174eae235 DBZ-7461 Remove redundant log 2024-03-01 14:12:31 +01:00
mfvitale
4ade54351f DBZ-7461 Rename SCHEMA_ONLY_RECOVERY to RECOVERY and SCHEMA_ONLY to NO_DATA 2024-03-01 14:12:31 +01:00
mfvitale
211675a355 DBZ-7461 Rename shouldSnapshot to shouldSnapshotData 2024-03-01 14:12:31 +01:00
mfvitale
562c999da4 DBZ-7461 Move out-of-the-box snapshotters to core 2024-03-01 14:12:31 +01:00
mfvitale
371905f135 DBZ-7461 Move snapshot mode validation logic to BaseSourceTask 2024-03-01 14:12:31 +01:00
mfvitale
ce1b5d7231 DBZ-7461 Support WhenNeededSnapshotter for PostgreSQL 2024-03-01 14:12:31 +01:00
akula
cd4c6958bd DBZ-7512 Support arbitrary payloads with outbox event router on 
debezium server

1. Support for string and binary serialization formats on debezium api.
2. Allow configuring separate key and value formats on embedded engine.

This change fixes the following issue using outbox event router on
embedded engine:

Outbox event router supports arbitrary payload formats with
BinaryDataConverter as the value.converter which passes payload
transparently. However this is  currently not supported with the
embedded engine which handles message conversion using value.format to
specify the format.

In addition, when we want to pass payload transparently, it makes
sense to also pass aggregateid i.e. the event key transparently. The
default outbox table configuration specifies aggregateid as a
varchar which is also not supported by embedded engine.
 2024-03-01 08:23:47 +01:00
Jiri Pechanec
6564579423 DBZ-7581 Log exception and stack trace 2024-03-01 08:14:16 +01:00
James Johnston
3d93516b53 DBZ-5071 Create new RowValueConstructorChunkQueryBuilder 
The default query builder has maximum SQL compatibility, but the query
plans it makes are not always optimal, especially in the case of
multi-column keys.  For example, PostgreSQL is unable to to effectively
create a query plan that does index scanning when faced with such a
query.

Some databases support the concept of row value constructors.  In
these cases, we can use these as an alternative to make a much more
simple and easier to understand query.  Not only is it easier for
humans to understand, but most importantly, the query planner also gets
the hint and finally uses the relevant multi-column index!  This commit
starts out with support for PostgreSQL and MySQL.
 2024-02-29 13:36:26 +01:00
James Johnston
f632fa081e DBZ-5071 Correctly handle NULL values in incremental snapshots 
It turns out that the existing code for chunking a table when taking
an incremental snapshot was buggy and did not correctly handle NULL
values when building the chunk query.  An example of such a situation
would be when the user has specified "message.key.columns" to reference
a column that is part of a PostgreSQL UNIQUE INDEX that was created with
the NULLS NOT DISTINCT option.

This commit updates the new AbstractChunkQueryBuilder so that it checks
whether a key column is optional.  If it is, then additional will
appropriately consider NULL values when generating a chunk query using
"IS [NOT] NULL" clauses.

One complication is that different database engines have different
sorting behavior of ORDER BY.  It is apparently not well-defined by the
SQL standard.  Some databases consider NULL values to be higher than any
non-NULL values, and others consider them to be lower.

To handle this situation, a new nullsSortLast() function is added to the
JdbcConnection class.  By default, it returns an empty value, indicating
that the behavior of the database engine is unknown.  When an optional
field is encountered by AbstractChunkQueryBuilder in this situation, we
throw an error because we don't actually know how to correctly chunk the
query: there's no safe assumption that can be made here.

Derived JdbcConnection classes can then override the nullsSortLast
function, and return a value indicating the actual behavior of that
database engine.  When this is done, the AbstractChunkQueryBuilder then
knows how to correctly build a chunk query that can handle NULL values.

To help test this, new tests have been added to
AbstractIncrementalSnapshotTest.  First, the existing insertsWithoutPks
test has been moved and deduplicated from MySQL and PostgreSQL so that
the test case can be reused on other engines.  Second, a new
insertsWithoutPksAndNull test is run, which inserts data with NULL
values in the message key columns.  To demonstrate that chunk queries
are being correctly generated for practically every case, the
INCREMENTAL_SNAPSHOT_CHUNK_SIZE is set to 1 so that NULL values are not
returned in the middle of a chunk, which can cause us to skip testing
the code we need to test.
 2024-02-29 13:36:26 +01:00
Chris Cranford
c83c3c3922 DBZ-7582 Correctly serialize toasted HSTORE values when unavailable 2024-02-29 11:18:58 +01:00
Chris Cranford
254fc2fa1b DBZ-7562 Add requested validate record 2024-02-27 22:40:15 -05:00
Chris Cranford
3fd66483d9 DBZ-7562 Add "default values" assertion check 2024-02-27 22:40:15 -05:00
Chris Cranford
45805eef8c DBZ-7562 Truncate values that exceed column scale 2024-02-27 22:40:15 -05:00
Chris Cranford
b6e45beab1 DBZ-7453 Exclude testEmptyChangesProducesHeartbeat for pgoutput 2024-02-23 13:56:26 -05:00
Chris Cranford
940b3cfbeb DBZ-7530 Make PG parallel snapshot test shouldGenerateSnapshotAndContinueStreaming stable 2024-02-23 13:56:26 -05:00
rkerner
1e69e40ec2 DBZ-7416 Fix duplicate SMTs sometimes returned by Kafka Connect. Moved deduplication from Map to LinkedHashSet. 
+ minor fixes added for cleanup and centralization of common code

closes https://issues.redhat.com/browse/DBZ-7416
 2024-02-22 13:34:08 -05:00
Jiri Pechanec
68b6591142 DBZ-7416 Fix duplicate SMTs sometimes returned by Kafka Connect. Moved deduplication from Map to LinkedHashSet. 
closes https://issues.redhat.com/browse/DBZ-7416
 2024-02-22 13:34:08 -05:00
mfvitale
65a34b10d3 DBZ-7526 Fix table name with escaped characters 2024-02-20 11:40:30 -05:00
mfvitale
7ed5649e07 DBZ-7302 Implement Snapshotter for Oracle 2024-02-20 14:45:59 +01:00
mfvitale
9fe60a698d DBZ-7302 Move snapshot.locking.mode.custom.name, snapshot.query.mode and snapshot.query.mode.custom.name to CommonConnectorConfig 2024-02-20 14:45:59 +01:00
harveyyue
82f5e6ea77 DBZ-7480 Allow special characters in signal table name 2024-02-19 11:43:20 +01:00
Chris Cranford
928aa26cdb DBZ-7107 Fix source info test failures 2024-02-16 12:52:20 +01:00
Debezium Builder
10e327602c [maven-release-plugin] prepare for next development iteration 2024-02-13 09:20:04 +00:00
Debezium Builder
0c5b05738c [maven-release-plugin] prepare release v2.6.0.Alpha2 2024-02-13 09:20:04 +00:00
mfvitale
cb5a4d7a1a DBZ-7481 SnapshotterServiceProvider will check if snapshot mode class is related to the running connector. 2024-02-13 08:42:34 +01:00
Vojtech Juranek
4452e3d095 DBZ-7024 Move EmbeddedConfig into separate class 
`EmbeddedConfig` needs to be shared with other implmentations of
`DebeziumEngine` as long as Debezium embedded depends on the Kafka
model.
 2024-02-12 13:43:21 +01:00
mfvitale
2c9b77fc1c DBZ-7452 defaultJdbcConfig will use correct hostname and port from parameters 2024-02-08 14:31:55 +01:00
Chris Cranford
8a2c098e78 [ci] Mark testEmptyChangesProducesHeartbeat as flaky 2024-02-06 09:49:05 -05:00
mfvitale
a8a07e35f1 DBZ-7301 Implement SnapshotLock for MySQL connector 2024-02-06 07:12:42 +01:00
mfvitale
f78d0cbd19 DBZ-7301 Add SnapshotterService to ChangeEventSourceFactory and SnapshotChangeEventSource 2024-02-06 07:12:42 +01:00
mfvitale
c9458f4f58 DBZ-7300 Snapshotter, SnapshotLock and SnapshotQuery are now services registered in the ServiceRegistry 2024-02-06 07:12:42 +01:00
mfvitale
57c4e2a876 DBZ-7300 Add new properties to config definition 2024-02-02 13:41:45 +01:00
mfvitale
881442de3d DBZ-7300 Fix duplicate field assignment for CONNECTOR_SNAPSHOT group 2024-02-02 13:41:45 +01:00
mfvitale
3b92786a41 DBZ-7300 Snapshotter, SnapshotLock and SnapshotQuery are now services registered in the ServiceRegistry 2024-02-02 13:41:45 +01:00
mfvitale
8e17724392 DBZ-7300 Refactor Snapshotter interface to be an SPI through SnapshotterProvider. 
Also, this will split snapshot locking and query behavior into a separate interface
 2024-02-02 13:41:45 +01:00
nicholas-fwang
7d99605886 DBZ-7143 Add case when EventConvertingFailureHandlingMode is null 2024-01-29 13:48:19 +01:00
nicholas-fwang
eeea0f1e70 DBZ-7143 Add event.converting.failure.handling.mode option 2024-01-29 13:48:19 +01:00
Vojtech Juranek
8365221719 DBZ-7382 Don't check record order in parallel snapshot tests 2024-01-26 08:11:52 +01:00
Debezium Builder
65d63ed42d [maven-release-plugin] prepare for next development iteration 2024-01-21 10:12:45 +00:00
Debezium Builder
485fa82a8f [maven-release-plugin] prepare release v2.6.0.Alpha1 2024-01-21 10:12:44 +00:00
Jiri Pechanec
2d1ce13d7c DBZ-7316 Formatting and cotnributor 2024-01-16 11:00:04 +01:00
James Johnston
20fbbd6452 DBZ-7316 Send heartbeats while searching for WAL position 
If the logical replication slot has absolutely zero events in it, then
the searchWalPosition could be looping for quite some time.  During
this time, it wasn't sending heartbeats.

This commit fixes that function to send heartbeats.
 2024-01-16 11:00:04 +01:00
mfvitale
5d088cffca DBZ-7312 On demand blocking snapshot will not set transaction snapshot 2024-01-15 14:29:31 +01:00
mfvitale
20fbe73bc8 DBZ-7311 Use a more meaningful name for identify an on demand blocking snapshot 2024-01-08 12:52:03 +01:00
mfvitale
47cbdee526 DBZ-7311 Permits to execute a blocking snapshot even if snapshot.mode=never 2024-01-08 12:52:03 +01:00
Mickael Maison
70e7186868 DBZ-7298 Add service loader manifests for all Connect plugins 
Add manifests for source/sink connectors, converters and transformations.
 2024-01-04 09:22:29 +01:00
Debezium Builder
3853d20f44 [maven-release-plugin] prepare for next development iteration 2023-12-21 06:52:01 +00:00
Debezium Builder
5d35e9caaa [maven-release-plugin] prepare release v2.5.0.Final 2023-12-21 06:52:01 +00:00