Commit Graph

381 Commits

Author SHA1 Message Date
Jakub Cechacek
dd5e6e1a16 DBZ-8135 Added Signaler support to async engine 2024-08-20 13:03:22 +02:00
Ryan van Huuksloot
8617f474c5 DBZ-8082: Pass Headers to Key/Value Converters 2024-08-15 08:44:31 +02:00
rkerner
42fbde8268 DBZ-8138 Re-add check if assembly profile is active as JUnit annotation
closes https://issues.redhat.com/browse/DBZ-8138
2024-08-15 08:15:43 +02:00
Lars M Johansson
ba39a27b89 DBZ-8114: waitForAvailableRecords with records.waittime 2024-08-15 08:04:26 +02:00
Lars M Johansson
da143ef8b2 DBZ-8114: start connector after altering schema to avoid having to stop and start it again for Informix 2024-08-15 08:04:26 +02:00
Vojtech Juranek
e583599d22 DBZ-8100 Wait for streaming instead of starting connector
If we stop the connector short after its start, we need to wait for
streaming phase, i.e. wait for starting all connector tasks.
Waiting for connector start return `true` when connector begins to start
its tasks, but to prevent leaking of the resources, async engine doesn't
allow stopping the connector until all its tasks are started. Therefore
when using async. engine, we have to wait for tasks to be started before
stopping the connector, otherwise exception is thrown.
2024-08-02 05:31:22 +02:00
Vojtech Juranek
fb94614419 DBZ-8106 Close transfomations upon embedded engine stop 2024-07-31 11:19:07 +02:00
Vojtech Juranek
63377d9d6a DBZ-8106 Stop header converter after stopping connector and its tasks 2024-07-31 11:19:07 +02:00
Ryan van Huuksloot
681db3ee03 DBZ-8040: Formatting 2024-07-16 17:26:51 -04:00
Ryan van Huuksloot
479a41af28 DBZ-8040: Custom Converter Support 2024-07-16 17:26:51 -04:00
mfvitale
3499f9e80d DBZ-8050 Skip data collection names that fails to be parsed as TableId 2024-07-16 16:56:30 -04:00
Ryan van Huuksloot
ad1ae7a124 [typo] Typo in AsyncDebeziumEngine 2024-07-10 06:22:39 +02:00
Vojtech Juranek
ee0045d962 DBZ-7976 Mark EmbeddedEngine as deprecated 2024-06-25 13:37:22 -04:00
mfvitale
05a04451ec DBZ-7917 Fix ReadOnlyIncrementalSnapshotIT.java test 2024-06-13 08:41:49 +02:00
mfvitale
9704f0c3e0 DBZ-7917 Add ReadOnlyIncrementalSnapshotIT.java test 2024-06-13 08:41:49 +02:00
Vojtech Juranek
8662f7ddd8 DBZ-7937 Add exception stack trace when engine fails to commit offset 2024-06-10 14:18:38 +02:00
Lars M Johansson
a7bfa8d11f DBZ-7813: waitForAvailableRecords using Awaitility 2024-06-10 07:20:20 +02:00
Lars M Johansson
145c42e700 DBZ-7813: waitForAvailableRecords using Awaitility 2024-06-06 13:36:36 +02:00
mfvitale
730eabc1f9 DBZ-7858 Initial snapshot completed/aborted notification will the correct offset 2024-05-14 12:14:53 +02:00
Michal Augustýn
5b14097a94 DBZ-7870 default value of -1 for connection error retries interpreted as no limit 2024-05-14 11:43:16 +02:00
mfvitale
70c526eb16 DBZ-7806 Skip search for WAl position when offset exist and is just after the initial snapshot. 2024-05-10 09:34:45 +02:00
Vojtech Juranek
294fdb9d3f DBZ-7777 Use explicit list iterator in for cycle 2024-04-19 15:11:48 +02:00
Vojtech Juranek
c4a54962ad DBZ-7777 Avoid using List#get() in ParallelSmtConsumerProcessor
Before the change:

    Benchmark                                        (processingOrder)  (recordCount)  (threadCount)  Mode  Cnt   Score   Error  Units
    DebeziumEnginePerf.processRecordsAsyncEngine               ORDERED         100000              1    ss       88.006           s/op
    DebeziumEnginePerf.processRecordsAsyncEngine               ORDERED         100000              2    ss       90.226           s/op
    DebeziumEnginePerf.processRecordsAsyncEngine               ORDERED         100000              4    ss       83.129           s/op
    DebeziumEnginePerf.processRecordsAsyncEngine               ORDERED         100000              8    ss       81.417           s/op
    DebeziumEnginePerf.processRecordsAsyncEngine               ORDERED         100000             16    ss       90.936           s/op
    DebeziumEnginePerf.processRecordsAsyncEngine             UNORDERED         100000              1    ss       83.645           s/op
    DebeziumEnginePerf.processRecordsAsyncEngine             UNORDERED         100000              2    ss       77.527           s/op
    DebeziumEnginePerf.processRecordsAsyncEngine             UNORDERED         100000              4    ss       88.495           s/op
    DebeziumEnginePerf.processRecordsAsyncEngine             UNORDERED         100000              8    ss       79.590           s/op
    DebeziumEnginePerf.processRecordsAsyncEngine             UNORDERED         100000             16    ss       90.246           s/op
    DebeziumEnginePerf.processRecordsEmbeddedEngine                N/A         100000            N/A    ss        0.125           s/op

After the change:

    Benchmark                                        (processingOrder)  (recordCount)  (threadCount)  Mode  Cnt  Score   Error  Units
    DebeziumEnginePerf.processRecordsAsyncEngine               ORDERED         100000              1    ss       0.514           s/op
    DebeziumEnginePerf.processRecordsAsyncEngine               ORDERED         100000              2    ss       0.391           s/op
    DebeziumEnginePerf.processRecordsAsyncEngine               ORDERED         100000              4    ss       0.467           s/op
    DebeziumEnginePerf.processRecordsAsyncEngine               ORDERED         100000              8    ss       0.393           s/op
    DebeziumEnginePerf.processRecordsAsyncEngine               ORDERED         100000             16    ss       0.462           s/op
    DebeziumEnginePerf.processRecordsAsyncEngine             UNORDERED         100000              1    ss       0.410           s/op
    DebeziumEnginePerf.processRecordsAsyncEngine             UNORDERED         100000              2    ss       0.431           s/op
    DebeziumEnginePerf.processRecordsAsyncEngine             UNORDERED         100000              4    ss       0.460           s/op
    DebeziumEnginePerf.processRecordsAsyncEngine             UNORDERED         100000              8    ss       0.419           s/op
    DebeziumEnginePerf.processRecordsAsyncEngine             UNORDERED         100000             16    ss       0.448           s/op
    DebeziumEnginePerf.processRecordsEmbeddedEngine                N/A         100000            N/A    ss       0.230           s/op
2024-04-19 15:11:48 +02:00
Vojtech Juranek
17d537c67f DBZ-7777 Avoid using List#get() in ParallelSmtAndConvertConsumerProcessor
Before the change:

    Benchmark                                        (processingOrder)  (recordCount)  (threadCount)  Mode  Cnt    Score   Error  Units
    DebeziumEnginePerf.processRecordsAsyncEngine               ORDERED         100000              1    ss       295.353           s/op
    DebeziumEnginePerf.processRecordsAsyncEngine               ORDERED         100000              2    ss       310.652           s/op
    DebeziumEnginePerf.processRecordsAsyncEngine               ORDERED         100000              4    ss       305.956           s/op
    DebeziumEnginePerf.processRecordsAsyncEngine               ORDERED         100000              8    ss       334.755           s/op
    DebeziumEnginePerf.processRecordsAsyncEngine               ORDERED         100000             16    ss       306.477           s/op
    DebeziumEnginePerf.processRecordsAsyncEngine             UNORDERED         100000              1    ss       257.661           s/op
    DebeziumEnginePerf.processRecordsAsyncEngine             UNORDERED         100000              2    ss        78.385           s/op
    DebeziumEnginePerf.processRecordsAsyncEngine             UNORDERED         100000              4    ss        75.899           s/op
    DebeziumEnginePerf.processRecordsAsyncEngine             UNORDERED         100000              8    ss        81.068           s/op
    DebeziumEnginePerf.processRecordsAsyncEngine             UNORDERED         100000             16    ss        94.506           s/op
    DebeziumEnginePerf.processRecordsEmbeddedEngine                N/A         100000            N/A    ss         0.857           s/op

After the change:

    Benchmark                                        (processingOrder)  (recordCount)  (threadCount)  Mode  Cnt  Score   Error  Units
    DebeziumEnginePerf.processRecordsAsyncEngine               ORDERED         100000              1    ss       0.738           s/op
    DebeziumEnginePerf.processRecordsAsyncEngine               ORDERED         100000              2    ss       0.404           s/op
    DebeziumEnginePerf.processRecordsAsyncEngine               ORDERED         100000              4    ss       0.524           s/op
    DebeziumEnginePerf.processRecordsAsyncEngine               ORDERED         100000              8    ss       0.505           s/op
    DebeziumEnginePerf.processRecordsAsyncEngine               ORDERED         100000             16    ss       0.428           s/op
    DebeziumEnginePerf.processRecordsAsyncEngine             UNORDERED         100000              1    ss       0.685           s/op
    DebeziumEnginePerf.processRecordsAsyncEngine             UNORDERED         100000              2    ss       0.400           s/op
    DebeziumEnginePerf.processRecordsAsyncEngine             UNORDERED         100000              4    ss       0.357           s/op
    DebeziumEnginePerf.processRecordsAsyncEngine             UNORDERED         100000              8    ss       0.508           s/op
    DebeziumEnginePerf.processRecordsAsyncEngine             UNORDERED         100000             16    ss       0.410           s/op
    DebeziumEnginePerf.processRecordsEmbeddedEngine                N/A         100000            N/A    ss       0.857           s/op
2024-04-19 15:11:48 +02:00
twthorn
ea8dddade3 DBZ-7698 Add ordered transaction metadata 2024-04-19 10:23:30 +02:00
mfvitale
72b3b4c3c0 DBZ-7732 Fix test wait condition 2024-04-11 07:53:48 +02:00
mfvitale
64454cc378 DBZ-7716 Correct IncrementalSnapshotContext dataCollectionsToSnapshot cache update 2024-04-11 07:53:48 +02:00
Vojtech Juranek
a2eaddffcf DBZ-7764 Use cache thread pool as the default for async. engine 2024-04-11 07:46:01 +02:00
Vojtech Juranek
230076205f DBZ-7763 Remove async engine records processing thread cap 2024-04-11 07:46:01 +02:00
mfvitale
1190389edc DBZ-7718 Support table name escaping in BlockingSnapshotIT 2024-04-04 14:58:24 +02:00
mfvitale
ba2f893168 DBZ-7718 Support table name escaping in data collection to be snapshotted 2024-04-04 14:58:24 +02:00
Vojtech Juranek
d88fd9e2e7 DBZ-7710 Remove unneded copying from RecordProcessors 2024-03-27 14:05:11 +01:00
Vojtech Juranek
6b35efc00a DBZ-7661 Don't propagate cancelation exption when polling is stopped
`CancellationException` should be thrown mostly in the test when we want
to stop as soon as possible and may not wait for polling task to finish.
2024-03-25 17:48:31 +01:00
Vojtech Juranek
8e0c6ad88e DBZ-7661 Close properly offset backing store 2024-03-25 17:48:31 +01:00
Vojtech Juranek
16a089abb6 DBZ-7661 Always shut down record processing thread pool 2024-03-25 17:48:31 +01:00
Chris Cranford
200f9ed28e DBZ-7596 Improved tests for reselect post processor 2024-03-18 13:44:14 +01:00
Chris Cranford
0e267d8ef6 DBZ-7596 Support reselection of PostgreSQL hstore values 2024-03-18 13:44:14 +01:00
Andreas Martens
adabb899f8 DBZ-7614: expand scope of catch furing Engine validation 2024-03-11 15:41:30 +01:00
Andreas Martens
c74c6d6188 DBZ-7614: indent changes in EmbeddedEngine 2024-03-11 15:41:30 +01:00
Andreas Martens
85aea68c9a DBZ-7614: catch exception during validation 2024-03-11 15:41:30 +01:00
mfvitale
9ad4273791 DBZ-7303 Align snapshot modes for SqlServer connector 2024-03-04 11:17:06 +01:00
mfvitale
211675a355 DBZ-7461 Rename shouldSnapshot to shouldSnapshotData 2024-03-01 14:12:31 +01:00
akula
cd4c6958bd DBZ-7512 Support arbitrary payloads with outbox event router on
debezium server

1. Support for string and binary serialization formats on debezium api.
2. Allow configuring separate key and value formats on embedded engine.

This change fixes the following issue using outbox event router on
embedded engine:

Outbox event router supports arbitrary payload formats with
BinaryDataConverter as the value.converter which passes payload
transparently. However this is  currently not supported with the
embedded engine which handles message conversion using value.format to
specify the format.

In addition, when we want to pass payload transparently, it makes
sense to also pass aggregateid i.e. the event key transparently. The
default outbox table configuration specifies aggregateid as a
varchar which is also not supported by embedded engine.
2024-03-01 08:23:47 +01:00
Vojtech Juranek
6cc68fdfe3 DBZ-7568 Use default engine wait time in all async engine latches 2024-02-29 13:44:54 +01:00
Vojtech Juranek
6928cd775c DBZ-7568 Switch waitTimeForEngine() to seconds and increase wait default time 2024-02-29 13:44:54 +01:00
James Johnston
f632fa081e DBZ-5071 Correctly handle NULL values in incremental snapshots
It turns out that the existing code for chunking a table when taking
an incremental snapshot was buggy and did not correctly handle NULL
values when building the chunk query.  An example of such a situation
would be when the user has specified "message.key.columns" to reference
a column that is part of a PostgreSQL UNIQUE INDEX that was created with
the NULLS NOT DISTINCT option.

This commit updates the new AbstractChunkQueryBuilder so that it checks
whether a key column is optional.  If it is, then additional will
appropriately consider NULL values when generating a chunk query using
"IS [NOT] NULL" clauses.

One complication is that different database engines have different
sorting behavior of ORDER BY.  It is apparently not well-defined by the
SQL standard.  Some databases consider NULL values to be higher than any
non-NULL values, and others consider them to be lower.

To handle this situation, a new nullsSortLast() function is added to the
JdbcConnection class.  By default, it returns an empty value, indicating
that the behavior of the database engine is unknown.  When an optional
field is encountered by AbstractChunkQueryBuilder in this situation, we
throw an error because we don't actually know how to correctly chunk the
query: there's no safe assumption that can be made here.

Derived JdbcConnection classes can then override the nullsSortLast
function, and return a value indicating the actual behavior of that
database engine.  When this is done, the AbstractChunkQueryBuilder then
knows how to correctly build a chunk query that can handle NULL values.

To help test this, new tests have been added to
AbstractIncrementalSnapshotTest.  First, the existing insertsWithoutPks
test has been moved and deduplicated from MySQL and PostgreSQL so that
the test case can be reused on other engines.  Second, a new
insertsWithoutPksAndNull test is run, which inserts data with NULL
values in the message key columns.  To demonstrate that chunk queries
are being correctly generated for practically every case, the
INCREMENTAL_SNAPSHOT_CHUNK_SIZE is set to 1 so that NULL values are not
returned in the middle of a chunk, which can cause us to skip testing
the code we need to test.
2024-02-29 13:36:26 +01:00
Xianming Zhou
86c1dac16a DBZ-7517 Remove the unused 'connector' parameter in the createSourceTask method in EmbeddedEngine.java 2024-02-22 09:07:55 +01:00
Jiri Pechanec
df18e00173 DBZ-7535 Use error level for error message 2024-02-22 08:31:58 +01:00
Vojtech Juranek
7edefef14c DBZ-7535 Ensure at least one task starts and one fails
Add also `INFO` log with number of failed tasks.
2024-02-22 08:31:58 +01:00
Chris Cranford
68f31f7662 [ci] Log records found when expecting no records 2024-02-21 09:00:45 -05:00