Vojtech Juranek
eef8ee4cea
DBZ-7024 Move async engine into separate package
2024-02-12 13:43:21 +01:00
Vojtech Juranek
425407331c
DBZ-7024 Add TODO item for improving ConnectorCallback API
2024-02-12 13:43:21 +01:00
Vojtech Juranek
3edc61e443
DBZ-7024 Improve processor instantiation
2024-02-12 13:43:21 +01:00
Vojtech Juranek
cdf5e0255a
DBZ-7024 Improve log level and log messges
2024-02-12 13:43:21 +01:00
Vojtech Juranek
a04dc84b3e
DBZ-7024 Embedde state comparions into State enum methods
...
It's more safe to have the comparions directly in the enum and also make
obvisou that the ordering of enum is important.
2024-02-12 13:43:21 +01:00
Vojtech Juranek
5c21d19815
DBZ-7024 Use enum for record processing order option
2024-02-12 13:43:21 +01:00
Vojtech Juranek
ee1f33fe33
DBZ-7024 Limit size of records processing thread pool
...
If the number of threads is provided as a number, limit it to 16 threads
to avoid possible overhead with context switching on a beefy machines
where the default value using all available cores may result in many
threads, which would be waiting most of the time anyway, as such machine
may run probably many other tasks, not only Debezium.
If the user really wants to use all available cores, it can be specified
using `AVAILABLE-CORES` placeholder.
2024-02-12 13:43:21 +01:00
Vojtech Juranek
e2d2cff7fd
DBZ-7024 Interrupt polling if needed
...
Some polling tasks may be stuck and we need to interrupt polling during
the shutdown not have to wait for TASK_MANAGEMENT_TIMEOUT_MS to timeout.
Also, when we start to interrput polling, we have to remove interruption
of the main thread in the `catch` part. It was a bug anyway as it
interrputed the main thread what we definitelly don't want to happen in
any case.
2024-02-12 13:43:21 +01:00
Vojtech Juranek
40131c0531
DBZ-7024 Increase task management timeout to 2min
...
Increase task management timeout to two minutes and make this option
internal. This timeout will be hopefully sufficient for most of the
deployments. If not, we will increase the timeout it make this option
public.
2024-02-12 13:43:21 +01:00
Vojtech Juranek
fc7381ad91
DBZ-7024 Improve javadoc and comments
2024-02-12 13:43:21 +01:00
Vojtech Juranek
8bb1a122b1
DBZ-7024 Add missing condition
2024-02-12 13:43:21 +01:00
Vojtech Juranek
0f7d3100b4
DBZ-7024 Add option for creating default ChangeConsumer
...
This option effective allowes the user to request serial processing of
the records byt the provided Consumer.
2024-02-12 13:43:21 +01:00
Vojtech Juranek
cc5f7aedd1
DBZ-7024 Don't provide default ChangeConsumer
...
To allow user to use different processors, don't provide the default
ChangeHandler.
2024-02-12 13:43:21 +01:00
Vojtech Juranek
de2e4aba9f
DBZ-7024 Add debug logging when selecting the processor
2024-02-12 13:43:21 +01:00
Vojtech Juranek
001cb2a640
DBZ-7024 Improve javadocs and comments, fix typos
2024-02-12 13:43:21 +01:00
Vojtech Juranek
4689db90c0
DBZ-7024 Switch abstract embedded tests to async engine
2024-02-12 13:43:21 +01:00
Vojtech Juranek
dfdeab7ab8
DBZ-7024 Add method to await engine shutdown
2024-02-12 13:43:21 +01:00
Vojtech Juranek
3ec22951ce
DBZ-7024 Create testing engine and base class for async engine tests
2024-02-12 13:43:21 +01:00
Vojtech Juranek
293b84645d
DBZ-7024 Make default RecordCommitter thread unsafe
...
Default implmentation of `RecordCommitter`, the `SourceRecordCommitter`,
is always created for each task and withing given task is called
sequentially, always in the same thread. There's no need to aquire locks
for each method call.
Make `SourceRecordCommitter` thread unsafe.
2024-02-12 13:43:21 +01:00
Vojtech Juranek
998f00f811
DBZ-7024 Create initial implementation of async embedded engine
...
Implementation is based on the proposed Debezium design document for
asynchronous embedded engine, which is currently still WIP:
https://github.com/debezium/debezium-design-documents/pull/8
2024-02-12 13:43:21 +01:00
Vojtech Juranek
69bbed1fa3
DBZ-7024 Allow to override type into AbstractConnectorTest
2024-02-12 13:43:21 +01:00
Vojtech Juranek
1eec31b3f3
DBZ-7024 Allow to specify multiple records when error should be thrown during processing
2024-02-12 13:43:21 +01:00
Vojtech Juranek
542b0fec7f
DBZ-7024 Intorduce retryable callable
2024-02-12 13:43:21 +01:00
Vojtech Juranek
d7b7768071
DBZ-7024 Add more testing connectors
...
Add connector which runs mutiple tasks and connector whose some of
the tasks fail.
2024-02-12 13:43:21 +01:00
Vojtech Juranek
7eaf0fc288
DBZ-7024 Move reuseable testing functions for DebeziumEngine into common util class
2024-02-12 13:43:21 +01:00
Vojtech Juranek
b8e16ee89f
DBZ-7024 Move reuseable interface implementations for DebeziumEngine into common class
2024-02-12 13:43:21 +01:00
Vojtech Juranek
ba35b395c5
DBZ-7024 Move required setup into EmbeddedWorkerConfig
...
Move required by `WorkerConfig` validators into `EmbeddedWorkerConfig`
so we have all Kafka related stuff in the same class.
2024-02-12 13:43:21 +01:00
Vojtech Juranek
4452e3d095
DBZ-7024 Move EmbeddedConfig into separate class
...
`EmbeddedConfig` needs to be shared with other implmentations of
`DebeziumEngine` as long as Debezium embedded depends on the Kafka
model.
2024-02-12 13:43:21 +01:00
Chris Cranford
a597a82c19
DBZ-7439 Remove unnecessary log/output entries
2024-02-07 15:00:27 +01:00
Chris Cranford
de9364bb4d
DBZ-7439 Fix test compatibility
2024-02-07 15:00:27 +01:00
Debezium Builder
65d63ed42d
[maven-release-plugin] prepare for next development iteration
2024-01-21 10:12:45 +00:00
Debezium Builder
485fa82a8f
[maven-release-plugin] prepare release v2.6.0.Alpha1
2024-01-21 10:12:44 +00:00
Artem Shubovych
27f42d101a
DBZ-7342 Replace temporary variable with immediate and early return
2024-01-17 09:50:54 +01:00
Artem Shubovych
d9de6ceba2
DBZ-7342 Respect the max error retries setting
2024-01-17 09:50:52 +01:00
Roman Kudryashov
0c80f1f38d
DBZ-7284 Provide config option to customize CloudEvents.data schema name
2024-01-11 13:20:35 +01:00
Jakub Cechacek
aa0c53ec2a
DBZ-7260 Offset consolidation test coverage
2024-01-11 09:58:24 +01:00
mfvitale
47cbdee526
DBZ-7311 Permits to execute a blocking snapshot even if snapshot.mode=never
2024-01-08 12:52:03 +01:00
Jiri Pechanec
7a80c9dae8
DBZ-7098 Converting engine should honor unsupported tombstone flag
2024-01-05 08:25:23 +01:00
Debezium Builder
3853d20f44
[maven-release-plugin] prepare for next development iteration
2023-12-21 06:52:01 +00:00
Debezium Builder
5d35e9caaa
[maven-release-plugin] prepare release v2.5.0.Final
2023-12-21 06:52:01 +00:00
Roman Kudryashov
516aa87fad
DBZ-7235 Add option to customize CloudEvents schema name
2023-12-20 06:53:37 +01:00
Debezium Builder
2c1def7241
[maven-release-plugin] prepare for next development iteration
2023-12-14 09:43:13 +00:00
Debezium Builder
ef8260f802
[maven-release-plugin] prepare release v2.5.0.CR1
2023-12-14 09:43:12 +00:00
Chris Cranford
2b02b3982e
DBZ-4321 Rework configuration options
2023-12-13 11:27:40 -05:00
Chris Cranford
59027ed5ed
DBZ-4321 New PostProcessor
contract and Column Reselection
2023-12-13 11:27:40 -05:00
mfvitale
5ed16284f4
DBZ-6834 Provide INSERT/DELETE semantics for MongoDb incremental snapshot watermarking
2023-12-06 14:10:26 +01:00
mfvitale
4fedfbba03
DBZ-6834 Provide INSERT/DELETE semantics for incremental snapshot watermarking
2023-12-06 14:10:26 +01:00
Debezium Builder
0fd1c0dc9a
[maven-release-plugin] prepare for next development iteration
2023-12-04 13:55:35 +00:00
Debezium Builder
3e2d75f0da
[maven-release-plugin] prepare release v2.5.0.Beta1
2023-12-04 13:55:35 +00:00
“vsantonastaso”
8c1c369449
DBZ-6878 add table specific notification in initial snapshot
2023-11-29 08:32:21 +01:00
Sebastiaan Knijnenburg
036eda0c64
DBZ-6723 Expose partition number in ChangeEvent interface
...
Based on feedback in https://github.com/debezium/debezium-server/pull/33 this
commit adds the partition() method to the ChangeEvent interface and implements
it in the EmbeddedEngineChangeEvent. This allows reading the assigned partition
for an event in downstream processors, for example in custom Sinks that need
the assigned partition for routing purposes.
Link: https://issues.redhat.com/browse/DBZ-6723
2023-11-28 09:38:18 +01:00
Roman Kudryashov
1992c1e7e4
DBZ-7159 Fail fast during deserialization if a value is not a CloudEvent
2023-11-23 14:19:37 +01:00
Vojtech Juranek
60939c8965
DBZ-7099 Provide default value for PeriodicCommitOffsetPolicy
...
In 7b4cf1901
deprecated `io.debezium.embedded.spi.OffsetCommitPolicy`,
which provided also constructor for `Configuration`, was removed.
This constructor was actually used for creating `OffsetCommitPolicy`.
`Configuration` provides default values for options which are not
explicitly set, while `Properties` based constructor cannot do that
and therefore with this switch the default value for
`offset.flush.interval.ms` is now missing.
As debezium-api package has no knowledge about `Configuration`
interface, which is part of debezium-core, and thus about the default
values, specify default value direcly in the `PeriodicCommitOffsetPolicy`
class.
2023-11-14 06:58:11 +01:00
Debezium Builder
1521445908
[maven-release-plugin] prepare for next development iteration
2023-11-10 10:26:05 +00:00
Debezium Builder
6c6f6e9138
[maven-release-plugin] prepare release v2.5.0.Alpha2
2023-11-10 10:26:05 +00:00
Vojtech Juranek
cf7f0f3801
DBZ-7110 Use better name for DebeziumEngine.Builder implementation
2023-11-07 10:13:50 +01:00
Vojtech Juranek
c34cf6920c
DBZ-7110 Remove deprecated EmbeddedEngine interface
2023-11-07 10:13:50 +01:00
Vojtech Juranek
11d2ff0b9b
DBZ-7007 Use better name for auxiliary variable detecing if engine is running
2023-11-06 10:50:21 +01:00
Vojtech Juranek
89054bf8a8
DBZ-7007 Introduce TestingDebeziumEngine and switch to it in AbstractConnectorTest
...
Postgres `RecordsStreamProducerIT` reliaes on
EmebeddedEngine.runWithTask(). As this method effectively expose
engine's internal task and tests do the asserts against the state
of the task, it's hard to replace it. If we want to keep the tests,
the most simple approach seems to expose engine task in similar way
how EmbeddedEngine does that.
Add interface for testing Debezium engine, which would define minimal
set of methods which needs to be exposed by the implementing classes to
be able to run the testsuite against the Debezium engine. The number of
such methods should be as low as possible. Implementing classes would
typically act as proxies to actual `DebeziumEngine` implementations.
Add `TestingDebeziumEngine` implementation for `EmbeddedEngine` and
switch to `TestingDebeziumEngine` in `AbstractConnectorTest`.
2023-11-06 10:50:21 +01:00
Vojtech Juranek
fdab2cd61b
DBZ-7007 Remove engine.isRunning() from testsuite
2023-11-06 10:50:21 +01:00
Vojtech Juranek
89277591dd
DBZ-7007 Replace engine.await() with Awaitility condition
...
In the second case replace infinite loop by one minute wait. If this is
not sufficient for any case, we can increase the timeout in the future.
2023-11-06 10:50:21 +01:00
Vojtech Juranek
9b1d5172ff
DBZ-6213 Remove unneeded code from the testsuite
...
After fixing start method to really start the connector, this wait loop
is not needed any more.
2023-11-06 10:50:21 +01:00
Vojtech Juranek
1a58845507
DBZ-7007 Use close() instead of EmbeddedEngine#stop() method
2023-11-06 10:50:21 +01:00
Vojtech Juranek
953f1c77aa
DBZ-7007 Switch to DebeziuEngine in ConnectorOutputTest
2023-11-06 10:50:21 +01:00
Vojtech Juranek
d0c2658c43
DBZ-7007 Switch to DebeziuEngine in EmbeddedEngineTest
2023-11-06 10:50:21 +01:00
Vojtech Juranek
3295844cc3
DBZ-7007 Use property based configuration in the emebdded tests
2023-11-06 10:50:21 +01:00
Vojtech Juranek
9d26dd4cf4
DBZ-7007 Move embedded engine config into separate interface
...
This config will be re-used by possible other implementations of
DebeiumEngine API in the embedded package. As DebeziumEngine API
can have completely different implementations and thus also config,
the class is called `EmbeddedEngineConfig` as it's assumed to be used
only by embedded engine "family" of implementations.
To keep backward compatibility, the config options are extracted into
an interface and `EmbeddedEngine` implements this interface, thus
allowing to use these options in custom classes without any need for the
code changes.
2023-11-06 10:50:21 +01:00
Debezium Builder
78c5204444
[maven-release-plugin] prepare for next development iteration
2023-10-26 15:39:35 +00:00
Debezium Builder
356e488e83
[maven-release-plugin] prepare release v2.5.0.Alpha1
2023-10-26 15:39:35 +00:00
Roman Kudryashov
b0d7d60b5d
DBZ-7016 Optimize retrieval of id and type
2023-10-19 11:35:18 +02:00
Roman Kudryashov
22f6b083b1
DBZ-7016 Change metadata.location property syntax
2023-10-19 11:35:18 +02:00
Roman Kudryashov
ee64c70546
DBZ-7016 id
and type
of a CloudEvent can be retrieved from headers
2023-10-19 11:35:18 +02:00
Vojtech Juranek
7b4cf1901c
DBZ-7013 Remove deprecated and unused code from embedded package
2023-10-10 06:36:05 +02:00
Roman Kudryashov
01485aceb3
DBZ-6982 Provide configuration option to exclude extension attributes from a CloudEvent
2023-10-04 13:58:53 +02:00
Debezium Builder
40a8f4026d
[maven-release-plugin] prepare for next development iteration
2023-10-03 14:18:54 +00:00
Debezium Builder
412c6f7979
[maven-release-plugin] prepare release v2.4.0.Final
2023-10-03 14:18:54 +00:00
mfvitale
993413955c
DBZ-6793 Add timestamp for notifications
2023-09-28 14:20:17 -04:00
Roman Kudryashov
98c426dec4
DBZ-3642 CloudEventsConverter can retrieve metadata info from headers
2023-09-26 13:06:09 +02:00
Animesh Kumar
fcbff074f4
DBZ-6787 Dedupe table list while incremental snapshot
2023-09-26 10:50:32 +02:00
Debezium Builder
cc1de0aa51
[maven-release-plugin] prepare for next development iteration
2023-09-22 09:00:45 +00:00
Debezium Builder
6809978083
[maven-release-plugin] prepare release v2.4.0.CR1
2023-09-22 09:00:45 +00:00
mfvitale
5d7ba3e951
DBZ-6957 Remove spaces in Signal and Notification JMX ObjectName
2023-09-21 12:04:33 -04:00
mfvitale
68eb4b2df2
DBZ-6828 Process drop table events during blocking snapshot
2023-09-20 08:18:33 +02:00
mfvitale
ae199d2053
DBZ-6828 Blocking snapshot will generate schema events only for signaled tables
2023-09-20 08:18:33 +02:00
Debezium Builder
65169a9315
[maven-release-plugin] prepare for next development iteration
2023-09-13 09:41:46 +00:00
Debezium Builder
845e3e9d09
[maven-release-plugin] prepare release v2.4.0.Beta2
2023-09-13 09:41:46 +00:00
Nir Levy
c748fc494b
DBZ-6866 Fix behaviour when errors.max.retries = 0
2023-09-05 15:03:01 -04:00
Jiri Pechanec
5515fb67f1
DBZ-6853 Use topic admin
2023-08-31 18:06:34 +02:00
Debezium Builder
d97a18e8dc
[maven-release-plugin] prepare for next development iteration
2023-08-29 11:41:08 +00:00
Debezium Builder
5ce25f2cfe
[maven-release-plugin] prepare release v2.4.0.Beta1
2023-08-29 11:41:08 +00:00
“vsantonastaso”
43adc59292
DBZ-6742 use JSON format for JMX Notitication userData
2023-08-22 06:42:51 +02:00
mfvitale
542b3619fa
DBZ-6731 Blocking snapshot takes configuration from signal payload
2023-08-15 12:43:43 +02:00
Debezium Builder
3c72f071b0
[maven-release-plugin] prepare for next development iteration
2023-08-09 10:33:06 +00:00
Debezium Builder
915c465440
[maven-release-plugin] prepare release v2.4.0.Alpha2
2023-08-09 10:33:06 +00:00
Vojtech Juranek
230c056577
DBZ-6715 Refactor polling of the records
2023-08-04 15:00:11 +02:00
Vojtech Juranek
9483b5ac76
DBZ-6715 Speed-up embedded engine test
...
This speeds up embedded engine test by 5 minutes.
2023-08-04 15:00:11 +02:00
Vojtech Juranek
226cd99e07
DBZ-6715 Use better method name
2023-08-04 15:00:11 +02:00
Vojtech Juranek
e4b34e771c
DBZ-6715 Make variables final
2023-08-04 15:00:11 +02:00
Vojtech Juranek
23adf86379
DBZ-6715 Refactor running connector and task and error handling
2023-08-04 15:00:11 +02:00
Vojtech Juranek
fae7b9b18d
DBZ-6715 Refactor initialization of the connector
...
Exiting from `run()` method is now done by throwing new runtime
exception.
2023-08-04 15:00:11 +02:00