Xianming Zhou
86c1dac16a
DBZ-7517 Remove the unused 'connector' parameter in the createSourceTask method in EmbeddedEngine.java
2024-02-22 09:07:55 +01:00
Jiri Pechanec
df18e00173
DBZ-7535 Use error level for error message
2024-02-22 08:31:58 +01:00
Vojtech Juranek
7edefef14c
DBZ-7535 Ensure at least one task starts and one fails
...
Add also `INFO` log with number of failed tasks.
2024-02-22 08:31:58 +01:00
Chris Cranford
68f31f7662
[ci] Log records found when expecting no records
2024-02-21 09:00:45 -05:00
Chris Cranford
93cf3b06bf
DBZ-7516 Correctly enable table CDC to avoid failures
2024-02-20 10:59:39 +01:00
Vojtech Juranek
71b04b351e
DBZ-7495 Remove low task stop timeout in testsuite
...
This leads to random test failures and moreover it eventually overrides
`task.management.timeout.ms` configured in concrete tests.
2024-02-19 08:45:33 +01:00
Vojtech Juranek
ae53895cd8
DBZ-7495 Define constant for executor shutdown timeout
...
Unify executor shutdown timeout for executor services in the code base.
2024-02-19 08:45:33 +01:00
Vojtech Juranek
6c7abe7317
DBZ-7496 Add info how long it takes to stop the task
...
Mostly to stabilize the testsuite and find the right value for the
default task start/stop timeout. This info may be inaccurate for
mutiple tasks as we eventually call `ConnectorCallback::taskStopped`
callback and this time is added to the next task stop time.
2024-02-19 08:45:33 +01:00
Vojtech Juranek
46fa5e79b9
DBZ-7496 Intorduce configurable async engine timeout in tests
...
The defaul async engine timeout to start and stop is 1 second, but it's
configurable via `debezium.test.engine.waittime` system property.
2024-02-19 08:45:33 +01:00
Vojtech Juranek
ecc4c096ab
DBZ-7496 Refactor run method to keep it short
2024-02-19 08:45:33 +01:00
Vojtech Juranek
a2c249ae33
DBZ-7496 Make sure completion callback is called after connector shutdown
...
Engine is typically run in a different thread when one from which the
`close()` method si called. During the call of `close()` method, we stop
task polling and `run()` method may move to `finally` block, calling
completetion callback before we return from `close()` method and thus
e.g. even before calling stop of the connector.
Make sure engine state is moved to `STOPPED` and completion callback is
called after engine is really stopped and `close()` method has finished.
2024-02-19 08:45:33 +01:00
Vojtech Juranek
2c61cc7293
DBZ-7496 Fix await conditions, add logging
2024-02-19 08:45:33 +01:00
Jiri Pechanec
7cc8459cd5
DBZ-7488 Keep assertion and switch expected result
2024-02-16 12:33:33 +01:00
Chris Cranford
09e1bf1df0
DBZ-7488 Skip re-selection on r
(read) events
2024-02-16 12:33:33 +01:00
Debezium Builder
10e327602c
[maven-release-plugin] prepare for next development iteration
2024-02-13 09:20:04 +00:00
Debezium Builder
0c5b05738c
[maven-release-plugin] prepare release v2.6.0.Alpha2
2024-02-13 09:20:04 +00:00
Vojtech Juranek
7789d995e5
DBZ-7024 Add possibility to specify engine builder factory
...
Also add converting builder factory for async engine into SPI service.
2024-02-12 13:43:21 +01:00
Vojtech Juranek
dbdb052535
DBZ-7024 Add converting builder for async engine
2024-02-12 13:43:21 +01:00
Vojtech Juranek
004ebeff16
DBZ-7024 Move creation of converters into dedicated class
2024-02-12 13:43:21 +01:00
Vojtech Juranek
b74a0eb2c2
DBZ-7024 Move RecordProcessors into separate classes
2024-02-12 13:43:21 +01:00
Vojtech Juranek
eef8ee4cea
DBZ-7024 Move async engine into separate package
2024-02-12 13:43:21 +01:00
Vojtech Juranek
425407331c
DBZ-7024 Add TODO item for improving ConnectorCallback API
2024-02-12 13:43:21 +01:00
Vojtech Juranek
3edc61e443
DBZ-7024 Improve processor instantiation
2024-02-12 13:43:21 +01:00
Vojtech Juranek
cdf5e0255a
DBZ-7024 Improve log level and log messges
2024-02-12 13:43:21 +01:00
Vojtech Juranek
a04dc84b3e
DBZ-7024 Embedde state comparions into State enum methods
...
It's more safe to have the comparions directly in the enum and also make
obvisou that the ordering of enum is important.
2024-02-12 13:43:21 +01:00
Vojtech Juranek
5c21d19815
DBZ-7024 Use enum for record processing order option
2024-02-12 13:43:21 +01:00
Vojtech Juranek
ee1f33fe33
DBZ-7024 Limit size of records processing thread pool
...
If the number of threads is provided as a number, limit it to 16 threads
to avoid possible overhead with context switching on a beefy machines
where the default value using all available cores may result in many
threads, which would be waiting most of the time anyway, as such machine
may run probably many other tasks, not only Debezium.
If the user really wants to use all available cores, it can be specified
using `AVAILABLE-CORES` placeholder.
2024-02-12 13:43:21 +01:00
Vojtech Juranek
e2d2cff7fd
DBZ-7024 Interrupt polling if needed
...
Some polling tasks may be stuck and we need to interrupt polling during
the shutdown not have to wait for TASK_MANAGEMENT_TIMEOUT_MS to timeout.
Also, when we start to interrput polling, we have to remove interruption
of the main thread in the `catch` part. It was a bug anyway as it
interrputed the main thread what we definitelly don't want to happen in
any case.
2024-02-12 13:43:21 +01:00
Vojtech Juranek
40131c0531
DBZ-7024 Increase task management timeout to 2min
...
Increase task management timeout to two minutes and make this option
internal. This timeout will be hopefully sufficient for most of the
deployments. If not, we will increase the timeout it make this option
public.
2024-02-12 13:43:21 +01:00
Vojtech Juranek
fc7381ad91
DBZ-7024 Improve javadoc and comments
2024-02-12 13:43:21 +01:00
Vojtech Juranek
8bb1a122b1
DBZ-7024 Add missing condition
2024-02-12 13:43:21 +01:00
Vojtech Juranek
0f7d3100b4
DBZ-7024 Add option for creating default ChangeConsumer
...
This option effective allowes the user to request serial processing of
the records byt the provided Consumer.
2024-02-12 13:43:21 +01:00
Vojtech Juranek
cc5f7aedd1
DBZ-7024 Don't provide default ChangeConsumer
...
To allow user to use different processors, don't provide the default
ChangeHandler.
2024-02-12 13:43:21 +01:00
Vojtech Juranek
de2e4aba9f
DBZ-7024 Add debug logging when selecting the processor
2024-02-12 13:43:21 +01:00
Vojtech Juranek
001cb2a640
DBZ-7024 Improve javadocs and comments, fix typos
2024-02-12 13:43:21 +01:00
Vojtech Juranek
4689db90c0
DBZ-7024 Switch abstract embedded tests to async engine
2024-02-12 13:43:21 +01:00
Vojtech Juranek
dfdeab7ab8
DBZ-7024 Add method to await engine shutdown
2024-02-12 13:43:21 +01:00
Vojtech Juranek
3ec22951ce
DBZ-7024 Create testing engine and base class for async engine tests
2024-02-12 13:43:21 +01:00
Vojtech Juranek
293b84645d
DBZ-7024 Make default RecordCommitter thread unsafe
...
Default implmentation of `RecordCommitter`, the `SourceRecordCommitter`,
is always created for each task and withing given task is called
sequentially, always in the same thread. There's no need to aquire locks
for each method call.
Make `SourceRecordCommitter` thread unsafe.
2024-02-12 13:43:21 +01:00
Vojtech Juranek
998f00f811
DBZ-7024 Create initial implementation of async embedded engine
...
Implementation is based on the proposed Debezium design document for
asynchronous embedded engine, which is currently still WIP:
https://github.com/debezium/debezium-design-documents/pull/8
2024-02-12 13:43:21 +01:00
Vojtech Juranek
69bbed1fa3
DBZ-7024 Allow to override type into AbstractConnectorTest
2024-02-12 13:43:21 +01:00
Vojtech Juranek
1eec31b3f3
DBZ-7024 Allow to specify multiple records when error should be thrown during processing
2024-02-12 13:43:21 +01:00
Vojtech Juranek
542b0fec7f
DBZ-7024 Intorduce retryable callable
2024-02-12 13:43:21 +01:00
Vojtech Juranek
d7b7768071
DBZ-7024 Add more testing connectors
...
Add connector which runs mutiple tasks and connector whose some of
the tasks fail.
2024-02-12 13:43:21 +01:00
Vojtech Juranek
7eaf0fc288
DBZ-7024 Move reuseable testing functions for DebeziumEngine into common util class
2024-02-12 13:43:21 +01:00
Vojtech Juranek
b8e16ee89f
DBZ-7024 Move reuseable interface implementations for DebeziumEngine into common class
2024-02-12 13:43:21 +01:00
Vojtech Juranek
ba35b395c5
DBZ-7024 Move required setup into EmbeddedWorkerConfig
...
Move required by `WorkerConfig` validators into `EmbeddedWorkerConfig`
so we have all Kafka related stuff in the same class.
2024-02-12 13:43:21 +01:00
Vojtech Juranek
4452e3d095
DBZ-7024 Move EmbeddedConfig into separate class
...
`EmbeddedConfig` needs to be shared with other implmentations of
`DebeziumEngine` as long as Debezium embedded depends on the Kafka
model.
2024-02-12 13:43:21 +01:00
Chris Cranford
a597a82c19
DBZ-7439 Remove unnecessary log/output entries
2024-02-07 15:00:27 +01:00
Chris Cranford
de9364bb4d
DBZ-7439 Fix test compatibility
2024-02-07 15:00:27 +01:00