If the number of threads is provided as a number, limit it to 16 threads
to avoid possible overhead with context switching on a beefy machines
where the default value using all available cores may result in many
threads, which would be waiting most of the time anyway, as such machine
may run probably many other tasks, not only Debezium.
If the user really wants to use all available cores, it can be specified
using `AVAILABLE-CORES` placeholder.
Some polling tasks may be stuck and we need to interrupt polling during
the shutdown not have to wait for TASK_MANAGEMENT_TIMEOUT_MS to timeout.
Also, when we start to interrput polling, we have to remove interruption
of the main thread in the `catch` part. It was a bug anyway as it
interrputed the main thread what we definitelly don't want to happen in
any case.
Increase task management timeout to two minutes and make this option
internal. This timeout will be hopefully sufficient for most of the
deployments. If not, we will increase the timeout it make this option
public.
Default implmentation of `RecordCommitter`, the `SourceRecordCommitter`,
is always created for each task and withing given task is called
sequentially, always in the same thread. There's no need to aquire locks
for each method call.
Make `SourceRecordCommitter` thread unsafe.
Based on feedback in https://github.com/debezium/debezium-server/pull/33 this
commit adds the partition() method to the ChangeEvent interface and implements
it in the EmbeddedEngineChangeEvent. This allows reading the assigned partition
for an event in downstream processors, for example in custom Sinks that need
the assigned partition for routing purposes.
Link: https://issues.redhat.com/browse/DBZ-6723
In 7b4cf1901 deprecated `io.debezium.embedded.spi.OffsetCommitPolicy`,
which provided also constructor for `Configuration`, was removed.
This constructor was actually used for creating `OffsetCommitPolicy`.
`Configuration` provides default values for options which are not
explicitly set, while `Properties` based constructor cannot do that
and therefore with this switch the default value for
`offset.flush.interval.ms` is now missing.
As debezium-api package has no knowledge about `Configuration`
interface, which is part of debezium-core, and thus about the default
values, specify default value direcly in the `PeriodicCommitOffsetPolicy`
class.
Postgres `RecordsStreamProducerIT` reliaes on
EmebeddedEngine.runWithTask(). As this method effectively expose
engine's internal task and tests do the asserts against the state
of the task, it's hard to replace it. If we want to keep the tests,
the most simple approach seems to expose engine task in similar way
how EmbeddedEngine does that.
Add interface for testing Debezium engine, which would define minimal
set of methods which needs to be exposed by the implementing classes to
be able to run the testsuite against the Debezium engine. The number of
such methods should be as low as possible. Implementing classes would
typically act as proxies to actual `DebeziumEngine` implementations.
Add `TestingDebeziumEngine` implementation for `EmbeddedEngine` and
switch to `TestingDebeziumEngine` in `AbstractConnectorTest`.
This config will be re-used by possible other implementations of
DebeiumEngine API in the embedded package. As DebeziumEngine API
can have completely different implementations and thus also config,
the class is called `EmbeddedEngineConfig` as it's assumed to be used
only by embedded engine "family" of implementations.
To keep backward compatibility, the config options are extracted into
an interface and `EmbeddedEngine` implements this interface, thus
allowing to use these options in custom classes without any need for the
code changes.
This channel permits to send Debezium notification to JMX bean
DBZ-6424 Add JMX Signal channel
This channel permits to send signal to Debezium through the JMX operation
DBZ-1973 Add more tests for other connectors
DBZ-1973 Add send method with offset parameter
DBZ-1973 Instantiate NotificationService in the task class
DBZ-1973 Instantiate NotificationService in the task class
DBZ-4027 Move specific sink channel configuration to SinkNotificationChannel
DBZ-4027 Remove not used SPI file
DBZ-1973 Moved SPI file definition to debezium-core
DBZ-1973 Rename KafkaNotificationChannel to more generic SinkNotificationChannel
DBZ-1973 Code refactor
DBZ-1973 Improve configuration property description
DBZ-1973 Improve test
DBZ-1973 Add close method to NotificationChannel
DBZ-1973 Implement KafkaNotificationChannel
DBZ-1973 Add NotificationService and LogNotificationChannel
DBZ-4027 Add an Incremental snapshot test with kafka signaling
DBZ-4027 Add an Incremental snapshot test with kafka signaling
DBZ-4027 Add an Incremental snapshot test with kafka signaling
DBZ-4027 Code style
DBZ-4027 Make SignalPayload more generic and extensible
DBZ-4027 Rename DatabaseSignalChannel to SourceSignalChannel
DBZ-4027 Improve logging
DBZ-4027 Moved SPI file definition to debezium-core
DBZ-4027 Move SignalProcessor synchronization point to be processed only when a signal cdc event arrives.
DBZ-4027 Add EventDispatcher constructor without signalProcessor for spanner connector
DBZ-4027 Fix NPE
DBZ-4027 Fix NPE
DBZ-4027 Formatting
DBZ-4027 Correctly manage signal on not supported connector
DBZ-4027 Use the correct MongoDbOffset
DBZ-4027 Correctly initialize offset for Oracle and SqlServer connectors
DBZ-4027 Register SPI implementations
DBZ-4027 Improve SignalProcessor instantiation
DBZ-4027 Pass source info in case of SchemaChanges action
DBZ-4027 Manage close event in a synchronous way
DBZ-4027 Correctly init offset context also in case of snapshot mode 'never'
DBZ-4027 Fix MySqlMetricsIT test
DBZ-4027 Move KafkaSignalChannel to core
DBZ-4027 Move KafkaSignalChannel to core
DBZ-4027 Set pass offset context after initial snapshot to SignalProcessor
DBZ-4027 Pass OffsetContext to signal processor
DBZ-4027 Pass CommonConnectorConfig to SignalChannelReader init method
DBZ-4027 Move Incremental snapshot window actions to dedicated package
DBZ-4027 Align SignalsIT test with new code
DBZ-4027 Fix SignalsIT test
DBZ-4027 Fix SignalProcessor scheduling
DBZ-4027 Moved DatabaseSignalChannel and SignalChannelReader to dedicated package
DBZ-4027 Start SignalProcessor from ChangeEventSourceCoordinator
DBZ-4027 Create SignalProcessor and renamed Signal to DatabaseSignalChannel
DBZ-4027 Initial refactoring of signal feature
