This channel permits to send Debezium notification to JMX bean
DBZ-6424 Add JMX Signal channel
This channel permits to send signal to Debezium through the JMX operation
DBZ-1973 Add more tests for other connectors
DBZ-1973 Add send method with offset parameter
DBZ-1973 Instantiate NotificationService in the task class
DBZ-1973 Instantiate NotificationService in the task class
DBZ-4027 Move specific sink channel configuration to SinkNotificationChannel
DBZ-4027 Remove not used SPI file
DBZ-1973 Moved SPI file definition to debezium-core
DBZ-1973 Rename KafkaNotificationChannel to more generic SinkNotificationChannel
DBZ-1973 Code refactor
DBZ-1973 Improve configuration property description
DBZ-1973 Improve test
DBZ-1973 Add close method to NotificationChannel
DBZ-1973 Implement KafkaNotificationChannel
DBZ-1973 Add NotificationService and LogNotificationChannel
DBZ-4027 Add an Incremental snapshot test with kafka signaling
DBZ-4027 Add an Incremental snapshot test with kafka signaling
DBZ-4027 Add an Incremental snapshot test with kafka signaling
DBZ-4027 Code style
DBZ-4027 Make SignalPayload more generic and extensible
DBZ-4027 Rename DatabaseSignalChannel to SourceSignalChannel
DBZ-4027 Improve logging
DBZ-4027 Moved SPI file definition to debezium-core
DBZ-4027 Move SignalProcessor synchronization point to be processed only when a signal cdc event arrives.
DBZ-4027 Add EventDispatcher constructor without signalProcessor for spanner connector
DBZ-4027 Fix NPE
DBZ-4027 Fix NPE
DBZ-4027 Formatting
DBZ-4027 Correctly manage signal on not supported connector
DBZ-4027 Use the correct MongoDbOffset
DBZ-4027 Correctly initialize offset for Oracle and SqlServer connectors
DBZ-4027 Register SPI implementations
DBZ-4027 Improve SignalProcessor instantiation
DBZ-4027 Pass source info in case of SchemaChanges action
DBZ-4027 Manage close event in a synchronous way
DBZ-4027 Correctly init offset context also in case of snapshot mode 'never'
DBZ-4027 Fix MySqlMetricsIT test
DBZ-4027 Move KafkaSignalChannel to core
DBZ-4027 Move KafkaSignalChannel to core
DBZ-4027 Set pass offset context after initial snapshot to SignalProcessor
DBZ-4027 Pass OffsetContext to signal processor
DBZ-4027 Pass CommonConnectorConfig to SignalChannelReader init method
DBZ-4027 Move Incremental snapshot window actions to dedicated package
DBZ-4027 Align SignalsIT test with new code
DBZ-4027 Fix SignalsIT test
DBZ-4027 Fix SignalProcessor scheduling
DBZ-4027 Moved DatabaseSignalChannel and SignalChannelReader to dedicated package
DBZ-4027 Start SignalProcessor from ChangeEventSourceCoordinator
DBZ-4027 Create SignalProcessor and renamed Signal to DatabaseSignalChannel
DBZ-4027 Initial refactoring of signal feature
Introduces a new PostgreSQL configuration parameter called `replica.autoset.type`. With this parameter, you can easily specify the Replica Identity value for each table that is captured by the connector
When the last operation before connector stop is tx BEGIN, subsequent
INSERT done after connector stop can have same LSN as BEGIN and thus it
was possible to skip this INSERT when we search for the resume LSN. This
issue was described and fixed in DBZ-5915. However, the same LSN can
have also previous tx COMMIT, so the WAL can look like this:
LSN | operation
------------------
123 | COMMIT
123 | BEGIN
123 | INSERT
124 | COMMIT
When the last stored LSN is 123 for COMMIT operation, we skip also BEGIN
and INSERT operations with LSN 123 and resume from COMMIT with LSN 124.
Adjust condition introduced in DBZ-5915 to consider also the case when
the last processed operation is COMMIT.
If the table contains back slash, which is ANSI SQL escape chracter,
in its name, querying column metadata would fail in some cases (*)
which would result into NPE. Fix table name before the query and
escape escape character if the table name contains it.
Default backslash works for all currently supported databases as it's
ANSI SQL standard,
(*) everything works when we collect column metadata for all tables,
i.e. when there are no excluded tables
Befre we start streamig, we adjust `confirmed_flush_lsn` in Postgres to
the position where we determined that the streaming should start. When
user disables LSN flushing, which is not desired as it's the user
reponsibility to flush LSN in Postgres.
Also adjust related tests, as when LSN flushing is not disabled, flushed
LSN in Postgres may change (as we e.g. determined start of streaming
from offset or from `xlogpos`) and can result in the test failure. When
flushing is enabled, only LSNs after start of streaming can be compared.
Run the tests alwyas in thr same order to make it more easy to debug
failures. If needed, the order can be changed (e.g. to `random`) by
overriding propeperty `runOrder`.
Add `EventDispatcher` constructor which accepts `TransactionMonior`
instance as a parameter and in case of Postgres pass into
`EventDispatcher` custom `PostgresTransactionMonitor` which adjusts
transactions IDs by adding LSN, i.e. Postgres transaction is now of
form `txId:LSN`.
Allow to override Docker maven plugin properties [1] from command line
to be able to change various Docker parameters more easily when starting
the container.
[1] https://dmp.fabric8.io/#combining-property-config
`TOPIC_PREFIX` is now mandatory to all connectors therefore it make
sense to have it in common config. Beside that, it also makes it more
easy to use it in Debezium UI without any workarounds - if the field is
not member of the given connector config, the field has to be
explicitely added into known fields otherwise is invisible for UI.
With this change the user cannot direcly set connector logical name and
thus in the future it can be remove without breaking user config.
If the turn out that the logical name is useful and user should be able
to configure it, dedicated config option can be added.
Originally it was prposed in the Jira to replace it with connector name,
but it turned out that logical name defaults to `database.server.name`
and is heavily used in the tests and JMX, so it would require another
big refactoring. Thus, use topic prefix for now. Once we know further
direction (remove logical name or add new option), do this refactoring.
During incremental snapshot fields `xmin`, `lsn` and `txId` are not
updated and therefore we stream obsolete values from previous records
in `source` struct. Don't include these obsolete value in the `source`
struct during incremental snapshot.
Add two JXM objects:
* `snapshotPaused` - detemines if the incremental snapshot is paused
or not
* `snapshotPausedDurationInSeconds` - overall time when the incremental
snapshot was paused. The time adds up - if the snapshot was paused
e.g. two times, the `snapshotPausedDurationInSeconds` is the sum of
these two paused times.
Postgres supports partitioned tables. Debezium requires tables to have
primary key to be able to snapshot them. Primary key support for
partitioned was added in Postgres 11, see [1]. Add partitioned table
type into supported table tables so that Debezium fetches the schema
and can do the snapshot for partitioned tables.
N.B.: On Postgres < 11 is still possible to define primary key
constraint on partitioned sub-tables. In such cases sub-tables are
snapshotted, but parent table is not.
Fix RecordsSnapshotProducerIT#shouldGenerateSnapshotsForPartitionedTables
test after adding support for partitioned tables. Test assumed that
the parent table is not snapshotted and filters out duplicate records
which results into lossing `LAST` snapshot record. Keepting original
test by filtering parent table.
[1] https://www.postgresql.org/docs/11/release-11.html
Currently we update first `lsn` and then update `lastCommitLsn` in
`PostgresOffsetContext` constructor. However, when `lastCommitLsn`
is for whatever reason `null`, previously updated `lsn` is reset to
`null` as `updateLastCommit()` updates also `lsn`. This can have
unwanted consequences like streaming again records which were already
streamed. To prevent this, update `lsn` in `updateLastCommit()` only
when `lastCommitLsn` is not `null`.
When a table is renamed with the ALTER TABLE statement, the schema
history record not only will reference the current table's unique id
in the "id" metadata, but will also refer to the old table name in
the "previousId" metadata field.