The cache mechanism had to be adapted in order to support non-versioned
and versioned schemas, a test now confirms it's the same valueSchema
instance created once.
To allow more flexibility we should proxy the new Event Schema fields
based on the original table columns which Debezium is capturing.
The Schema is now built one time during the first Record in order to
detect those types which are only available within the Record.
It's a good practice to depend on the Kafka metadata instead of custom
dates in the payload, this way for instance when using KStreams with
Tumbling Window the dates are correctly matched.
This commit does a few things:
- Refactors snapshot modes to be encapsulated by an interface and
to only use that interface in determining when to snapshot and in
determing the type of the `RecordProducer` interface to instantiate
- Refactors the configuration of existing snapshot modes to tie the
existing snapshot modes to their aligned implementation
- Adds a new snapshot.mode, custom, and a new configuration option to
specify a custom implementation that will be loaded by the class loader
- Changes the visibility of some classes to allow for custom snapshot
modes to get enough context to make an informed choice
- Adds some metadata about slots (the catalog_xmin) to give a full idea
of the state of slots which can be useful in implementing snapshot
modes (which is also configurable, as it can add some overhead)
Together, these changes allow for a much broader flexibility got end
users to implement a snapshot mode that can do more advanced snapshots,
such as partial recovery or for partial snapshots for tables where not
all records are needed.
This could also be seen as superseeding the
`snapshot.select.statement.overrides` to allow for users to dynamically
build queries based on the state of the slot and the offsets consumed.
DBZ-1181 Adding support for processing WAL2JSON events that have no changes;
This allows to produce heartbeats when _only_ receiving such events for a longer period of time, e.g. updates of materialized views.
The previous statement was never achievable since `ReplicaSets.parse(hosts)` can never return null
Also removed some code duplication for better readability
- wal2json sends the txn commitTime using a function from PostgreSQL's C
library. The value that Debezium recevies is in nanoseconds.
- decoderbufs sends the txn commitTime in microseconds.
- RecordsSnapshotProducer updates SourceInfo.ts_usec by converting
System.currentTimeMillis() to microseconds.
- RecordsSnapshotProducer updates the SourceInfo's ts_usec field using
message.getCommitTime().
This means that when using wal2json, the value of SourceInfo.ts_usec
is in microseconds since epoch during snapshot but is in nanoseconds
during streaming. To fix this, we changed
Wal2JsonReplicationMessage.getCommitTime() to return in microseconds.
