There is a corner case where it's possible the Oracle connector may query
the Oracle metadata tables quicker than the ARC process can generate an
archive log history record in V$ARCHIVED_LOG, and this can lead to a race
condition where we may incorrectly advance the connector forward to start
mining a group of logs when a log sequence gap exists in the log ranges.
For users who use the online_catalog strategy, there are some checks that
LogMiner does automatically which it skips, and one is with log sequence
gaps. This fix enforces that check by Debezium even for users who may use
the faster online_catalog mode so that no logs are omitted and events
could be missed.
There was a possible situation where if a long transaction consisted of
updating and inserting into the same table with identical keys with a
given sequence that the commit handler would merge several events for a
table without LOB columns, resulting in a difference in expected events
in the Kafka topic vs what was seen in LogMiner.
Per https://jira.mariadb.org/browse/CONJ-977, the MariaDB team does not
believe that the Integer.MIN_VALUE solution for triggering streaming is
spec compliant and as of MariaDB 3.x+ drivers, they explicitly will not
allow this.
This change explicitly sets the fetchSize to 1 to mimic the same style
of behavior that the default fetchSize uses when using MySQL. If a user
provides a custom fetchSize, that will take precedences.
Postgres `RecordsStreamProducerIT` reliaes on
EmebeddedEngine.runWithTask(). As this method effectively expose
engine's internal task and tests do the asserts against the state
of the task, it's hard to replace it. If we want to keep the tests,
the most simple approach seems to expose engine task in similar way
how EmbeddedEngine does that.
Add interface for testing Debezium engine, which would define minimal
set of methods which needs to be exposed by the implementing classes to
be able to run the testsuite against the Debezium engine. The number of
such methods should be as low as possible. Implementing classes would
typically act as proxies to actual `DebeziumEngine` implementations.
Add `TestingDebeziumEngine` implementation for `EmbeddedEngine` and
switch to `TestingDebeziumEngine` in `AbstractConnectorTest`.
This config will be re-used by possible other implementations of
DebeiumEngine API in the embedded package. As DebeziumEngine API
can have completely different implementations and thus also config,
the class is called `EmbeddedEngineConfig` as it's assumed to be used
only by embedded engine "family" of implementations.
To keep backward compatibility, the config options are extracted into
an interface and `EmbeddedEngine` implements this interface, thus
allowing to use these options in custom classes without any need for the
code changes.
In order to stabilize testsuite on Docker and Kuberetes following changes were made:
- Remove port forwarding from testsuite
- Increase poll timeout
- Separate Mongo test classes based on deployment and streaming mode
- Allow both Docker and Kubernetes Mongo deployments use connection string
- Create container for testsuite
- Create kubernetes deployment for the testsuite
