We hypothesize that there could be a situation where we may be mining precisely
around the CURRENT_SCN and this may lead to situations where LGWR may not have
flushed all records for the same SCN before being mined by the connector.
Introduces a new configuration option, `log.mining.restart.connection`,
which closes and re-opens the JDBC connection when an Oracle Log switch
occurs or when the optionally configured log mining session max lifetime
is reached.
Currently we scan all the tables, which may result into a substantial
delay in initial snapshot when the database is very large. We need to
filter out tables which we are not interested in.
Add back table filter when loading schema of tables. As per comment of
this block of code, passing all tables and table filter should be faster
than passing only list of tables we are interested in.
DBZ-1973 Add more tests for other connectors
DBZ-1973 Add send method with offset parameter
DBZ-1973 Instantiate NotificationService in the task class
DBZ-1973 Instantiate NotificationService in the task class
DBZ-4027 Move specific sink channel configuration to SinkNotificationChannel
DBZ-4027 Remove not used SPI file
DBZ-1973 Moved SPI file definition to debezium-core
DBZ-1973 Rename KafkaNotificationChannel to more generic SinkNotificationChannel
DBZ-1973 Code refactor
DBZ-1973 Improve configuration property description
DBZ-1973 Improve test
DBZ-1973 Add close method to NotificationChannel
DBZ-1973 Implement KafkaNotificationChannel
DBZ-1973 Add NotificationService and LogNotificationChannel
This change explicitly captures the state of two critical tables when the connector
fails to start a LogMiner session, namely V$LOGMNR_LOGS and V$LOGMNR_PARAMETERS.
While these were previously captured by the LogMinerDatabaseStateWriter, the old
code required DEBUG log level and that isn't always feasible, so this change will
capture these in ERROR log level regardless moving forward.
DBZ-4027 Add an Incremental snapshot test with kafka signaling
DBZ-4027 Add an Incremental snapshot test with kafka signaling
DBZ-4027 Add an Incremental snapshot test with kafka signaling
DBZ-4027 Code style
DBZ-4027 Make SignalPayload more generic and extensible
DBZ-4027 Rename DatabaseSignalChannel to SourceSignalChannel
DBZ-4027 Improve logging
DBZ-4027 Moved SPI file definition to debezium-core
DBZ-4027 Move SignalProcessor synchronization point to be processed only when a signal cdc event arrives.
DBZ-4027 Add EventDispatcher constructor without signalProcessor for spanner connector
DBZ-4027 Fix NPE
DBZ-4027 Fix NPE
DBZ-4027 Formatting
DBZ-4027 Correctly manage signal on not supported connector
DBZ-4027 Use the correct MongoDbOffset
DBZ-4027 Correctly initialize offset for Oracle and SqlServer connectors
DBZ-4027 Register SPI implementations
DBZ-4027 Improve SignalProcessor instantiation
DBZ-4027 Pass source info in case of SchemaChanges action
DBZ-4027 Manage close event in a synchronous way
DBZ-4027 Correctly init offset context also in case of snapshot mode 'never'
DBZ-4027 Fix MySqlMetricsIT test
DBZ-4027 Move KafkaSignalChannel to core
DBZ-4027 Move KafkaSignalChannel to core
DBZ-4027 Set pass offset context after initial snapshot to SignalProcessor
DBZ-4027 Pass OffsetContext to signal processor
DBZ-4027 Pass CommonConnectorConfig to SignalChannelReader init method
DBZ-4027 Move Incremental snapshot window actions to dedicated package
DBZ-4027 Align SignalsIT test with new code
DBZ-4027 Fix SignalsIT test
DBZ-4027 Fix SignalProcessor scheduling
DBZ-4027 Moved DatabaseSignalChannel and SignalChannelReader to dedicated package
DBZ-4027 Start SignalProcessor from ChangeEventSourceCoordinator
DBZ-4027 Create SignalProcessor and renamed Signal to DatabaseSignalChannel
DBZ-4027 Initial refactoring of signal feature
There is the potential when using multiple connector deployments on Oracle that
there could be some level of lock contention with all connectors using the same
table. This allows users to configure the flush table name themselves, reducing
the lock contention across multiple connector deployments.
The default Debezium Oracle images are pre-configured with archive log mode
enabled and this isn't something we can simply turn off and back on in the
test suite proper. This test must be invoked manually and separately, so it
is disabled by default in the overall test suite execution.
This avoids a scenario where we want to increment the batch size; however, the
current batch size is equal to the max and therefore triggers a decrement and
the next iteration increments. The new behavior will be that we don't trigger
this ping pong. With this change, we can track specifically when we reach the
max batch size and only log the warning once. If the batch size drops, and it
later increments, the warning will be logged again but this should be expected.
Using current batch size for comparison is wrong in case that when currentScn is topScn as described in DBZ-6155.
In the other case (when topScn is behind currentScn) can eventually lead to this situation.
when topScn would fall behind currentScn more and more as we would compare currentScn - topScn with bigger and bigger number (current batch size).
Comapring with logMiningBatchSizeMin could result in very small window which would mean we will send many small queries instead of several bigger ones during mining.
Therefore reverting back to do the adjustments based on the defaultBatchSize.
The "fetch-state" attribute was deprecated and is no longer a valid option
with Infinispan 14, which causes the tests to fail to execute. Additionally,
the "segmented" attribute must be set to true going forward as the file
store implementation no longer supports non-segmented configurations.
We load all the schemas of the captured tables when the connector
starts. If we process a record from a table which schema is not
available, this means we have some bug in the intial schema loading.
Don't fail in such case, but print a warning about that.
When taking a snapshot, the Oracle connector was converting the TIMESTAMP
WITH TIME ZONE value to GMT and per the documentation, the value should
be emitted in the time zone of the data.
The snapshot emitted value in GMT is temporally accurate, so there is no
data inconsistency, but the emitted format itself was inconsistent when
looking at how the column data was emitted during a snapshot versus in a
streaming event.
DBZ-5648 introduced a regression where transaction start, commit, and rollback
events were only being read from within the scope of the configured PDB that
the connector was capturing changes instead of the entire Oracle database.
This can lead to situations where the offsets may not be advanced as quickly
in a low traffic PDB environment, potentially causing stale offsets.
* Add annotation and jUnit rule for skipping test when it's run against
Apicurio registry.
* Skip `AbstractOracleDatatypesTest#intTypes` tests which fails with
Apicurio. Which is partially tests in `OracleNumberNegativeScaleIT`
Oracle allows the scale to be negatove for `NUMBER` data type.
Conversion of such number to Avro would fail as Avro doesn't allow
negative scales. Provide a converter which converts the number to
zero scale number. For completeness the converter provides also
conversion to other supported types - string and double.
N.B.: if the conversion to Avro fails actually depends on
implementation, e.g. Kafka schema registy allows also negative scales.
Run the tests alwyas in thr same order to make it more easy to debug
failures. If needed, the order can be changed (e.g. to `random`) by
overriding propeperty `runOrder`.
While `ALWAYS` mode allows also snapshot on schema error, it allows
snapshot also on other ocassions. Don't throw exception when offset
is missing and schema is set to `ALWAYS`.
