ggaborg
664fd9e1b0
DBZ-3293 unit tests for filters
2022-09-01 20:16:47 +02:00
ggaborg
244de2bd96
DBZ-3293 Add signal table automatically to include list
2022-09-01 20:16:47 +02:00
thangdc94
57210095d2
DBZ-5439 Fix testcase
2022-09-01 09:32:17 +02:00
Jiri Pechanec
de53591dba
DBZ-5244 Retry all communication exceptions by default
2022-08-31 09:45:09 -04:00
harveyyue
7ccd40129c
DBZ-5569 Oracle connector's max.queue.size.in.bytes is invalid
2022-08-31 09:39:29 -04:00
Jiri Pechanec
1c291d75fd
DBZ-5568 Use only Java 11 for tests
2022-08-31 09:19:23 +02:00
pkgonan
fd309193c6
DBZ-5544 Add document
2022-08-24 09:50:23 +02:00
pkgonan
b031770300
DBZ-5544 Support BASE64_URL_SAFE in BinaryHandlingMode
2022-08-24 09:50:23 +02:00
tjwornjs
ca009d8492
DBZ-5489 add column comment to schema
2022-08-24 09:46:01 +02:00
Anisha Mohanty
669024ed46
DBZ-5484 Better error reporting in SKIP/WARN cases
2022-08-23 10:02:29 -04:00
harveyyue
0308a1cbb6
DBZ-5535 Disable preferring DDL before logical schema in history recovery
2022-08-23 14:16:35 +02:00
Jiri Pechanec
34b78e6bce
DBZ-2766 Make queue more generic
2022-08-23 13:43:41 +02:00
Jiri Pechanec
f770e0afb1
DBZ-2766 Remove Guava dependency
2022-08-23 13:43:41 +02:00
ggaborg
bf20a9c70f
DBZ-5533 Unit test fails on Windows
2022-08-19 09:44:26 -04:00
Sergei Morozov
1c99437890
DBZ-5470: Remove the dependency of JdbcConnection on DatabaseSchema
2022-08-19 11:06:59 +02:00
Chris Cranford
37b97970a6
DBZ-5497 Skip truncate operations by default
...
* Promote truncate skip default from PostgreSQL to core library
* Adjust behavior for Oracle with truncates skipped by default
2022-08-19 11:05:29 +02:00
Vivek Wassan
d89e71ebeb
DBZ-5327 Smart Backfills
2022-08-17 10:05:55 +02:00
harveyyue
a08420b21c
DBZ-5511 Keep consistence with method serialize of Connect/Json Table Change Serializer
2022-08-16 16:58:07 +02:00
Vojtech Juranek
8e0ad245e5
DBZ-5475 Ensure Outbox JSON NUMBER arrays are of the same type
2022-08-05 11:00:46 -04:00
Vojtech Juranek
7256f62f6b
DBZ-5475 Remove unneeded checks from Outbox payload conversion
...
* remove float check as we return `float64` by default anyway
* simplify/narrow the checks when detemining array type, also add a test
for array having the first element `null`
2022-08-05 11:00:46 -04:00
Chris Cranford
d4261bd2fd
DBZ-5476 Fix record accumulator during statistics log entry
2022-08-04 15:28:46 -04:00
harveyyue
2481172309
DBZ-5471 Missing the regex properties validation before start connector of DefaultRegexTopicNamingStrategy
2022-08-04 10:49:29 -04:00
Vojtech Juranek
c7d8353d7b
DBZ-4727 Add snapshot paused JXM metrics
...
Add two JXM objects:
* `snapshotPaused` - detemines if the incremental snapshot is paused
or not
* `snapshotPausedDurationInSeconds` - overall time when the incremental
snapshot was paused. The time adds up - if the snapshot was paused
e.g. two times, the `snapshotPausedDurationInSeconds` is the sum of
these two paused times.
2022-07-29 12:41:58 -04:00
Vojtech Juranek
ed504173e1
DBZ-4727 Implement incremental snaphost pause/resume
...
Intorduce two new signals:
* `pause-snapshot` - pauses running incremental snapshot
* `resume-snapshot` - resumes paused incremental snapshot
If the incremental snapshot is running and pause signal is sent,
currently processed chunk is finished and then no other chunk is read.
Snapshot state is kept in running state. Once snapshot is resume,
chunk is reverted to position which was last sent to the broker and
process of reading chunks and emitting `read` events continues until
all tables are read and sent.
This patch handles only table-based signals. MySQL Kafka-based signals
will be addressed later.
2022-07-29 12:41:58 -04:00
harveyyue
5f430f1a8e
DBZ-5457 Deprecate legacy topic selector for all connectors
2022-07-28 12:58:59 -04:00
ggaborg
2c8d71a36f
DBZ-5452 Fix unit test failing on Windows platform
2022-07-26 15:50:28 -04:00
Debezium Builder
1ab51ae8d0
[maven-release-plugin] prepare for next development iteration
2022-07-26 18:13:10 +00:00
Debezium Builder
d247a48b79
[maven-release-plugin] prepare release v2.0.0.Beta1
2022-07-26 18:13:09 +00:00
harveyyue
7730610320
DBZ-5425 Remove the duplicated SimpleDdlParserListener from mysql connector
2022-07-20 14:35:34 +02:00
Chris Cranford
75ad893d67
DBZ-5396 Scope attributes inside table document
2022-07-19 15:41:02 -04:00
Chris Cranford
76dafedea7
DBZ-5396 Add multi-type accessors for attribute values
2022-07-19 15:41:02 -04:00
Chris Cranford
f412c2e6a4
DBZ-5396 Add support for connector-specific relational model attributes
2022-07-19 15:41:02 -04:00
harveyyue
daf329bc06
DBZ-5413 Update the topic naming strategy doc to all connectors
2022-07-18 10:18:12 -04:00
harveyyue
77c7db163d
DBZ-5414 Validate topic naming strategy relative topic name properties
2022-07-18 09:58:51 -04:00
Chris Cranford
3acf317668
DBZ-5404 Restore build compatibility with Kafka 2.x and 3.0.x
2022-07-14 08:40:42 +02:00
harveyyue
6ea7abc8fc
DBZ-4180 Pluggable topic selector
2022-07-13 14:45:25 +02:00
Jiri Pechanec
d8baac7c46
DBZ-5229 Reorganize code for inheritance use
2022-07-12 17:31:19 +02:00
Hossein Torabi
1ea02be9c2
DBZ-5229 define centralized and modular aproach for debezium storage
2022-07-12 17:31:19 +02:00
Chris Cranford
d2a76a7ffc
DBZ-5295 Reselect LOB columns on primary key update
2022-07-12 11:59:34 +02:00
Chris Cranford
7b08fb8d3d
DBZ-5356 Rewrite loop
2022-07-12 11:01:10 +02:00
Chris Cranford
deaa34d71e
DBZ-5356 Avoid NPE during index evaluation
2022-07-12 11:01:10 +02:00
harveyyue
0842e5cd0e
DBZ-5366 MySQL Connector column hash v2 does not work
2022-07-06 14:23:57 -04:00
Paweł Malon
707a43d3bc
DBZ-5367 Fixed the problem where JSON expanding failed for Outbox event payloads that contained nested arrays where first array was empty and further array contained a Struct
...
With proposed changes empty arrays are skipped, as it allows us to delay defining the schema for them until an array with elements is found, as it seems that there is no way to modify a schema that was already defined for a given field
2022-07-06 13:47:54 -04:00
Debezium Builder
6fd1441af3
[maven-release-plugin] prepare for next development iteration
2022-07-01 09:21:49 +00:00
Debezium Builder
99402af634
[maven-release-plugin] prepare release v2.0.0.Alpha3
2022-07-01 09:21:48 +00:00
Vojtech Juranek
ac06623459
DBZ-5240 Move table types into separate method
...
This will allow connectors to override the default values and return
types the connector supports.
2022-06-30 10:46:54 +02:00
Gunnar Morling
cb53332561
DBZ-5335 Formatting
2022-06-30 09:54:13 +02:00
Mickael Maison
5ff71aa992
DBZ-5335 Fix formatting issues in the connectors configdefs
2022-06-30 09:35:54 +02:00
harveyyue
20dc353262
DBZ-5170 Mysql Commit Timestamp
2022-06-27 15:50:52 +02:00
Naveen Kumar KR
b75c9e635b
DBZ-5277 ChangeEventQueue - Support Multiple Writer (doEnqueue) and Reader (poll) Threads
...
* ReentrantLock based write and read coordination
* Condition based Inter Thread communication for isFull and isNotFull conditions
2022-06-27 15:37:00 +02:00
Naveen Kumar KR
823a49fff4
DBZ-5277 ChangeEventQueue Test with Multiple Writer and Reader Threads
...
Assertion would fail if one or both of the scenario occurs
1) Deadlock between Writer and Reader Threads
2) Thread Starvation between Writer and Reader Threads
2022-06-27 15:37:00 +02:00
Jiri Pechanec
9ada2f28ce
DBZ-4902 Change for exponential strategy
2022-06-27 14:29:51 +02:00
kanha gupta
1b17372d2d
DBZ-4902 Replace long durations with Duration
2022-06-27 14:29:51 +02:00
Chris Cranford
ef99595d91
DBZ-5270 Reduce table schema capture logging verbosity
...
When enabling the `database.history.store.only.captured.tables.ddl`,
the logging in RelationalSnapshotChangeEventSource does not take into
account this setting when logging, leading to a false impression of
what tables will actually be captured in the schema history.
2022-06-27 08:47:34 +02:00
Jiri Pechanec
5e1ae24e92
DBZ-5272 Notify all threads in case of mutliple writers
2022-06-21 07:08:39 +02:00
Chris Cranford
3fb5bf1339
DBZ-4451 Introduce SchemaChangeEvent factory methods
2022-06-16 09:19:41 +02:00
Chris Cranford
863abcd081
DBZ-4451 Store previous table id in alter schema change events
...
When a table is renamed with the ALTER TABLE statement, the schema
history record not only will reference the current table's unique id
in the "id" metadata, but will also refer to the old table name in
the "previousId" metadata field.
2022-06-16 09:19:41 +02:00
Chris Cranford
20bb2adea5
DBZ-4834 Fix incremental snapshot of table added to include list
2022-06-16 08:39:16 +02:00
Vojtech Juranek
25eba9749d
DBZ-5198 Add support for SQL server specific table delimiters
...
Add suppoprt for `TableId` delimiters and provide implementation for
SQL server. SQL server allows to use reserved words in table names or
names with spaces, but they have to wrapped by `[]` characters, e.g.
`[dbname].[table name]`.
Debezium can handle spaces e.g. in table include list, but fails when
parsing snapshot select, therefore the parsing with predicates is used
only for parsing spanshot select for now. If useful, it can be used
later on on other places as well.
2022-06-13 10:57:15 +02:00
Vojtech Juranek
796e17b2c6
DBZ-5198 Move predicates for parsing TableId into dedicate class
...
Moving the predicates into parsing context decouples the predicates from
the private `TableIdTokenizer` class. Moving the predicates into
separate class would allow us to provide database specific
implementations.
2022-06-13 10:57:15 +02:00
Debezium Builder
6fff3a2876
[maven-release-plugin] prepare for next development iteration
2022-06-09 11:49:26 +00:00
Debezium Builder
b2e2e351df
[maven-release-plugin] prepare release v2.0.0.Alpha2
2022-06-09 11:49:26 +00:00
Vojtech Juranek
4a48eb33df
DBZ-5045 Remove legacy V1 source info
2022-06-08 09:11:00 +02:00
Vojtech Juranek
250e160b81
DBZ-5045 Removed unused methods
2022-06-08 09:11:00 +02:00
Vojtech Juranek
ef8d7f7af7
DBZ-5045 Remove deprecated OBSOLETE_NAME_FOR_SKIP_FAILURE_HANDLING option value
2022-06-08 09:11:00 +02:00
Vojtech Juranek
ff5ddd7959
DBZ-5045 Remove deprecated FIELD_PAYLOAD_ID
...
`table.field.event.payload.id` is replaced by `table.field.event.key`.
Also fix default value of `table.field.event.key` which was missing.
2022-06-08 09:11:00 +02:00
Vojtech Juranek
0530dc2de9
DBZ-5045 Remove deprecated STORE_ONLY_MONITORED_TABLES_DDL
2022-06-08 09:11:00 +02:00
Vojtech Juranek
4e7b30b718
DBZ-5045 Remove deprecated DEBEZIUM_OPERATION_INVALID_BEHAVIOR
2022-06-08 09:11:00 +02:00
Vojtech Juranek
b0b3066ed5
DBZ-5045 Remove deprecated COLUMN_BLACKLIST
2022-06-08 09:11:00 +02:00
Vojtech Juranek
5031594b75
DBZ-5045 Remove deprecated COLUMN_WHITELIST
2022-06-08 09:11:00 +02:00
Vojtech Juranek
b3a6131cee
DBZ-5045 Remove deprecated DATABASE_BLACKLIST
2022-06-08 09:11:00 +02:00
Vojtech Juranek
0cbb57f4bf
DBZ-5045 Remove deprecated DATABASE_WHITELIST
2022-06-08 09:11:00 +02:00
Vojtech Juranek
f84893a8b4
DBZ-5045 Remove deprecated SCHEMA_BLACKLIST
2022-06-08 09:11:00 +02:00
Vojtech Juranek
261188482f
DBZ-5045 Remove deprecated SCHEMA_WHITELIST
2022-06-08 09:11:00 +02:00
Vojtech Juranek
6abbcee6a3
DBZ-5045 Remove deprecated TABLE_BLACKLIST
2022-06-08 09:11:00 +02:00
Vojtech Juranek
db08e894fc
DBZ-5045 Remove deprecated TABLE_WHITELIST
2022-06-08 09:11:00 +02:00
harveyyue
acb743bd34
DBZ-5206 Remove the deprecated monitored tables metrics
2022-06-07 16:13:29 -04:00
Chris Cranford
3e820c4047
DBZ-4251 Adjust semantics with collection removal
2022-06-02 07:30:07 +02:00
Chris Cranford
187e3250a3
DBZ-4251 Add regex support for stop signal
2022-06-02 07:30:07 +02:00
Chris Cranford
9d71296b5c
DBZ-4251 Support stopping incremental snapshots
2022-06-02 07:30:07 +02:00
Jiri Pechanec
88b7ae2050
DBZ-5136 Move sourceInfo to common class
2022-06-01 16:56:29 +02:00
Mark Bereznitsky
c64b8af31a
DBZ-5136 reintroduce the CommonOffsetContext class and refactor common functionality into it
2022-06-01 16:56:29 +02:00
Mark Bereznitsky
2d72dae649
DBZ-5136 further refining of the algorithm for determining position in snapshot and IT test fixes
2022-06-01 16:56:29 +02:00
Jiri Pechanec
08e8c3d795
DBZ-5136 Move common class to common package
2022-06-01 16:56:29 +02:00
Andrew Walker
d0759461a4
DBZ-5136: Convert RelationalOffsetContext to CommonOffsetContext
2022-06-01 16:56:29 +02:00
Andrew Walker
01a81f8bc1
DBZ-5136: Refactored SnapshotContext setters
2022-06-01 16:56:29 +02:00
Andrew Walker
b6222d42bf
DBZ-5136: Add first flag and refactor OffsetContext
2022-06-01 16:56:29 +02:00
Andrew Walker
593c2769e6
DBZ-5136: Add last in table flag
2022-06-01 16:56:29 +02:00
Chris Cranford
15e9231c30
DBZ-5179 Use LRUCacheMap
to avoid high memory usage
2022-06-01 15:53:55 +02:00
Chris Cranford
9fa5a48274
DBZ-5011 Register MongoDB metrics with task id when more than one task
2022-06-01 09:41:36 +02:00
harveyyue
9a3afac243
DBZ-5180 Put the MysqlPartition values to MDC getting NPE in logging context phase
2022-06-01 09:17:04 +02:00
Vojtech Juranek
5550057034
DBZ-5042 Add abstract partition with DB name
...
DB names is used by logging facilities. As the DB name is required only
for relational DBs, also this abstract class is provided only for
relational DB connectors.
Use this abstract class for SQL server connector, where DB name logging
is already implemented.
2022-05-27 14:20:30 -04:00
Jiri Pechanec
b85cb275b1
DBZ-5089 Use atomic reference
2022-05-27 12:46:39 +02:00
connorszczepaniak-wk
8693b4f008
DBZ-5089: add partition to the outbox event router in additional fields
...
DBZ-5089: undo format change
DBZ-5089: add tests for the partition routing
DBZ-5089: remove nonsensical comment
DBZ-5089: partition fieldname is a string; test properly
DBZ-5089: fix one of the failing tests
DBZ-5089: pass args through
DBZ-5089: fix oracle implementation; add configs for mongo connector
DBZ-5089: put partition in additional placement
DBZ-5089: restore formatting
2022-05-27 12:46:39 +02:00
harveyyue
e45e8bd43b
DBZ-5172 Expose more useful exception info with building the field default value schema
2022-05-27 12:41:12 +02:00
Rotem Adhoh
8ef7224543
DBZ-5166 Fixed ExtractNewRecordState SMT to pass all columns' data as is, i.e: without potential unwanted values manipulation.
...
Prior to this fix, when inserting an explicit NULL value to a nullable column which has a default value, resulted in a record which contains the corresponding field with the default value (instead of NULL) after applying ExtractNewRecordState transformation.
A new unit test was added as well.
2022-05-27 12:33:23 +02:00
harveyyue
3ada04b522
DBZ-3865 Debezium MySql connector does not capture floating point numbers with the right precision
2022-05-25 12:54:42 -04:00
Chris Cranford
7d8c9486d6
DBZ-5119 Move Heartbeat creation to ConnectorConfig & descendants
2022-05-25 09:59:59 +02:00
Chris Cranford
7ff7826743
DBZ-5119 Pull heartbeat.action.query
support into debezium-core
2022-05-24 12:54:40 +02:00
Chris Lee
2a8456d21b
DBZ-5123 data-collection regex snapshot signal
2022-05-24 12:46:28 +02:00