alexey.rassokhin/tet123
1
0
Fork 0
Code Issues Pull Requests 1 Packages Projects Releases Wiki Activity
9,499 Commits 3 Branches 0 Tags 51 MiB
8265e4e42e
Commit Graph

1910 Commits

Author SHA1 Message Date
harveyyue
3b4d19c174 DBZ-5639 Logging enhancement for non-incremental snapshot in postgres connector 2022-09-22 13:14:28 +02:00
Ismail Simsek
b966f4b65b DBZ-5591 Snapshot process setting source.ts_ms 2022-09-20 17:12:54 -04:00
Jiri Pechanec
29fcb0e047 DBZ-5585 Add last resort classloader 2022-09-19 17:15:29 +02:00
harveyyue
288c93434f DBZ-5585 Remove classloader parameter from JdbcConnection 2022-09-19 17:15:29 +02:00
harveyyue
20d56734d4 DBZ-5585 Modify the Instantiator to not require classloader 2022-09-19 17:15:29 +02:00
Vojtech Juranek
2a0e2a7e81 DBZ-5603 Move test containers to quay.io 2022-09-19 09:12:46 +02:00
Debezium Builder
218d414552 [maven-release-plugin] prepare for next development iteration 2022-09-16 06:36:40 +00:00
Debezium Builder
567f57c47b [maven-release-plugin] prepare release v2.0.0.Beta2 2022-09-16 06:36:39 +00:00
harveyyue
09b99840d8 DBZ-4617 Deprecate internal key/value converter options 2022-09-14 12:49:30 +02:00
Vojtech Juranek
8b8d4828f3 DBZ-5043 Make topic prefix field manadatory and with proper group 2022-09-14 12:44:38 +02:00
harveyyue
665edab844 DBZ-5589 Mysql connector can't handle the case sensitive of rename/change column statement 2022-09-13 09:36:25 +02:00
Vojtech Juranek
acc6be6df3 DBZ-5043 Move TOPIC_PREFIX into CommonConnectorConfig 
`TOPIC_PREFIX` is now mandatory to all connectors therefore it make
sense to have it in common config. Beside that, it also makes it more
easy to use it in Debezium UI without any workarounds - if the field is
not member of the given connector config, the field has to be
explicitely added into known fields otherwise is invisible for UI.
 2022-09-13 09:25:54 +02:00
Anisha Mohanty
f41b6dd458 DBZ-5044 Get rid of schema factory instances 2022-09-08 10:35:20 +02:00
Anisha Mohanty
b0dc7e557a DBZ-5044 Update schemaHistoryColumnSchema to add new fields 2022-09-08 10:35:20 +02:00
Anisha Mohanty
ec792527e9 DBZ-5044 Use SchemaFactory for centralizing Postgres schemas 2022-09-08 10:35:20 +02:00
Anisha Mohanty
cfd0242e28 DBZ-5044 Sort imports & fix test failures 2022-09-08 10:35:20 +02:00
Anisha Mohanty
91ccb7578f DBZ-5044 Centralize datatype schemas 2022-09-08 10:35:20 +02:00
Jiri Pechanec
77efd818da DBZ-5044 More descriptive names for schema history schemas 2022-09-08 10:35:20 +02:00
Jiri Pechanec
b8bcac1aef DBZ-5044 Ignore versions for schema comparison 2022-09-08 10:35:20 +02:00
Jiri Pechanec
fa2a7ba60c DBZ-5044 Fix transaction metadata schema 2022-09-08 10:35:20 +02:00
Anisha Mohanty
aea7c1ffa6 DBZ-5044 Use SchemaFactory for ConnectTable & EvenyDispatcher schemas 2022-09-08 10:35:20 +02:00
Anisha Mohanty
8fb2ffd70a DBZ-5044 Add review suggestions 2022-09-08 10:35:20 +02:00
Jiri Pechanec
cd07b8a06d DBZ-5044 Rename and reorder schema attributes 2022-09-08 10:35:20 +02:00
Anisha Mohanty
1a55869f5b DBZ-5044 Reword schema names 2022-09-08 10:35:20 +02:00
Anisha Mohanty
867ff0a117 DBZ-5044 Use seperate version for each schema 2022-09-08 10:35:20 +02:00
Anisha Mohanty
ec0ac3fea5 DBZ-5044 Centralize creation of Message Schemas 2022-09-08 10:35:20 +02:00
Vojtech Juranek
402df78381 DBZ-5043 Rename schema.history to schema.history.internal 2022-09-07 12:57:31 +02:00
Vojtech Juranek
b6c31f3da5 DBZ-5043 Replace 'database history' with 'database schema history' 
Reflect renaming database history to schema history in comments,
configuration descriptions and log and error messages.
 2022-09-07 12:57:31 +02:00
Vojtech Juranek
5bffd7839a DBZ-5043 Change default schema history suffix 
Change default schema history name from `$LOGICAL_NAME-dbhistory` to
`$LOGICAL_NAME-schemahistory`.
 2022-09-07 12:57:31 +02:00
Vojtech Juranek
0494a16a79 DBZ-5043 Rename DatabaseHistory to SchemaHistory 
Rename also all related classes and varibales.
 2022-09-07 12:57:31 +02:00
Vojtech Juranek
0823cedf25 DBZ-5043 Replace database.server.name with topic.prefix 2022-09-07 12:57:31 +02:00
Vojtech Juranek
a7a293c5c8 DBZ-5043 Move driver pass-through parameters into dedicated namespace 
Currently we mix database parameters like DB hostname, username or
password with other JDBC driver parameters which are passed to JDBC
driver as it is. Move these pass-through parameters into dedicated
namespace `driver.*` to separate them from common database parameters.
JDBC parameters which are defined on the connector level still stay
in `database` namespace.
 2022-09-07 12:57:31 +02:00
Vojtech Juranek
5d1b72e95f DBZ-5043 Simplify code after renaming database.history namespace 
Filtering `database.history` namespace from pass-through parameters is now
not needed as it doesn't interfere with other `database.*` parameters.
 2022-09-07 12:57:31 +02:00
Vojtech Juranek
25aa6c4acc DBZ-5043 Change namespace to database.history to schema.history 2022-09-07 12:57:31 +02:00
harveyyue
48d37857b6 DBZ-5588 Field validation errors are misleading for positive, non-zero expectations 2022-09-07 09:18:38 +02:00
harveyyue
58f4d3d912 DBZ-5480 Augment a uniqueness key filed/value in regex topic naming strategy 2022-09-06 14:58:57 +02:00
Claus Ibsen
1f48bfc515
[docs] Typo in description (remove space before dot) 2022-09-06 09:48:57 +02:00
Alexey Loubyansky
c4b95f4ea2 DBZ-5561 Use TCCL to load configuration classes by default 2022-09-06 08:42:45 +02:00
Vadzim Ramanenka
917de37a01 DBZ-5557: Supply partition when comiting offsets with source database 
Keep track of partitions offsets being committed belong to in
`BaseSourceTask`. Supply the partition along with the offset in calls to
`commitOffset`.
 2022-09-05 14:04:23 +02:00
Jiri Pechanec
6e53798407 DBZ-5538 Remove duplicated formatting message 2022-09-05 10:35:55 +02:00
harveyyue
f4c7b52451 DBZ-5538 Unclear validation error when required field is missing 2022-09-05 10:35:55 +02:00
ggaborg
664fd9e1b0 DBZ-3293 unit tests for filters 2022-09-01 20:16:47 +02:00
ggaborg
244de2bd96 DBZ-3293 Add signal table automatically to include list 2022-09-01 20:16:47 +02:00
thangdc94
57210095d2 DBZ-5439 Fix testcase 2022-09-01 09:32:17 +02:00
Jiri Pechanec
de53591dba DBZ-5244 Retry all communication exceptions by default 2022-08-31 09:45:09 -04:00
harveyyue
7ccd40129c DBZ-5569 Oracle connector's max.queue.size.in.bytes is invalid 2022-08-31 09:39:29 -04:00
Jiri Pechanec
1c291d75fd DBZ-5568 Use only Java 11 for tests 2022-08-31 09:19:23 +02:00
pkgonan
fd309193c6 DBZ-5544 Add document 2022-08-24 09:50:23 +02:00
pkgonan
b031770300 DBZ-5544 Support BASE64_URL_SAFE in BinaryHandlingMode 2022-08-24 09:50:23 +02:00
tjwornjs
ca009d8492 DBZ-5489 add column comment to schema 2022-08-24 09:46:01 +02:00
Anisha Mohanty
669024ed46 DBZ-5484 Better error reporting in SKIP/WARN cases 2022-08-23 10:02:29 -04:00
harveyyue
0308a1cbb6 DBZ-5535 Disable preferring DDL before logical schema in history recovery 2022-08-23 14:16:35 +02:00
Jiri Pechanec
34b78e6bce DBZ-2766 Make queue more generic 2022-08-23 13:43:41 +02:00
Jiri Pechanec
f770e0afb1 DBZ-2766 Remove Guava dependency 2022-08-23 13:43:41 +02:00
ggaborg
bf20a9c70f DBZ-5533 Unit test fails on Windows 2022-08-19 09:44:26 -04:00
Sergei Morozov
1c99437890 DBZ-5470: Remove the dependency of JdbcConnection on DatabaseSchema 2022-08-19 11:06:59 +02:00
Chris Cranford
37b97970a6 DBZ-5497 Skip truncate operations by default 
* Promote truncate skip default from PostgreSQL to core library
* Adjust behavior for Oracle with truncates skipped by default
 2022-08-19 11:05:29 +02:00
Vivek Wassan
d89e71ebeb DBZ-5327 Smart Backfills 2022-08-17 10:05:55 +02:00
harveyyue
a08420b21c DBZ-5511 Keep consistence with method serialize of Connect/Json Table Change Serializer 2022-08-16 16:58:07 +02:00
Vojtech Juranek
8e0ad245e5 DBZ-5475 Ensure Outbox JSON NUMBER arrays are of the same type 2022-08-05 11:00:46 -04:00
Vojtech Juranek
7256f62f6b DBZ-5475 Remove unneeded checks from Outbox payload conversion 
* remove float check as we return `float64` by default anyway
* simplify/narrow the checks when detemining array type, also add a test
  for array having the first element `null`
 2022-08-05 11:00:46 -04:00
Chris Cranford
d4261bd2fd DBZ-5476 Fix record accumulator during statistics log entry 2022-08-04 15:28:46 -04:00
harveyyue
2481172309 DBZ-5471 Missing the regex properties validation before start connector of DefaultRegexTopicNamingStrategy 2022-08-04 10:49:29 -04:00
Vojtech Juranek
c7d8353d7b DBZ-4727 Add snapshot paused JXM metrics 
Add two JXM objects:
* `snapshotPaused` - detemines if the incremental snapshot is paused
  or not
* `snapshotPausedDurationInSeconds` - overall time when the incremental
  snapshot was paused. The time adds up - if the snapshot was paused
  e.g. two times, the `snapshotPausedDurationInSeconds` is the sum of
  these two paused times.
 2022-07-29 12:41:58 -04:00
Vojtech Juranek
ed504173e1 DBZ-4727 Implement incremental snaphost pause/resume 
Intorduce two new signals:
* `pause-snapshot` - pauses running incremental snapshot
* `resume-snapshot` - resumes paused incremental snapshot

If the incremental snapshot is running and pause signal is sent,
currently processed chunk is finished and then no other chunk is read.
Snapshot state is kept in running state. Once snapshot is resume,
chunk is reverted to position which was last sent to the broker and
process of reading chunks and emitting `read` events continues until
all tables are read and sent.

This patch handles only table-based signals. MySQL Kafka-based signals
will be addressed later.
 2022-07-29 12:41:58 -04:00
harveyyue
5f430f1a8e DBZ-5457 Deprecate legacy topic selector for all connectors 2022-07-28 12:58:59 -04:00
ggaborg
2c8d71a36f DBZ-5452 Fix unit test failing on Windows platform 2022-07-26 15:50:28 -04:00
Debezium Builder
1ab51ae8d0 [maven-release-plugin] prepare for next development iteration 2022-07-26 18:13:10 +00:00
Debezium Builder
d247a48b79 [maven-release-plugin] prepare release v2.0.0.Beta1 2022-07-26 18:13:09 +00:00
harveyyue
7730610320 DBZ-5425 Remove the duplicated SimpleDdlParserListener from mysql connector 2022-07-20 14:35:34 +02:00
Chris Cranford
75ad893d67 DBZ-5396 Scope attributes inside table document 2022-07-19 15:41:02 -04:00
Chris Cranford
76dafedea7 DBZ-5396 Add multi-type accessors for attribute values 2022-07-19 15:41:02 -04:00
Chris Cranford
f412c2e6a4 DBZ-5396 Add support for connector-specific relational model attributes 2022-07-19 15:41:02 -04:00
harveyyue
daf329bc06 DBZ-5413 Update the topic naming strategy doc to all connectors 2022-07-18 10:18:12 -04:00
harveyyue
77c7db163d DBZ-5414 Validate topic naming strategy relative topic name properties 2022-07-18 09:58:51 -04:00
Chris Cranford
3acf317668 DBZ-5404 Restore build compatibility with Kafka 2.x and 3.0.x 2022-07-14 08:40:42 +02:00
harveyyue
6ea7abc8fc DBZ-4180 Pluggable topic selector 2022-07-13 14:45:25 +02:00
Jiri Pechanec
d8baac7c46 DBZ-5229 Reorganize code for inheritance use 2022-07-12 17:31:19 +02:00
Hossein Torabi
1ea02be9c2 DBZ-5229 define centralized and modular aproach for debezium storage 2022-07-12 17:31:19 +02:00
Chris Cranford
d2a76a7ffc DBZ-5295 Reselect LOB columns on primary key update 2022-07-12 11:59:34 +02:00
Chris Cranford
7b08fb8d3d DBZ-5356 Rewrite loop 2022-07-12 11:01:10 +02:00
Chris Cranford
deaa34d71e DBZ-5356 Avoid NPE during index evaluation 2022-07-12 11:01:10 +02:00
harveyyue
0842e5cd0e DBZ-5366 MySQL Connector column hash v2 does not work 2022-07-06 14:23:57 -04:00
Paweł Malon
707a43d3bc DBZ-5367 Fixed the problem where JSON expanding failed for Outbox event payloads that contained nested arrays where first array was empty and further array contained a Struct 
With proposed changes empty arrays are skipped, as it allows us to delay defining the schema for them until an array with elements is found, as it seems that there is no way to modify a schema that was already defined for a given field
 2022-07-06 13:47:54 -04:00
Debezium Builder
6fd1441af3 [maven-release-plugin] prepare for next development iteration 2022-07-01 09:21:49 +00:00
Debezium Builder
99402af634 [maven-release-plugin] prepare release v2.0.0.Alpha3 2022-07-01 09:21:48 +00:00
Vojtech Juranek
ac06623459 DBZ-5240 Move table types into separate method 
This will allow connectors to override the default values and return
types the connector supports.
 2022-06-30 10:46:54 +02:00
Gunnar Morling
cb53332561 DBZ-5335 Formatting 2022-06-30 09:54:13 +02:00
Mickael Maison
5ff71aa992 DBZ-5335 Fix formatting issues in the connectors configdefs 2022-06-30 09:35:54 +02:00
harveyyue
20dc353262 DBZ-5170 Mysql Commit Timestamp 2022-06-27 15:50:52 +02:00
Naveen Kumar KR
b75c9e635b DBZ-5277 ChangeEventQueue - Support Multiple Writer (doEnqueue) and Reader (poll) Threads 
* ReentrantLock based write and read coordination
* Condition based Inter Thread communication for isFull and isNotFull conditions
 2022-06-27 15:37:00 +02:00
Naveen Kumar KR
823a49fff4 DBZ-5277 ChangeEventQueue Test with Multiple Writer and Reader Threads 
Assertion would fail if one or both of the scenario occurs

1) Deadlock between Writer and Reader Threads
2) Thread Starvation between Writer and Reader Threads
 2022-06-27 15:37:00 +02:00
Jiri Pechanec
9ada2f28ce DBZ-4902 Change for exponential strategy 2022-06-27 14:29:51 +02:00
kanha gupta
1b17372d2d DBZ-4902 Replace long durations with Duration 2022-06-27 14:29:51 +02:00
Chris Cranford
ef99595d91 DBZ-5270 Reduce table schema capture logging verbosity 
When enabling the `database.history.store.only.captured.tables.ddl`,
the logging in RelationalSnapshotChangeEventSource does not take into
account this setting when logging, leading to a false impression of
what tables will actually be captured in the schema history.
 2022-06-27 08:47:34 +02:00
Jiri Pechanec
5e1ae24e92 DBZ-5272 Notify all threads in case of mutliple writers 2022-06-21 07:08:39 +02:00
Chris Cranford
3fb5bf1339 DBZ-4451 Introduce SchemaChangeEvent factory methods 2022-06-16 09:19:41 +02:00
Chris Cranford
863abcd081 DBZ-4451 Store previous table id in alter schema change events 
When a table is renamed with the ALTER TABLE statement, the schema
history record not only will reference the current table's unique id
in the "id" metadata, but will also refer to the old table name in
the "previousId" metadata field.
 2022-06-16 09:19:41 +02:00
Chris Cranford
20bb2adea5 DBZ-4834 Fix incremental snapshot of table added to include list 2022-06-16 08:39:16 +02:00
Vojtech Juranek
25eba9749d DBZ-5198 Add support for SQL server specific table delimiters 
Add suppoprt for `TableId` delimiters and provide implementation for
SQL server. SQL server allows to use reserved words in table names or
names with spaces, but they have to wrapped by `[]` characters, e.g.
`[dbname].[table name]`.

Debezium can handle spaces e.g. in table include list, but fails when
parsing snapshot select, therefore the parsing with predicates is used
only for parsing spanshot select for now. If useful, it can be used
later on on other places as well.
 2022-06-13 10:57:15 +02:00
Vojtech Juranek
796e17b2c6 DBZ-5198 Move predicates for parsing TableId into dedicate class 
Moving the predicates into parsing context decouples the predicates from
the private `TableIdTokenizer` class. Moving the predicates into
separate class would allow us to provide database specific
implementations.
 2022-06-13 10:57:15 +02:00
Debezium Builder
6fff3a2876 [maven-release-plugin] prepare for next development iteration 2022-06-09 11:49:26 +00:00
Debezium Builder
b2e2e351df [maven-release-plugin] prepare release v2.0.0.Alpha2 2022-06-09 11:49:26 +00:00
Vojtech Juranek
4a48eb33df DBZ-5045 Remove legacy V1 source info 2022-06-08 09:11:00 +02:00
Vojtech Juranek
250e160b81 DBZ-5045 Removed unused methods 2022-06-08 09:11:00 +02:00
Vojtech Juranek
ef8d7f7af7 DBZ-5045 Remove deprecated OBSOLETE_NAME_FOR_SKIP_FAILURE_HANDLING option value 2022-06-08 09:11:00 +02:00
Vojtech Juranek
ff5ddd7959 DBZ-5045 Remove deprecated FIELD_PAYLOAD_ID 
`table.field.event.payload.id` is replaced by `table.field.event.key`.
Also fix default value of `table.field.event.key` which was missing.
 2022-06-08 09:11:00 +02:00
Vojtech Juranek
0530dc2de9 DBZ-5045 Remove deprecated STORE_ONLY_MONITORED_TABLES_DDL 2022-06-08 09:11:00 +02:00
Vojtech Juranek
4e7b30b718 DBZ-5045 Remove deprecated DEBEZIUM_OPERATION_INVALID_BEHAVIOR 2022-06-08 09:11:00 +02:00
Vojtech Juranek
b0b3066ed5 DBZ-5045 Remove deprecated COLUMN_BLACKLIST 2022-06-08 09:11:00 +02:00
Vojtech Juranek
5031594b75 DBZ-5045 Remove deprecated COLUMN_WHITELIST 2022-06-08 09:11:00 +02:00
Vojtech Juranek
b3a6131cee DBZ-5045 Remove deprecated DATABASE_BLACKLIST 2022-06-08 09:11:00 +02:00
Vojtech Juranek
0cbb57f4bf DBZ-5045 Remove deprecated DATABASE_WHITELIST 2022-06-08 09:11:00 +02:00
Vojtech Juranek
f84893a8b4 DBZ-5045 Remove deprecated SCHEMA_BLACKLIST 2022-06-08 09:11:00 +02:00
Vojtech Juranek
261188482f DBZ-5045 Remove deprecated SCHEMA_WHITELIST 2022-06-08 09:11:00 +02:00
Vojtech Juranek
6abbcee6a3 DBZ-5045 Remove deprecated TABLE_BLACKLIST 2022-06-08 09:11:00 +02:00
Vojtech Juranek
db08e894fc DBZ-5045 Remove deprecated TABLE_WHITELIST 2022-06-08 09:11:00 +02:00
harveyyue
acb743bd34 DBZ-5206 Remove the deprecated monitored tables metrics 2022-06-07 16:13:29 -04:00
Chris Cranford
3e820c4047 DBZ-4251 Adjust semantics with collection removal 2022-06-02 07:30:07 +02:00
Chris Cranford
187e3250a3 DBZ-4251 Add regex support for stop signal 2022-06-02 07:30:07 +02:00
Chris Cranford
9d71296b5c DBZ-4251 Support stopping incremental snapshots 2022-06-02 07:30:07 +02:00
Jiri Pechanec
88b7ae2050 DBZ-5136 Move sourceInfo to common class 2022-06-01 16:56:29 +02:00
Mark Bereznitsky
c64b8af31a DBZ-5136 reintroduce the CommonOffsetContext class and refactor common functionality into it 2022-06-01 16:56:29 +02:00
Mark Bereznitsky
2d72dae649 DBZ-5136 further refining of the algorithm for determining position in snapshot and IT test fixes 2022-06-01 16:56:29 +02:00
Jiri Pechanec
08e8c3d795 DBZ-5136 Move common class to common package 2022-06-01 16:56:29 +02:00
Andrew Walker
d0759461a4 DBZ-5136: Convert RelationalOffsetContext to CommonOffsetContext 2022-06-01 16:56:29 +02:00
Andrew Walker
01a81f8bc1 DBZ-5136: Refactored SnapshotContext setters 2022-06-01 16:56:29 +02:00
Andrew Walker
b6222d42bf DBZ-5136: Add first flag and refactor OffsetContext 2022-06-01 16:56:29 +02:00
Andrew Walker
593c2769e6 DBZ-5136: Add last in table flag 2022-06-01 16:56:29 +02:00
Chris Cranford
15e9231c30 DBZ-5179 Use LRUCacheMap to avoid high memory usage 2022-06-01 15:53:55 +02:00
Chris Cranford
9fa5a48274 DBZ-5011 Register MongoDB metrics with task id when more than one task 2022-06-01 09:41:36 +02:00
harveyyue
9a3afac243 DBZ-5180 Put the MysqlPartition values to MDC getting NPE in logging context phase 2022-06-01 09:17:04 +02:00
Vojtech Juranek
5550057034 DBZ-5042 Add abstract partition with DB name 
DB names is used by logging facilities. As the DB name is required only
for relational DBs, also this abstract class is provided only for
relational DB connectors.

Use this abstract class for SQL server connector, where DB name logging
is already implemented.
 2022-05-27 14:20:30 -04:00
Jiri Pechanec
b85cb275b1 DBZ-5089 Use atomic reference 2022-05-27 12:46:39 +02:00
connorszczepaniak-wk
8693b4f008 DBZ-5089: add partition to the outbox event router in additional fields 
DBZ-5089: undo format change

DBZ-5089: add tests for the partition routing

DBZ-5089: remove nonsensical comment

DBZ-5089: partition fieldname is a string; test properly

DBZ-5089: fix one of the failing tests

DBZ-5089: pass args through

DBZ-5089: fix oracle implementation; add configs for mongo connector

DBZ-5089: put partition in additional placement

DBZ-5089: restore formatting
 2022-05-27 12:46:39 +02:00
harveyyue
e45e8bd43b DBZ-5172 Expose more useful exception info with building the field default value schema 2022-05-27 12:41:12 +02:00
Rotem Adhoh
8ef7224543 DBZ-5166 Fixed ExtractNewRecordState SMT to pass all columns' data as is, i.e: without potential unwanted values manipulation. 
Prior to this fix, when inserting an explicit NULL value to a nullable column which has a default value, resulted in a record which contains the corresponding field with the default value (instead of NULL) after applying ExtractNewRecordState transformation.
A new unit test was added as well.
 2022-05-27 12:33:23 +02:00
harveyyue
3ada04b522 DBZ-3865 Debezium MySql connector does not capture floating point numbers with the right precision 2022-05-25 12:54:42 -04:00
Chris Cranford
7d8c9486d6 DBZ-5119 Move Heartbeat creation to ConnectorConfig & descendants 2022-05-25 09:59:59 +02:00
Chris Cranford
7ff7826743 DBZ-5119 Pull heartbeat.action.query support into debezium-core 2022-05-24 12:54:40 +02:00
Chris Lee
2a8456d21b DBZ-5123 data-collection regex snapshot signal 2022-05-24 12:46:28 +02:00
harveyyue
52ba58ccc3 DBZ-5098 Introduce a new field "ts_ms" to identify the process time for schema change event 2022-05-18 14:21:39 +02:00
harveyyue
f66b07cd80 DBZ-5141 Should be sleep with REGISTRATION_RETRY_DELAY when registry MBean failed 2022-05-18 12:19:31 +02:00
Jiri Pechanec
8643ad3a22 DBZ-5138 Unify happy/unhappy common path 2022-05-18 07:42:33 +02:00
Chris Cranford
e075d6b2eb DBZ-5138 Fix tracking MBean registration 2022-05-17 20:02:11 -04:00
Jiri Pechanec
d56f6ef6e0 DBZ-5138 Retry failed metrics registration; allow starting without metrics 2022-05-17 20:02:11 -04:00
Sergei Morozov
1ad97b9d66 DBZ-5130: Log history recover progress every two seconds 2022-05-16 09:17:24 +02:00
Sergei Morozov
65e97796e7 DBZ-5129: Use buffered reader when reading history from file 2022-05-13 09:38:43 +02:00
Jiri Pechanec
4396fc3ec3 DBZ-4600 Remove deprecated class 2022-05-10 15:25:35 +02:00
Jiri Pechanec
5e94c1d498 DBZ-4600 Fix formatting 2022-05-10 15:08:46 +02:00
Nathan Bradshaw
c76e697f2f DBZ-4600 - Add BinaryDataConverter to support byte[] and ByteBuffer source data 2022-05-10 15:08:46 +02:00
Nathan Bradshaw
b8b8dffac4 DBZ-4600 - Deprecate ByteBufferConverter from 1.9 2022-05-10 15:08:46 +02:00
Nathan Bradshaw
4bf01a95a2 DBZ-4600 - Start to rough-out ByteArrayConverter 
- Add delegator support to ByteArrayConverter. Add missing tests
- Add required headers, correct typos
 2022-05-10 15:08:46 +02:00
Mark Bereznitsky
ae00729aa2 DBZ-5087 Added a decorator for result sets that can cancel the parent statements 2022-05-06 07:52:00 +02:00
Sergei Morozov
f6546e4d87 DBZ-5075: Remove unused legacy MySQL code 2022-05-03 13:46:23 +02:00
harveyyue
d03e555144 DBZ-5072 Update the two properties to topic-routing doc 2022-05-03 13:44:58 +02:00
harveyyue
25205b233c DBZ-5072 Config the cache size property for ByLogicalTableRouter caches 2022-05-03 13:44:58 +02:00
Debezium Builder
901825810a [maven-release-plugin] prepare for next development iteration 2022-04-28 13:51:13 +00:00
Debezium Builder
05dcb45647 [maven-release-plugin] prepare release v2.0.0.Alpha1 2022-04-28 13:51:12 +00:00
Chris Cranford
09e58c5a5c DBZ-5006 Allow "/" characters in Oracle table names 2022-04-27 08:52:29 -04:00
Sergei Morozov
fd46a670ad DBZ-5033: Ignore null offsets when recovering database history 2022-04-26 11:40:21 +02:00
Sergei Morozov
e79305cf93 DBZ-5033: Update Javadocs in DatabaseHistory 2022-04-26 11:40:21 +02:00
Vojtech Juranek
da5378d095 DBZ-4949 Adjust one test to use features > Java 11 
Adjust one test to use multiline strings, which is a feature added in
Java 15 and thus verify Java source level > 11 works for tests.
 2022-04-22 05:53:51 +02:00
Brad Morgan
8e25c52585 DBZ-4989 Passthrough payload schema in EventRouterDelegate 2022-04-14 12:17:53 +02:00
Jiri Pechanec
053bf602af DBZ-4979 Guard against NPE for non-table related DDLs 2022-04-12 09:36:14 -04:00
Debezium Builder
4a6de64c25 [maven-release-plugin] prepare for next development iteration 2022-04-05 14:49:52 +00:00
Debezium Builder
683851fd90 [maven-release-plugin] prepare release v1.9.0.Final 2022-04-05 14:49:51 +00:00
Debezium Builder
ab7fe3b96a [maven-release-plugin] prepare for next development iteration 2022-03-25 16:45:06 +00:00
Debezium Builder
d19fc234ee [maven-release-plugin] prepare release v1.9.0.CR1 2022-03-25 16:45:06 +00:00
Sergei Morozov
010f8c9d50 DBZ-3535: Rework CloudEventsConverter 2022-03-24 09:37:05 +01:00
Sergei Morozov
854186e846 DBZ-3535: Rework ByLogicalTableRouter 2022-03-24 09:37:05 +01:00
Sergei Morozov
139b129cf5 DBZ-3535: Rework TransactionMonitor 2022-03-24 09:37:05 +01:00
Sergei Morozov
fccc31c35b DBZ-3535: Rework ConnectTableChangeSerializer 2022-03-24 09:37:05 +01:00
Sergei Morozov
b258674782 DBZ-3535: Introduce schema.name.adjustment.mode 2022-03-24 09:37:05 +01:00
Sergei Morozov
3a03ff0186 DBZ-3535: Remove unused EventDispatcher constructor parameter 2022-03-24 09:37:05 +01:00
Vojtech Juranek
0163c07800 DBZ-4786 Avoid NPE when removing column which doesn't exists 2022-03-23 07:53:06 +01:00
Vojtech Juranek
fdfc7a6647 DBZ-3762 Filter out MySQL inline comments by default 2022-03-21 16:57:43 +01:00
kate
4b2d5c29be DBZ-4878 Re-read incremental snapshot chunk on DDL event 2022-03-18 07:55:43 +01:00
harveyyue
044e454e0a DBZ-4832 Don't set truststore/keystore parameters to system variables 2022-03-15 11:25:28 +01:00
Josh Ribera
020e845f44 DBZ-4809 Add task id and partition to the logging context for 
multi-partition connectors.

Co-authored-by: Sergei Morozov <morozov@tut.by>
 2022-03-09 20:46:38 +01:00
Gunnar Morling
d20a6f00e7 DBZ-4801 Using JdbcConfiguration in JdbcConnection 2022-03-07 17:53:45 +01:00
Gunnar Morling
c1c322ecf0 DBZ-4801 Simplifying JdbcConnection constructors 2022-03-07 17:53:45 +01:00
harveyyue
856ab85cef DBZ-4824 Expect the null value with snapshot CapturedTables metric when skipping snapshotting 2022-03-07 12:19:13 +01:00
Debezium Builder
fece16039c [maven-release-plugin] prepare for next development iteration 2022-03-03 17:00:08 +00:00
Debezium Builder
e0d8a0d809 [maven-release-plugin] prepare release v1.9.0.Beta1 2022-03-03 17:00:08 +00:00
yingyingtang-brex
eb3f02b9c0 DBZ-4813 indicating message id and key in outbox log messages 2022-03-03 10:47:20 +01:00
Josh Ribera
fc6d55e935 DBZ-4783: Add task id label to metrics 2022-03-01 14:21:39 +01:00
Mike Kamornikov
37a67d5dcf DBZ-4783: Manage change tables per partition 2022-03-01 14:21:39 +01:00
Chris Cranford
e472a06da4 DBZ-4773 Fix formatting 2022-02-23 08:49:31 +01:00
Gunnar Morling
a144d84257 DBZ-4773 Using right constructor signature based on Kafka client version 2022-02-23 08:49:31 +01:00
Chris Cranford
9103a220b7 DBZ-4744 Correctly parse signal table name when using domain name 2022-02-22 15:27:21 +01:00
Jiri Pechanec
d13a1635d7 DBZ-3952 Store schema for tables not in schema collection list 2022-02-17 07:41:00 +01:00
Sergei Morozov
2d14b5b09f DBZ-4478: Multi-partition metrics for SQL Server 2022-02-16 11:31:38 +01:00
Sergei Morozov
6d0aa9408c DBZ-4478: Add Partition parameter to io.debezium.pipeline.source.spi.*Listener interfaces 2022-02-16 11:31:38 +01:00
Sergei Morozov
af8ea4d73d DBZ-4478: Move logging from metrics to coordinator 2022-02-16 11:31:38 +01:00
Sergei Morozov
24ff3c9eba DBZ-4478: Remove throws InterruptedException from DataChangeEventListener::onEvent 2022-02-16 11:31:38 +01:00
Jiri Pechanec
54ba2cd598 DBZ-4736 Handle string NULL default values correctly 2022-02-11 12:06:55 +01:00
harveyyue
8431a614a2 DBZ-4730 Expect plain value instead of scientific exponential notation when using decimal string mode 2022-02-10 17:40:12 +01:00
Debezium Builder
7a47f478a4 [maven-release-plugin] prepare for next development iteration 2022-02-09 07:23:10 +00:00
Debezium Builder
b1dd35025e [maven-release-plugin] prepare release v1.9.0.Alpha2 2022-02-09 07:23:10 +00:00
Farid Uyar
a559e51b08 DBZ-4547 Create History Topic with optional RF 2022-02-08 16:58:13 +01:00
Paweł Malon
7f7e71b36a DBZ-4704 Replaced hasNotField with hasField and negated the result of it 2022-02-07 20:39:52 +01:00
Paweł Malon
abd8c4202f DBZ-4704 Fixed the problem with expanding JSON payload that contains arrays of objects 2022-02-07 20:39:52 +01:00
Chris Cranford
055d5ba474 DBZ-4701 executeWithoutCommitting throw exception when auto-commit is enabled 2022-02-07 06:21:11 +01:00
Snigdhajyoti Ghosh
0ab45ecab0 DBZ-4518 Make kafka query timeout configurable 2022-02-04 14:08:15 +01:00
Chris Cranford
3980210c0c DBZ-4648 Relax case requirement for SnapshotType 2022-02-03 18:15:23 +01:00
nathan
d5dcde8f7a DBZ-4646 Changing exists method for FileDataBaseHistory to test if file is empty. 2022-02-03 16:47:22 +01:00
Mark Drilling
8a5279d304
DBZ-4314 Define Oracle category grouping and ordering (#3151) 2022-02-02 15:33:01 -06:00
Jiri Pechanec
74ea147152 DBZ-3679 Clarify log messages 2022-02-02 14:33:41 -05:00
Chris Cranford
088df2db8e DBZ-4584 Fix test failures 2022-01-31 09:14:46 +01:00
Chris Cranford
dcd8607108 DBZ-4584 Support case-sensitive Oracle incremental snapshots 2022-01-31 09:14:46 +01:00
Debezium Builder
33e2f5cd0f [maven-release-plugin] prepare for next development iteration 2022-01-26 19:00:20 +00:00
Debezium Builder
a77b6f8b8e [maven-release-plugin] prepare release v1.9.0.Alpha1 2022-01-26 19:00:20 +00:00
Jiri Pechanec
55894e57c2 DBZ-4610 Upgrade to Kafka 3.1.0 2022-01-25 15:26:42 +01:00
Chris Cranford
bd604a23cd DBZ-4419 Add "none" option to skipped.operations 2022-01-22 00:33:21 +01:00
Chris Cranford
773debc340 DBZ-4419 Deprecate TruncateHandlingMode, favor skipped.operations 2022-01-22 00:33:21 +01:00
Jiri Pechanec
664d989e42 DBZ-4577 Provide custom error message handling 2022-01-21 10:24:19 +01:00
Jiri Pechanec
8f5bb31eab DBZ-3700 Strings a bytes only when converters present 
When database was used with non-UTF8 charset the snapshot was sending
byte array with UTF-8 bytes but MySQL converter was using database based
charset to convert it to String leading to malformed data.
It is necessary to convert strings using the database charset but it
might be a problem in case of non-charset matching.
The current solution
1) Uses strings by default
2) Switches to byte array only when converters are present to cover
original use case
3) If the conversion fails it switches back to string
 2022-01-19 16:20:32 +01:00
Jiri Pechanec
8fd3243a86 DBZ-2224 Test logging based on logback 2022-01-18 11:24:21 +01:00
Sergei Morozov
17fba7258b DBZ-4541: Promote failure to register metrics to exception 2022-01-14 19:50:57 +01:00
Sergei Morozov
6d97433d18 DBZ-4541: Remove logger from Metrics register/unregister signatures 2022-01-14 19:50:57 +01:00
Chris Cranford
610e8eba73 DBZ-4548 Fix DatabaseVersionResolver comparison logic 2022-01-13 13:29:34 -05:00
Sergei Morozov
0329859cda DBZ-4459: Introduce interfaces and default implementations for change event source metrics 2022-01-12 14:32:19 +01:00
Gunnar Morling
f0b4a5ba9f DBZ-4351 Misc. clean-up 2022-01-06 19:03:46 +01:00
harveyyue
291e8ac221 DBZ-4351 Add create/update/delete event seen metrics for monitor upstream dml operation 2022-01-06 19:03:46 +01:00
harveyyue
831217053c DBZ-4503 Drop the primary key column getting exception 2022-01-03 10:57:07 +01:00
harveyyue
3670b0c6ef DBZ-4497 Parse multiple signed/unsigned keyword from ddl statement failed 2022-01-03 10:47:03 +01:00
Gunnar Morling
92665f1799 DBZ-4472 Improving log output 2021-12-20 15:58:14 +01:00
Jiri Pechanec
d13e6f546c DBZ-4467 Skip unsupported test for Db2 2021-12-20 11:29:30 +01:00
Debezium Builder
67c9663c98 [maven-release-plugin] prepare for next development iteration 2021-12-16 10:11:29 +00:00
Debezium Builder
8d532d895f [maven-release-plugin] prepare release v1.8.0.Final 2021-12-16 10:11:28 +00:00
Chris Cranford
f99ad7512f DBZ-3940 Add connector-driven EventRouter configuration 2021-12-13 13:52:31 +01:00
Jiri Pechanec
740b6b21f6 DBZ-4298 Support non-JDBC logical values in keys for incremental snapshots 
Incremental snapshot source gets the PK values after they are processed
by value converter. This is usally correct solution. The values are
passed to chunk queries. There are few datatypes like unconstrained
NUMERIC in PostgreSQL that must be converted to a Debezium specific
class to keep and enrich the value. Such class is not known to JDBC
driver. This commit intorduce a marker interface that enables the code
to identify the value not recognized by JDBC and provides a method to
obtain the original value.
 2021-12-13 12:29:21 +01:00
Gunnar Morling
0023cb10a5
DBZ-4393 Create a Debezium Schema Generator for Debezium connectors 
* added an API generator for Debezium connectors and static API definitions for connectors in a separate module
* added Maven plug-in
* added GH workflow for debezium-schema-generator

Co-authored-by: rkerner <rkerner.mobil@gmail.com>
Co-authored-by: Anisha Mohanty <anishamohanty23@gmail.com>
 2021-12-10 12:35:09 +01:00
Gunnar Morling
73cfe71342 DBZ-4423 Micro-optimizing Strings::duration() 2021-12-10 11:30:18 +01:00
Debezium Builder
f53e6c00df [maven-release-plugin] prepare for next development iteration 2021-12-09 20:16:04 +00:00
Debezium Builder
826837701a [maven-release-plugin] prepare release v1.8.0.CR1 2021-12-09 20:16:04 +00:00
Jiri Pechanec
b1dcfcb0b8 DBZ-3342 Incremental snapshot support for MongoDB 2021-12-09 10:03:41 +01:00
Tom Billiet
ec9f122e0a DBZ-4385 Truncate support for Oracle 2021-12-08 15:00:53 -05:00
Gunnar Morling
1f8fd22685 DBZ-4341 Improving logging; 
Also fixing incorrect JavaDoc.
 2021-12-08 13:12:54 +01:00
Chris Cranford
8f007e08a2 DBZ-4388 Don't set default value if resolution fails 2021-12-07 16:38:22 +01:00
Debezium Builder
b5439b54bf [maven-release-plugin] prepare for next development iteration 2021-11-30 09:38:47 +00:00
Debezium Builder
125f6e5aa2 [maven-release-plugin] prepare release v1.8.0.Beta1 2021-11-30 09:38:47 +00:00
Jiri Pechanec
2789dd610a DBZ-4272 Throw exception for non matching schema 2021-11-30 09:28:44 +01:00
Jiri Pechanec
5e40e1cdf5 DBZ-4272 Skip connector test using annotations 2021-11-30 09:28:44 +01:00
Jiri Pechanec
358eb0379b DBZ-4272 Disable tests for SQL Server 2021-11-30 09:28:44 +01:00
Chris Cranford
0a10d5b484 DBZ-4272 Trigger schema refresh before snapshot; only for PostgreSQL 2021-11-30 09:28:44 +01:00
Chris Cranford
3a8d1ff838 DBZ-4272 Incrementally snapshot after schema change without NPE 2021-11-30 09:28:44 +01:00
Gunnar Morling
2157256398 DBZ-4364 Awaiting table metadata to become queryable 2021-11-29 09:23:18 +01:00
Jiri Pechanec
1c5f3b2330 DBZ-4364 Make byte[] and ByteBuffer comparable 2021-11-29 09:23:18 +01:00
Jiri Pechanec
4a2a60db74 DBZ-4364 Fix missing method call 2021-11-29 09:23:18 +01:00
Gunnar Morling
560ef2babd DBZ-2363 Misc. adjustments; 
* Adding Strings::isBlank()
* Simplifying test
* Using long in cases where we know it's present
* Removing some unused code
* Testing binary handling mode
* Avoiding TX id value overflow
* Assigning names to schemas to avoid issues as in DBZ-650
 2021-11-26 14:29:15 +01:00
lairen
a0898ed440 DBZ-2363 Adding support for pg_logical_emit messages in Postgres connector for the pgoutput plugin 2021-11-26 14:29:15 +01:00
Jacob Gminder
f7f8748062 DBZ-2975: Move the poll interval logic to the ChangeEventSourceCoordinator. 2021-11-22 12:30:31 +01:00
Gunnar Morling
a7c6d18489 DBZ-4309 Removing legacy logger 2021-11-19 08:04:44 +01:00
Andrey Yegorov
b6d2c41fb0 DBZ-4309 Avoid calling 'LoggerFactory.getLogger(getClass())' for each new instance of the RelationalChangeRecordEmitter 2021-11-18 22:37:51 +01:00
Chris Cranford
2730187a4b DBZ-4012 Deprecate table.field.event.payload.id 2021-11-18 21:45:41 +01:00
Chris Cranford
ab86cd18a9 DBZ-4012 Suggested changes 2021-11-18 21:45:41 +01:00
Chris Cranford
349b9690f4 DBZ-4012 Suggested changes 2021-11-18 21:45:41 +01:00
Chris Cranford
0890202539 DBZ-4012 Remove incubating from EventRouter 2021-11-18 21:45:41 +01:00
Chris Cranford
0df5c6219b DBZ-4012 Bubble up SMT validation failures to Kafka Connect 2021-11-18 21:45:41 +01:00
Chris Cranford
970792bc40 DBZ-4012 Deprecate/Rename debezium namespace config options 2021-11-18 21:45:41 +01:00
Chris Cranford
7a4140c6aa DBZ-4012 ActivateTracingSpan is pass-through when OpenTracing unavailable 2021-11-18 21:45:41 +01:00
Gunnar Morling
ca95b424fb DBZ-3528 Misc. clean-up 2021-11-17 16:02:06 +01:00
Sungho Hwang
78c92c3772 DBZ-3528 turned access modifier of jsonNodeToSchemaBuilder() back to private 2021-11-17 16:02:06 +01:00
Gunnar Morling
03433b663b DBZ-3528 Misc. fixes 2021-11-17 16:02:06 +01:00
Sungho Hwang
7c95636668 DBZ-3528 Chaged variable naming for better readability 2021-11-17 16:02:06 +01:00
Sungho Hwang
c9eb0f9157 DBZ-3528 Refactored several parts according to the feedbacks 
A delegate class for common logic between two Outbox Event Routers has been created.
Documentation has been fixed.
The configuration properties for MongoDB Outbox Event Router have been changed to use existing Outbox Event Router's.
 2021-11-17 16:02:06 +01:00
Sungho Hwang
0b9fc88ea6 DBZ-3528 Applied the code style rules 2021-11-17 16:02:06 +01:00
Sungho Hwang
2c279b07f6 DBZ-3528 Wrote MongoDB Outbox Event Router with documentation and its unit & integration tests 2021-11-17 16:02:06 +01:00
Chris Cranford
e3da449303 DBZ-4276 Expose unavailable value placeholder for PG and Oracle only 2021-11-16 20:29:59 +01:00
Chris Cranford
ca17352221 DBZ-4276 Unify unavailable value placeholder handling 
Deprecate the `toasted.value.placeholder` option in PostgreSQL in favor
of the new `unavailable.value.placeholder` option that is shared by all
relational connectors.
 2021-11-16 20:29:59 +01:00
Kate
44073cf7d8 DBZ-4196 Support schema changes during incremental snapshot 2021-11-16 14:56:06 +01:00
camilesing
dfabe8cda8 DBZ-4275 fix AbstractDatabaseHistory typo 2021-11-14 19:21:37 +01:00
Debezium Builder
f93e901dce [maven-release-plugin] prepare for next development iteration 2021-11-11 09:21:59 +00:00
Debezium Builder
895a16bf19 [maven-release-plugin] prepare release v1.8.0.Alpha2 2021-11-11 09:21:59 +00:00
Chris Cranford
85101f0ad7 DBZ-3611 Avoid NPE with database include/exclude lists 2021-11-11 09:24:05 +01:00
Kate
992ae537c8 DBZ-4244 Quote signal table name in MySQL connector 2021-11-10 10:18:26 +01:00
Chris Cranford
5afda302ec DBZ-3710 Javadoc updates 2021-11-10 09:43:25 +01:00
Chris Cranford
5ec9381702 DBZ-3710 Unify DefaultValueMapper (Oracle/SQL Server/PostgreSQL) 2021-11-10 09:43:25 +01:00
Chris Cranford
343318b5d4 DBZ-3710 Support Oracle column default values 2021-11-10 09:43:25 +01:00
harveyyue
aa3ae05e22 DBZ-4107 Incremental snapshot doesn't work without primary key 2021-11-05 13:31:26 +01:00
Gunnar Morling
1226c3b2ab DBZ-4029 Misc. clean-up; 
* Exposing heartbeatInterval() on CommonConnectorConfig
* Closing heartbeat connection
* Making test more resilient
* Correct member ordering and visibility
* Doc fixes
 2021-11-05 13:27:09 +01:00
ahodavdekar
91b7ef2c36 DBZ-4029 Adding support for heartbeat action queries to MySQL connector 2021-11-05 13:27:09 +01:00
Gunnar Morling
b78e318399 DBZ-3966 Expanding test 2021-11-04 10:47:24 +01:00
jiabao.sun
2987e96c44 DBZ-3966 Serializing hasDefaultValue in JsonTableChangeSerializer 2021-11-04 10:47:17 +01:00
Gunnar Morling
d1cae3f3b7 DBZ-3966 Making defaultValueExpression() return Optional<String> from editor too 2021-11-04 10:47:17 +01:00
jiabao.sun
24ead16c24 DBZ-3966 Remove defaultValue and keep defaultValueExpression 2021-11-04 10:47:17 +01:00
jiabao.sun
77c4113697 DBZ-3966 JsonTableChangeSerializer support serialization for defaultValue and enumValues 2021-11-04 10:47:17 +01:00
Kate
a645d0bc7e DBZ-4197 Process transaction started/committed in R/O snapshot 2021-11-04 06:19:58 +01:00
ahodavdekar
1d81b6db19 DBZ-4077 Exposing the transaction metadata topic name as a config option 2021-11-02 06:07:05 +01:00
Gunnar Morling
fd06a4e2a4 DBZ-2975 Using Offsets in recover() contract; 
Keeping legacy method until the legacy MySQL connector implementation has been removed.
 2021-11-01 10:17:49 +01:00
Mike Kamornikov
16c07e915b DBZ-2975: Recover database schema from multiple partitions 2021-11-01 10:17:49 +01:00
Debezium Builder
2026a48302 [maven-release-plugin] prepare for next development iteration 2021-10-27 06:47:27 +00:00
Debezium Builder
97106c1fd9 [maven-release-plugin] prepare release v1.8.0.Alpha1 2021-10-27 06:47:26 +00:00
Jiri Pechanec
a1084360da DBZ-435 Too fast delete can prevent from obtaining the full document 2021-10-26 18:13:16 +02:00
Jiri Pechanec
cb36b575db DBZ-435 Support for Change Streams capture 2021-10-26 18:13:16 +02:00
Sungho Hwang
54d5c16b69 DBZ-4191 Fixed a variable name in an unit test of Outbox Event Router to enhance readability. 2021-10-22 07:57:24 +02:00
harveyyue
90e9bea93f DBZ-4000 Support parse table and columns comment 2021-10-20 15:19:31 +02:00
Anisha Mohanty
2eae7a742c DBZ-4045 Use listener props for deprecated host & port props 2021-10-20 11:51:34 +02:00
Gunnar Morling
3d093e2a3b DBZ-4164 Import ordering 2021-10-18 14:45:00 +02:00
lujiefsi
12ccc24e17
DBZ-4164 Fixing resource leak due to Files::lines 2021-10-18 14:31:39 +02:00
Juan Fiallo
346e0f3af2 DBZ-3517 Making database.user optional for SQL Server. 
SQL Server doesn't require username/password when using Kerberos authentication, which can be configured using pass-through database properties. Changing the database.user config to be optional for SQL Server Connector to reflect that.
 2021-10-18 12:06:56 +02:00
Juan Fiallo
5ac93ac20e DBZ-3517 Making database.user optional for SQL Server. 
SQL Server doesn't require username/password when using Kerberos authentication, which can be configured using pass-through database properties. Changing the database.user config to be optional for SQL Server Connector to reflect that.
 2021-10-18 12:06:56 +02:00
Juan Fiallo
e041b599d8 DBZ-3517 Making database.user optional for SQL Server. 
SQL Server doesn't require username/password when using Kerberos authentication, which can be configured using pass-through database properties. Changing the database.user config to be optional for SQL Server Connector to reflect that.
 2021-10-18 12:06:56 +02:00
Jiri Pechanec
0ccc83db21 DBZ-3860 Provide correct boundaries for multi PK tables 2021-10-14 17:26:50 +02:00
Gunnar Morling
26f13d83d5 DBZ-3993 Reflectively invoking Runtime.version(); 
As that API is only available from Java 9 onwards, calling it will cause
a compilation failure when compiling with --release.
 2021-10-14 09:46:06 -04:00
Gunnar Morling
6b3d2c3b76 DBZ-3993 Formatting fix 2021-10-13 10:59:23 +02:00
Gunnar Morling
c3eed6051b DBZ-3993 Skipping object size based test on Java 16+ 2021-10-13 10:59:23 +02:00
Sergei Morozov
4a03c5df17 DBZ-2975: Include database name into topic and schema names 
Co-authored-by: Mike Kamornikov <mikekamornikov@gmail.com>
 2021-10-12 15:40:02 +02:00
Jiri Pechanec
657c5540b3 DBZ-4083 Add filter to MongoDB cloud converter 2021-10-11 11:19:30 +02:00
Jiri Pechanec
aec2080415 DBZ-3895 Config returns Patterns not Strings 2021-10-11 10:57:33 +02:00
Jiri Pechanec
68f148c3f3 DBZ-3895 Make snapshot collection list matching case insensitive 2021-10-11 10:57:33 +02:00
Chris Cranford
abc3bb42d6 DBZ-4106 Don't continuously reconfigure DatabaseHistory 2021-10-05 20:15:58 +02:00
Gunnar Morling
62715c4e1d DBZ-4033 Doc update 2021-10-05 10:54:01 +02:00
unalsurmeli
9a13d80074 DBZ-4033 A new hash version has been added so that it can be used to have the same hash value in different places; 
In V2, the byte array only has the actual value, unlike the serialization based V1 implementation which exposed Java's internal serialization format.
 2021-10-05 10:52:30 +02:00
Vivek Wassan
82dad32e2a DBZ-4078 Checking connection is valid before commit in readChunk() 2021-10-04 10:00:27 +02:00
Laurent Broudoux
77eaa0a293 DBZ-1297 Add boolean type in test 2021-10-01 22:29:53 +02:00
Laurent Broudoux
7b7a1f3064 DBZ-1297 Expand JSON escaped string as JSON 2021-10-01 22:29:53 +02:00
Jiri Pechanec
f910fc930b [release] Switch to 1.8 snapshot 2021-10-01 14:50:16 +02:00
Debezium Builder
12b0ae9046 [maven-release-plugin] prepare for next development iteration 2021-10-01 13:26:33 +02:00
Debezium Builder
652162fd85 [maven-release-plugin] prepare release v1.7.0.Final 2021-10-01 13:26:33 +02:00
Jiri Pechanec
a2c6058404 DBZ-4057 Initialize Postgres schema before incremental snapshot restart 2021-09-30 09:10:49 +02:00
Debezium Builder
bec0b6b67e [maven-release-plugin] prepare for next development iteration 2021-09-23 06:31:25 +00:00
Debezium Builder
cf8da39725 [maven-release-plugin] prepare release v1.7.0.CR2 2021-09-23 06:31:25 +00:00
Naveen Kumar KR
62045f6430 DBZ-3887 1) Replace notifyAll with notify as atmost only one thread would be in wait state at any time 
2) Remove synchronized from jmh metric methods as it doesn't strictly require lock due to nature of metrics
3) Always wait by predefined timeout to gurantee that thread would never stuck
 2021-09-22 19:12:21 +02:00
Naveen Kumar KR
858557d184 DBZ-3887 Major improvements for Throughput Bottleneck and Inefficient Batching in ChangeEventQueue 
* Replaced naive metronome.pause and Thread.sleep with efficient wait() and notifyAll()
* Replaced LinkedBlockingDeque with ArrayDeque for better performance
* Replaced Atomic and Map operation related to maxQueueSize with primitive long and ArrayDeque for better performance

* x10 - x100 Improvements in overall throughput
* Efficient batching based on maxBatchSize
 2021-09-22 19:12:21 +02:00
Kate
0099a50c53 DBZ-3948 Support kafka topic based signals 2021-09-22 13:08:45 +02:00
Vivek Wassan
b6e891ec7d DBZ-3951 Moving preReadChunk() to AbstractIncrementalSnapshotChangeEventSource 2021-09-22 07:01:29 +02:00
Vivek Wassan
65ac2387dc DBZ-3951 Adding a separate isValid method 2021-09-22 07:01:29 +02:00
Vivek Wassan
5f37e6630f DBZ-3951 Adding a safety check, isValid, for connection 2021-09-22 07:01:29 +02:00
Naveen Kumar KR
b5ccd249c9 DBZ-4015 1) TableIdParser - Initialize ArrayList with optimal capacity of 3 (at max three parts in an identifier - catalog.schema.table) 
2) TableId - Replace .stream().toArray() with .toArray() to eliminate stream overhead
 2021-09-21 10:35:08 +02:00
Naveen Kumar KR
4c5264f7d2 DBZ-4015 Efficient String Replacement using String.replace in TableIdParser (improvement of x4-x5 times) 
(https://medium.com/javarevisited/micro-optimizations-in-java-string-replaceall-c6d0edf2ef6)
 2021-09-21 10:35:08 +02:00
Jiri Pechanec
26d0e78023 DBZ-4013 Skip invalid table 2021-09-17 14:20:42 +02:00
Debezium Builder
c15e0b303b [maven-release-plugin] prepare for next development iteration 2021-09-16 08:10:45 +00:00
Debezium Builder
23d9790121 [maven-release-plugin] prepare release v1.7.0.CR1 2021-09-16 08:10:44 +00:00
Anisha Mohanty
f57545d7b1 DBZ-3787 Add tests for invisible & visible columns using testcontainers 
*Using user "debezium" to have right grants provided
 2021-09-15 15:13:18 +02:00
René Kerner
b92e5c5efe
DBZ-3943 Add ConfigDef unit tests to verify all Debezium Field instances have a proper documentation/description set; 
Also adding missing descriptions.
 2021-09-09 13:35:41 +02:00
Gunnar Morling
6f1577a70b DBZ-3958 Gracefully handling history file in sym-linked directory; 
Also removing redundant test method definition.
 2021-09-09 12:18:21 +02:00
Gunnar Morling
7c150de943 DBZ-2525 Don't qualify selected columns with table name; 
It shouldn't be needed as we select from a single table, and it caused the columns to not be found for the Db2 connector.
 2021-08-31 17:39:36 +05:30
Anisha Mohanty
153e0920e1 DBZ-2525 Generifying exclusion of columns from snapshotting in all relational connectors; 
* Using RelationalDatabaseSchema in RelationalSnapshotChangeEventSource for consistency.
* Adding test for table/column names which are keywords
 2021-08-30 12:20:56 +02:00
Jiri Pechanec
a36eb45dff
DBZ-3912 Added Binary padding for hex/base64 handling mode 
Co-authored-by: Gunnar Morling <gunnar.morling@googlemail.com>
 2021-08-27 09:48:19 +02:00
Debezium Builder
2ce6bcb2ab [maven-release-plugin] prepare for next development iteration 2021-08-25 05:20:56 -04:00
Debezium Builder
d3a7efe423 [maven-release-plugin] prepare release v1.7.0.Beta1 2021-08-25 05:20:56 -04:00
Jiri Pechanec
4396513795 DBZ-3901 Guard against chunk signal coming when none expected 2021-08-24 10:26:32 +02:00
rkerner
52333596de DBZ-3364 add additional metadata to fields for API spec generator for the Debezium UI 
part of https://issues.redhat.com/browse/DBZ-3364
DBZ-2653 User rollback from parent class
 2021-08-23 17:57:38 +02:00
Chris Cranford
973fa248ed DBZ-3902 Fix test failure shouldNotEmitDdlEventsForNonTableObjects 2021-08-23 11:44:14 -04:00
Gunnar Morling
36af61b789 DBZ-3889 Moving ContainerImageVersions to core for broader re-use and using it in two more places 2021-08-23 12:34:03 +02:00
Sergei Morozov
b06b5aecbc DBZ-2975: Introduce opt-in configuration for multi-partition mode 2021-08-23 12:01:11 +02:00
Jiri Pechanec
11e3ef4325 DBZ-3857 Exclude database name in snapshot tables filtering 2021-08-16 14:12:53 +02:00
Gunnar Morling
c2b13037fb DBZ-3444 Upgrading to Apache Kafka 2.8 2021-08-12 11:57:57 +02:00
Gunnar Morling
9b0f784fcd DBZ-3444 Removing unused dependency to Apache Curator 2021-08-12 11:57:57 +02:00
Jiri Pechanec
a4fba49184 DBZ-3688 Expose min/max key for incrementally snapshotted table 2021-08-11 14:40:15 +02:00
Chris Cranford
46590a3bc6 DBZ-3839 Simplify version checks 2021-08-11 07:35:57 +02:00
Chris Cranford
c15cf81e18 DBZ-3839 Correctly skip tests with minor/patch not specified 2021-08-11 07:35:57 +02:00
Sergei Morozov
a40ad7132d DBZ-2975: Remove partition from offset context 2021-08-11 07:34:37 +02:00
Sergei Morozov
77e2ac8a12 DBZ-2975: Introduce io.debezium.pipeline.spi.Offsets 2021-08-11 07:34:37 +02:00
Sergei Morozov
c7ac5fca52 DBZ-2975: Move Partition to io.debezium.pipeline.spi 2021-08-11 07:34:37 +02:00
Chris Cranford
85a732d7dc DBZ-3842 Validate keySchema is null and not valueSchema 2021-08-10 11:15:55 +02:00
pkgonan
fae4f45acc DBZ-3765 Add dot pattern test code for database.server.name 2021-08-08 16:25:15 -04:00
pkgonan
5e0ffa28d2 DBZ-3765 Allow dot in database.server.name 2021-08-08 16:25:15 -04:00
Debezium Builder
d3c4482d06 [maven-release-plugin] prepare for next development iteration 2021-07-30 02:39:08 -04:00
Debezium Builder
35dcfc5426 [maven-release-plugin] prepare release v1.7.0.Alpha1 2021-07-30 02:39:08 -04:00
Chris Cranford
26461bc07d DBZ-3692 Introduce completion callback for incremental snapshots 2021-07-29 13:51:42 +02:00
Chris Cranford
393a44a55f DBZ-3692 Oracle incremental snapshot support 2021-07-29 13:51:42 +02:00
Kate
9b66ab374a DBZ-3577 Support read-only MySQL incremental snapshots 2021-07-29 07:10:46 +02:00
Gunnar Morling
09ebc833f3 DBZ-3655 Gracefully handling DB shutdown 2021-07-27 10:21:54 +02:00
Anisha Mohanty
bd47fecdbd DBZ-3717 Throw MultipleParsingExceptions to handle multiple parsing errors & smooth running of database history 2021-07-22 14:12:38 -04:00
Hossein Torabi
0ad8ededa3 DBZ-3725 Validate table before start incemental processing 2021-07-21 14:53:22 -04:00
Gunnar Morling
b84c3c870d DBZ-3668 Extracting SPI package 2021-07-03 16:31:52 +07:00
Gunnar Morling
dcb7336902 DBZ-3668 Making provoiders map unmodifiable 2021-07-03 16:31:52 +07:00
Chris Cranford
2e84427603 DBZ-3668 Consolidate ServiceLoader lookup 2021-07-03 16:31:52 +07:00
Chris Cranford
7f7e23b110 DBZ-3668 Add ServiceLoader hook CloudEvents converter 2021-07-03 16:31:52 +07:00
Chris Cranford
219ed17a28 DBZ-3668 Add CloudEventsConverter support for Vitess connector 2021-07-03 16:31:52 +07:00
Chris Cranford
80a4e003d8 DBZ-3668 Add CloudEventsConverter support for Db2 connector 2021-07-03 16:31:52 +07:00
Chris Cranford
3f759a8daa DBZ-3668 Add CloudEventsConverter support for Oracle connector 2021-07-03 16:31:52 +07:00
Sergei Morozov
db105baeba DBZ-2975: Add partition awareness to source task components 
Co-authored-by: Mike Kamornikov <mikekamornikov@gmail.com>
 2021-07-03 10:56:34 +02:00
Sergei Morozov
0c0c4f5e33 DBZ-2975: Introduce partition-aware API components 2021-07-03 10:56:34 +02:00
Gunnar Morling
5b76bf1b71 DBZ-3632 Pulling up getJdbcConfig() to RelationalDatabaseConnectorConfig 2021-07-01 15:11:01 +02:00
Debezium Builder
9d3f6b2a86 [maven-release-plugin] prepare for next development iteration 2021-06-30 03:45:06 -04:00
Debezium Builder
c8c5fc4b56 [maven-release-plugin] prepare release v1.6.0.Final 2021-06-30 03:45:06 -04:00
Jiri Pechanec
472b82abb7 DBZ-3622 Decouple database and schema history record writes 2021-06-29 10:36:41 +02:00
Jiri Pechanec
443c0ce089 DBZ-3622 Fix non-captured database storing; make the code more robust 2021-06-29 10:36:41 +02:00
Debezium Builder
ae053414ac [maven-release-plugin] prepare for next development iteration 2021-06-24 01:50:07 -04:00
Debezium Builder
af3f942a74 [maven-release-plugin] prepare release v1.6.0.CR1 2021-06-24 01:50:07 -04:00
Chris Cranford
da2e142460 DBZ-3593 Only emit commit event if there is an active transaction 2021-06-17 09:25:30 +02:00
Debezium Builder
48ca58a609 [maven-release-plugin] prepare for next development iteration 2021-06-10 06:03:12 -04:00
Debezium Builder
0961517859 [maven-release-plugin] prepare release v1.6.0.Beta2 2021-06-10 06:03:12 -04:00
Gunnar Morling
205c63a56a DBZ-3608 Fixing version handling; 
- wrong minor version for Postgres
- minor/patch versions of -1 were not actually ignored in comparison
 2021-06-10 07:39:49 +02:00
Gunnar Morling
dc865cd82d DBZ-2975 Formatting 2021-06-09 16:02:33 +02:00
Gunnar Morling
1ce65fa628 DBZ-2975 Avoiding raw types in some places; 
Also parameterizing offset loader contract, so to avoid casts later on.
 2021-06-09 16:02:33 +02:00
Sergei Morozov
657a6d2542 DBZ-2975: Extract offset context from object states to method signatures 2021-06-09 16:02:33 +02:00
Jiri Pechanec
06ceac1b24 DBZ-3607 Document incremental snapshot SPI 2021-06-09 14:15:09 +02:00
Sergei Morozov
92e0488874 DBZ-3562: Allow hyphen in database.server.name 2021-05-27 11:03:30 +02:00
Sergei Morozov
3b72e5149c DBZ-3486: Replace @see with @link where a URL is referenced 2021-05-25 07:19:43 +02:00
Debezium Builder
ead4ee7226 [maven-release-plugin] prepare for next development iteration 2021-05-20 19:07:44 +02:00
Debezium Builder
9e16df7d4b [maven-release-plugin] prepare release v1.6.0.Beta1 2021-05-20 19:07:44 +02:00
Jiri Pechanec
ba581ed323 DBZ-3500 Added Javadoc 2021-05-20 08:06:37 +02:00
Jiri Pechanec
cbaf13841f DBZ-3500 Incremental snapshots for Db2 2021-05-20 08:06:37 +02:00
Jiri Pechanec
258757410d DBZ-3500 Fix tests 2021-05-20 08:06:37 +02:00
Jiri Pechanec
4929397022 DBZ-3500 Per-connector table name quoting 2021-05-20 08:06:37 +02:00
Jiri Pechanec
a879b7b3e1 DBZ-3500 Re-use ResultSet to Object code 2021-05-20 08:06:37 +02:00
Chris Cranford
d147d1f517
DBZ-3240 Remove database.tablename.case.insensitive deprecated option 2021-05-19 20:44:59 +02:00
Thomas Aregger
d1670b636c
DBZ-3456 Timezone difference not considered in LagFromSourceInMilliseconds calculation; 
* Retrieve v$logmnr_contents.change_time in UTC
* Respect time zones when calculating time difference
* Allow constructors to be annotated with VisibleForTesting
 2021-05-19 13:41:02 +02:00
Chris Cranford
b13cbb1077 DBZ-2916 Add support for Oracle DDL statement parsing 2021-05-18 14:58:20 +02:00
Jiri Pechanec
6086eaf642 DBZ-3469 Emit null keys without schema 2021-05-13 08:04:50 -04:00
Alfusainey Jallow
1a3a752ddb DBZ-3514 prevent configuration modification after creation 
Signed-off-by: Alfusainey Jallow <alf.jallow@gmail.com>
 2021-05-12 09:31:29 +02:00
Anisha Mohanty
64d2cde825 DBZ-3438 Add mapped diagnostic contexts for snapshotting & streaming 2021-05-11 11:43:58 +02:00
Debezium Builder
4dd3a146ca [maven-release-plugin] prepare for next development iteration 2021-05-06 02:40:15 -04:00
Debezium Builder
b3c5bb281c [maven-release-plugin] prepare release v1.6.0.Alpha1 2021-05-06 02:40:15 -04:00
Jiri Pechanec
5c722341e2 DBZ-3473 Extract incremental snapshot contract 2021-05-06 05:37:36 +02:00
Jiri Pechanec
7daae11600 DBZ-3473 Postgres has its own schema instance 2021-05-06 05:37:36 +02:00
Jiri Pechanec
13334ee72f DBZ-3473 Add current chunk metric 2021-05-06 05:37:36 +02:00
Jiri Pechanec
39d21806eb DBZ-3473 Metrics enabled 2021-05-06 05:37:36 +02:00
Jiri Pechanec
3d940ca14a DBZ-3473 Support for cursor-based fetching 2021-05-06 05:37:36 +02:00
Jiri Pechanec
c53785dde7 DBZ-3473 Incremental Snapshot support for MySQL 2021-05-06 05:37:36 +02:00
Jiri Pechanec
d10bf3f7ef DBZ-3473 Log chunk id 2021-05-06 05:37:36 +02:00
Jiri Pechanec
49c5a00bbf DBZ-3473 Share read JDBC value between regular and incremental snapshot 2021-05-06 05:37:36 +02:00
Jiri Pechanec
91ecbcc81a DBZ-3473 Share read statement creation between regular and incremental snapshot 2021-05-06 05:37:36 +02:00
Jiri Pechanec
dd2bc6354b DBZ-3473 Pass the main clock form connector 2021-05-06 05:37:36 +02:00
Jiri Pechanec
3371e6ccb3 DBZ-3473 Enable per-dialect query limit 2021-05-06 05:37:36 +02:00
Jiri Pechanec
260c32cf0e DBZ-3473 Resume parallel snapshot after connector restart 2021-05-06 05:37:36 +02:00
Jiri Pechanec
4d52b21a76 DBZ-3473 Ignore parallel snapshot for connector not supporting it 2021-05-06 05:37:36 +02:00
Jiri Pechanec
fd9c9d02b4 DBZ-3473 Handle exceptions 2021-05-06 05:37:36 +02:00
Jiri Pechanec
37626c0dfe DBZ-3473 Extract incremental snapshot context 2021-05-06 05:37:36 +02:00
Jiri Pechanec
07371c085b DBZ-3473 Watermark signals must be in separate tx 2021-05-06 05:37:36 +02:00
Jiri Pechanec
a52c536a68 DBZ-3473 First prototype of incremental snapshotting 2021-05-06 05:37:36 +02:00
Anisha Mohanty
a6b54c7000 DBZ-3427 Validate the server name field 2021-04-28 13:18:52 +02:00
Gunnar Morling
8f891248ad DBZ-3429 Making option retrieval lenient towards trailing/leading whitespace 2021-04-28 11:01:27 +02:00
kyleyj
99536f21d2 DBZ-3429 Support spaces in the SQL Server properties 
Many of the SQL Server properties support spaces in the comma-separated lists.
However, the snapshot.select.statement.overrides property doesn't support
spaces after the commas.
 2021-04-28 10:23:02 +02:00
Gunnar Morling
91b784c3b6 DBZ-3161 Avoid failing when legacy option is given; also 
* adding all public history options on config surface
* adding missing whitespace in field docs
* recommending non-deprecated option in docs
 2021-04-26 13:07:23 +02:00
Anisha Mohanty
11fb079582 DBZ-3161 Add database.history.store.only.captured.tables.ddl option & CapturedTables metrics to documentation 2021-04-26 13:07:23 +02:00
Anisha Mohanty
f0f8d5d180 DBZ-3161 Refactor/Rename all references of MonitoredTables to CapturedTables 2021-04-26 13:07:23 +02:00
Anisha Mohanty
062c1c3860 DBZ-3161 Add new option for CapturedTables, deprecate existing MonitoredTables option & emit warnings 2021-04-26 13:07:23 +02:00
Anisha Mohanty
4a86330c0a DBZ-3161 Add new CapturedTables metrics 2021-04-26 13:07:23 +02:00
Gunnar Morling
42a7b74725 DBZ-3403 Misc. improvements 2021-04-14 11:14:07 +02:00
Hossein Torabi
ba048cf47d DBZ-3403 Semi Optimisation on MongoDB and MySQL connector for skipped.operations 2021-04-14 11:14:07 +02:00
Hossein Torabi
5d6705c3fe DBZ-2699 implement skipped.operation in core 2021-04-08 11:59:08 +02:00
Debezium Builder
8a88ba9b45 [maven-release-plugin] prepare for next development iteration 2021-04-08 04:04:00 -04:00
Debezium Builder
74152afe74 [maven-release-plugin] prepare release v1.5.0.Final 2021-04-08 04:04:00 -04:00
Chris Egerton
67f044ff87 DBZ-3371: Adjust scale of fallback BigDecimal values to match column schema 2021-04-06 10:35:23 +02:00
Jiri Pechanec
f50c3e8222 DBZ-3361 Prefer DDL over logical schema for MySQL 2021-03-30 11:35:12 +02:00
Chris Cranford
776180a0df DBZ-3347 Emit schema change events with correct source partition 2021-03-30 07:26:06 +02:00
Jiri Pechanec
a02e530c80 DBZ-2911 Use sequence only when available 2021-03-26 09:19:17 +01:00
JLDLaughlin
06b0475f17 DBZ-2911 Add additional LSN information to all sources 
Adds a new "sequence" field to all sources via the AbstractSourceInfo
class. Specifically, this field is required to deduplicate records
from PostgreSQL sources in O(1) time. The sequence field is a stringified
list of metadata. For PostgreSQL sources, this will include the last
committed LSN and the current LSN. For other sources, this will be NULL
until implemented. A new integration test was added for PostgreSQL.
 2021-03-26 09:19:17 +01:00
Debezium Builder
6b4d88bb51 [maven-release-plugin] prepare for next development iteration 2021-03-24 06:35:02 -04:00
Debezium Builder
c578715233 [maven-release-plugin] prepare release v1.5.0.CR1 2021-03-24 06:35:02 -04:00
Gunnar Morling
c7b7d5303a DBZ-3218 Avoiding duplicated validation (again ;) 2021-03-19 15:04:20 +01:00
Gunnar Morling
178140ae9c DBZ-3218 Removing superfluous method parameter 2021-03-19 15:04:20 +01:00
Anisha Mohanty
c761d92b2a DBZ-3218 Add separate method for logging warnings 2021-03-19 15:04:20 +01:00