alexey.rassokhin/tet123
1
0
Fork 0
Code Issues Pull Requests 1 Packages Projects Releases Wiki Activity
7,840 Commits 3 Branches 0 Tags 51 MiB
f0a1fa8a67
Commit Graph

1561 Commits

Author SHA1 Message Date
Jiri Pechanec
70d5b2db50 DBZ-5639 Inform about snapshot reexcution 2022-09-22 13:14:28 +02:00
harveyyue
3b4d19c174 DBZ-5639 Logging enhancement for non-incremental snapshot in postgres connector 2022-09-22 13:14:28 +02:00
Ismail Simsek
b966f4b65b DBZ-5591 Snapshot process setting source.ts_ms 2022-09-20 17:12:54 -04:00
Jiri Pechanec
29fcb0e047 DBZ-5585 Add last resort classloader 2022-09-19 17:15:29 +02:00
harveyyue
288c93434f DBZ-5585 Remove classloader parameter from JdbcConnection 2022-09-19 17:15:29 +02:00
harveyyue
20d56734d4 DBZ-5585 Modify the Instantiator to not require classloader 2022-09-19 17:15:29 +02:00
Vojtech Juranek
2a0e2a7e81 DBZ-5603 Move test containers to quay.io 2022-09-19 09:12:46 +02:00
Debezium Builder
218d414552 [maven-release-plugin] prepare for next development iteration 2022-09-16 06:36:40 +00:00
Debezium Builder
567f57c47b [maven-release-plugin] prepare release v2.0.0.Beta2 2022-09-16 06:36:39 +00:00
harveyyue
09b99840d8 DBZ-4617 Deprecate internal key/value converter options 2022-09-14 12:49:30 +02:00
Vojtech Juranek
8b8d4828f3 DBZ-5043 Make topic prefix field manadatory and with proper group 2022-09-14 12:44:38 +02:00
harveyyue
665edab844 DBZ-5589 Mysql connector can't handle the case sensitive of rename/change column statement 2022-09-13 09:36:25 +02:00
Vojtech Juranek
acc6be6df3 DBZ-5043 Move TOPIC_PREFIX into CommonConnectorConfig 
`TOPIC_PREFIX` is now mandatory to all connectors therefore it make
sense to have it in common config. Beside that, it also makes it more
easy to use it in Debezium UI without any workarounds - if the field is
not member of the given connector config, the field has to be
explicitely added into known fields otherwise is invisible for UI.
 2022-09-13 09:25:54 +02:00
Anisha Mohanty
f41b6dd458 DBZ-5044 Get rid of schema factory instances 2022-09-08 10:35:20 +02:00
Anisha Mohanty
b0dc7e557a DBZ-5044 Update schemaHistoryColumnSchema to add new fields 2022-09-08 10:35:20 +02:00
Anisha Mohanty
ec792527e9 DBZ-5044 Use SchemaFactory for centralizing Postgres schemas 2022-09-08 10:35:20 +02:00
Anisha Mohanty
cfd0242e28 DBZ-5044 Sort imports & fix test failures 2022-09-08 10:35:20 +02:00
Anisha Mohanty
91ccb7578f DBZ-5044 Centralize datatype schemas 2022-09-08 10:35:20 +02:00
Jiri Pechanec
77efd818da DBZ-5044 More descriptive names for schema history schemas 2022-09-08 10:35:20 +02:00
Jiri Pechanec
b8bcac1aef DBZ-5044 Ignore versions for schema comparison 2022-09-08 10:35:20 +02:00
Jiri Pechanec
fa2a7ba60c DBZ-5044 Fix transaction metadata schema 2022-09-08 10:35:20 +02:00
Anisha Mohanty
aea7c1ffa6 DBZ-5044 Use SchemaFactory for ConnectTable & EvenyDispatcher schemas 2022-09-08 10:35:20 +02:00
Anisha Mohanty
8fb2ffd70a DBZ-5044 Add review suggestions 2022-09-08 10:35:20 +02:00
Jiri Pechanec
cd07b8a06d DBZ-5044 Rename and reorder schema attributes 2022-09-08 10:35:20 +02:00
Anisha Mohanty
1a55869f5b DBZ-5044 Reword schema names 2022-09-08 10:35:20 +02:00
Anisha Mohanty
867ff0a117 DBZ-5044 Use seperate version for each schema 2022-09-08 10:35:20 +02:00
Anisha Mohanty
ec0ac3fea5 DBZ-5044 Centralize creation of Message Schemas 2022-09-08 10:35:20 +02:00
Vojtech Juranek
402df78381 DBZ-5043 Rename schema.history to schema.history.internal 2022-09-07 12:57:31 +02:00
Vojtech Juranek
b6c31f3da5 DBZ-5043 Replace 'database history' with 'database schema history' 
Reflect renaming database history to schema history in comments,
configuration descriptions and log and error messages.
 2022-09-07 12:57:31 +02:00
Vojtech Juranek
5bffd7839a DBZ-5043 Change default schema history suffix 
Change default schema history name from `$LOGICAL_NAME-dbhistory` to
`$LOGICAL_NAME-schemahistory`.
 2022-09-07 12:57:31 +02:00
Vojtech Juranek
0494a16a79 DBZ-5043 Rename DatabaseHistory to SchemaHistory 
Rename also all related classes and varibales.
 2022-09-07 12:57:31 +02:00
Vojtech Juranek
0823cedf25 DBZ-5043 Replace database.server.name with topic.prefix 2022-09-07 12:57:31 +02:00
Vojtech Juranek
a7a293c5c8 DBZ-5043 Move driver pass-through parameters into dedicated namespace 
Currently we mix database parameters like DB hostname, username or
password with other JDBC driver parameters which are passed to JDBC
driver as it is. Move these pass-through parameters into dedicated
namespace `driver.*` to separate them from common database parameters.
JDBC parameters which are defined on the connector level still stay
in `database` namespace.
 2022-09-07 12:57:31 +02:00
Vojtech Juranek
5d1b72e95f DBZ-5043 Simplify code after renaming database.history namespace 
Filtering `database.history` namespace from pass-through parameters is now
not needed as it doesn't interfere with other `database.*` parameters.
 2022-09-07 12:57:31 +02:00
Vojtech Juranek
25aa6c4acc DBZ-5043 Change namespace to database.history to schema.history 2022-09-07 12:57:31 +02:00
harveyyue
48d37857b6 DBZ-5588 Field validation errors are misleading for positive, non-zero expectations 2022-09-07 09:18:38 +02:00
harveyyue
58f4d3d912 DBZ-5480 Augment a uniqueness key filed/value in regex topic naming strategy 2022-09-06 14:58:57 +02:00
Claus Ibsen
1f48bfc515
[docs] Typo in description (remove space before dot) 2022-09-06 09:48:57 +02:00
Alexey Loubyansky
c4b95f4ea2 DBZ-5561 Use TCCL to load configuration classes by default 2022-09-06 08:42:45 +02:00
Vadzim Ramanenka
917de37a01 DBZ-5557: Supply partition when comiting offsets with source database 
Keep track of partitions offsets being committed belong to in
`BaseSourceTask`. Supply the partition along with the offset in calls to
`commitOffset`.
 2022-09-05 14:04:23 +02:00
Jiri Pechanec
6e53798407 DBZ-5538 Remove duplicated formatting message 2022-09-05 10:35:55 +02:00
harveyyue
f4c7b52451 DBZ-5538 Unclear validation error when required field is missing 2022-09-05 10:35:55 +02:00
ggaborg
664fd9e1b0 DBZ-3293 unit tests for filters 2022-09-01 20:16:47 +02:00
ggaborg
244de2bd96 DBZ-3293 Add signal table automatically to include list 2022-09-01 20:16:47 +02:00
thangdc94
57210095d2 DBZ-5439 Fix testcase 2022-09-01 09:32:17 +02:00
Jiri Pechanec
de53591dba DBZ-5244 Retry all communication exceptions by default 2022-08-31 09:45:09 -04:00
harveyyue
7ccd40129c DBZ-5569 Oracle connector's max.queue.size.in.bytes is invalid 2022-08-31 09:39:29 -04:00
Jiri Pechanec
1c291d75fd DBZ-5568 Use only Java 11 for tests 2022-08-31 09:19:23 +02:00
pkgonan
fd309193c6 DBZ-5544 Add document 2022-08-24 09:50:23 +02:00
pkgonan
b031770300 DBZ-5544 Support BASE64_URL_SAFE in BinaryHandlingMode 2022-08-24 09:50:23 +02:00
tjwornjs
ca009d8492 DBZ-5489 add column comment to schema 2022-08-24 09:46:01 +02:00
Anisha Mohanty
669024ed46 DBZ-5484 Better error reporting in SKIP/WARN cases 2022-08-23 10:02:29 -04:00
harveyyue
0308a1cbb6 DBZ-5535 Disable preferring DDL before logical schema in history recovery 2022-08-23 14:16:35 +02:00
Jiri Pechanec
34b78e6bce DBZ-2766 Make queue more generic 2022-08-23 13:43:41 +02:00
Jiri Pechanec
f770e0afb1 DBZ-2766 Remove Guava dependency 2022-08-23 13:43:41 +02:00
ggaborg
bf20a9c70f DBZ-5533 Unit test fails on Windows 2022-08-19 09:44:26 -04:00
Sergei Morozov
1c99437890 DBZ-5470: Remove the dependency of JdbcConnection on DatabaseSchema 2022-08-19 11:06:59 +02:00
Chris Cranford
37b97970a6 DBZ-5497 Skip truncate operations by default 
* Promote truncate skip default from PostgreSQL to core library
* Adjust behavior for Oracle with truncates skipped by default
 2022-08-19 11:05:29 +02:00
Vivek Wassan
d89e71ebeb DBZ-5327 Smart Backfills 2022-08-17 10:05:55 +02:00
harveyyue
a08420b21c DBZ-5511 Keep consistence with method serialize of Connect/Json Table Change Serializer 2022-08-16 16:58:07 +02:00
Vojtech Juranek
8e0ad245e5 DBZ-5475 Ensure Outbox JSON NUMBER arrays are of the same type 2022-08-05 11:00:46 -04:00
Vojtech Juranek
7256f62f6b DBZ-5475 Remove unneeded checks from Outbox payload conversion 
* remove float check as we return `float64` by default anyway
* simplify/narrow the checks when detemining array type, also add a test
  for array having the first element `null`
 2022-08-05 11:00:46 -04:00
Chris Cranford
d4261bd2fd DBZ-5476 Fix record accumulator during statistics log entry 2022-08-04 15:28:46 -04:00
harveyyue
2481172309 DBZ-5471 Missing the regex properties validation before start connector of DefaultRegexTopicNamingStrategy 2022-08-04 10:49:29 -04:00
Vojtech Juranek
c7d8353d7b DBZ-4727 Add snapshot paused JXM metrics 
Add two JXM objects:
* `snapshotPaused` - detemines if the incremental snapshot is paused
  or not
* `snapshotPausedDurationInSeconds` - overall time when the incremental
  snapshot was paused. The time adds up - if the snapshot was paused
  e.g. two times, the `snapshotPausedDurationInSeconds` is the sum of
  these two paused times.
 2022-07-29 12:41:58 -04:00
Vojtech Juranek
ed504173e1 DBZ-4727 Implement incremental snaphost pause/resume 
Intorduce two new signals:
* `pause-snapshot` - pauses running incremental snapshot
* `resume-snapshot` - resumes paused incremental snapshot

If the incremental snapshot is running and pause signal is sent,
currently processed chunk is finished and then no other chunk is read.
Snapshot state is kept in running state. Once snapshot is resume,
chunk is reverted to position which was last sent to the broker and
process of reading chunks and emitting `read` events continues until
all tables are read and sent.

This patch handles only table-based signals. MySQL Kafka-based signals
will be addressed later.
 2022-07-29 12:41:58 -04:00
harveyyue
5f430f1a8e DBZ-5457 Deprecate legacy topic selector for all connectors 2022-07-28 12:58:59 -04:00
ggaborg
2c8d71a36f DBZ-5452 Fix unit test failing on Windows platform 2022-07-26 15:50:28 -04:00
Debezium Builder
1ab51ae8d0 [maven-release-plugin] prepare for next development iteration 2022-07-26 18:13:10 +00:00
Debezium Builder
d247a48b79 [maven-release-plugin] prepare release v2.0.0.Beta1 2022-07-26 18:13:09 +00:00
harveyyue
7730610320 DBZ-5425 Remove the duplicated SimpleDdlParserListener from mysql connector 2022-07-20 14:35:34 +02:00
Chris Cranford
75ad893d67 DBZ-5396 Scope attributes inside table document 2022-07-19 15:41:02 -04:00
Chris Cranford
76dafedea7 DBZ-5396 Add multi-type accessors for attribute values 2022-07-19 15:41:02 -04:00
Chris Cranford
f412c2e6a4 DBZ-5396 Add support for connector-specific relational model attributes 2022-07-19 15:41:02 -04:00
harveyyue
daf329bc06 DBZ-5413 Update the topic naming strategy doc to all connectors 2022-07-18 10:18:12 -04:00
harveyyue
77c7db163d DBZ-5414 Validate topic naming strategy relative topic name properties 2022-07-18 09:58:51 -04:00
Chris Cranford
3acf317668 DBZ-5404 Restore build compatibility with Kafka 2.x and 3.0.x 2022-07-14 08:40:42 +02:00
harveyyue
6ea7abc8fc DBZ-4180 Pluggable topic selector 2022-07-13 14:45:25 +02:00
Jiri Pechanec
d8baac7c46 DBZ-5229 Reorganize code for inheritance use 2022-07-12 17:31:19 +02:00
Hossein Torabi
1ea02be9c2 DBZ-5229 define centralized and modular aproach for debezium storage 2022-07-12 17:31:19 +02:00
Chris Cranford
d2a76a7ffc DBZ-5295 Reselect LOB columns on primary key update 2022-07-12 11:59:34 +02:00
Chris Cranford
7b08fb8d3d DBZ-5356 Rewrite loop 2022-07-12 11:01:10 +02:00
Chris Cranford
deaa34d71e DBZ-5356 Avoid NPE during index evaluation 2022-07-12 11:01:10 +02:00
harveyyue
0842e5cd0e DBZ-5366 MySQL Connector column hash v2 does not work 2022-07-06 14:23:57 -04:00
Paweł Malon
707a43d3bc DBZ-5367 Fixed the problem where JSON expanding failed for Outbox event payloads that contained nested arrays where first array was empty and further array contained a Struct 
With proposed changes empty arrays are skipped, as it allows us to delay defining the schema for them until an array with elements is found, as it seems that there is no way to modify a schema that was already defined for a given field
 2022-07-06 13:47:54 -04:00
Debezium Builder
6fd1441af3 [maven-release-plugin] prepare for next development iteration 2022-07-01 09:21:49 +00:00
Debezium Builder
99402af634 [maven-release-plugin] prepare release v2.0.0.Alpha3 2022-07-01 09:21:48 +00:00
Vojtech Juranek
ac06623459 DBZ-5240 Move table types into separate method 
This will allow connectors to override the default values and return
types the connector supports.
 2022-06-30 10:46:54 +02:00
Gunnar Morling
cb53332561 DBZ-5335 Formatting 2022-06-30 09:54:13 +02:00
Mickael Maison
5ff71aa992 DBZ-5335 Fix formatting issues in the connectors configdefs 2022-06-30 09:35:54 +02:00
harveyyue
20dc353262 DBZ-5170 Mysql Commit Timestamp 2022-06-27 15:50:52 +02:00
Naveen Kumar KR
b75c9e635b DBZ-5277 ChangeEventQueue - Support Multiple Writer (doEnqueue) and Reader (poll) Threads 
* ReentrantLock based write and read coordination
* Condition based Inter Thread communication for isFull and isNotFull conditions
 2022-06-27 15:37:00 +02:00
Naveen Kumar KR
823a49fff4 DBZ-5277 ChangeEventQueue Test with Multiple Writer and Reader Threads 
Assertion would fail if one or both of the scenario occurs

1) Deadlock between Writer and Reader Threads
2) Thread Starvation between Writer and Reader Threads
 2022-06-27 15:37:00 +02:00
Jiri Pechanec
9ada2f28ce DBZ-4902 Change for exponential strategy 2022-06-27 14:29:51 +02:00
kanha gupta
1b17372d2d DBZ-4902 Replace long durations with Duration 2022-06-27 14:29:51 +02:00
Chris Cranford
ef99595d91 DBZ-5270 Reduce table schema capture logging verbosity 
When enabling the `database.history.store.only.captured.tables.ddl`,
the logging in RelationalSnapshotChangeEventSource does not take into
account this setting when logging, leading to a false impression of
what tables will actually be captured in the schema history.
 2022-06-27 08:47:34 +02:00
Jiri Pechanec
5e1ae24e92 DBZ-5272 Notify all threads in case of mutliple writers 2022-06-21 07:08:39 +02:00
Chris Cranford
3fb5bf1339 DBZ-4451 Introduce SchemaChangeEvent factory methods 2022-06-16 09:19:41 +02:00
Chris Cranford
863abcd081 DBZ-4451 Store previous table id in alter schema change events 
When a table is renamed with the ALTER TABLE statement, the schema
history record not only will reference the current table's unique id
in the "id" metadata, but will also refer to the old table name in
the "previousId" metadata field.
 2022-06-16 09:19:41 +02:00
Chris Cranford
20bb2adea5 DBZ-4834 Fix incremental snapshot of table added to include list 2022-06-16 08:39:16 +02:00