alexey.rassokhin/tet123
1
0
Fork 0
Code Issues Pull Requests 1 Packages Projects Releases Wiki Activity
9,015 Commits 3 Branches 0 Tags 51 MiB
6f04e9ce52
Commit Graph

1276 Commits

Author SHA1 Message Date
mfvitale
7429b4e6f1 DBZ-1973 Add incremental snapshot notifications 2023-05-10 12:22:02 +02:00
mfvitale
811b800f55 DBZ-1973 Add support for Notifications 
DBZ-1973 Add more tests for other connectors

DBZ-1973 Add send method with offset parameter

DBZ-1973 Instantiate NotificationService in the task class

DBZ-1973 Instantiate NotificationService in the task class

DBZ-4027 Move specific sink channel configuration to SinkNotificationChannel

DBZ-4027 Remove not used SPI file

DBZ-1973 Moved SPI file definition to debezium-core

DBZ-1973 Rename KafkaNotificationChannel to more generic SinkNotificationChannel

DBZ-1973 Code refactor

DBZ-1973 Improve configuration property description

DBZ-1973 Improve test

DBZ-1973 Add close method to NotificationChannel

DBZ-1973 Implement KafkaNotificationChannel

DBZ-1973 Add NotificationService and LogNotificationChannel
 2023-05-10 12:22:02 +02:00
kate
6c338754b3 DBZ-6440 Init Kafka signal channel 2023-05-09 06:48:39 +02:00
mfvitale
952340286e DBZ-4027 Make signalling channel configurable 
DBZ-4027 Add an Incremental snapshot test with kafka signaling

DBZ-4027 Add an Incremental snapshot test with kafka signaling

DBZ-4027 Add an Incremental snapshot test with kafka signaling

DBZ-4027 Code style

DBZ-4027 Make SignalPayload more generic and extensible

DBZ-4027 Rename DatabaseSignalChannel to SourceSignalChannel

DBZ-4027 Improve logging

DBZ-4027 Moved SPI file definition to debezium-core

DBZ-4027 Move SignalProcessor synchronization point to be processed only when a signal cdc event arrives.

DBZ-4027 Add EventDispatcher constructor without signalProcessor for spanner connector

DBZ-4027 Fix NPE

DBZ-4027 Fix NPE

DBZ-4027 Formatting

DBZ-4027 Correctly manage signal on not supported connector

DBZ-4027 Use the correct MongoDbOffset

DBZ-4027 Correctly initialize offset for Oracle and SqlServer connectors

DBZ-4027 Register SPI implementations

DBZ-4027 Improve SignalProcessor instantiation

DBZ-4027 Pass source info in case of SchemaChanges action

DBZ-4027 Manage close event in a synchronous way

DBZ-4027 Correctly init offset context also in case of snapshot mode 'never'

DBZ-4027 Fix MySqlMetricsIT test

DBZ-4027 Move KafkaSignalChannel to core

DBZ-4027 Move KafkaSignalChannel to core

DBZ-4027 Set pass offset context after initial snapshot to SignalProcessor

DBZ-4027 Pass OffsetContext to signal processor

DBZ-4027 Pass CommonConnectorConfig to SignalChannelReader init method

DBZ-4027 Move Incremental snapshot window actions to dedicated package

DBZ-4027 Align SignalsIT test with new code

DBZ-4027 Fix SignalsIT test

DBZ-4027 Fix SignalProcessor scheduling

DBZ-4027 Moved DatabaseSignalChannel and SignalChannelReader to dedicated package

DBZ-4027 Start SignalProcessor from ChangeEventSourceCoordinator

DBZ-4027 Create SignalProcessor and renamed Signal to DatabaseSignalChannel

DBZ-4027 Initial refactoring of signal feature
 2023-05-05 13:35:40 +02:00
Jiri Pechanec
58ef4f0b98 DBZ-6157 Disable potentially dangerous MySQL JDBC props 2023-05-04 11:54:04 +02:00
harveyyue
3860a4faca DBZ-6388 __source_ts_ms r (read) operation date is set to future for SQL Server 2023-05-02 08:11:02 -04:00
Nir Levy
f5555a894e DBZ-6357 Fix parsing of NATIONAL CHAR 2023-05-02 08:02:32 -04:00
ani-sha
a453c3ca19 DBZ-6259 Add missing description of various snapshot modes in the connector field definitions 2023-05-01 11:03:04 -04:00
Frederic Laurent
84b2e29698 DBZ-6340 Make preferred the new default SSL mode for the MySQL connector 2023-04-26 12:48:04 +02:00
hiscat
f07ec7ae5f DBZ-6361 Remove superfluous signal prefix 2023-04-19 13:08:18 +02:00
Jiri Pechanec
f4c200ac3e DBZ-2979 Fix tests 2023-04-19 09:04:29 +02:00
Ronak Jain
465b355dd0 DBZ-2979 If column.include.list/column.exclude.list are used and the target table receives an update for the excluded (or not included) column - such events should be ignored 
* Add More Test Cases

Addresses/Closes DBZ-2979
 2023-04-19 09:04:29 +02:00
Ronak Jain
762122ecf5 DBZ-2979 If column.include.list/column.exclude.list are used and the target table receives an update for the excluded (or not included) column - such events should be ignored 
* Add Mysql and SqlServer Connector Changes
* Change Docs
* Clean Test Case
* Formatting and Checkstyle

Addresses/Closes DBZ-2979
 2023-04-19 09:04:29 +02:00
Vojtech Juranek
44f2f40c56 DBZ-6216 Update testcontainers to use quay.io images 
When needed update also version.
Update also leftovers in readme files.
 2023-04-12 14:07:35 +02:00
harveyyue
c525f34cd0 DBZ-6070 DDL events not stored in schema history topic for excluded tables 2023-04-06 12:10:30 +02:00
Mark Bereznitsky
d4d116e45b DBZ-6271 add throwing of stack trace and a warn log 2023-04-04 06:57:57 +02:00
Mark Bereznitsky
ded4707108 DBZ-6271 allow for quiet failure of bin log retrieval 2023-04-04 06:57:57 +02:00
Mark Bereznitsky
4b41e0e649 DBZ-6271 allow binlog to be absent when doing a snapshot only 2023-04-04 06:57:57 +02:00
Chris Cranford
621ee17f81 DBZ-6225 DBZ-6226 DBZ-6231 Fix test failure 2023-03-29 08:27:03 +02:00
harveyyue
c9c1ce8a2f DBZ-6185 Zerofill property failed for different int types 2023-03-28 09:34:58 +02:00
Chris Cranford
1c5f14f7fd DBZ-6225 DBZ-6226 DBZ-6231 Introduce CustomConverter for JDBC sink consistency 
In DBZ-6225, it was identified that the schema type associated with TINYINT(1)
and BOOLEAN data types wasn't consistent. This converter implementation will
always emit BOOLEAN data types as INT16, similar to TINYINT(1).

In DBZ-6226, it was identified that the schema type associated with REAL data
types wasn't consistent, emitted as FLOAT64 during snapshot and FLOAT32 in
the streaming phase. This converter implementation will always emit such types
as FLOAT64 for consistency.

In DBZ-6231, during streaming the DDL parser will get the column type as the
literal type supplied by the user's DDL, i.e. NCHAR or NVARCHAR but when the
DDL is sourced from SHOW CREATE TABLE, MySQL returns these columns as types
of CHAR/VARCHAR but with a character set.  By passing the character_set in
the schema consistency, we can deduce nationalized vs non-nationalized.
 2023-03-21 14:53:56 +01:00
harveyyue
1877906dd8 DBZ-6197 Remove duplicated createDdlFilter method from historized connector config 2023-03-13 10:56:43 +01:00
PlugaruT
472162e46e DBZ-6023 Add support for providing a surrogate key when triggering incremental snapshots 
For tables that have a composite Primary Key, SQL queries for incremental snapshot perform bad. This change allows to provide a surrogate key that will be used for the queries. The key should be unique in order for the snapshot to guarantee consistency.
 2023-03-06 10:25:59 +01:00
Jiri Pechanec
d492ec892c DBZ-823 Use enum for enumeration 2023-03-03 12:47:15 +01:00
Jiri Pechanec
3da2434c6c DBZ-823 Restrict the type with generics 2023-03-03 12:47:15 +01:00
Jiri Pechanec
5e666f00ee DBZ-823 Add JavaDoc to new interfaces 2023-03-03 12:47:15 +01:00
Jiri Pechanec
7d92dc0211 DBZ-823 Don't use getter based name 2023-03-03 12:47:15 +01:00
Jiri Pechanec
0f4adb9fe6 DBZ-823 Use better name for CF with main connection 2023-03-03 12:47:15 +01:00
ggaborg
e875e784a0 DBZ-823 CR changes 2023-03-03 12:47:15 +01:00
ggaborg
065dc997b9 DBZ-823 Optionally parallelize initial snapshots 2023-03-03 12:47:15 +01:00
harveyyue
57b902ff8c DBZ-5743 Introduce alternative AVRO_UNICODE mode for field name selector 2023-02-02 13:21:45 +01:00
harveyyue
003bef0736 DBZ-5743 Introduce a new unicode topic naming strategy 2023-02-02 13:21:45 +01:00
Jiri Pechanec
c487d3aee2 DBZ-6059 Revert "DBZ-4663 Remove option for specifying driver class from MySQL Connector" 
This reverts commit 115d7c2d81.
 2023-01-31 13:14:01 +01:00
Liz Chatman
8a6fdab5d7 DBZ-2296 return false instead of true from isGtidModeEnabled for GTID_MODEs of OFF_PERMISSIVE and ON_PERMISSIVE 2023-01-26 07:43:13 +01:00
harveyyue
bd47d7396d DBZ-6003 Add more case to method to MySqlConnectorIT.shouldProcessCreateUniqueIndex 2023-01-26 07:39:28 +01:00
harveyyue
735b763075 DBZ-6003 Nullable columns marked with "optional: false" in DDL events 2023-01-26 07:39:28 +01:00
harveyyue
9ee39691bf DBZ-5798 Data type conversion failed for mysql bigint 2023-01-25 12:25:24 +01:00
harveyyue
0aed5d5fe9 DBZ-6040 Remove the redundancy "schema.history.internal" from MySqlConnectorConfig 2023-01-25 09:13:20 +01:00
Jiri Pechanec
acc9203c5c DBZ-6029 Use single regex 2023-01-25 06:10:18 +01:00
harveyyue
184e9725ab DBZ-6029 Unexpected format for TIME column: 8:00 2023-01-25 06:10:18 +01:00
Anil Dasari
e1f9d64fbb DBZ-5852:Snapshotter#snapshotCompleted is invoked regardless of snapshot result 2023-01-19 09:24:02 +01:00
Jeremy Ford
8da86b78e8 DBZ-5879 Fix unit tests 2023-01-18 14:46:14 +01:00
Jiri Pechanec
bcd6787cdc DBZ-6008 Revert "DBZ-3653 Introduce incremental.snapshot.isolation.mode config" 
This reverts commit 6d8bcd42fd.
 2023-01-17 10:40:20 +01:00
Jiri Pechanec
0c5e4f0051 DBZ-6002 Fix tests now requiring fractions 2023-01-13 12:00:00 +01:00
akshansh
66c9ab20c8 DBZ-5996 Removed * import 2023-01-11 12:09:26 +01:00
akshansh
444dab3ffa DBZ-5996 Handled case where fractionalWidth was -1 2023-01-11 12:09:26 +01:00
akshansh
b87d80f462 DBZ-5996 Implemented custom DateTimeFormatter to parse ZonedDateTime while preserving length of column. 2023-01-11 12:09:26 +01:00
harveyyue
5e56ad75f8 DBZ-5985 Disable to print invalid estimated row count when config "snapshot.select.statement.overrides" 2023-01-11 09:20:15 +01:00
Jochen Schalanda
44bcedaa72 DBZ-5993 Remove superfluous placeholder in log message (MySqlDatabaseSchema) 
The warning message emitted when the Debezium Connector for MySQL
encounters an unparseable DDL statement contains a superfluous
placeholder.
 2023-01-11 08:57:12 +01:00
govi20
6d8bcd42fd DBZ-3653 Introduce incremental.snapshot.isolation.mode config 2023-01-11 08:55:33 +01:00
Jiri Pechanec
8ca5e0a9b5 DBZ-4669 Reformat after plugin version consolidation 2023-01-11 06:44:53 +01:00
Mark Lambert
e5d16b14a6 DBZ-2439 Remove redundant modifiers 2023-01-11 06:44:53 +01:00
harveyyue
115d7c2d81 DBZ-4663 Remove option for specifying driver class from MySQL Connector 2023-01-06 09:23:34 -05:00
erdinc
0e8163daca DBZ-5976 Use as much as possible "replace" method rather than "replaceAll" 2023-01-05 07:30:49 +01:00
Raul Estrada
c1493c8e20 DBZ-3575 Comments review 2022-12-21 12:34:17 +01:00
Raul Estrada
282717fa55 DBZ-3575 Implement support for JSON_TABLE in MySQL parser 2022-12-21 12:34:17 +01:00
Raul Estrada
dd96ccc27d DBZ-5888 MariaDB-grammar 2022-12-14 14:42:29 +01:00
Vojtech Juranek
163ea307aa DBZ-5838 Skip Avro validation in tests which fail due to apicurio-registry #2980 2022-12-12 14:44:05 +01:00
mfvitale
4c941b63ed DBZ-5847: Add support for all connectors 
Support for qualified table names
 2022-12-12 14:40:56 +01:00
mfvitale
d12811e9aa DBZ-5847: Add documentation 2022-12-12 14:40:56 +01:00
mfvitale
bd680224da DBZ-5847: Add ComputePartition Transformation. 
This new SMT permits to assign record to a specific partition based on configured table field name hash.
 2022-12-12 14:40:56 +01:00
Marcelo Avancini
7754163c5d DBZ-2432: Fixing misuse of assertj 
- Some usages of assertj was not callig the actual assert of the DSL, fixing it
- Changing the Assertions to be staticlly imported
 2022-12-08 13:10:26 +01:00
Nir Levy
39393299e4 DBZ-5904 Suppport INSERT INTO statements with dots in column names 2022-12-08 13:04:41 +01:00
Erdinç Taşkın
681f2a7c43 [optimization] Simplify binlog validation conditions 2022-11-29 14:17:49 +01:00
Erdinç Taşkın
9f0224526a Remove nested if check 
Simplify to flow. 
- Remove nested if check
- Postpone possible unused stamement which is connection.isBinlogRowImageFull()
 2022-11-29 14:17:49 +01:00
Erdinç Taşkın
77e1cfc9a8 [optimization] Remove unnecessary null check 
Remove Unnecessary null checks
 2022-11-29 14:09:50 +01:00
Erdinç Taşkın
07d93497fe [optimization] Remove Unreachable if statement 
if statement never will be true
 2022-11-23 12:35:45 +01:00
Anisha Mohanty
5c2c0f54be DBZ-5836 COMMENT string for CREATE_USER & ALTER_USER clause 2022-11-23 08:46:12 +01:00
ggaborg
73691db6b1 DBZ-4720 fix mysql tls test 2022-11-22 14:46:55 +01:00
harveyyue
21fddc8515 DBZ-5843 Invalid Java object for schema with type FLOAT64: class java.lang.Float 2022-11-21 11:21:49 -05:00
Vojtech Juranek
0510102db5 DBZ-2131 Don't skip Avro validation 
Original issue seems to be resolved, at least don't with with Avro
validation.
 2022-11-09 08:44:45 +01:00
harveyyue
e6f0551451 DBZ-5801 The merge method of configuration is not work 2022-11-08 14:19:51 +01:00
harveyyue
6ba46b97cb DBZ-5790 Enhance the ability to sanitize topic name 2022-11-03 15:45:54 +01:00
jcechace
740715e2ee DBZ-5779 Migrated debezium-connector-mysql to AssertJ 2022-11-02 08:50:26 +01:00
Chris Cranford
dc5de5ae87 DBZ-5760 Only log full row data at TRACE level 2022-10-26 08:13:34 +02:00
harveyyue
a37d3332df DBZ-5610 MySQL Connector capture TRUNCATE command as message in table topic 2022-10-20 08:57:28 +02:00
harveyyue
299f93afd7 DBZ-5709 The DDL_FILTER of SchemaHistory doesn't work for including break lines ddl statement 2022-10-14 09:21:40 +02:00
Jochen Schalanda
45ec33c17b DBZ-5724 Add PRIMARY as keyword to MySQL grammar to fix index hints 
Index hints using the primary index (`USE INDEX(PRIMARY)`) are currently
failing because `PRIMARY` is missing from the list of keywords in the
ANTLR-based MySQL parser.

According to https://dev.mysql.com/doc/refman/8.0/en/keywords.html#keywords-8-0-detailed-P,
`PRIMARY` is a reserved keyword.

Index hints: https://dev.mysql.com/doc/refman/8.0/en/index-hints.html
 2022-10-14 08:35:53 +02:00
harveyyue
0cbd14e67f DBZ-5728 MySqlSchemaMigrationIT runs failed 2022-10-14 07:20:24 +02:00
Vojtech Juranek
73f056df85 DBZ-5710 Remove whitelist/blacklist from log messages 2022-10-11 15:30:28 +02:00
Jiri Pechanec
99dff302f1 DBZ-5612 Support for formatted messages 2022-10-04 13:45:43 +02:00
rajdangwal
58ab632ea3 DBZ-5612 log records only at trace level 2022-10-04 13:45:43 +02:00
rajdangwal
c03bcf8606 DBZ-5612 Remove records from being logged at all levels 2022-10-04 13:45:43 +02:00
harveyyue
f6b72f060c DBZ-5671 Remove logic name parameter from sub connector config 2022-10-03 11:40:17 -04:00
Sergei Morozov
e4d46c81b1 DBZ-5541: Default Debezium schema.name.adjustment.mode to "none" 2022-09-29 09:47:25 +02:00
harveyyue
d1153e8ad4 DBZ-5594 Clean up "logical name" config 2022-09-22 14:29:37 +02:00
Sergei Morozov
1a4cb1458a DBZ-5630: Unquote double quotes from default values on MySQL 2022-09-22 13:24:36 +02:00
harveyyue
f341724fa9 DBZ-5623 DDL Parsing Error 2022-09-20 17:18:57 -04:00
harveyyue
288c93434f DBZ-5585 Remove classloader parameter from JdbcConnection 2022-09-19 17:15:29 +02:00
harveyyue
665edab844 DBZ-5589 Mysql connector can't handle the case sensitive of rename/change column statement 2022-09-13 09:36:25 +02:00
Vojtech Juranek
acc6be6df3 DBZ-5043 Move TOPIC_PREFIX into CommonConnectorConfig 
`TOPIC_PREFIX` is now mandatory to all connectors therefore it make
sense to have it in common config. Beside that, it also makes it more
easy to use it in Debezium UI without any workarounds - if the field is
not member of the given connector config, the field has to be
explicitely added into known fields otherwise is invisible for UI.
 2022-09-13 09:25:54 +02:00
Jiri Pechanec
c928f7a99e DBZ-5587 Formatting; add contributor 2022-09-08 13:31:04 +02:00
Hang Ruan
3477d53d47 DBZ-5587 fix default value parser when default value containing COLLATE 2022-09-08 13:31:04 +02:00
Anisha Mohanty
9fd204b383 DBZ-5044 Introduce respective SchemaFactory for all connectors 2022-09-08 10:35:20 +02:00
Anisha Mohanty
cfd0242e28 DBZ-5044 Sort imports & fix test failures 2022-09-08 10:35:20 +02:00
Anisha Mohanty
91ccb7578f DBZ-5044 Centralize datatype schemas 2022-09-08 10:35:20 +02:00
Vojtech Juranek
402df78381 DBZ-5043 Rename schema.history to schema.history.internal 2022-09-07 12:57:31 +02:00
Vojtech Juranek
b797201263 DBZ-5043 Rename DB_HISTORY_PATH to SCHEMA_HISTORY_PATH 2022-09-07 12:57:31 +02:00
Vojtech Juranek
b6c31f3da5 DBZ-5043 Replace 'database history' with 'database schema history' 
Reflect renaming database history to schema history in comments,
configuration descriptions and log and error messages.
 2022-09-07 12:57:31 +02:00
Vojtech Juranek
0494a16a79 DBZ-5043 Rename DatabaseHistory to SchemaHistory 
Rename also all related classes and varibales.
 2022-09-07 12:57:31 +02:00