debjeetsarkar
b824c138db
DBZ-2663 | Updated with checkstyle fixes.
...
More Ref:
https://debezium.zulipchat.com/#narrow/stream/302533-dev/topic/Handling.20Transaction.20Payload.20Events.20for.20Mysql/near/289650440
https://github.com/osheroff/mysql-binlog-connector-java/pull/76#issuecomment-1185629635
2022-08-17 10:22:23 +02:00
debjeetsarkar
f41474847d
DBZ-2663 | Update mysql binlog connector library and add support for TRANSACTION PAYLOAD EVENT.
...
More Ref:
https://debezium.zulipchat.com/#narrow/stream/302533-dev/topic/Handling.20Transaction.20Payload.20Events.20for.20Mysql/near/289650440
https://github.com/osheroff/mysql-binlog-connector-java/pull/76#issuecomment-1185629635
2022-08-17 10:22:23 +02:00
Vivek Wassan
d89e71ebeb
DBZ-5327 Smart Backfills
2022-08-17 10:05:55 +02:00
Chris Cranford
fcb845024f
DBZ-5483 Fix testStopSnapshotKafkaSignal random failure
2022-08-08 15:55:56 -04:00
Chris Cranford
f8e67e3ff9
DBZ-5453 Add stop-snapshot
MySQL read-only Kafka signal
2022-08-04 16:16:42 -04:00
Vojtech Juranek
2c505c08f5
DBZ-4727 Add Kafka based pause/resume signals
...
Add pause and resume snapshot signals. See previous commit for more
details. These signal do the same, but are used by read-only MySQL
where sending signals is done by sending Kafka messages.
2022-07-29 12:41:58 -04:00
Vojtech Juranek
3968b6b75f
DBZ-4727 Intorduce KafkaSignal interface
...
In the follow-up commit signal for pause and resume snapshot will be
added. Intorduce `KafkaSignal` interface which would server as general
kafka-based signal.
2022-07-29 12:41:58 -04:00
Vojtech Juranek
c7d8353d7b
DBZ-4727 Add snapshot paused JXM metrics
...
Add two JXM objects:
* `snapshotPaused` - detemines if the incremental snapshot is paused
or not
* `snapshotPausedDurationInSeconds` - overall time when the incremental
snapshot was paused. The time adds up - if the snapshot was paused
e.g. two times, the `snapshotPausedDurationInSeconds` is the sum of
these two paused times.
2022-07-29 12:41:58 -04:00
harveyyue
5f430f1a8e
DBZ-5457 Deprecate legacy topic selector for all connectors
2022-07-28 12:58:59 -04:00
harveyyue
8d76ba217f
DBZ-5333 Caused by: java.io.EOFException: Failed to read next byte from position 2005308603
2022-07-26 11:10:29 -04:00
harveyyue
9683b0fa77
DBZ-5424 Verify the unique index whether including function or arbitrary expression
2022-07-20 14:38:25 +02:00
harveyyue
7730610320
DBZ-5425 Remove the duplicated SimpleDdlParserListener from mysql connector
2022-07-20 14:35:34 +02:00
Chris Cranford
f412c2e6a4
DBZ-5396 Add support for connector-specific relational model attributes
2022-07-19 15:41:02 -04:00
Vojtech Juranek
5315dbd9cb
DBZ-5047 Remove setting last snapshot event in MySQL connector
...
MySQL connector sets last snapshot event twice - the first time during
the streaming and the second time in
`MySqlSnapshotChangeEventSource.postSnapshot()` [1]. Remove setting the
last snapshot event during the streaming and keep it only in one place
in `postSnapshot()` method.
[1] https://github.com/debezium/debezium/blob/v2.0.0.Alpha3/debezium-connector-mysql/src/main/java/io/debezium/connector/mysql/MySqlSnapshotChangeEventSource.java#L563
2022-07-19 11:19:52 +02:00
Vojtech Juranek
109c3d1e0d
DBZ-5047 Remove deprecated unused method from SourceInfo
2022-07-19 11:19:52 +02:00
harveyyue
77c7db163d
DBZ-5414 Validate topic naming strategy relative topic name properties
2022-07-18 09:58:51 -04:00
harveyyue
6ea7abc8fc
DBZ-4180 Pluggable topic selector
2022-07-13 14:45:25 +02:00
Jiri Pechanec
d8baac7c46
DBZ-5229 Reorganize code for inheritance use
2022-07-12 17:31:19 +02:00
Hossein Torabi
1ea02be9c2
DBZ-5229 define centralized and modular aproach for debezium storage
2022-07-12 17:31:19 +02:00
Jiri Pechanec
2a14e2fff7
DBZ-5343 Skip test for MySQL 8
2022-07-11 12:12:26 +02:00
harveyyue
35bd0107ce
DBZ-5343 Unsigned tinyint conversion fails for MySQL 8.x
2022-07-11 12:12:26 +02:00
harveyyue
7ba7929d7e
DBZ-5386 Missing "previousId" property with parsing the rename statement in kafka history topic
2022-07-11 07:44:14 +02:00
Mickael Maison
5ff71aa992
DBZ-5335 Fix formatting issues in the connectors configdefs
2022-06-30 09:35:54 +02:00
Mike Kamornikov
cd84fb17ed
DBZ-5291: Fix binlog filename comparison
2022-06-28 06:08:59 +02:00
harveyyue
20dc353262
DBZ-5170 Mysql Commit Timestamp
2022-06-27 15:50:52 +02:00
harveyyue
638076a899
DBZ-5267 Invalid date 'SEPTEMBER 31'
2022-06-24 16:27:46 +02:00
Chris Cranford
0bbf6e811c
DBZ-5251 Align DefaultValueListener usage across enter/exit handlers
2022-06-24 11:31:08 +02:00
Chris Cranford
3fb5bf1339
DBZ-4451 Introduce SchemaChangeEvent factory methods
2022-06-16 09:19:41 +02:00
Chris Cranford
863abcd081
DBZ-4451 Store previous table id in alter schema change events
...
When a table is renamed with the ALTER TABLE statement, the schema
history record not only will reference the current table's unique id
in the "id" metadata, but will also refer to the old table name in
the "previousId" metadata field.
2022-06-16 09:19:41 +02:00
Chris Cranford
20bb2adea5
DBZ-4834 Fix incremental snapshot of table added to include list
2022-06-16 08:39:16 +02:00
harveyyue
f6a04e2db8
DBZ-5236 Cannot convert field type tinyint(1) unsigned to boolean
2022-06-13 14:42:20 +02:00
Jiri Pechanec
db52977626
DBZ-5241 More specific pattern
2022-06-13 14:23:12 +02:00
Chris Cranford
93af38c189
DBZ-5241 Parse character set introducers on default values
2022-06-13 14:23:12 +02:00
Jiri Pechanec
48072682c6
DBZ-5201 Handle expression in DEFAULT as NULL
2022-06-08 13:33:10 +02:00
Vojtech Juranek
4a48eb33df
DBZ-5045 Remove legacy V1 source info
2022-06-08 09:11:00 +02:00
Vojtech Juranek
ef8d7f7af7
DBZ-5045 Remove deprecated OBSOLETE_NAME_FOR_SKIP_FAILURE_HANDLING option value
2022-06-08 09:11:00 +02:00
Vojtech Juranek
1edab1d1a5
DBZ-5045 Remove deprecated MySQL GTID_NEW_CHANNEL_POSITION option
2022-06-08 09:11:00 +02:00
Vojtech Juranek
b0b3066ed5
DBZ-5045 Remove deprecated COLUMN_BLACKLIST
2022-06-08 09:11:00 +02:00
Vojtech Juranek
b3a6131cee
DBZ-5045 Remove deprecated DATABASE_BLACKLIST
2022-06-08 09:11:00 +02:00
Vojtech Juranek
0cbb57f4bf
DBZ-5045 Remove deprecated DATABASE_WHITELIST
2022-06-08 09:11:00 +02:00
Vojtech Juranek
f84893a8b4
DBZ-5045 Remove deprecated SCHEMA_BLACKLIST
2022-06-08 09:11:00 +02:00
Vojtech Juranek
261188482f
DBZ-5045 Remove deprecated SCHEMA_WHITELIST
2022-06-08 09:11:00 +02:00
Vojtech Juranek
6abbcee6a3
DBZ-5045 Remove deprecated TABLE_BLACKLIST
2022-06-08 09:11:00 +02:00
Vojtech Juranek
db08e894fc
DBZ-5045 Remove deprecated TABLE_WHITELIST
2022-06-08 09:11:00 +02:00
harveyyue
acb743bd34
DBZ-5206 Remove the deprecated monitored tables metrics
2022-06-07 16:13:29 -04:00
Chris Cranford
9d71296b5c
DBZ-4251 Support stopping incremental snapshots
2022-06-02 07:30:07 +02:00
Jiri Pechanec
88b7ae2050
DBZ-5136 Move sourceInfo to common class
2022-06-01 16:56:29 +02:00
Mark Bereznitsky
c64b8af31a
DBZ-5136 reintroduce the CommonOffsetContext class and refactor common functionality into it
2022-06-01 16:56:29 +02:00
Mark Bereznitsky
2d72dae649
DBZ-5136 further refining of the algorithm for determining position in snapshot and IT test fixes
2022-06-01 16:56:29 +02:00
Jiri Pechanec
08e8c3d795
DBZ-5136 Move common class to common package
2022-06-01 16:56:29 +02:00
Andrew Walker
d0759461a4
DBZ-5136: Convert RelationalOffsetContext to CommonOffsetContext
2022-06-01 16:56:29 +02:00
Andrew Walker
01a81f8bc1
DBZ-5136: Refactored SnapshotContext setters
2022-06-01 16:56:29 +02:00
Andrew Walker
b6222d42bf
DBZ-5136: Add first flag and refactor OffsetContext
2022-06-01 16:56:29 +02:00
Andrew Walker
593c2769e6
DBZ-5136: Add last in table flag
2022-06-01 16:56:29 +02:00
Vojtech Juranek
a2110f7c02
DBZ-5042 Add DB name logging support into MySQL connector
2022-05-27 14:20:30 -04:00
Vojtech Juranek
f8d4307155
DBZ-5052 Send tombstones only when consumer is able to consume them
...
Change event consumer may specify if is able to consume tombstones or
not. If the it's not, don't send them. However, connector configuration
takes precence and thus cunsumer capability is taken into account only
when `tombstones.on.delete` is not explicitely configured for the
connector.
Also fix debezium server - setting notifier has to be done once embedded
config already exists, not before it.
2022-05-26 12:40:02 +02:00
harveyyue
3ada04b522
DBZ-3865 Debezium MySql connector does not capture floating point numbers with the right precision
2022-05-25 12:54:42 -04:00
Chris Cranford
7d8c9486d6
DBZ-5119 Move Heartbeat creation to ConnectorConfig & descendants
2022-05-25 09:59:59 +02:00
Chris Cranford
7ff7826743
DBZ-5119 Pull heartbeat.action.query
support into debezium-core
2022-05-24 12:54:40 +02:00
Chris Lee
2a8456d21b
DBZ-5123 data-collection regex snapshot signal
2022-05-24 12:46:28 +02:00
harveyyue
52ba58ccc3
DBZ-5098 Introduce a new field "ts_ms" to identify the process time for schema change event
2022-05-18 14:21:39 +02:00
Vojtech Juranek
80aad894a2
DBZ-5131 Use fetch size only in tests which need it
...
Cuncurrent reading and modification of MySQL table can result into MySQL
JDBC driver ArrayIndexOutOfBoundsException when it tried to merge table
fields. This is result fo caching parepared statements while table
schema is modified. It seems that caching of prepared statements is
triggered by using fetch size. Use it only in test which needs this
option to work properly and which don't do any schema changes.
2022-05-17 05:56:42 +02:00
harveyyue
10632a3489
DBZ-5134 java.lang.NumberFormatException: For input string: \"0.000000000000000000\"
2022-05-16 10:53:32 +02:00
Jiabao Sun
6a2071acee
DBZ-5126 Clear tableMapEventByTableId on rotate event
2022-05-16 09:19:47 +02:00
Vojtech Juranek
fd6be945b5
DBZ-5099 Allow zero-length dates
...
When date is zero and binary protocol use comperssion, date has zero
length [1]:
type to store a DATE, DATETIME and TIMESTAMP fields in the binary protocol.
to save space the packet can be compressed:
if year, month, day, hour, minutes, seconds and micro_seconds are all 0, length is 0 and no other field is sent
Allow zero-lenght date and return null if column is optional, epoch date
otherwise (this conforms to MySqlDefaultValueConverter.convertToLocalDate()).
[1] https://dev.mysql.com/doc/internals/en/binary-protocol-value.html#packet-ProtocolBinary::MYSQL_TYPE_TIME
2022-05-09 13:57:04 +02:00
Chris Cranford
65017dc2d6
DBZ-5100 Do not default database.server.id
2022-05-09 08:52:24 +02:00
Vojtech Juranek
dfd8afab7b
DBZ-4939 Fall back to JDBC driver if we fail to read timestamp or date
...
MySQL protocol can change during connection and as a result our custom
field reader may fail. In such case fall back to JDBC to read the value.
This is workaround until proper solution, which will require bigger
refactoring, is implemented, see DBZ-5084 for more details.
2022-05-06 07:55:58 +02:00
Sergei Morozov
194934b74a
DBZ-5075: Clean up Javadocs
2022-05-03 13:46:23 +02:00
Sergei Morozov
f6546e4d87
DBZ-5075: Remove unused legacy MySQL code
2022-05-03 13:46:23 +02:00
Sergei Morozov
1e8181e5c4
DBZ-5075: Fix typo in a private property name
2022-05-03 13:46:23 +02:00
harveyyue
7fbc53df9d
DBZ-5078 Rename "Mysql" to "MySql" in related MysqlFieldReader interface
2022-05-02 07:21:47 +02:00
harveyyue
7f60baa6e1
DBZ-4998 Should store event header timestamp in HistoryRecord
2022-04-19 13:50:09 +02:00
LarsWerkman
dbcb6e27c8
DBZ-4983 fix incorrect sorted import
2022-04-14 14:39:40 +02:00
LarsWerkman
6522809719
DBZ-4983 Move CharsetMappingWrapper to MysqlConnection class and expose using a static method
2022-04-14 14:39:40 +02:00
LarsWerkman
fec5aef1ec
DBZ-4983 Use correct Java encoding Charset in AbstractMysqlFieldReader
2022-04-14 14:39:40 +02:00
Anisha Mohanty
b57b5e66dd
DBZ-4965 Remove JDBC legacy date time properties
2022-04-08 08:31:30 +02:00
Anisha Mohanty
7d92a137e8
DBZ-4950 Remove MySQL legacy implementation
2022-04-08 08:31:30 +02:00
Nenad Stojanovikj
2d89f296de
DBZ-4933 Fix NPE on null column data in MySQL
...
When doing a snapshot on a MySQL database which has char type columns
(VARCHAR, TEXT, CHAR) and they are null, and a custom converter is
present, the MySQL reader throws an NPE.
This fix addresses the problem by returning an object from the `null`
value.
https://issues.redhat.com/browse/DBZ-4933
2022-04-01 08:36:05 +02:00
Jiri Pechanec
586657f715
DBZ-3541 Test scientific as string
2022-03-29 09:12:17 +02:00
Sergei Morozov
a0533227cb
DBZ-3541 Test case for scientific notation
2022-03-29 09:12:17 +02:00
Sergei Morozov
d766d806af
DBZ-3541 Round decimal default values of integer columns
2022-03-29 09:12:17 +02:00
Sergei Morozov
b258674782
DBZ-3535: Introduce schema.name.adjustment.mode
2022-03-24 09:37:05 +01:00
Sergei Morozov
3a03ff0186
DBZ-3535: Remove unused EventDispatcher constructor parameter
2022-03-24 09:37:05 +01:00
Vojtěch Juránek
49d3a10f1e
DBZ-3949 Ignore DML records in binlog in MySQL connector;
...
In some cases (e.g. in some cases when MEMORY engine is used), DML
records can appear in binlog in spite of the fact it's set to ROW
format. Always ignore these records.
Co-authored-by: Gunnar Morling <gunnar.morling@googlemail.com>
2022-03-23 08:12:02 +01:00
Vojtech Juranek
0163c07800
DBZ-4786 Avoid NPE when removing column which doesn't exists
2022-03-23 07:53:06 +01:00
harveyyue
e1706802a6
DBZ-4787 Improving MySQL SSL truststore/keystore property descriptions
...
Co-authored-by: Gunnar Morling <gunnar.morling@googlemail.com>
2022-03-18 09:11:23 +01:00
kate
4b2d5c29be
DBZ-4878 Re-read incremental snapshot chunk on DDL event
2022-03-18 07:55:43 +01:00
harveyyue
044e454e0a
DBZ-4832 Don't set truststore/keystore parameters to system variables
2022-03-15 11:25:28 +01:00
harveyyue
dac2e1350f
DBZ-4841 Debezium Mysql connector can't handle CREATE INDEX IF NOT EXISTS (MariaDB)
2022-03-11 15:11:00 +01:00
harveyyue
ba3438c860
DBZ-4822 Mysql: Getting ERROR `Failed due to error: connect.errors.ConnectException: For input string: "false"
2022-03-11 10:22:43 +01:00
harveyyue
759e52c423
DBZ-4787 Skip some tests with SSL auth enabled
2022-03-11 10:10:18 +01:00
harveyyue
66d413c114
DBZ-4833 MySQL 5.7 - no viable alternative at input 'ALTER TABLE ORD_ALLOCATION_CONFIG CHANGE RANK'
2022-03-10 15:09:13 +01:00
Gunnar Morling
b46773e5e4
DBZ-4787 Making one more test pass with SSL auth enabled, removing another, now obsolete, one
2022-03-09 14:42:51 +01:00
Gunnar Morling
76b365e1cb
DBZ-4787 Skipping one test when SSL authentication is enabled
2022-03-09 12:12:25 +01:00
Gunnar Morling
8896feb635
DBZ-4787 Adjusting several tests to run with SSL enabled
2022-03-09 11:24:49 +01:00
Gunnar Morling
7e987cd806
DBZ-4787 Simplifying the execution of tests with SSL authentication in the IDE;
...
When starting a MySQL test with -Ddatabase.ssl.mode=verify_ca, the right
defaults for truststore/keystore location and password are used, matching
the set-up created and executed by the Maven build.
2022-03-09 09:23:13 +01:00
Gunnar Morling
8125b7d8b8
DBZ-4787 Allowing to run entire MySQL test suite with SSL authentication
2022-03-09 09:23:13 +01:00
harveyyue
e7dc03fe6d
DBZ-4787 Add mysql connector ssl it and build the relation mysql docker container
2022-03-09 09:23:13 +01:00
harveyyue
49963a5688
DBZ-4787 Not reading the keystore/truststore when enable mysql ssl configurations
2022-03-09 09:23:13 +01:00
harveyyue
da67ba332e
DBZ-4823 Add event type for filtering trace message
2022-03-08 09:43:01 +01:00