alexey.rassokhin/tet123
1
0
Fork 0
Code Issues Pull Requests 1 Packages Projects Releases Wiki Activity
7,290 Commits 3 Branches 0 Tags 51 MiB
1f48bfc515
Commit Graph

1265 Commits

Author SHA1 Message Date
Chris Cranford
970792bc40 DBZ-4012 Deprecate/Rename debezium namespace config options 2021-11-18 21:45:41 +01:00
Chris Cranford
7a4140c6aa DBZ-4012 ActivateTracingSpan is pass-through when OpenTracing unavailable 2021-11-18 21:45:41 +01:00
Gunnar Morling
ca95b424fb DBZ-3528 Misc. clean-up 2021-11-17 16:02:06 +01:00
Sungho Hwang
78c92c3772 DBZ-3528 turned access modifier of jsonNodeToSchemaBuilder() back to private 2021-11-17 16:02:06 +01:00
Gunnar Morling
03433b663b DBZ-3528 Misc. fixes 2021-11-17 16:02:06 +01:00
Sungho Hwang
7c95636668 DBZ-3528 Chaged variable naming for better readability 2021-11-17 16:02:06 +01:00
Sungho Hwang
c9eb0f9157 DBZ-3528 Refactored several parts according to the feedbacks 
A delegate class for common logic between two Outbox Event Routers has been created.
Documentation has been fixed.
The configuration properties for MongoDB Outbox Event Router have been changed to use existing Outbox Event Router's.
 2021-11-17 16:02:06 +01:00
Sungho Hwang
0b9fc88ea6 DBZ-3528 Applied the code style rules 2021-11-17 16:02:06 +01:00
Sungho Hwang
2c279b07f6 DBZ-3528 Wrote MongoDB Outbox Event Router with documentation and its unit & integration tests 2021-11-17 16:02:06 +01:00
Chris Cranford
e3da449303 DBZ-4276 Expose unavailable value placeholder for PG and Oracle only 2021-11-16 20:29:59 +01:00
Chris Cranford
ca17352221 DBZ-4276 Unify unavailable value placeholder handling 
Deprecate the `toasted.value.placeholder` option in PostgreSQL in favor
of the new `unavailable.value.placeholder` option that is shared by all
relational connectors.
 2021-11-16 20:29:59 +01:00
Kate
44073cf7d8 DBZ-4196 Support schema changes during incremental snapshot 2021-11-16 14:56:06 +01:00
camilesing
dfabe8cda8 DBZ-4275 fix AbstractDatabaseHistory typo 2021-11-14 19:21:37 +01:00
Chris Cranford
85101f0ad7 DBZ-3611 Avoid NPE with database include/exclude lists 2021-11-11 09:24:05 +01:00
Kate
992ae537c8 DBZ-4244 Quote signal table name in MySQL connector 2021-11-10 10:18:26 +01:00
Chris Cranford
5afda302ec DBZ-3710 Javadoc updates 2021-11-10 09:43:25 +01:00
Chris Cranford
5ec9381702 DBZ-3710 Unify DefaultValueMapper (Oracle/SQL Server/PostgreSQL) 2021-11-10 09:43:25 +01:00
Chris Cranford
343318b5d4 DBZ-3710 Support Oracle column default values 2021-11-10 09:43:25 +01:00
harveyyue
aa3ae05e22 DBZ-4107 Incremental snapshot doesn't work without primary key 2021-11-05 13:31:26 +01:00
Gunnar Morling
1226c3b2ab DBZ-4029 Misc. clean-up; 
* Exposing heartbeatInterval() on CommonConnectorConfig
* Closing heartbeat connection
* Making test more resilient
* Correct member ordering and visibility
* Doc fixes
 2021-11-05 13:27:09 +01:00
ahodavdekar
91b7ef2c36 DBZ-4029 Adding support for heartbeat action queries to MySQL connector 2021-11-05 13:27:09 +01:00
Gunnar Morling
b78e318399 DBZ-3966 Expanding test 2021-11-04 10:47:24 +01:00
jiabao.sun
2987e96c44 DBZ-3966 Serializing hasDefaultValue in JsonTableChangeSerializer 2021-11-04 10:47:17 +01:00
Gunnar Morling
d1cae3f3b7 DBZ-3966 Making defaultValueExpression() return Optional<String> from editor too 2021-11-04 10:47:17 +01:00
jiabao.sun
24ead16c24 DBZ-3966 Remove defaultValue and keep defaultValueExpression 2021-11-04 10:47:17 +01:00
jiabao.sun
77c4113697 DBZ-3966 JsonTableChangeSerializer support serialization for defaultValue and enumValues 2021-11-04 10:47:17 +01:00
Kate
a645d0bc7e DBZ-4197 Process transaction started/committed in R/O snapshot 2021-11-04 06:19:58 +01:00
ahodavdekar
1d81b6db19 DBZ-4077 Exposing the transaction metadata topic name as a config option 2021-11-02 06:07:05 +01:00
Gunnar Morling
fd06a4e2a4 DBZ-2975 Using Offsets in recover() contract; 
Keeping legacy method until the legacy MySQL connector implementation has been removed.
 2021-11-01 10:17:49 +01:00
Mike Kamornikov
16c07e915b DBZ-2975: Recover database schema from multiple partitions 2021-11-01 10:17:49 +01:00
Jiri Pechanec
a1084360da DBZ-435 Too fast delete can prevent from obtaining the full document 2021-10-26 18:13:16 +02:00
Jiri Pechanec
cb36b575db DBZ-435 Support for Change Streams capture 2021-10-26 18:13:16 +02:00
Sungho Hwang
54d5c16b69 DBZ-4191 Fixed a variable name in an unit test of Outbox Event Router to enhance readability. 2021-10-22 07:57:24 +02:00
harveyyue
90e9bea93f DBZ-4000 Support parse table and columns comment 2021-10-20 15:19:31 +02:00
Anisha Mohanty
2eae7a742c DBZ-4045 Use listener props for deprecated host & port props 2021-10-20 11:51:34 +02:00
Gunnar Morling
3d093e2a3b DBZ-4164 Import ordering 2021-10-18 14:45:00 +02:00
lujiefsi
12ccc24e17
DBZ-4164 Fixing resource leak due to Files::lines 2021-10-18 14:31:39 +02:00
Juan Fiallo
346e0f3af2 DBZ-3517 Making database.user optional for SQL Server. 
SQL Server doesn't require username/password when using Kerberos authentication, which can be configured using pass-through database properties. Changing the database.user config to be optional for SQL Server Connector to reflect that.
 2021-10-18 12:06:56 +02:00
Juan Fiallo
5ac93ac20e DBZ-3517 Making database.user optional for SQL Server. 
SQL Server doesn't require username/password when using Kerberos authentication, which can be configured using pass-through database properties. Changing the database.user config to be optional for SQL Server Connector to reflect that.
 2021-10-18 12:06:56 +02:00
Juan Fiallo
e041b599d8 DBZ-3517 Making database.user optional for SQL Server. 
SQL Server doesn't require username/password when using Kerberos authentication, which can be configured using pass-through database properties. Changing the database.user config to be optional for SQL Server Connector to reflect that.
 2021-10-18 12:06:56 +02:00
Jiri Pechanec
0ccc83db21 DBZ-3860 Provide correct boundaries for multi PK tables 2021-10-14 17:26:50 +02:00
Gunnar Morling
26f13d83d5 DBZ-3993 Reflectively invoking Runtime.version(); 
As that API is only available from Java 9 onwards, calling it will cause
a compilation failure when compiling with --release.
 2021-10-14 09:46:06 -04:00
Gunnar Morling
6b3d2c3b76 DBZ-3993 Formatting fix 2021-10-13 10:59:23 +02:00
Gunnar Morling
c3eed6051b DBZ-3993 Skipping object size based test on Java 16+ 2021-10-13 10:59:23 +02:00
Sergei Morozov
4a03c5df17 DBZ-2975: Include database name into topic and schema names 
Co-authored-by: Mike Kamornikov <mikekamornikov@gmail.com>
 2021-10-12 15:40:02 +02:00
Jiri Pechanec
657c5540b3 DBZ-4083 Add filter to MongoDB cloud converter 2021-10-11 11:19:30 +02:00
Jiri Pechanec
aec2080415 DBZ-3895 Config returns Patterns not Strings 2021-10-11 10:57:33 +02:00
Jiri Pechanec
68f148c3f3 DBZ-3895 Make snapshot collection list matching case insensitive 2021-10-11 10:57:33 +02:00
Chris Cranford
abc3bb42d6 DBZ-4106 Don't continuously reconfigure DatabaseHistory 2021-10-05 20:15:58 +02:00
Gunnar Morling
62715c4e1d DBZ-4033 Doc update 2021-10-05 10:54:01 +02:00
unalsurmeli
9a13d80074 DBZ-4033 A new hash version has been added so that it can be used to have the same hash value in different places; 
In V2, the byte array only has the actual value, unlike the serialization based V1 implementation which exposed Java's internal serialization format.
 2021-10-05 10:52:30 +02:00
Vivek Wassan
82dad32e2a DBZ-4078 Checking connection is valid before commit in readChunk() 2021-10-04 10:00:27 +02:00
Laurent Broudoux
77eaa0a293 DBZ-1297 Add boolean type in test 2021-10-01 22:29:53 +02:00
Laurent Broudoux
7b7a1f3064 DBZ-1297 Expand JSON escaped string as JSON 2021-10-01 22:29:53 +02:00
Jiri Pechanec
a2c6058404 DBZ-4057 Initialize Postgres schema before incremental snapshot restart 2021-09-30 09:10:49 +02:00
Naveen Kumar KR
62045f6430 DBZ-3887 1) Replace notifyAll with notify as atmost only one thread would be in wait state at any time 
2) Remove synchronized from jmh metric methods as it doesn't strictly require lock due to nature of metrics
3) Always wait by predefined timeout to gurantee that thread would never stuck
 2021-09-22 19:12:21 +02:00
Naveen Kumar KR
858557d184 DBZ-3887 Major improvements for Throughput Bottleneck and Inefficient Batching in ChangeEventQueue 
* Replaced naive metronome.pause and Thread.sleep with efficient wait() and notifyAll()
* Replaced LinkedBlockingDeque with ArrayDeque for better performance
* Replaced Atomic and Map operation related to maxQueueSize with primitive long and ArrayDeque for better performance

* x10 - x100 Improvements in overall throughput
* Efficient batching based on maxBatchSize
 2021-09-22 19:12:21 +02:00
Kate
0099a50c53 DBZ-3948 Support kafka topic based signals 2021-09-22 13:08:45 +02:00
Vivek Wassan
b6e891ec7d DBZ-3951 Moving preReadChunk() to AbstractIncrementalSnapshotChangeEventSource 2021-09-22 07:01:29 +02:00
Vivek Wassan
65ac2387dc DBZ-3951 Adding a separate isValid method 2021-09-22 07:01:29 +02:00
Vivek Wassan
5f37e6630f DBZ-3951 Adding a safety check, isValid, for connection 2021-09-22 07:01:29 +02:00
Naveen Kumar KR
b5ccd249c9 DBZ-4015 1) TableIdParser - Initialize ArrayList with optimal capacity of 3 (at max three parts in an identifier - catalog.schema.table) 
2) TableId - Replace .stream().toArray() with .toArray() to eliminate stream overhead
 2021-09-21 10:35:08 +02:00
Naveen Kumar KR
4c5264f7d2 DBZ-4015 Efficient String Replacement using String.replace in TableIdParser (improvement of x4-x5 times) 
(https://medium.com/javarevisited/micro-optimizations-in-java-string-replaceall-c6d0edf2ef6)
 2021-09-21 10:35:08 +02:00
Jiri Pechanec
26d0e78023 DBZ-4013 Skip invalid table 2021-09-17 14:20:42 +02:00
Anisha Mohanty
f57545d7b1 DBZ-3787 Add tests for invisible & visible columns using testcontainers 
*Using user "debezium" to have right grants provided
 2021-09-15 15:13:18 +02:00
René Kerner
b92e5c5efe
DBZ-3943 Add ConfigDef unit tests to verify all Debezium Field instances have a proper documentation/description set; 
Also adding missing descriptions.
 2021-09-09 13:35:41 +02:00
Gunnar Morling
6f1577a70b DBZ-3958 Gracefully handling history file in sym-linked directory; 
Also removing redundant test method definition.
 2021-09-09 12:18:21 +02:00
Gunnar Morling
7c150de943 DBZ-2525 Don't qualify selected columns with table name; 
It shouldn't be needed as we select from a single table, and it caused the columns to not be found for the Db2 connector.
 2021-08-31 17:39:36 +05:30
Anisha Mohanty
153e0920e1 DBZ-2525 Generifying exclusion of columns from snapshotting in all relational connectors; 
* Using RelationalDatabaseSchema in RelationalSnapshotChangeEventSource for consistency.
* Adding test for table/column names which are keywords
 2021-08-30 12:20:56 +02:00
Jiri Pechanec
a36eb45dff
DBZ-3912 Added Binary padding for hex/base64 handling mode 
Co-authored-by: Gunnar Morling <gunnar.morling@googlemail.com>
 2021-08-27 09:48:19 +02:00
Jiri Pechanec
4396513795 DBZ-3901 Guard against chunk signal coming when none expected 2021-08-24 10:26:32 +02:00
rkerner
52333596de DBZ-3364 add additional metadata to fields for API spec generator for the Debezium UI 
part of https://issues.redhat.com/browse/DBZ-3364
DBZ-2653 User rollback from parent class
 2021-08-23 17:57:38 +02:00
Chris Cranford
973fa248ed DBZ-3902 Fix test failure shouldNotEmitDdlEventsForNonTableObjects 2021-08-23 11:44:14 -04:00
Gunnar Morling
36af61b789 DBZ-3889 Moving ContainerImageVersions to core for broader re-use and using it in two more places 2021-08-23 12:34:03 +02:00
Sergei Morozov
b06b5aecbc DBZ-2975: Introduce opt-in configuration for multi-partition mode 2021-08-23 12:01:11 +02:00
Jiri Pechanec
11e3ef4325 DBZ-3857 Exclude database name in snapshot tables filtering 2021-08-16 14:12:53 +02:00
Gunnar Morling
c2b13037fb DBZ-3444 Upgrading to Apache Kafka 2.8 2021-08-12 11:57:57 +02:00
Jiri Pechanec
a4fba49184 DBZ-3688 Expose min/max key for incrementally snapshotted table 2021-08-11 14:40:15 +02:00
Chris Cranford
46590a3bc6 DBZ-3839 Simplify version checks 2021-08-11 07:35:57 +02:00
Chris Cranford
c15cf81e18 DBZ-3839 Correctly skip tests with minor/patch not specified 2021-08-11 07:35:57 +02:00
Sergei Morozov
a40ad7132d DBZ-2975: Remove partition from offset context 2021-08-11 07:34:37 +02:00
Sergei Morozov
77e2ac8a12 DBZ-2975: Introduce io.debezium.pipeline.spi.Offsets 2021-08-11 07:34:37 +02:00
Sergei Morozov
c7ac5fca52 DBZ-2975: Move Partition to io.debezium.pipeline.spi 2021-08-11 07:34:37 +02:00
Chris Cranford
85a732d7dc DBZ-3842 Validate keySchema is null and not valueSchema 2021-08-10 11:15:55 +02:00
pkgonan
fae4f45acc DBZ-3765 Add dot pattern test code for database.server.name 2021-08-08 16:25:15 -04:00
pkgonan
5e0ffa28d2 DBZ-3765 Allow dot in database.server.name 2021-08-08 16:25:15 -04:00
Chris Cranford
26461bc07d DBZ-3692 Introduce completion callback for incremental snapshots 2021-07-29 13:51:42 +02:00
Chris Cranford
393a44a55f DBZ-3692 Oracle incremental snapshot support 2021-07-29 13:51:42 +02:00
Kate
9b66ab374a DBZ-3577 Support read-only MySQL incremental snapshots 2021-07-29 07:10:46 +02:00
Gunnar Morling
09ebc833f3 DBZ-3655 Gracefully handling DB shutdown 2021-07-27 10:21:54 +02:00
Anisha Mohanty
bd47fecdbd DBZ-3717 Throw MultipleParsingExceptions to handle multiple parsing errors & smooth running of database history 2021-07-22 14:12:38 -04:00
Hossein Torabi
0ad8ededa3 DBZ-3725 Validate table before start incemental processing 2021-07-21 14:53:22 -04:00
Gunnar Morling
b84c3c870d DBZ-3668 Extracting SPI package 2021-07-03 16:31:52 +07:00
Gunnar Morling
dcb7336902 DBZ-3668 Making provoiders map unmodifiable 2021-07-03 16:31:52 +07:00
Chris Cranford
2e84427603 DBZ-3668 Consolidate ServiceLoader lookup 2021-07-03 16:31:52 +07:00
Chris Cranford
7f7e23b110 DBZ-3668 Add ServiceLoader hook CloudEvents converter 2021-07-03 16:31:52 +07:00
Chris Cranford
219ed17a28 DBZ-3668 Add CloudEventsConverter support for Vitess connector 2021-07-03 16:31:52 +07:00
Chris Cranford
80a4e003d8 DBZ-3668 Add CloudEventsConverter support for Db2 connector 2021-07-03 16:31:52 +07:00
Chris Cranford
3f759a8daa DBZ-3668 Add CloudEventsConverter support for Oracle connector 2021-07-03 16:31:52 +07:00
Sergei Morozov
db105baeba DBZ-2975: Add partition awareness to source task components 
Co-authored-by: Mike Kamornikov <mikekamornikov@gmail.com>
 2021-07-03 10:56:34 +02:00
Sergei Morozov
0c0c4f5e33 DBZ-2975: Introduce partition-aware API components 2021-07-03 10:56:34 +02:00
Gunnar Morling
5b76bf1b71 DBZ-3632 Pulling up getJdbcConfig() to RelationalDatabaseConnectorConfig 2021-07-01 15:11:01 +02:00
Jiri Pechanec
472b82abb7 DBZ-3622 Decouple database and schema history record writes 2021-06-29 10:36:41 +02:00
Jiri Pechanec
443c0ce089 DBZ-3622 Fix non-captured database storing; make the code more robust 2021-06-29 10:36:41 +02:00
Chris Cranford
da2e142460 DBZ-3593 Only emit commit event if there is an active transaction 2021-06-17 09:25:30 +02:00
Gunnar Morling
205c63a56a DBZ-3608 Fixing version handling; 
- wrong minor version for Postgres
- minor/patch versions of -1 were not actually ignored in comparison
 2021-06-10 07:39:49 +02:00
Gunnar Morling
dc865cd82d DBZ-2975 Formatting 2021-06-09 16:02:33 +02:00
Gunnar Morling
1ce65fa628 DBZ-2975 Avoiding raw types in some places; 
Also parameterizing offset loader contract, so to avoid casts later on.
 2021-06-09 16:02:33 +02:00
Sergei Morozov
657a6d2542 DBZ-2975: Extract offset context from object states to method signatures 2021-06-09 16:02:33 +02:00
Jiri Pechanec
06ceac1b24 DBZ-3607 Document incremental snapshot SPI 2021-06-09 14:15:09 +02:00
Sergei Morozov
92e0488874 DBZ-3562: Allow hyphen in database.server.name 2021-05-27 11:03:30 +02:00
Sergei Morozov
3b72e5149c DBZ-3486: Replace @see with @link where a URL is referenced 2021-05-25 07:19:43 +02:00
Jiri Pechanec
ba581ed323 DBZ-3500 Added Javadoc 2021-05-20 08:06:37 +02:00
Jiri Pechanec
cbaf13841f DBZ-3500 Incremental snapshots for Db2 2021-05-20 08:06:37 +02:00
Jiri Pechanec
258757410d DBZ-3500 Fix tests 2021-05-20 08:06:37 +02:00
Jiri Pechanec
4929397022 DBZ-3500 Per-connector table name quoting 2021-05-20 08:06:37 +02:00
Jiri Pechanec
a879b7b3e1 DBZ-3500 Re-use ResultSet to Object code 2021-05-20 08:06:37 +02:00
Chris Cranford
d147d1f517
DBZ-3240 Remove database.tablename.case.insensitive deprecated option 2021-05-19 20:44:59 +02:00
Thomas Aregger
d1670b636c
DBZ-3456 Timezone difference not considered in LagFromSourceInMilliseconds calculation; 
* Retrieve v$logmnr_contents.change_time in UTC
* Respect time zones when calculating time difference
* Allow constructors to be annotated with VisibleForTesting
 2021-05-19 13:41:02 +02:00
Chris Cranford
b13cbb1077 DBZ-2916 Add support for Oracle DDL statement parsing 2021-05-18 14:58:20 +02:00
Jiri Pechanec
6086eaf642 DBZ-3469 Emit null keys without schema 2021-05-13 08:04:50 -04:00
Alfusainey Jallow
1a3a752ddb DBZ-3514 prevent configuration modification after creation 
Signed-off-by: Alfusainey Jallow <alf.jallow@gmail.com>
 2021-05-12 09:31:29 +02:00
Anisha Mohanty
64d2cde825 DBZ-3438 Add mapped diagnostic contexts for snapshotting & streaming 2021-05-11 11:43:58 +02:00
Jiri Pechanec
5c722341e2 DBZ-3473 Extract incremental snapshot contract 2021-05-06 05:37:36 +02:00
Jiri Pechanec
7daae11600 DBZ-3473 Postgres has its own schema instance 2021-05-06 05:37:36 +02:00
Jiri Pechanec
13334ee72f DBZ-3473 Add current chunk metric 2021-05-06 05:37:36 +02:00
Jiri Pechanec
39d21806eb DBZ-3473 Metrics enabled 2021-05-06 05:37:36 +02:00
Jiri Pechanec
3d940ca14a DBZ-3473 Support for cursor-based fetching 2021-05-06 05:37:36 +02:00
Jiri Pechanec
c53785dde7 DBZ-3473 Incremental Snapshot support for MySQL 2021-05-06 05:37:36 +02:00
Jiri Pechanec
d10bf3f7ef DBZ-3473 Log chunk id 2021-05-06 05:37:36 +02:00
Jiri Pechanec
49c5a00bbf DBZ-3473 Share read JDBC value between regular and incremental snapshot 2021-05-06 05:37:36 +02:00
Jiri Pechanec
91ecbcc81a DBZ-3473 Share read statement creation between regular and incremental snapshot 2021-05-06 05:37:36 +02:00
Jiri Pechanec
dd2bc6354b DBZ-3473 Pass the main clock form connector 2021-05-06 05:37:36 +02:00
Jiri Pechanec
3371e6ccb3 DBZ-3473 Enable per-dialect query limit 2021-05-06 05:37:36 +02:00
Jiri Pechanec
260c32cf0e DBZ-3473 Resume parallel snapshot after connector restart 2021-05-06 05:37:36 +02:00
Jiri Pechanec
4d52b21a76 DBZ-3473 Ignore parallel snapshot for connector not supporting it 2021-05-06 05:37:36 +02:00
Jiri Pechanec
fd9c9d02b4 DBZ-3473 Handle exceptions 2021-05-06 05:37:36 +02:00
Jiri Pechanec
37626c0dfe DBZ-3473 Extract incremental snapshot context 2021-05-06 05:37:36 +02:00
Jiri Pechanec
07371c085b DBZ-3473 Watermark signals must be in separate tx 2021-05-06 05:37:36 +02:00
Jiri Pechanec
a52c536a68 DBZ-3473 First prototype of incremental snapshotting 2021-05-06 05:37:36 +02:00
Anisha Mohanty
a6b54c7000 DBZ-3427 Validate the server name field 2021-04-28 13:18:52 +02:00
Gunnar Morling
8f891248ad DBZ-3429 Making option retrieval lenient towards trailing/leading whitespace 2021-04-28 11:01:27 +02:00
kyleyj
99536f21d2 DBZ-3429 Support spaces in the SQL Server properties 
Many of the SQL Server properties support spaces in the comma-separated lists.
However, the snapshot.select.statement.overrides property doesn't support
spaces after the commas.
 2021-04-28 10:23:02 +02:00
Gunnar Morling
91b784c3b6 DBZ-3161 Avoid failing when legacy option is given; also 
* adding all public history options on config surface
* adding missing whitespace in field docs
* recommending non-deprecated option in docs
 2021-04-26 13:07:23 +02:00
Anisha Mohanty
11fb079582 DBZ-3161 Add database.history.store.only.captured.tables.ddl option & CapturedTables metrics to documentation 2021-04-26 13:07:23 +02:00
Anisha Mohanty
f0f8d5d180 DBZ-3161 Refactor/Rename all references of MonitoredTables to CapturedTables 2021-04-26 13:07:23 +02:00
Anisha Mohanty
062c1c3860 DBZ-3161 Add new option for CapturedTables, deprecate existing MonitoredTables option & emit warnings 2021-04-26 13:07:23 +02:00
Anisha Mohanty
4a86330c0a DBZ-3161 Add new CapturedTables metrics 2021-04-26 13:07:23 +02:00
Gunnar Morling
42a7b74725 DBZ-3403 Misc. improvements 2021-04-14 11:14:07 +02:00
Hossein Torabi
ba048cf47d DBZ-3403 Semi Optimisation on MongoDB and MySQL connector for skipped.operations 2021-04-14 11:14:07 +02:00
Hossein Torabi
5d6705c3fe DBZ-2699 implement skipped.operation in core 2021-04-08 11:59:08 +02:00
Chris Egerton
67f044ff87 DBZ-3371: Adjust scale of fallback BigDecimal values to match column schema 2021-04-06 10:35:23 +02:00
Jiri Pechanec
f50c3e8222 DBZ-3361 Prefer DDL over logical schema for MySQL 2021-03-30 11:35:12 +02:00
Chris Cranford
776180a0df DBZ-3347 Emit schema change events with correct source partition 2021-03-30 07:26:06 +02:00
Jiri Pechanec
a02e530c80 DBZ-2911 Use sequence only when available 2021-03-26 09:19:17 +01:00
JLDLaughlin
06b0475f17 DBZ-2911 Add additional LSN information to all sources 
Adds a new "sequence" field to all sources via the AbstractSourceInfo
class. Specifically, this field is required to deduplicate records
from PostgreSQL sources in O(1) time. The sequence field is a stringified
list of metadata. For PostgreSQL sources, this will include the last
committed LSN and the current LSN. For other sources, this will be NULL
until implemented. A new integration test was added for PostgreSQL.
 2021-03-26 09:19:17 +01:00
Gunnar Morling
c7b7d5303a DBZ-3218 Avoiding duplicated validation (again ;) 2021-03-19 15:04:20 +01:00
Gunnar Morling
178140ae9c DBZ-3218 Removing superfluous method parameter 2021-03-19 15:04:20 +01:00
Anisha Mohanty
c761d92b2a DBZ-3218 Add separate method for logging warnings 2021-03-19 15:04:20 +01:00
Anisha Mohanty
6856dd9a80 DBZ-3218 Handle duplicate warnings for deprecated options 2021-03-19 15:04:20 +01:00
Jiri Pechanec
f7b6750ce3 DBZ-2511 Fix tests 2021-03-18 14:42:34 +01:00
Jiri Pechanec
495be2659b DBZ-2511 Dedicated logger for schema name adjuster 2021-03-18 14:42:34 +01:00
Jiri Pechanec
11e4880018 DBZ-3309 Revert to MySQL specific default snapshot fetch size 2021-03-18 11:08:26 +01:00
Gunnar Morling
647076ea69 DBZ-2957 Misc. fixes; 
* Simplifying reg exp
* Adding leniency towards whitespace
* Reporting *all* problems including correct count
* Fixing incorrect actual/expected order
 2021-03-17 14:04:16 +01:00
Anisha Mohanty
2e5a5ce001 DBZ-2957 Handle message.key.columns regex validation time complexity 2021-03-17 14:04:16 +01:00
Gunnar Morling
1f02d25738 DBZ-3271 Removing unused methods 2021-03-17 09:54:25 +01:00
Gunnar Morling
911b5e9a3b DBZ-3271 Unifying filter handling across relational connectors 2021-03-17 09:54:25 +01:00
Gunnar Morling
4d61c57206 DBZ-3157 Adding test 2021-03-11 16:10:08 +01:00
Vladimir Osin
88f751ad3a DBZ-3157 Add event_serial_no to CloudEvent Id 2021-03-11 16:10:08 +01:00
Jiri Pechanec
c8eec4995b DBZ-3141 Changes after review 2021-03-10 19:12:34 +01:00
Jiri Pechanec
f23421920b DBZ-2919 Remove race condition 2021-03-10 19:12:34 +01:00
Jiri Pechanec
de0d540a9f DBZ-3141 Change schema signal for Oracle 2021-03-10 19:12:34 +01:00
Jiri Pechanec
15c7b11a1f DBZ-3141 Change schema signal tested for Postgres 2021-03-10 19:12:34 +01:00
Jiri Pechanec
ff7cb44801 DBZ-3141 Offsets are propagated to signal actions 2021-03-10 19:12:34 +01:00
Jiri Pechanec
7bcf9338a0 DBZ-3141 Added unit tests 2021-03-10 19:12:34 +01:00
Jiri Pechanec
7fdad2ed49 DBZ-3141 Support for signalling table 2021-03-10 19:12:34 +01:00
Anisha Mohanty
323b23498e DBZ-3084 Handle wrong warnings about deprecated options 2021-03-09 07:10:45 +01:00
Kewei Shang
158b616bfe DBZ-3199 Add support for SET column to Vitess Connector 2021-02-26 21:23:12 +01:00
Chris Cranford
7e34c918e2 DBZ-3034 Add validate methods to CommonConnectorConfig 2021-02-23 10:28:17 +01:00
Chris Cranford
85f7688527 DBZ-3151 Add filter hook for table unique index resolution 2021-02-22 15:19:25 +01:00
Kewei Shang
92659d6a27 DBZ-3124 Overload builder() and schema() to take multiple values 2021-02-19 12:26:44 +01:00
Jiri Pechanec
1edad510ff DBZ-2614 Support for infinite timestamps 2021-02-19 10:44:06 +01:00
Gunnar Morling
440364e781 DBZ-2984 Removing unused field 2021-02-16 14:49:35 +01:00
Jiri Pechanec
8c0b87c667 DBZ-2984 Guess struct name only when unavailable 2021-02-16 14:49:35 +01:00
Gunnar Morling
184d1a2399 DBZ-3056 Misc. clean-up 2021-02-16 12:45:09 +01:00
Yilong Chang
a0b3704832 DBZ-3056 Validate all fields in subclasses. 2021-02-16 12:34:06 +01:00
Yilong Chang
76c7e0d426 DBZ-3056 Extract basic valdiation in parent class. 2021-02-16 12:34:06 +01:00
Gunnar Morling
cb2b2fc07a DBZ-3099 Doc fix 2021-02-16 10:11:23 +01:00
Jiri Pechanec
1c9f7a1c1c DBZ-3099 Document non-volatile fields accessed by a single thread 2021-02-16 10:02:17 +01:00
Jiri Pechanec
745bb6c11d DBZ-3099 Address review comments; extract history comparator for testability 2021-02-16 10:02:17 +01:00
Jiri Pechanec
cb1476ce18 DBZ-3099 Address review comments 2021-02-16 10:02:17 +01:00
Jiri Pechanec
21b15facb2 DBZ-3099 Snapshot buffering to handle config corner cases 2021-02-16 10:02:17 +01:00
Jiri Pechanec
6a9ca7d544 DBZ-3099 Fix schema recovery; Add table level locking 2021-02-16 10:02:17 +01:00
Jiri Pechanec
08e4988c72 DBZ-3093 Unify snapshot and streaming parsing 2021-02-16 10:02:17 +01:00
Jiri Pechanec
6607310f13 DBZ-3093 Separate parsing and change applying phase in MySQL schema 2021-02-16 10:02:17 +01:00
ani-sha
5f8a22a5df DBZ-2858 Make field descriptions consistent for time values 2021-02-16 09:55:35 +01:00
Gunnar Morling
b5b73b4409 DBZ-3057 Formatting fix 2021-02-16 09:40:13 +01:00
Chris Cranford
09de77cf7d DBZ-3057 Add some TableId helper methods 2021-02-16 09:40:13 +01:00
harveyyue
3a4bcb104b DBZ-3096 Getting RejectedExecutionException when checking topic settings from KafkaDatabaseHistory.checkTopicSettings 2021-02-12 08:37:16 +01:00
Kevin Pullin
2426de849a DBZ-3074 PgOutputMessageDecoder - Fix unordered keys 
Fix issue where `PgOutputMessageDecoder` does not define primary key metadata in key-order.

`PgOutputMessageDecoder.getTablePrimaryKeyColumnNamesFromDatabase()` returns a `Set`, leading to unordered
primary keys in the metadata. Switch the logic to call `JdbcConnection.readPrimaryKeyNames()` which
_does_ correctly return an ordered primary key list.
 2021-02-10 10:58:26 +01:00
Yilong Chang
0e63a5a651 DBZ-3075 Close Kafka Database History producer with a timeout of 30s. 2021-02-10 09:08:02 +01:00
Jiri Pechanec
f57343dc3b DBZ-1865 Use OptionaLong; add comments 2021-02-04 09:07:55 +01:00
Jiri Pechanec
2374b6b045 DBZ-1865 First draft of MySQL rewrite 2021-02-04 09:07:55 +01:00
Gunnar Morling
6c5fb793af DBZ-2575 Marking option as private 2021-02-04 08:28:55 +01:00
Martin Sillence
164ab1330c DBZ-2575 Improve performance where tables we fetch are much less than total tables 2021-02-04 08:28:49 +01:00
Naveen Kumar KR
75882f35ac DBZ-2382 Support emitting TRUNCATE events in PG11+ PGOUTPUT plugin 
default - TruncateHandlingMode.SKIP
supported configs - TruncateHandlingMode.SKIP / TruncateHandlingMode.INCLUDE
 2021-02-03 13:53:01 +01:00
ani-sha
d4003c9172 DBZ-2420 Define database configs under RelationalDatabaseConnectorConfig 2021-02-03 08:05:42 +01:00
Gunnar Morling
f76d8e0869 DBZ-2938 Calling snapshot listener only after complete() hook 2021-01-19 15:16:46 +01:00
Gunnar Morling
6d441bc538 DBZ-2938 Centralizing listener handling; avoiding some empty methods 2021-01-19 15:16:46 +01:00
rkerner
d7ac0f7e6b DBZ-2572 Fix docs for message.key.columns and skipped.operations 2021-01-12 14:08:21 +01:00
Gunnar Morling
21c3003187 DBZ-2889 Don't access payload id field if not present 2021-01-06 14:46:58 +01:00
Chris Cranford
d32e4b3720 DBZ-2849 Added test 2021-01-05 12:42:08 +01:00
Chris Cranford
8409282739 DBZ-2849 Fix value converter field lookups with sanitized field names 2021-01-05 12:42:08 +01:00
Jiri Pechanec
58713599ba DBZ-2167 Remove legacy parser completely; tests depending on parser move to MySQL 2020-12-21 16:30:15 +01:00
Jiri Pechanec
00f9e3b261 DBZ-2167 Use legacy parser for testing purposes only 2020-12-21 16:30:15 +01:00
Jiri Pechanec
8e333d9b1d DBZ-1818 DBZ-559 Changes after review 
OpenTracing not used in names
Span for event time
Span SMT embedded in outbox
Handling of missing tracing libs in Connect runtime
Aligned tracing tag names with the standard ones
 2020-12-08 16:09:09 +01:00
Jiri Pechanec
318340078b DBZ-1818 DBZ-559 Integration with OpenTracing 2020-12-08 16:09:09 +01:00
Jiri Pechanec
b7fede7d53 DBZ-735 Verify Avro serialization 2020-11-27 17:33:23 +01:00
vanhoale
17a424d9ac DBZ-2662 Handle ChangeEventQueue by the size in bytes 2020-11-16 08:39:33 +01:00
rkerner
a611979268 DBZ-2580 apply PR feedback 2020-11-12 07:42:48 +01:00
rkerner
9c5db0cd2f DBZ-2580 Fix "The primary key cannot reference a non-existant column" error from MySQL DDL parser when CREATE TABLE statement starts with a primary key definition like "CREATE TABLE Products (PRIMARY KEY (id), ..." where the referenced primary key column is not yet defined 2020-11-12 07:42:48 +01:00
Chris Cranford
aea26a2949 DBZ-2742 Migrate initial.sync.max.threads to snapshot.max.threads 2020-11-11 09:08:05 +01:00
Ganesh Ramasubramanian
4716194ff9 DBZ-2606: Map new name for the fields and headers 
Added ability to map new name for the fields and headers.
The "new name" mapping is completely optional and we will not
be affecting the existing functionality.

The optional "new name" is case-sensitive.
 2020-11-04 08:57:16 +01:00
ani-sha
fd6c8f9502 DBZ-2690 get oracle specific changes into debezium-core 2020-11-04 07:21:10 +01:00
rkerner
c274c2b39b DBZ-2617 refactor to proper HeartbeatErrorHandler injection 2020-11-03 07:23:02 +01:00
rkerner
2a3f44ebe5 DBZ-2617 failing heartbeats should put Postgres connector into FAILED state to allow proper shutdown of Postgres when shutdown was triggered with shutdown mode "fast" 2020-11-03 07:23:02 +01:00
Sergei Morozov
344acb55c0 DBZ-2682: Mapper converters should leave empty dates as zero 2020-10-27 16:21:53 +01:00
Jiri Pechanec
2cded66d30 DBZ-2632 Minor improvements 2020-10-20 16:28:23 +02:00
Arik Cohen
2e8c43bd60 DBZ-2632: Forcibly abort the JDBC connection if failed to close it gracefully. 2020-10-20 16:28:23 +02:00
Jiri Pechanec
ee5353ef6f DBZ-2653 Process commits even with tx monitoring 2020-10-20 10:42:10 +02:00
Kaushik Iyer
7d9e8fdc46 DBZ-2456 fixes after rebase 2020-10-16 10:13:48 +02:00
Kaushik Iyer
f752b3ab39 DBZ-2456 updated common connector config with generalized naming 2020-10-16 10:13:48 +02:00
Kaushik Iyer
e2e6dc0932 DBZ-2456 selective snapshot implementation 2020-10-16 10:13:48 +02:00
Gunnar Morling
b792ac3274 DBZ-2495 Misc. fixes 2020-09-30 12:13:45 +02:00
Gunnar Morling
6b5a3100e9 DBZ-2495 Misc. clean-up 2020-09-30 12:13:45 +02:00
James Gormley
9e780ac481 DBZ-2495 Updating per PR comments 2020-09-30 12:13:45 +02:00
James Gormley
1b845535ab DBZ-2495 Updating tables to utilize Table.editor for creating new tables 2020-09-30 12:13:45 +02:00
James Gormley
c7b53e1992 DBZ-2495 Updating docs and comments 2020-09-30 12:13:45 +02:00
James Gormley
0d46ec47da DBZ-2495 Updating to handle captured_column_list without specify column.exclude.list 2020-09-30 12:13:45 +02:00
Gunnar Morling
c1b057f792 DBZ-2588 Introducing dedicated functional interface for key/value generators; 
Also keys are always KC Struct instead of generic j.l.Object.
 2020-09-25 06:01:59 +02:00
Chris Cranford
df41cd8bfe DBZ-2396 Suggested changes 2020-09-24 07:41:36 +02:00
Chris Cranford
0db4997102 DBZ-2396 Support Converter delegation in ByteBufferConverter 
In the even that the value supplied to the ByteBufferConverter
is not supported, e.g. not BYTES, the converter should then
support delegation to a configured converter.
 2020-09-24 07:41:36 +02:00
Jiri Pechanec
73c74189fa DBZ-2549 Move scripting code into a separate module 2020-09-18 11:17:17 +02:00
Ganesh Ramasubramanian
6ca025b1bb DBZ-2504: Add support for field and header prefix 
Added support for field and header prefix.
Updated the documentation to include the options
 'add.fields.prefix' and 'add.headers.prefix'.
 2020-09-16 11:31:05 +02:00
rkerner
011c3be948 DBZ-2527 Add missing ListOfRegex validator to all regex list fields and fix dependent fields to not include @Deprecated whitelist/blacklist fields 2020-09-15 10:05:04 +02:00
rkerner
385a873172 DBZ-2526 add DisplayName for MASK and TRUNCATE COLUMN fields 2020-09-15 10:05:04 +02:00
rkerner
5c5fb5dc2c DBZ-2516 add config fields for datatype.propagate.source.type and column.propagate.source.type properties 2020-09-15 10:05:04 +02:00
Gunnar Morling
73ba6d42a0 DBZ-137 Docs update 2020-09-14 12:33:43 +02:00
Chris Cranford
a686c1c875 DBZ-137 Suggested changes 2020-09-14 12:33:43 +02:00
Chris Cranford
5ca36a84ad DBZ-137 Reimagine stream.fetch.size as query.fetch.size 2020-09-14 12:33:43 +02:00