alexey.rassokhin/tet123
1
0
Fork 0
Code Issues Pull Requests 1 Packages Projects Releases Wiki Activity
10,666 Commits 3 Branches 0 Tags 51 MiB
27bb2d261b
Commit Graph

2042 Commits

Author SHA1 Message Date
mfvitale
4fedfbba03 DBZ-6834 Provide INSERT/DELETE semantics for incremental snapshot watermarking 2023-12-06 14:10:26 +01:00
Debezium Builder
0fd1c0dc9a [maven-release-plugin] prepare for next development iteration 2023-12-04 13:55:35 +00:00
Debezium Builder
3e2d75f0da [maven-release-plugin] prepare release v2.5.0.Beta1 2023-12-04 13:55:35 +00:00
Jakub Cechacek
8a5007a738 DBZ-7184 Refactored streaming loop of MongoDB connector 2023-12-04 06:58:28 +01:00
Jakub Cechacek
93e8570fb6 DBZ-7184 Implementation of buffering change stream cursor 2023-12-04 06:58:28 +01:00
Chris Cranford
17a9245772 DBZ-7197 Checkstyle fixes 2023-11-29 12:10:04 -05:00
Matt Vance
f26c24e82f DBZ-7197 Add "table.fields.additional.error.on.missing" config support to Outbox EventRouter 2023-11-29 12:10:04 -05:00
Roman Kudryashov
6474c68e42 DBZ-7159 Simplify CE validator API by setting serializer type at the configuration phase 2023-11-29 15:09:10 +01:00
“vsantonastaso”
8c1c369449 DBZ-6878 add table specific notification in initial snapshot 2023-11-29 08:32:21 +01:00
Roman Kudryashov
cf3b74bcfb DBZ-7159 Code review 2023-11-23 14:19:37 +01:00
Roman Kudryashov
1992c1e7e4 DBZ-7159 Fail fast during deserialization if a value is not a CloudEvent 2023-11-23 14:19:37 +01:00
rkerner
aeaba3ae70 DBZ-6762 Add "validate filters" endpoint/s for connector-specific Connect REST Extensions 
closes to https://issues.redhat.com/browse/DBZ-6762
 2023-11-22 06:10:40 +01:00
ruslan
daf7f57d77 DBZ-7142 Improved Outbox Event Router SMT to handle whitespaces inside of fields.additional.placement setting 2023-11-16 11:57:34 +01:00
mfvitale
a049ea92fa DBZ-7141 Remove ComputePartition SMT related code 2023-11-15 12:57:11 +01:00
Jiri Pechanec
e654dc228f DBZ-7066 Code reorganization 2023-11-14 11:56:16 +01:00
harveyyue
223b6e4335 DBZ-7066 Extract common creating transformation methods to ConnectRecordUtil 2023-11-14 11:56:16 +01:00
harveyyue
4c3b5540ec DBZ-7066 Add NewExtractNewDocumentStateTest and NewExtractNewDocumentStateTestIT for new option "delete.tombstone.handling.mode" 2023-11-14 11:56:16 +01:00
harveyyue
d21d5838fc DBZ-7066 Add NewExtractNewRecordStateTest for new option "delete.tombstone.handling.mode" 2023-11-14 11:56:16 +01:00
harveyyue
5e8287eb40 DBZ-7066 The DefaultDeleteHandlingStrategy couldn't add the rewrite "__deleted" field to a non-struct value 2023-11-14 11:56:16 +01:00
Debezium Builder
1521445908 [maven-release-plugin] prepare for next development iteration 2023-11-10 10:26:05 +00:00
Debezium Builder
6c6f6e9138 [maven-release-plugin] prepare release v2.5.0.Alpha2 2023-11-10 10:26:05 +00:00
mfvitale
33dc8e0ec6 DBZ-6317 Document tombstone behaviour 2023-11-10 06:43:18 +01:00
rkerner
68481038b3 DBZ-7105 Fix exceptions that are triggered by RelationalBaseSourceConnector#validate when calling #validateConnection() with an invalid configuration. Other, not-checked fields can cause an exception when the ConnectorConfig class is instantiated to setup the connection. Now we are only allowing connection validation when all config options are free of errors. 
closes to https://issues.redhat.com/browse/DBZ-7105
 2023-11-06 15:02:51 +01:00
Roman Kudryashov
3d56341375 DBZ-7060 Rename metadata.location -> metadata.source 2023-10-27 13:55:18 +02:00
harveyyue
767c342506 DBZ-7062 Update transformation property "delete.tombstone.handling.mode" to debezium doc 2023-10-27 11:17:02 +02:00
Debezium Builder
78c5204444 [maven-release-plugin] prepare for next development iteration 2023-10-26 15:39:35 +00:00
Debezium Builder
356e488e83 [maven-release-plugin] prepare release v2.5.0.Alpha1 2023-10-26 15:39:35 +00:00
Jiri Pechanec
13d1b4e0b2 DBZ-4999 Provide hook to allow Informix to avoid blocking 2023-10-20 11:15:17 +02:00
Roman Kudryashov
215f3098b7 DBZ-7016 Refactoring 2023-10-19 11:35:18 +02:00
Roman Kudryashov
b0d7d60b5d DBZ-7016 Optimize retrieval of id and type 2023-10-19 11:35:18 +02:00
Roman Kudryashov
89e2700cbf DBZ-7016 Fix dataschema() 2023-10-19 11:35:18 +02:00
Roman Kudryashov
22f6b083b1 DBZ-7016 Change metadata.location property syntax 2023-10-19 11:35:18 +02:00
Roman Kudryashov
ee64c70546 DBZ-7016 id and type of a CloudEvent can be retrieved from headers 2023-10-19 11:35:18 +02:00
Jiri Pechanec
9b098615ab DBZ-6907 Change package; use better strategy names 2023-10-19 09:35:15 +02:00
harveyyue
4836fcf1ea DBZ-6907 Introduce a new handling deleted record option "delete.tombstone.handling.mode" 2023-10-19 09:35:15 +02:00
harveyyue
f64ea1a5f8 DBZ-6907 Extract an abstract transformation class from ExtractNewRecordState 2023-10-19 09:35:15 +02:00
Jiri Pechanec
6dfbdb4f69 DBZ-7033 Upgrade Kafka to 3.6.0 2023-10-18 07:23:42 +02:00
ani-sha
49ff78ae9a DBZ-7020 Modify the updated schema envelope, adjust tests 2023-10-16 10:06:38 +02:00
ani-sha
d28bedb710 DBZ-7020 Use single record schema definition 2023-10-16 10:06:38 +02:00
ani-sha
c4fe510f00 DBZ-7020 Add source block struct for transformed records 2023-10-16 10:06:38 +02:00
ani-sha
8cb02eb16a DBZ-7020 Use optional schema for Timezone Converter tests 2023-10-16 10:06:38 +02:00
Vojtech Juranek
3cc70c8a52 DBZ-6778 Remove unused code 2023-10-05 11:57:59 +02:00
Vojtech Juranek
d7e8943d4e DBZ-6778 Initialize ElapsedTimeStrategy upon its creation 
Currently, newly created `ElapsedTimeStrategy` is uninitialized and its
`hasElapsed()` has to be called once `ElapsedTimeStrategy` is created to
initialize the strategy. This is confusing and error prone.

Move initialization of `ElapsedTimeStrategy` into it's constructor, so
it's initialized once it's created.
 2023-10-05 11:57:59 +02:00
Roman Kudryashov
e8e7c705af DBZ-6982 Fix property name 2023-10-04 13:58:53 +02:00
Roman Kudryashov
01485aceb3 DBZ-6982 Provide configuration option to exclude extension attributes from a CloudEvent 2023-10-04 13:58:53 +02:00
harveyyue
e80c82622f DBZ-6983 Add the ability to sanitize field name when inferencing json schema 2023-10-04 13:53:18 +02:00
Debezium Builder
40a8f4026d [maven-release-plugin] prepare for next development iteration 2023-10-03 14:18:54 +00:00
Debezium Builder
412c6f7979 [maven-release-plugin] prepare release v2.4.0.Final 2023-10-03 14:18:54 +00:00
Jakub Cechacek
baafe602ce DBZ-6973 Literal matching mode for database/collection filters and ability to override internal aggregation pipeline for MongoDB conenctor 2023-10-02 08:25:53 +02:00
Jiri Pechanec
b65582f2c0 DBZ-6966 Fix reverted filtering logic 2023-09-28 14:27:16 -04:00
mfvitale
56795732b5 DBZ-6793 Add timestamp for notifications into documentation 2023-09-28 14:20:17 -04:00
mfvitale
993413955c DBZ-6793 Add timestamp for notifications 2023-09-28 14:20:17 -04:00
ani-sha
c20f297fe1 DBZ-6940 Refactor & add check for unsupported schema types 2023-09-28 13:58:13 -04:00
Jiri Pechanec
c7caa6726f DBZ-3642 Use Java naming conventions 2023-09-26 13:06:09 +02:00
Roman Kudryashov
98c426dec4 DBZ-3642 CloudEventsConverter can retrieve metadata info from headers 2023-09-26 13:06:09 +02:00
Animesh Kumar
fcbff074f4 DBZ-6787 Dedupe table list while incremental snapshot 2023-09-26 10:50:32 +02:00
nancyxu123
b08acc6043 DBZ-6870 throw producer exception during poll() even if the queue isn't empty yet 2023-09-26 10:05:48 +02:00
Jiri Pechanec
cd2df01fe8 DBZ-6956 Add parenthesis around additional condition 2023-09-25 06:10:35 +02:00
Debezium Builder
cc1de0aa51 [maven-release-plugin] prepare for next development iteration 2023-09-22 09:00:45 +00:00
Debezium Builder
6809978083 [maven-release-plugin] prepare release v2.4.0.CR1 2023-09-22 09:00:45 +00:00
mfvitale
5d7ba3e951 DBZ-6957 Remove spaces in Signal and Notification JMX ObjectName 2023-09-21 12:04:33 -04:00
mfvitale
b4661881c9 DBZ-6945 Remove Table object for DROP schema event 2023-09-21 13:32:38 +02:00
rkerner
32161c9874 DBZ-4395 Add connector specific Debezium Connect REST Extension/s and move logic from UI backend to the Debezium Connect REST Extension (part 1 of the re-architecturing/re-factoring) 
closes to https://issues.redhat.com/browse/DBZ-4395
 2023-09-21 13:29:59 +02:00
harveyyue
e341910d2c DBZ-6910 Values being omitted from list of JSON object 2023-09-21 05:31:36 +02:00
mfvitale
68eb4b2df2 DBZ-6828 Process drop table events during blocking snapshot 2023-09-20 08:18:33 +02:00
mfvitale
ae199d2053 DBZ-6828 Blocking snapshot will generate schema events only for signaled tables 2023-09-20 08:18:33 +02:00
mfvitale
19ee1070f5 DBZ-6937 Incremental snapshot context is initialized in processSchemaChange method to avoid NPE when no context from initial snapshot exists 2023-09-20 08:14:36 +02:00
Jiri Pechanec
d83139a54a DBZ-6901 Fix formatting 2023-09-19 07:00:14 +02:00
harveyyue
07abd6dbc2 DBZ-6901 ExtractNewRecordState's schema cache is not updated with arrival of the ddl change event 2023-09-19 07:00:14 +02:00
Debezium Builder
65169a9315 [maven-release-plugin] prepare for next development iteration 2023-09-13 09:41:46 +00:00
Debezium Builder
845e3e9d09 [maven-release-plugin] prepare release v2.4.0.Beta2 2023-09-13 09:41:46 +00:00
Chris Cranford
46fa3bef4d DBZ-2543 Refactor value conversion logic 2023-09-12 18:20:07 -04:00
Jiri Pechanec
f1e7999431 DBZ-6481 Support Postgres enum as key in incremental snapshot 2023-09-11 06:08:57 +02:00
ani-sha
087362bfcc DBZ-6567 Use optional seconds in TIMEZONE_OFFSET_PATTERN, handle sub-milliseconds precision loss, correct the DST time values 2023-09-07 11:00:26 +02:00
ani-sha
d46744aa06 DBZ-6567 Add a test case for checking DST 2023-09-07 11:00:26 +02:00
ani-sha
78477e7622 DBZ-6567 Handle DST for test cases, use correct zone rules 2023-09-07 11:00:26 +02:00
ani-sha
22500839de DBZ-6567 SMT for handling timezone based fields 2023-09-07 11:00:26 +02:00
Chris Cranford
c86b2648a5 DBZ-6864 Checkstyle fixes 2023-09-04 09:02:16 +02:00
Breno Moreira
3a2e141c56 DBZ-6864 - Debezium-MySQL not filtering AWS RDS internal events 2023-09-04 09:02:16 +02:00
Jiri Pechanec
0dd0ff7306 DBZ-6865 Clarify error message 2023-09-04 08:59:31 +02:00
harveyyue
fe300784b0 DBZ-6865 Avoid getting NPE when executing the arrived method in ExecuteSnapshot 2023-09-04 08:59:31 +02:00
Debezium Builder
d97a18e8dc [maven-release-plugin] prepare for next development iteration 2023-08-29 11:41:08 +00:00
Debezium Builder
5ce25f2cfe [maven-release-plugin] prepare release v2.4.0.Beta1 2023-08-29 11:41:08 +00:00
“vsantonastaso”
3ec6a3f18e DBZ-6843 change skipped change event log to debug 2023-08-29 12:18:17 +02:00
Chris Cranford
ed88a73510 DBZ-6840 Add test case 2023-08-29 11:56:12 +02:00
harveyyue
c03b2249d8 DBZ-6840 Wrong filtered comments 2023-08-29 11:56:12 +02:00
Vojtech Juranek
c2b48312a6 DBZ-6841 Fix randomly failing BaseSourceTaskTest#verifyTaskRestartsSuccessfully 
Test was passing locally and in the CI as the start was very fast and
after the first start retriable wait condition wasn't met, so we
actually sleep and while the sleep we doesn't increment the counter
and the state is still `RESTARTING`. On slower machines when start
takes longer time and we don't wait, the test is actualy wrong.
The correct test either should remove last assert for `RESTARTING`
state or increase number of failures and add a sleep time after initial
start, so that there is no retry wait time.
This PR implements the later option.
 2023-08-29 10:27:56 +02:00
“vsantonastaso”
43adc59292 DBZ-6742 use JSON format for JMX Notitication userData 2023-08-22 06:42:51 +02:00
Chris Beard
2d6490e3f8 [docs] Fix minor typos in ExtractNewRecordState docs 2023-08-20 22:37:03 -04:00
Matan Cohen
40ac4662fb DBZ-6712 fix snapshot to history internal topic to respect schema.history.internal.store.only.captured.databases.ddl flag 2023-08-15 19:34:40 +02:00
mfvitale
542b3619fa DBZ-6731 Blocking snapshot takes configuration from signal payload 2023-08-15 12:43:43 +02:00
“vsantonastaso”
64eaf72665 DBZ-6689 make kafka signal consumer group configurable 2023-08-11 13:20:07 +02:00
Anil Dasari
b7e0e1cd2a DBZ-6641: Schema name changed with Custom topic naming strategy 2023-08-10 11:53:31 +02:00
Anil Dasari
a4c0bf5614 DBZ-6641: Schema name changed with Custom topic naming strategy 2023-08-10 11:53:31 +02:00
Jiri Pechanec
ec100d6127 DBZ-6240 Rename vars to reflect the semantics 2023-08-10 07:04:17 +02:00
sunxiaojian
43c7c0a3f6 DBZ-6240 Rename the configuration name 'schema.change.event.include.list' to 'schema.change.event.exclude.list' 2023-08-10 07:04:17 +02:00
sunxiaojian
6451c93256 DBZ-6240 Add an STM to filter schema change event 2023-08-10 07:04:17 +02:00
Debezium Builder
3c72f071b0 [maven-release-plugin] prepare for next development iteration 2023-08-09 10:33:06 +00:00
Debezium Builder
915c465440 [maven-release-plugin] prepare release v2.4.0.Alpha2 2023-08-09 10:33:06 +00:00
david remy
565075a320 DBZ-2862 Improved OpenTelemetry Handling in Debezium 
- Implemented a dynamic solution for handling version changes in OpenTelemetry, using Java introspection to determine the correct interceptor class at runtime.
- Utilized a custom "ProducerInterceptor" to delegate logic based on the presence of specific OpenTelemetry classes.
- Removed direct usage of OpenTelemetry's internal API 'ConfigUtils'.
- Aligned solution with Debezium's aim of ensuring compatibility with Strimzi.

This update ensures a more robust handling of OpenTelemetry's version transitions.
 2023-08-08 21:34:04 +02:00
REMY David
15fdcb5027 DBZ-2862 Performance optimization 
- Changing the traceRecord method to be static. This avoids creating an instance on each call.
- Caching the results of isOpenTelemetryJavaagentEnable() and isOpenTelemetryApiEnable().
- Using isDebugEnabled() to avoid the costs of constructing strings when it's not necessary.
- Reorganizing some conditions to minimize the operations performed.
- Moving the static instances TEXT_MAP_PROPAGATOR, SETTER, and GETTER into the traceRecord method. This will prevent them from being held in memory when not in use.
 2023-08-08 21:34:04 +02:00
REMY David
86e5288656 DBZ-2862 Removed reference to Strimzi in comment 2023-08-08 21:34:04 +02:00
REMY David
76c536ec23 DBZ-2862 Adaptation of traceability with OpenTelemetry dependencies in the scope provided. Moved the interceptor to a new module. 2023-08-08 21:34:04 +02:00
REMY David
5846446672 DBZ-2862 Switch tracing to OpenTelemetry, Rewrite SMT with Otel API, Add Interceptor for e2e tracing, add Otel TextMap getter and setter 2023-08-08 21:34:04 +02:00
mfvitale
df52a1068a DBZ-6777 Fix wrong static notification and signals lists that lead to be wrongly shared between connectors instances 2023-08-08 13:01:37 +02:00
Vojtech Juranek
41cdaeb170 DBZ-6213 Adjust connector restart 
* Try to start the connector in BaseSourceTask::start(). This is the
  root cause of DBZ-6213 as EmbeddedEngine calls start() and assumes
  it really tries to start the connector.
* Simplify the code by removing unnecessary checks.
 2023-08-08 10:27:33 +02:00
Jiri Pechanec
e5ca8c5000 DBZ-6684 Use more descriptive class name 2023-08-08 10:22:10 +02:00
harveyyue
b16d991a25 DBZ-6684 Rename back fields with avro_unicode 2023-08-08 10:22:10 +02:00
tyrantlucifer
32d2bf1fac DBZ-6745 Remove redundant filter logic from SingleProcessor 2023-08-03 16:52:41 +02:00
mfvitale
524987227f DBZ-6566 Shutdown blockingSnapshotExecutor when ChangeEventSourceCoordinator stops 2023-08-03 09:22:24 +02:00
mfvitale
61651a94b5 DBZ-6566 Force blocking snapshot even if a snapshot has already executed 2023-08-03 09:22:24 +02:00
mfvitale
def7379a0d DBZ-6566 Support Blocking snapshot for MySQL 2023-08-03 09:22:24 +02:00
mfvitale
6e0b323e83 DBZ-6566 Set correct log context during blocking snapshot 2023-08-03 09:22:24 +02:00
mfvitale
70f16af7c4 DBZ-6566 Support Blocking snapshot for PostgreSQL 2023-08-03 09:22:24 +02:00
jcechace
699a1612ae DBZ-6670 Replace error handler instead of recreate in order to keep retry counter 2023-07-25 13:46:56 +02:00
mfvitale
8ea2b3c5b7 DBZ-6669 Snapshot will get data when only signal data collection table is specified 2023-07-24 10:02:13 -04:00
Debezium Builder
267dfda8a6 [maven-release-plugin] prepare for next development iteration 2023-07-17 10:40:52 +00:00
Debezium Builder
0d3cef6a29 [maven-release-plugin] prepare release v2.4.0.Alpha1 2023-07-17 10:40:52 +00:00
Chris Cranford
de8cf36b2c DBZ-6660 Introduce internal.log.mining.max.scn.deviation.ms 
We hypothesize that there could be a situation where we may be mining precisely
around the CURRENT_SCN and this may lead to situations where LGWR may not have
flushed all records for the same SCN before being mined by the connector.
 2023-07-17 10:22:46 +02:00
Chris Cranford
56ddbe21d5 DBZ-6650 Avoid ConcurrentModificationException during dispatch 2023-07-14 11:21:39 +02:00
harveyyue
e3f684170e DBZ-6472 MySqlSnapshotChangeEventSource parallel execute createSchemaEventsForTables 2023-07-13 13:21:59 +02:00
Adam Strickland
0b15fe4556 [docs] Doc of type of return value 2023-07-13 13:08:41 +02:00
Jiri Pechanec
764427081b DBZ-6517 DBZ-6654 Make CE converter compatible with Kafka 3.5 2023-07-13 11:26:21 +02:00
harveyyue
aa139203aa DBZ-6603 Update property "custom.metric.tags" to connector docs 2023-07-13 09:52:33 +02:00
harveyyue
5bc7fd1f7c DBZ-6603 Support for custom tags in the connector metrics 2023-07-13 09:52:33 +02:00
jcechace
5cb579bf12 DBZ-6525 Removing runtime dependency on core for testcontainers module in order to prevent logback.xml downstream conflicts 2023-07-11 12:07:04 +02:00
Chris Cranford
6a41761f85 DBZ-6573 Restore spacing 2023-06-30 09:39:32 +02:00
tjwornjs
2767f469aa DBZ-6573 add max retries config to common config 2023-06-30 09:39:32 +02:00
bdbene
6f29ce08ef DBZ-6605 Fix DataCollections for snapshot completion notification 
DBZ-6605 Use new 'scanned_collection' field for snapshot completion signal

Updated the docs for fixed snapshot completion notification
 2023-06-27 12:01:26 +02:00
mfvitale
d5febcc054 DBZ-6417 Provide extensible signal action mechanism through SPI 2023-06-23 15:03:27 +02:00
mfvitale
9ed928a1c9 DBZ-6416 Improve tests stability 2023-06-22 13:22:34 +02:00
mfvitale
34e28ac52d DBZ-6416 Add initial snapshot notifications 2023-06-22 13:22:34 +02:00
mfvitale
7d28833ba8 DBZ-6588 Skip tombstone record on HeaderToValue SMT 2023-06-21 14:11:44 +02:00
jcechace
8f70de42c1 DBZ-6578 Cleaning up connection string propagation between discovery and tasks 2023-06-21 12:22:23 +02:00
Debezium Builder
60f730d383 [maven-release-plugin] prepare for next development iteration 2023-06-20 13:21:31 +00:00
Debezium Builder
150396e4c8 [maven-release-plugin] prepare release v2.3.0.Final 2023-06-20 13:21:31 +00:00
mfvitale
91152c4c5d DBZ-6559 Add test to cover topic name strategy 2023-06-16 11:48:34 +02:00
Jiri Pechanec
7f81f4bc6b DBZ-6559 Handle keeping the first char as exception 2023-06-16 11:48:34 +02:00
mfvitale
87217a1474 DBZ-6559 Restore backward compatibility for field that stars with a digit 2023-06-16 11:48:34 +02:00
Jiri Pechanec
067683e56a DBZ-6076 Consistent config for all connectors 2023-06-15 11:55:28 +02:00
Anil Dasari
0431a8b9fc DBZ-6076: Add support for custom SourceInfoStructMaker for adding new fields to source field 2023-06-15 11:55:28 +02:00
Chris Cranford
ef8059f3ff DBZ-6528 Fix Oracle NPE when storing only captured table schemas with signals 2023-06-14 10:48:19 +02:00
mfvitale
b7eb4b5c62 DBZ-6543 Fix NPE due to optional fields in PartitionRouting SMT 2023-06-14 07:22:27 +02:00
Andrei Isac
573a91be07 DBZ-6533: fix table order on partial match when snapshotting 2023-06-12 15:40:03 +02:00
kate
24d3425fdf DBZ-6552 Notify when incremental snapshot fails 2023-06-12 15:14:40 +02:00
mfvitale
3aaca16024 DBZ-6543 Fix NPE due to optional fields in PartitionRouting SMT 2023-06-09 13:47:18 +02:00
Debezium Builder
5d1b00588a [maven-release-plugin] prepare for next development iteration 2023-06-09 09:23:38 +00:00
Debezium Builder
d1e6107517 [maven-release-plugin] prepare release v2.3.0.CR1 2023-06-09 09:23:37 +00:00
Vojtech Juranek
b795519514 DBZ-6529 Use Murmur hash for PartitionRouting SMT 
Add option to use better hash function than default Java Object::hash
function to get better hashes which would be more equally spred over
the hash space and thus more equally over the Kafka partitions.
To preserve backward compatibility, previous Java `hashCode` function
is used a default.

Add Murmur3 hash function for computing hashes of the fields.
Murmur3 implementation is taken from from Infinispan project code base.

To allow users eventually use their own make `computePartition()`
protected so it can be overriden in the subclasses.
 2023-06-07 11:48:30 +02:00
mfvitale
61e75cc00d DBZ-6424 Move JMX register/unregister code from Metrics to JmxUtils 2023-06-07 07:36:58 +02:00
mfvitale
b0b968fcce DBZ-6424 Move JMX register/unregister code from Metrics to JmxUtils 2023-06-07 07:36:58 +02:00
mfvitale
394a9e32d8 DBZ-6424 Move JMX register/unregister code from Metrics to JmxUtils 2023-06-07 07:36:58 +02:00
mfvitale
7338fde84e DBZ-6424 Emit JMX Notification when Debezium generate its own notification 
DBZ-6424 Emit JMX Notification when Debezium generate its own notification

DBZ-6424 Emit JMX Notification when Debezium generate its own notification
 2023-06-07 07:36:58 +02:00
mfvitale
769d9361e7 DBZ-6424 Add JMX Notification channel 
This channel permits to send Debezium notification to JMX bean

DBZ-6424 Add JMX Signal channel

This channel permits to send signal to Debezium through the JMX operation
 2023-06-07 07:36:58 +02:00
mfvitale
f93affc3ca DBZ-6511 Add java docs for SignalProcessor getSignalChannel method 2023-06-05 17:33:07 +02:00
mfvitale
ba8a3e5062 DBZ-6511 Expose reset method on SignalChannelReader to reset processed signals 2023-06-05 17:33:07 +02:00
mfvitale
4d56bbd15a DBZ-6511 Restore Kafka signal offset only when KafkaSignalChannel is enabled 2023-06-05 17:33:07 +02:00
Ronak Jain
3f3967c2bb DBZ-6366: Code Improvements for skip.messages.without.change 
Addresses/Closes DBZ-6366
 2023-06-05 14:50:25 +02:00
Jiri Pechanec
8a1cd95086 DBZ-6515 Add units to the description 2023-06-05 11:27:57 +02:00
mfvitale
155207dc9b DBZ-6509 Add FileSignalChannel to META-INF/services 2023-05-31 14:39:35 +02:00
mfvitale
e1c1e65336 DBZ-6408 Add documentation for notifications 2023-05-30 17:17:52 +02:00
Debezium Builder
aa5b2131fc [maven-release-plugin] prepare for next development iteration 2023-05-26 10:35:20 +00:00
Debezium Builder
9cc9b3c29a [maven-release-plugin] prepare release v2.3.0.Beta1 2023-05-26 10:35:20 +00:00
mfvitale
ed2157bfc9 DBZ-6486 ExtractNewRecordState skip fields that not exists on payload 2023-05-26 11:30:37 +02:00
Jiri Pechanec
b72f084eac DBZ-6486 Check schema fields 2023-05-26 11:30:37 +02:00
mfvitale
08122371eb DBZ-6486 ExtractNewRecordState can now take arbitrary fields from payload 2023-05-26 11:30:37 +02:00
mfvitale
7ba9f57035 DBZ-6496 Fix signal.poll.interval.ms default value 2023-05-24 18:51:49 +02:00
mfvitale
dc00f64bbf DBZ-6447 Incremental snapshot notification ids are now equal to the id sent in the execute-snapshot signal. 2023-05-23 09:57:05 +02:00
harveyyue
016daf6fd8 DBZ-6490 Convert the field BigDecimal value to bytes failure of ApproximateStructSizeCalculator 2023-05-23 08:38:06 +02:00
Chris Cranford
d40da4f810 DBZ-6457 Use PDB for parallel snapshot connections 2023-05-19 14:04:14 +02:00
Ismail Simsek
2683b31394 DBZ-4027 implement FileSignalChannel 2023-05-15 15:34:43 +02:00
Vojtech Juranek
ee4ebaa957 DBZ-6439 Don't scan all the tables when loading schema 
Currently we scan all the tables, which may result into a substantial
delay in initial snapshot when the database is very large. We need to
filter out tables which we are not interested in.

Add back table filter when loading schema of tables. As per comment of
this block of code, passing all tables and table filter should be faster
than passing only list of tables we are interested in.
 2023-05-15 14:11:47 +02:00
Vojtech Juranek
f4596fa2e0 DBZ-6460 Enable setting log level in tests via log interceptor 2023-05-15 14:11:47 +02:00
mfvitale
85278e76af DBZ-6462 Stop SignalProcessor after test execution in SignalProcessorTest 2023-05-15 08:51:02 +02:00
mfvitale
e41bbe7191 DBZ-6462 Schedule SignalProcessor task with a more relaxed scheduling. Removes parallelStreams while processing signalChannelReaders 2023-05-15 08:51:02 +02:00
Debezium Builder
a5b44767bc [maven-release-plugin] prepare for next development iteration 2023-05-11 11:05:24 +00:00
Debezium Builder
1b1a7531f2 [maven-release-plugin] prepare release v2.3.0.Alpha1 2023-05-11 11:05:24 +00:00
mfvitale
7429b4e6f1 DBZ-1973 Add incremental snapshot notifications 2023-05-10 12:22:02 +02:00
mfvitale
811b800f55 DBZ-1973 Add support for Notifications 
DBZ-1973 Add more tests for other connectors

DBZ-1973 Add send method with offset parameter

DBZ-1973 Instantiate NotificationService in the task class

DBZ-1973 Instantiate NotificationService in the task class

DBZ-4027 Move specific sink channel configuration to SinkNotificationChannel

DBZ-4027 Remove not used SPI file

DBZ-1973 Moved SPI file definition to debezium-core

DBZ-1973 Rename KafkaNotificationChannel to more generic SinkNotificationChannel

DBZ-1973 Code refactor

DBZ-1973 Improve configuration property description

DBZ-1973 Improve test

DBZ-1973 Add close method to NotificationChannel

DBZ-1973 Implement KafkaNotificationChannel

DBZ-1973 Add NotificationService and LogNotificationChannel
 2023-05-10 12:22:02 +02:00
mfvitale
0ca87d8031 DBZ-4027 During event dispatch process only signal from source channel 2023-05-05 13:35:40 +02:00
mfvitale
952340286e DBZ-4027 Make signalling channel configurable 
DBZ-4027 Add an Incremental snapshot test with kafka signaling

DBZ-4027 Add an Incremental snapshot test with kafka signaling

DBZ-4027 Add an Incremental snapshot test with kafka signaling

DBZ-4027 Code style

DBZ-4027 Make SignalPayload more generic and extensible

DBZ-4027 Rename DatabaseSignalChannel to SourceSignalChannel

DBZ-4027 Improve logging

DBZ-4027 Moved SPI file definition to debezium-core

DBZ-4027 Move SignalProcessor synchronization point to be processed only when a signal cdc event arrives.

DBZ-4027 Add EventDispatcher constructor without signalProcessor for spanner connector

DBZ-4027 Fix NPE

DBZ-4027 Fix NPE

DBZ-4027 Formatting

DBZ-4027 Correctly manage signal on not supported connector

DBZ-4027 Use the correct MongoDbOffset

DBZ-4027 Correctly initialize offset for Oracle and SqlServer connectors

DBZ-4027 Register SPI implementations

DBZ-4027 Improve SignalProcessor instantiation

DBZ-4027 Pass source info in case of SchemaChanges action

DBZ-4027 Manage close event in a synchronous way

DBZ-4027 Correctly init offset context also in case of snapshot mode 'never'

DBZ-4027 Fix MySqlMetricsIT test

DBZ-4027 Move KafkaSignalChannel to core

DBZ-4027 Move KafkaSignalChannel to core

DBZ-4027 Set pass offset context after initial snapshot to SignalProcessor

DBZ-4027 Pass OffsetContext to signal processor

DBZ-4027 Pass CommonConnectorConfig to SignalChannelReader init method

DBZ-4027 Move Incremental snapshot window actions to dedicated package

DBZ-4027 Align SignalsIT test with new code

DBZ-4027 Fix SignalsIT test

DBZ-4027 Fix SignalProcessor scheduling

DBZ-4027 Moved DatabaseSignalChannel and SignalChannelReader to dedicated package

DBZ-4027 Start SignalProcessor from ChangeEventSourceCoordinator

DBZ-4027 Create SignalProcessor and renamed Signal to DatabaseSignalChannel

DBZ-4027 Initial refactoring of signal feature
 2023-05-05 13:35:40 +02:00
Jiri Pechanec
58ef4f0b98 DBZ-6157 Disable potentially dangerous MySQL JDBC props 2023-05-04 11:54:04 +02:00
Jiri Pechanec
6b1455666a DBZ-6411 Message content logged only at TRACE 2023-05-03 06:51:28 +02:00
mfvitale
e5d4332835 DBZ-6411 Fix HeaderToValue schema update cache. 
When the connector starts for the first time with initial snapshot record with read operation will be created. In this phase the ExtractChangedRecordState will produce no headers and since HeaderToValue was not skipping record without headers the cache of the new schema will be created only with the old fields.
 2023-05-03 06:51:28 +02:00
harveyyue
3860a4faca DBZ-6388 __source_ts_ms r (read) operation date is set to future for SQL Server 2023-05-02 08:11:02 -04:00
Chris Cranford
4664dd6df6 DBZ-6355 Add log.mining.transaction.retention.ms; deprecate log.mining.transaction.retention.hours 2023-04-27 09:28:29 -04:00
eizners
fa42a1b0ce DBZ-6355 fix unit tests errors 2023-04-27 09:28:29 -04:00
Sergey Eizner
cac5485176 DBZ-6355 Change to the last offset 2023-04-27 09:28:29 -04:00
Jiri Pechanec
22e19c0ff3 DBZ-6209 Extract additional common code 2023-04-27 08:13:58 +02:00
harveyyue
795f0dd977 DBZ-6209 Refactor storage implementations 2023-04-27 08:13:58 +02:00
Jiri Pechanec
8c52ccfd0c DBZ-6391 Use incdluded term for consistency 2023-04-26 12:31:15 +02:00
zalmane
2b8bf33876 DBZ-6391 change logging level 2023-04-26 12:31:15 +02:00
Debezium Builder
bb1b79f3bd [maven-release-plugin] prepare for next development iteration 2023-04-20 09:09:43 +02:00
Debezium Builder
5483b4de99 [maven-release-plugin] prepare release v2.2.0.Final 2023-04-20 09:09:43 +02:00
Ronak Jain
762122ecf5 DBZ-2979 If column.include.list/column.exclude.list are used and the target table receives an update for the excluded (or not included) column - such events should be ignored 
* Add Mysql and SqlServer Connector Changes
* Change Docs
* Clean Test Case
* Formatting and Checkstyle

Addresses/Closes DBZ-2979
 2023-04-19 09:04:29 +02:00
Jiri Pechanec
5eb5e3555d DBZ-6324 Rename flaky test config option 2023-04-18 08:22:43 +02:00
Jiri Pechanec
ab66f038e4 DBZ-6324 Add retry for flaking tests 2023-04-18 08:22:43 +02:00
Jiri Pechanec
f64c35de29 DBZ-6324 Provide @Flaky annotation for CI unstable tests 2023-04-18 08:22:43 +02:00
Debezium Builder
976a47a3a3 [maven-release-plugin] prepare for next development iteration 2023-04-14 08:46:06 +00:00
Debezium Builder
1ca8399e8b [maven-release-plugin] prepare release v2.2.0.CR1 2023-04-14 08:46:06 +00:00
Jiri Pechanec
8a648fbb4d DBZ-4488 Refactor code to keep error handler immutable 2023-04-13 08:22:45 +02:00
Jacob Gminder
5e615ff796 DBZ-4488: Reuse the ErrorHandler between retries 2023-04-13 08:22:45 +02:00
Jiri Pechanec
84d0d580bc DBZ-4488 Refactor the code 2023-04-13 08:22:45 +02:00
Jacob Gminder
9d5b6df0f7 DBZ-4488: Allow SQL Server connectors to set a maximum number of retriable exception restarts. 2023-04-13 08:22:45 +02:00
Jiri Pechanec
943b9485d7 DBZ-6216 Remove registry name for tag query 2023-04-12 14:07:35 +02:00
harveyyue
c525f34cd0 DBZ-6070 DDL events not stored in schema history topic for excluded tables 2023-04-06 12:10:30 +02:00
Jiri Pechanec
39ee498e64 DBZ-5902 Fix wording 2023-04-06 10:55:19 +02:00
mfvitale
bad6ad6bae DBZ-5902 Change fields configuration from 'partition.payload.field' to 'partition.payload.fields' 2023-04-06 10:55:19 +02:00
mfvitale
41416293ca DBZ-5902 Refactor and better field validation 2023-04-06 10:55:19 +02:00
mfvitale
fdf79b0379 DBZ-5902 Add warn log for deprecation 2023-04-06 10:55:19 +02:00
mfvitale
34f021988e DBZ-5902 Add new PartitionRouting SMT that will replace ComputePartition 2023-04-06 10:55:19 +02:00
mfvitale
6e9528a47d DBZ-5902 Deprecate ComputePartition SMT 2023-04-06 10:55:19 +02:00
mfvitale
80ac1b362f DBZ-5902 Moved list validation to CommonConnectorConfig 2023-04-06 10:55:19 +02:00
Debezium Builder
5c650fd054 [maven-release-plugin] prepare for next development iteration 2023-03-31 10:00:16 +00:00
Debezium Builder
68ffc4833b [maven-release-plugin] prepare release v2.2.0.Beta1 2023-03-31 10:00:16 +00:00
Jiri Pechanec
b6c984e4de DBZ-6072 Sort imports 2023-03-28 17:06:58 +02:00
Jiri Pechanec
0e42935b35 DBZ-6072 Use more descriptive name 2023-03-28 16:59:05 +02:00
harveyyue
5dd4b53ec3 DBZ-6072 NPE when setting schema.history.internal.store.only.captured.tables.ddl=true 2023-03-28 16:59:05 +02:00
harveyyue
2f46c83b2a DBZ-6104 java.lang.NullPointerException in MySQL connector with max.queue.size.in.bytes 2023-03-28 14:51:52 +02:00
harveyyue
c9c1ce8a2f DBZ-6185 Zerofill property failed for different int types 2023-03-28 09:34:58 +02:00
Jiri Pechanec
e3fdd873e1 DBZ-6201 Skip expensive logging 2023-03-22 14:02:44 +01:00
mfvitale
2d70f47c40 DBZ-6201: Add more fail test scenarios 2023-03-22 14:02:44 +01:00
mfvitale
16191a5a35 DBZ-6201: Add logs 2023-03-22 14:02:44 +01:00
mfvitale
039a3f04e8 DBZ-6201: Add support for move and nested fields for HeaderToValue SMT 2023-03-22 14:02:44 +01:00
mfvitale
c45bf39eb0 DBZ-6201: Add new HeaderToValue SMT to copy headers to value 2023-03-22 14:02:44 +01:00
mfvitale
4e544debdc DBZ-6201: Fix validation message that reports null as value 2023-03-22 14:02:44 +01:00
Chris Cranford
1c5f14f7fd DBZ-6225 DBZ-6226 DBZ-6231 Introduce CustomConverter for JDBC sink consistency 
In DBZ-6225, it was identified that the schema type associated with TINYINT(1)
and BOOLEAN data types wasn't consistent. This converter implementation will
always emit BOOLEAN data types as INT16, similar to TINYINT(1).

In DBZ-6226, it was identified that the schema type associated with REAL data
types wasn't consistent, emitted as FLOAT64 during snapshot and FLOAT32 in
the streaming phase. This converter implementation will always emit such types
as FLOAT64 for consistency.

In DBZ-6231, during streaming the DDL parser will get the column type as the
literal type supplied by the user's DDL, i.e. NCHAR or NVARCHAR but when the
DDL is sourced from SHOW CREATE TABLE, MySQL returns these columns as types
of CHAR/VARCHAR but with a character set.  By passing the character_set in
the schema consistency, we can deduce nationalized vs non-nationalized.
 2023-03-21 14:53:56 +01:00
harveyyue
1877906dd8 DBZ-6197 Remove duplicated createDdlFilter method from historized connector config 2023-03-13 10:56:43 +01:00
Nir Levy
16f460d2d7 DBZ-6190: Fix support for ddl filter 2023-03-09 08:48:43 +01:00
Debezium Builder
55d65e9618 [maven-release-plugin] prepare for next development iteration 2023-03-08 11:38:17 +00:00
Debezium Builder
6d183c3dc1 [maven-release-plugin] prepare release v2.2.0.Alpha3 2023-03-08 11:38:16 +00:00
subodh
b781960019 DBZ-6179 Fix swallowing of exception cause of finally block in RelationalSnapshotChangeEventSource 2023-03-06 10:28:57 +01:00
Jiri Pechanec
98dcdb8f55 DBZ-6023 Remove nnecessary code 2023-03-06 10:25:59 +01:00
PlugaruT
7d0492fee5 DBZ-6023 Unit tests for buildChunkQuery and buildMaxPrimaryKeyQuery methods 2023-03-06 10:25:59 +01:00
PlugaruT
472162e46e DBZ-6023 Add support for providing a surrogate key when triggering incremental snapshots 
For tables that have a composite Primary Key, SQL queries for incremental snapshot perform bad. This change allows to provide a surrogate key that will be used for the queries. The key should be unique in order for the snapshot to guarantee consistency.
 2023-03-06 10:25:59 +01:00
Jiri Pechanec
0585c6beb8 DBZ-823 Add NULL instance for the class 2023-03-03 12:47:15 +01:00
Jiri Pechanec
d492ec892c DBZ-823 Use enum for enumeration 2023-03-03 12:47:15 +01:00
Jiri Pechanec
3da2434c6c DBZ-823 Restrict the type with generics 2023-03-03 12:47:15 +01:00
Jiri Pechanec
5e666f00ee DBZ-823 Add JavaDoc to new interfaces 2023-03-03 12:47:15 +01:00
Jiri Pechanec
7d92dc0211 DBZ-823 Don't use getter based name 2023-03-03 12:47:15 +01:00
Jiri Pechanec
0f4adb9fe6 DBZ-823 Use better name for CF with main connection 2023-03-03 12:47:15 +01:00
ggaborg
e875e784a0 DBZ-823 CR changes 2023-03-03 12:47:15 +01:00
ggaborg
065dc997b9 DBZ-823 Optionally parallelize initial snapshots 2023-03-03 12:47:15 +01:00
Jiri Pechanec
0656de70c9 DBZ-6156 Add an extra test case 2023-03-03 11:20:21 +01:00
jcechace
7d3cca89bd DBZ-6032 MongoConnection allways rethrows InterruptedException 2023-03-03 05:46:33 +01:00
jcechace
922191cdd7 DBZ-6032 Connection management refactoring 2023-03-03 05:46:33 +01:00
harveyyue
440192851d DBZ-6156 Add hostname validator to connector configuration 2023-02-28 13:22:14 +01:00
Vojtech Juranek
0868bfd55e DBZ-3594 Remove unused deprecated method 2023-02-23 09:26:07 -05:00
Debezium Builder
2245ceda44 [maven-release-plugin] prepare for next development iteration 2023-02-16 11:28:25 +00:00
Debezium Builder
20f3b16e8a [maven-release-plugin] prepare release v2.2.0.Alpha2 2023-02-16 11:28:25 +00:00