Jakub Cechacek
acdb8bee8d
DBZ-7260 Merge now unnecessary client factory into MongoDbConnectionContext
2024-01-11 09:58:24 +01:00
Jakub Cechacek
26e46c9f55
DBZ-7260 Streamlined MongoDbConnection handling
2024-01-11 09:58:24 +01:00
Jakub Cechacek
6e946cfa60
DBZ-7260 Replaced raw config with typed connector configuration
2024-01-11 09:58:24 +01:00
Jakub Cechacek
4397468baa
DBZ-7260 MongoDB connector validation now requires RS name to be specified when connecting to RS cluster
2024-01-11 09:58:24 +01:00
Jakub Cechacek
a5c24787ba
DBZ-7260 MongoDbConnector javadoc + making LOGGER static to allign with the prevalent style used elsewhere
2024-01-11 09:58:24 +01:00
Jakub Cechacek
e7c7131e05
DBZ-7260 Ability to consolidate previous offsets from replica_set connection mode and some surounding refactorings
2024-01-11 09:58:24 +01:00
Jakub Cechacek
e01b847f7d
DBZ-7260 Ability to consolidate offsets previously recorded by the removed replica_set connection mode
2024-01-11 09:58:24 +01:00
Jakub Cechacek
0442fcf6fa
DBZ-7260 Removed unused code
2024-01-11 09:58:24 +01:00
Jakub Cechacek
99439c3cfa
DBZ-7260 Removed RS name from CollectionId
2024-01-11 09:58:24 +01:00
Jakub Cechacek
4c64e28832
DBZ-7260 Merged connection string and task connection string properties to futher simplify RS name usage
2024-01-11 09:58:24 +01:00
Jakub Cechacek
8def3bf82c
DBZ-7260 Moved offset management from SourceInfo to MongoDbOffsetContext
2024-01-11 09:58:24 +01:00
Jakub Cechacek
7ad4701253
DBZ-7260 Further elimination of RS references, mainly from SourceInfo
2024-01-11 09:58:24 +01:00
Jakub Cechacek
15d9b8a791
DBZ-7260 Removed ReplicaSet class
2024-01-11 09:58:24 +01:00
Jakub Cechacek
85536fbf59
DBZ-7260 Flattening offsetContext and partition structue
2024-01-11 09:58:24 +01:00
Jakub Cechacek
2aa072830a
DBZ-7260 Adjusted snapshot threads to reflect RS connection mode removal
2024-01-11 09:58:24 +01:00
Jakub Cechacek
1c413db228
DBZ-7260 Removed redundant RS parameters from event sources
2024-01-11 09:58:24 +01:00
Jakub Cechacek
d564261e6e
DBZ-7260 Compeltely removed ReplicaSetDiscovery
2024-01-11 09:58:24 +01:00
Jakub Cechacek
1f61140e40
DBZ-7260 Removed RS from connection management
2024-01-11 09:58:24 +01:00
Jakub Cechacek
86ce26bf4a
DBZ-7260 Removed RS from MongoDbChangeEventSourceFactory
2024-01-11 09:58:24 +01:00
Jakub Cechacek
85fdf034e5
DBZ-7260 Removed ReplicaSets class
2024-01-11 09:58:24 +01:00
Jakub Cechacek
17cd2158bd
DBZ-7260 Removed RS monitoring as there is now only single RS which is always a wrapper around provided connection string
2024-01-11 09:58:24 +01:00
Jakub Cechacek
72e88ffc7d
DBZ-7260 Removed RS connection mode, there is always only single RS now
2024-01-11 09:58:24 +01:00
mfvitale
20fbe73bc8
DBZ-7311 Use a more meaningful name for identify an on demand blocking snapshot
2024-01-08 12:52:03 +01:00
mfvitale
47cbdee526
DBZ-7311 Permits to execute a blocking snapshot even if snapshot.mode=never
2024-01-08 12:52:03 +01:00
Richard Harrington
da5143a7ee
DBZ-7287 Clarify comment about how we extract and serialize document ids
2024-01-05 11:20:24 +01:00
Mickael Maison
70e7186868
DBZ-7298 Add service loader manifests for all Connect plugins
...
Add manifests for source/sink connectors, converters and transformations.
2024-01-04 09:22:29 +01:00
Debezium Builder
3853d20f44
[maven-release-plugin] prepare for next development iteration
2023-12-21 06:52:01 +00:00
Debezium Builder
5d35e9caaa
[maven-release-plugin] prepare release v2.5.0.Final
2023-12-21 06:52:01 +00:00
ani-sha
4b36ed9975
DBZ-7177 DBZ-6764 Use correct context name and add suggestions for code review
2023-12-20 08:56:56 +01:00
ani-sha
5a27d42257
DBZ-7177 Add isStreaming check for all tests, remove unused files
2023-12-20 08:56:56 +01:00
ani-sha
a12b1a9872
DBZ-6764 Connector metrics implementation for REST extension
2023-12-20 08:56:56 +01:00
ani-sha
e535e5b78d
DBZ-7177 Add suggestions from code review, adjust code style changes
2023-12-20 08:56:56 +01:00
ani-sha
2555a1cee2
DBZ-6764 DBZ-7178 DBZ-7177 Use MBeanServer to query metrics for REST endpoint and create new metrics descriptor
2023-12-20 08:56:56 +01:00
ani-sha
2165d9bed1
DBZ-6764 Connector metrics implementation for REST extension
2023-12-20 08:56:56 +01:00
Roman Kudryashov
516aa87fad
DBZ-7235 Add option to customize CloudEvents schema name
2023-12-20 06:53:37 +01:00
harveyyue
225277c5c0
DBZ-7251 Return generic DataCollectionId instead of REST response DataCollection in getMatchingCollections method
2023-12-19 12:53:57 +01:00
harveyyue
4719396ffc
DBZ-7145 Introduce a common parent connect for all Debezium source connectors
2023-12-19 12:53:57 +01:00
harveyyue
5553059fb8
DBZ-7167 Provide a public API from the connector implementations to retrieve the list of matching collections or tables based on the different include-/exclude lists
2023-12-19 12:53:57 +01:00
Jakub Cechacek
67260f8286
DBZ-7272 Guarding against implicit offset invalidation cause by the change of default connection mode
2023-12-19 06:45:02 +01:00
Jakub Cechacek
0dc97b4d79
DBZ-7264: Propagating exception when using multiple snapshotting threads
2023-12-19 06:25:22 +01:00
Debezium Builder
2c1def7241
[maven-release-plugin] prepare for next development iteration
2023-12-14 09:43:13 +00:00
Debezium Builder
ef8260f802
[maven-release-plugin] prepare release v2.5.0.CR1
2023-12-14 09:43:12 +00:00
Chris Cranford
ada157d117
DBZ-4321 Introduce BeanRegistry
and ServiceRegistry
...
As a part of this work to handle injection in a cleaner way, this commit
adds two new broad concepts called `BeanRegistry` and `ServiceRegistry`.
A BeanRegistry is a glorified registry of different objects that are not
necessarily services but may be desired by a service. This contract will
allow Debezium to integrate in the future with other CDI providers.
A ServiceRegistry is more of an internal concept, where various systems
can be started based on their dependency order and provides a universal
way to split larger parts of the code into smaller, focused modules that
can be accessed using the Service Locator pattern.
2023-12-13 11:27:40 -05:00
Chris Cranford
26b3881941
DBZ-4321 Add MongoDB hook & polish injection handler
2023-12-13 11:27:40 -05:00
Jakub Cechacek
bde91ecf0f
DBZ-7252 Using correct offsets in table/collection notifications for MongoDB
2023-12-12 19:52:14 +01:00
Chris Cranford
aea6cc003d
[ci] Use a relaxed property check for tests
2023-12-06 18:04:59 -05:00
mfvitale
5ed16284f4
DBZ-6834 Provide INSERT/DELETE semantics for MongoDb incremental snapshot watermarking
2023-12-06 14:10:26 +01:00
Debezium Builder
0fd1c0dc9a
[maven-release-plugin] prepare for next development iteration
2023-12-04 13:55:35 +00:00
Debezium Builder
3e2d75f0da
[maven-release-plugin] prepare release v2.5.0.Beta1
2023-12-04 13:55:35 +00:00
Jakub Cechacek
355041dbaf
DBZ-7184 buffer lock is now acquired with timeout and reattempted on failure
2023-12-04 06:58:28 +01:00
Jakub Cechacek
88a64bf3d5
DBZ-7184 Resumable events are now guaranteed to contain resume token
2023-12-04 06:58:28 +01:00
Jakub Cechacek
9ad6f9c7db
DBZ-7184 Streaming loop now always uses resumable event to get resume token
2023-12-04 06:58:28 +01:00
Jakub Cechacek
8a5007a738
DBZ-7184 Refactored streaming loop of MongoDB connector
2023-12-04 06:58:28 +01:00
Jakub Cechacek
c7e87c5382
DBZ-7184 Using buffering cursor in straming change event source
2023-12-04 06:58:28 +01:00
Jakub Cechacek
93e8570fb6
DBZ-7184 Implementation of buffering change stream cursor
2023-12-04 06:58:28 +01:00
Jakub Cechacek
2c8277fcf6
DBZ-7162 Removed duplicit source event count metric for MongoDB
2023-12-01 11:38:21 +01:00
Jakub Cechacek
79230c6014
DBZ-7206 Always check for blocking snapshot pause request when processing change events in MongoDB connector
2023-11-29 14:56:41 +01:00
“vsantonastaso”
8c1c369449
DBZ-6878 add table specific notification in initial snapshot
2023-11-29 08:32:21 +01:00
Jakub Cechacek
0ff0ff54ae
DBZ-7186 Removing unsued offset parameter
2023-11-28 07:27:34 +01:00
Jakub Cechacek
03e305e835
DBZ-7184 Extracted split event handling from streaming source
2023-11-28 07:27:34 +01:00
Jakub Cechacek
3f0eeebf10
DBZ-7184 Extracted change stream initialisation
2023-11-28 07:27:34 +01:00
Roman Kudryashov
1992c1e7e4
DBZ-7159 Fail fast during deserialization if a value is not a CloudEvent
2023-11-23 14:19:37 +01:00
Jiri Pechanec
e57c607bcf
DBZ-6762 Rename to general name
2023-11-22 06:10:40 +01:00
rkerner
aeaba3ae70
DBZ-6762 Add "validate filters" endpoint/s for connector-specific Connect REST Extensions
...
closes to https://issues.redhat.com/browse/DBZ-6762
2023-11-22 06:10:40 +01:00
Jakub Cechacek
ae6e7d03cc
DBZ-7157 Unified MongoDB document key serialisation for snapshot and streaming
2023-11-21 13:29:12 +01:00
Jakub Cechacek
fb23ce13a1
DBZ-7157 JsonSerialization#getDocumentIdChangeStream now uses the same implementation as getDocumentIdSnaphsot
2023-11-21 13:29:12 +01:00
Jakub Cechacek
d2eca450d5
DBZ-7157 Added test case for key generation with composite document ids
2023-11-21 13:29:12 +01:00
mfvitale
2f8482d284
DBZ-7139 Use namespace instead of identifier for matching collections to be snapshotted
2023-11-21 11:13:39 +01:00
Jakub Cechacek
687f2858e3
DBZ-7162 Added various streaming metrics to MongoDB connector
2023-11-21 11:09:09 +01:00
Jakub Cechacek
7d4ca05ef6
DBZ-7162 Passing streaming metrics to MongoDbStreamingChangeEventSource
2023-11-21 11:09:09 +01:00
Jakub Cechacek
17464f5fe2
DBZ-7146 Configurable inactivity pause duration in MongoDB connector
2023-11-20 13:59:44 +01:00
Jiri Pechanec
e654dc228f
DBZ-7066 Code reorganization
2023-11-14 11:56:16 +01:00
harveyyue
41f9423caf
DBZ-7066 Add test cases synchronously from "DBZ-6809"
2023-11-14 11:56:16 +01:00
harveyyue
223b6e4335
DBZ-7066 Extract common creating transformation methods to ConnectRecordUtil
2023-11-14 11:56:16 +01:00
harveyyue
4c3b5540ec
DBZ-7066 Add NewExtractNewDocumentStateTest and NewExtractNewDocumentStateTestIT for new option "delete.tombstone.handling.mode"
2023-11-14 11:56:16 +01:00
harveyyue
5e8287eb40
DBZ-7066 The DefaultDeleteHandlingStrategy couldn't add the rewrite "__deleted" field to a non-struct value
2023-11-14 11:56:16 +01:00
Jiri Pechanec
ef69dbe91c
DBZ-6809 Fix imports
2023-11-10 16:06:43 +01:00
Jakub Cechacek
e88d00badb
DBZ-6809 Added test for ExtractNewDocumentState with avro enabled on connector and transformation levels
2023-11-10 16:06:43 +01:00
Debezium Builder
1521445908
[maven-release-plugin] prepare for next development iteration
2023-11-10 10:26:05 +00:00
Debezium Builder
6c6f6e9138
[maven-release-plugin] prepare release v2.5.0.Alpha2
2023-11-10 10:26:05 +00:00
Jiri Pechanec
9d84d01f64
DBZ-6726 Switch to ArrayList for repeated iteration
2023-11-09 07:57:12 +01:00
Jakub Cechacek
89b07a013e
DBZ-6726 support for $changeStreamSplitLargeEvent aggregation in MongoDB connector
2023-11-09 07:57:12 +01:00
Jakub Cechacek
bddf8802ec
DBZ-7108 Switched default connection mode for sharded clusters from replica_set to sharded
2023-11-07 10:36:53 +01:00
Vojtech Juranek
11d2ff0b9b
DBZ-7007 Use better name for auxiliary variable detecing if engine is running
2023-11-06 10:50:21 +01:00
Vojtech Juranek
fdab2cd61b
DBZ-7007 Remove engine.isRunning() from testsuite
2023-11-06 10:50:21 +01:00
Jakub Cechacek
a3affaf6a8
DBZ-7058 Fixed index handling in FieldSelector
2023-11-02 13:29:39 +01:00
Debezium Builder
78c5204444
[maven-release-plugin] prepare for next development iteration
2023-10-26 15:39:35 +00:00
Debezium Builder
356e488e83
[maven-release-plugin] prepare release v2.5.0.Alpha1
2023-10-26 15:39:35 +00:00
Roman Kudryashov
b0d7d60b5d
DBZ-7016 Optimize retrieval of id and type
2023-10-19 11:35:18 +02:00
Roman Kudryashov
22f6b083b1
DBZ-7016 Change metadata.location property syntax
2023-10-19 11:35:18 +02:00
Roman Kudryashov
ee64c70546
DBZ-7016 id
and type
of a CloudEvent can be retrieved from headers
2023-10-19 11:35:18 +02:00
harveyyue
4836fcf1ea
DBZ-6907 Introduce a new handling deleted record option "delete.tombstone.handling.mode"
2023-10-19 09:35:15 +02:00
harveyyue
fccaf37549
DBZ-6907 Remove unused method from ExtractNewDocumentState
2023-10-19 09:35:15 +02:00
harveyyue
f64ea1a5f8
DBZ-6907 Extract an abstract transformation class from ExtractNewRecordState
2023-10-19 09:35:15 +02:00
Roman Kudryashov
01485aceb3
DBZ-6982 Provide configuration option to exclude extension attributes from a CloudEvent
2023-10-04 13:58:53 +02:00
Debezium Builder
40a8f4026d
[maven-release-plugin] prepare for next development iteration
2023-10-03 14:18:54 +00:00
Debezium Builder
412c6f7979
[maven-release-plugin] prepare release v2.4.0.Final
2023-10-03 14:18:54 +00:00
Jakub Cechacek
9b3b718612
DBZ-6973 Fixed insufficient test isolation
2023-10-02 08:25:53 +02:00
Jakub Cechacek
baafe602ce
DBZ-6973 Literal matching mode for database/collection filters and ability to override internal aggregation pipeline for MongoDB conenctor
2023-10-02 08:25:53 +02:00
mfvitale
56795732b5
DBZ-6793 Add timestamp for notifications into documentation
2023-09-28 14:20:17 -04:00
Roman Kudryashov
98c426dec4
DBZ-3642 CloudEventsConverter can retrieve metadata info from headers
2023-09-26 13:06:09 +02:00
Debezium Builder
cc1de0aa51
[maven-release-plugin] prepare for next development iteration
2023-09-22 09:00:45 +00:00
Debezium Builder
6809978083
[maven-release-plugin] prepare release v2.4.0.CR1
2023-09-22 09:00:45 +00:00
mfvitale
5d7ba3e951
DBZ-6957 Remove spaces in Signal and Notification JMX ObjectName
2023-09-21 12:04:33 -04:00
rkerner
6d3528d4fb
DBZ-4395 Add connector specific Debezium Connect REST Extension/s and move logic from UI backend to the Debezium Connect REST Extension (part 2 of the re-architecturing/re-factoring)
...
DBZ-6761 Add "validate connection" endpoints in connector specific Connect REST extensions
closes to https://issues.redhat.com/browse/DBZ-4395
closes to https://issues.redhat.com/browse/DBZ-6761
2023-09-21 13:29:59 +02:00
mfvitale
ae199d2053
DBZ-6828 Blocking snapshot will generate schema events only for signaled tables
2023-09-20 08:18:33 +02:00
Jiri Pechanec
a064fc42e5
DBZ-6518 Fix imports
2023-09-13 13:00:03 +02:00
Yue Wang
199bf775ed
DBZ-6518 use multipe threads to query chunks
2023-09-13 13:00:03 +02:00
Debezium Builder
65169a9315
[maven-release-plugin] prepare for next development iteration
2023-09-13 09:41:46 +00:00
Debezium Builder
845e3e9d09
[maven-release-plugin] prepare release v2.4.0.Beta2
2023-09-13 09:41:46 +00:00
Jakub Cechacek
fd5c8353b2
DBZ-6883 Documenting oversized document handling
2023-09-13 11:04:07 +02:00
Jakub Cechacek
9aadc38414
DBZ-6892 Removing all code related to previously deprecated MongoDB conenctor configuration
2023-09-12 13:03:40 +02:00
Jakub Cechacek
eaa24ea91e
DBZ-6871 MongoDB connector is now able to skip events for oversized documents
2023-09-11 15:28:27 +02:00
Jakub Cechacek
3efe8c0e51
DBZ-6872 Ability to switch the order of user defined and internal aggregation pipeline used by MongoDB connector when streaming changes
2023-09-11 13:15:10 +02:00
Nir Levy
454a9ac914
DBZ-6741 Support custom authentication on MongoDB connector
2023-09-07 14:49:04 +02:00
Jiri Pechanec
41135efa74
DBZ-6773 Remove typo
2023-09-06 15:08:24 +02:00
Jakub Cechacek
9fad9b6895
DBZ-6773 Adding support for new field/header mappings to ExtractNewDocumentState transformation
2023-09-06 15:08:24 +02:00
Jakub Cechacek
842e92bc52
DBZ-6773 Cleaning unused fields and variables in tests
2023-09-06 15:08:24 +02:00
Jakub Cechacek
5a20419a98
DBZ-6867 streaming pipeline should match events from collection include list as well as signal collection
2023-09-04 13:28:20 +02:00
Debezium Builder
d97a18e8dc
[maven-release-plugin] prepare for next development iteration
2023-08-29 11:41:08 +00:00
Debezium Builder
5ce25f2cfe
[maven-release-plugin] prepare release v2.4.0.Beta1
2023-08-29 11:41:08 +00:00
“vsantonastaso”
43adc59292
DBZ-6742 use JSON format for JMX Notitication userData
2023-08-22 06:42:51 +02:00
Jiri Pechanec
fdafbc07cf
DBZ-6774 Use private modifier
2023-08-16 10:20:49 +02:00
harveyyue
625eec5aa0
DBZ-6774 MongoDB New Document State Extraction: nonexistent field for add.headers
2023-08-16 10:20:49 +02:00
ericpangiawan
5407d42014
DBZ-6725 Add Formatting
2023-08-15 13:34:17 +02:00
ericpangiawan
bc77f2488e
DBZ-6725 Include beforeRecord if the delete handling mode is REWRITE
...
Adding Integration Test and the feature to add beforeRecord in delete events of event flattening with delete handling mode rewrite. Works only if the collection has ChangeStreamPreAndPostImagesOptions turned on and capture mode is *_with_pre_image (MongoDB 6.0 or later)
2023-08-15 13:34:17 +02:00
mfvitale
542b3619fa
DBZ-6731 Blocking snapshot takes configuration from signal payload
2023-08-15 12:43:43 +02:00
harveyyue
063244a0ad
DBZ-6760 Outbox transformation can cause connector to crash
2023-08-10 08:59:43 +02:00
Jiri Novotny
0d18eb6768
DBZ-6779 Change approach to version parsing
2023-08-10 06:59:36 +02:00
Jiri Novotny
554163a3f1
DBZ-6779 Fix version parsing of non-numerical chars
2023-08-10 06:59:36 +02:00
Debezium Builder
3c72f071b0
[maven-release-plugin] prepare for next development iteration
2023-08-09 10:33:06 +00:00
Debezium Builder
915c465440
[maven-release-plugin] prepare release v2.4.0.Alpha2
2023-08-09 10:33:06 +00:00
mfvitale
524987227f
DBZ-6566 Shutdown blockingSnapshotExecutor when ChangeEventSourceCoordinator stops
2023-08-03 09:22:24 +02:00
mfvitale
61651a94b5
DBZ-6566 Force blocking snapshot even if a snapshot has already executed
2023-08-03 09:22:24 +02:00
mfvitale
3718d9818e
DBZ-6566 Support Blocking snapshot for MongoDb
2023-08-03 09:22:24 +02:00
jcechace
699a1612ae
DBZ-6670 Replace error handler instead of recreate in order to keep retry counter
2023-07-25 13:46:56 +02:00
jcechace
0fbad0c60c
DBZ-6700 using hello command instead of ping as resume backup in order to ensure that operationTime field is present across supported MongoDB platforms
2023-07-24 10:00:16 -04:00
Debezium Builder
267dfda8a6
[maven-release-plugin] prepare for next development iteration
2023-07-17 10:40:52 +00:00
Debezium Builder
0d3cef6a29
[maven-release-plugin] prepare release v2.4.0.Alpha1
2023-07-17 10:40:52 +00:00
harveyyue
5bc7fd1f7c
DBZ-6603 Support for custom tags in the connector metrics
2023-07-13 09:52:33 +02:00
jcechace
c620081cd1
DBZ-6182 Simplifying inheritance hierarchy in MongoDB tests
2023-06-28 09:51:51 +02:00
jcechace
d1e77e6165
DBZ-6182 Allowing mongodb connector to run with database restricted privileges
2023-06-28 09:51:51 +02:00
mfvitale
d5febcc054
DBZ-6417 Provide extensible signal action mechanism through SPI
2023-06-23 15:03:27 +02:00
mfvitale
34e28ac52d
DBZ-6416 Add initial snapshot notifications
2023-06-22 13:22:34 +02:00
jcechace
ee7586e6d8
DBZ-6578 Configurable connection string parameters for individual shard connections when using connection.mode=replica_set with sharded clusters
2023-06-21 12:22:23 +02:00
jcechace
8f70de42c1
DBZ-6578 Cleaning up connection string propagation between discovery and tasks
2023-06-21 12:22:23 +02:00
jcechace
dd97456877
DBZ-6578 Honoring connection string read preference configuration
2023-06-21 12:22:23 +02:00
Debezium Builder
60f730d383
[maven-release-plugin] prepare for next development iteration
2023-06-20 13:21:31 +00:00
Debezium Builder
150396e4c8
[maven-release-plugin] prepare release v2.3.0.Final
2023-06-20 13:21:31 +00:00
Jiri Pechanec
067683e56a
DBZ-6076 Consistent config for all connectors
2023-06-15 11:55:28 +02:00