chadthamn
1b23507504
DBZ-5015 Get the json value of a Document when parsing signaling message
2022-04-21 07:09:18 +02:00
jchipmunk
2c2a81b12b
DBZ-5003 Can't use 'local' database through mongos
...
MongoDb connector should use "config" database if the database exists. Otherwise, "local" database should be used to get MongoDB version during connector validation.
This is similar to the approach implemented in ReplicaSetDiscovery class.
2022-04-20 09:36:50 +02:00
Sergei Morozov
b258674782
DBZ-3535: Introduce schema.name.adjustment.mode
2022-03-24 09:37:05 +01:00
Sergei Morozov
6d0aa9408c
DBZ-4478: Add Partition parameter to io.debezium.pipeline.source.spi.*Listener interfaces
2022-02-16 11:31:38 +01:00
rkerner
8127985da0
[docs] fix Mongo DB Connector Config typo for mongodb.name
field description
2022-02-08 15:14:28 +01:00
Chris Cranford
bbd963549e
DBZ-4676 Fix MongoUtilIT test failure
2022-02-04 12:14:06 -05:00
Jiri Pechanec
664d989e42
DBZ-4577 Provide custom error message handling
2022-01-21 10:24:19 +01:00
Jiri Pechanec
ce41b0734c
DBZ-4575 Filter events based on database name
2022-01-20 19:13:44 +01:00
Jiri Pechanec
022f480ebf
DBZ-4415 oplog mode is not supported for MongoDB 5+
...
Co-authored-by: Gunnar Morling <gunnar.morling@googlemail.com>
2022-01-18 14:09:11 +01:00
Jiri Pechanec
8fd3243a86
DBZ-2224 Test logging based on logback
2022-01-18 11:24:21 +01:00
Sergei Morozov
0329859cda
DBZ-4459: Introduce interfaces and default implementations for change event source metrics
2022-01-12 14:32:19 +01:00
Anisha Mohanty
06860b3691
DBZ-4460 Apply the plug-in to all connector modules
2022-01-07 15:05:34 +01:00
harveyyue
291e8ac221
DBZ-4351 Add create/update/delete event seen metrics for monitor upstream dml operation
2022-01-06 19:03:46 +01:00
Gunnar Morling
92665f1799
DBZ-4472 Improving log output
2021-12-20 15:58:14 +01:00
harveyyue
792187f669
DBZ-4472 Make connector task partition readability for log
2021-12-20 15:58:14 +01:00
Mario Mueller
9330efaaa2
DBZ-4413 - Do not remove _id and do not overwrite any given field named "id".
...
IT test expectations are adjusted accordingly.
Removed leading zero from ZonedDateTime
Signed-off-by: Mario Mueller <mario@xenji.com>
2021-12-16 09:48:26 +01:00
Gunnar Morling
0023cb10a5
DBZ-4393 Create a Debezium Schema Generator for Debezium connectors
...
* added an API generator for Debezium connectors and static API definitions for connectors in a separate module
* added Maven plug-in
* added GH workflow for debezium-schema-generator
Co-authored-by: rkerner <rkerner.mobil@gmail.com>
Co-authored-by: Anisha Mohanty <anishamohanty23@gmail.com>
2021-12-10 12:35:09 +01:00
Jiri Pechanec
9e5661b573
DBZ-3342 Addressed review comments
2021-12-09 10:03:41 +01:00
Jiri Pechanec
b1dcfcb0b8
DBZ-3342 Incremental snapshot support for MongoDB
2021-12-09 10:03:41 +01:00
Jiri Pechanec
4cd5b092e9
DBZ-4205 Address review comments
2021-12-08 15:48:49 +01:00
Gunnar Morling
72a8662bd3
DBZ-4205 Document fixes and other misc. adjustments
2021-12-08 15:48:49 +01:00
Jiri Pechanec
25b29673db
DBZ-4205 Support for cloud events
2021-12-08 15:48:49 +01:00
Jiri Pechanec
9901c43c8a
DBZ-4205 Switch oplug to change streams is safe
2021-12-08 15:48:49 +01:00
Jiri Pechanec
a181d3c910
DBZ-4205 Test safety against capture mode switch
2021-12-08 15:48:49 +01:00
Jiri Pechanec
f037b35954
DBZ-4205 Document change streams capture mode
2021-12-08 15:48:49 +01:00
Jiri Pechanec
af401a0d71
DBZ-4205 Handle new transaction id format
2021-12-08 15:48:49 +01:00
Jiri Pechanec
6dcd3c0eab
DBZ-4205 Decouple change stream session TX id
2021-12-08 15:48:49 +01:00
Jiri Pechanec
39140a07b4
DBZ-4205 Add missing assertions
2021-12-08 15:48:49 +01:00
Jiri Pechanec
a516e78d94
DBZ-4205 Make config class immutable again
2021-12-08 15:48:49 +01:00
lairen
a0898ed440
DBZ-2363 Adding support for pg_logical_emit messages in Postgres connector for the pgoutput plugin
2021-11-26 14:29:15 +01:00
Sungho Hwang
d811b5ed1d
DBZ-4337 Fixed a bug for configuration field for expanding JSON payload in MongoDB Outbox Event Router
2021-11-22 09:14:11 +01:00
Sungho Hwang
64a9c91abb
DBZ-4318 Modified default type of aggregateid In outbox collection expected by MongoDB Outbox Event Router for both tests and documentation.
2021-11-21 09:16:58 +01:00
Jiri Pechanec
74db20de7d
DBZ-3938 Add config database among builtins
2021-11-19 09:54:52 +01:00
Chris Cranford
2730187a4b
DBZ-4012 Deprecate table.field.event.payload.id
2021-11-18 21:45:41 +01:00
Chris Cranford
ab86cd18a9
DBZ-4012 Suggested changes
2021-11-18 21:45:41 +01:00
Chris Cranford
0df5c6219b
DBZ-4012 Bubble up SMT validation failures to Kafka Connect
2021-11-18 21:45:41 +01:00
Chris Cranford
970792bc40
DBZ-4012 Deprecate/Rename debezium namespace config options
2021-11-18 21:45:41 +01:00
Jiri Pechanec
212fdb7912
DBZ-3938 Make assertions order independent
2021-11-18 20:24:53 +01:00
Sungho Hwang
c9eb0f9157
DBZ-3528 Refactored several parts according to the feedbacks
...
A delegate class for common logic between two Outbox Event Routers has been created.
Documentation has been fixed.
The configuration properties for MongoDB Outbox Event Router have been changed to use existing Outbox Event Router's.
2021-11-17 16:02:06 +01:00
Sungho Hwang
0cbf324e31
DBZ-3528 Refactored codes
2021-11-17 16:02:06 +01:00
Sungho Hwang
482cc2d529
DBZ-3528 Modified code formatting
2021-11-17 16:02:06 +01:00
Sungho Hwang
0b9fc88ea6
DBZ-3528 Applied the code style rules
2021-11-17 16:02:06 +01:00
Sungho Hwang
2c279b07f6
DBZ-3528 Wrote MongoDB Outbox Event Router with documentation and its unit & integration tests
2021-11-17 16:02:06 +01:00
ahodavdekar
1d81b6db19
DBZ-4077 Exposing the transaction metadata topic name as a config option
2021-11-02 06:07:05 +01:00
Gunnar Morling
fd06a4e2a4
DBZ-2975 Using Offsets in recover() contract;
...
Keeping legacy method until the legacy MySQL connector implementation has been removed.
2021-11-01 10:17:49 +01:00
Jiri Pechanec
6163421199
DBZ-435 Fixes after review
2021-10-26 18:13:16 +02:00
Jiri Pechanec
a1084360da
DBZ-435 Too fast delete can prevent from obtaining the full document
2021-10-26 18:13:16 +02:00
Jiri Pechanec
17df080717
DBZ-435 Fix test failures
2021-10-26 18:13:16 +02:00
Jiri Pechanec
8cb1954e08
DBZ-435 Converting FieldRenamesIT test
2021-10-26 18:13:16 +02:00
Jiri Pechanec
8733e39643
DBZ-435 Converting SMT tests
2021-10-26 18:13:16 +02:00
Jiri Pechanec
c8317857db
DBZ-435 Converting FieldExludeListIT test
2021-10-26 18:13:16 +02:00
Jiri Pechanec
955afddd56
DBZ-435 Converting FieldBlacklistIT and MongoMetricsIT tests
2021-10-26 18:13:16 +02:00
Jiri Pechanec
c60412dee5
DBZ-435 Aligned CloudEventsConverterIT test
2021-10-26 18:13:16 +02:00
Jiri Pechanec
2140ff51be
DBZ-435 Aligned MongoDbConnectorIT test
2021-10-26 18:13:16 +02:00
Jiri Pechanec
a0523186b5
DBZ-435 Support for field manipulation in change streams
2021-10-26 18:13:16 +02:00
Jiri Pechanec
cb36b575db
DBZ-435 Support for Change Streams capture
2021-10-26 18:13:16 +02:00
Jiri Pechanec
657c5540b3
DBZ-4083 Add filter to MongoDB cloud converter
2021-10-11 11:19:30 +02:00
René Kerner
b92e5c5efe
DBZ-3943 Add ConfigDef unit tests to verify all Debezium Field instances have a proper documentation/description set;
...
Also adding missing descriptions.
2021-09-09 13:35:41 +02:00
Anisha Mohanty
153e0920e1
DBZ-2525 Generifying exclusion of columns from snapshotting in all relational connectors;
...
* Using RelationalDatabaseSchema in RelationalSnapshotChangeEventSource for consistency.
* Adding test for table/column names which are keywords
2021-08-30 12:20:56 +02:00
rkerner
52333596de
DBZ-3364 add additional metadata to fields for API spec generator for the Debezium UI
...
part of https://issues.redhat.com/browse/DBZ-3364
DBZ-2653 User rollback from parent class
2021-08-23 17:57:38 +02:00
Chris Cranford
ee04d7f918
DBZ-3836 Specify oplog cursor to never time out
2021-08-13 05:58:19 +02:00
Chris Cranford
2d066513dd
DBZ-3836 Allow configurable execution timeout
2021-08-13 05:58:19 +02:00
Sergei Morozov
a40ad7132d
DBZ-2975: Remove partition from offset context
2021-08-11 07:34:37 +02:00
Sergei Morozov
3682ae9197
DBZ-2975: Introduce replica set partition for MongoDB
2021-08-11 07:34:37 +02:00
Sergei Morozov
77e2ac8a12
DBZ-2975: Introduce io.debezium.pipeline.spi.Offsets
2021-08-11 07:34:37 +02:00
Sergei Morozov
c7ac5fca52
DBZ-2975: Move Partition to io.debezium.pipeline.spi
2021-08-11 07:34:37 +02:00
Anisha Mohanty
f7268eb61a
DBZ-3028 Add test cases supporting filters to FilterTest
2021-08-06 18:12:45 +02:00
Anisha Mohanty
389a8a0a83
DBZ-3028 Validate field.exclude.list & field.renames for MongoDB connector
2021-08-06 18:12:45 +02:00
Chris Cranford
151f0777ec
DBZ-3788 Remove superfluous sort
2021-08-05 12:25:37 -04:00
Chris Cranford
3e72fb7e79
DBZ-3788 Avoid oplog collection scan
2021-08-05 12:25:37 -04:00
Chris Cranford
ed31ea9ffb
DBZ-3668 Adjust ServiceLoader for SPI change
2021-07-03 16:31:52 +07:00
Gunnar Morling
b84c3c870d
DBZ-3668 Extracting SPI package
2021-07-03 16:31:52 +07:00
Chris Cranford
7f7e23b110
DBZ-3668 Add ServiceLoader hook CloudEvents converter
2021-07-03 16:31:52 +07:00
Sergei Morozov
db105baeba
DBZ-2975: Add partition awareness to source task components
...
Co-authored-by: Mike Kamornikov <mikekamornikov@gmail.com>
2021-07-03 10:56:34 +02:00
Sergei Morozov
657a6d2542
DBZ-2975: Extract offset context from object states to method signatures
2021-06-09 16:02:33 +02:00
chriscollingwood
6225e124bb
DBZ-3522 Ensure that there is an active session on the replica set mongo client before using clusterDescription and ServerDescriptions to determine primary node.
2021-06-03 08:28:58 +02:00
Chris Cranford
d31d562376
DBZ-3331 Do not stream if offset no longer in the MongoDB oplog
2021-05-14 09:07:32 +02:00
Gunnar Morling
42a7b74725
DBZ-3403 Misc. improvements
2021-04-14 11:14:07 +02:00
Hossein Torabi
ba048cf47d
DBZ-3403 Semi Optimisation on MongoDB and MySQL connector for skipped.operations
2021-04-14 11:14:07 +02:00
JLDLaughlin
06b0475f17
DBZ-2911 Add additional LSN information to all sources
...
Adds a new "sequence" field to all sources via the AbstractSourceInfo
class. Specifically, this field is required to deduplicate records
from PostgreSQL sources in O(1) time. The sequence field is a stringified
list of metadata. For PostgreSQL sources, this will include the last
committed LSN and the current LSN. For other sources, this will be NULL
until implemented. A new integration test was added for PostgreSQL.
2021-03-26 09:19:17 +01:00
Gunnar Morling
c7b7d5303a
DBZ-3218 Avoiding duplicated validation (again ;)
2021-03-19 15:04:20 +01:00
Gunnar Morling
178140ae9c
DBZ-3218 Removing superfluous method parameter
2021-03-19 15:04:20 +01:00
Anisha Mohanty
c761d92b2a
DBZ-3218 Add separate method for logging warnings
2021-03-19 15:04:20 +01:00
Jiri Pechanec
495be2659b
DBZ-2511 Dedicated logger for schema name adjuster
2021-03-18 14:42:34 +01:00
Frank Koornstra
cdc7abdf28
DBZ-3278 Add timeout exceptions to MongoDBErrorHandler
...
When connecting to the server or executing some queries, due to
networking issues, timeouts sometimes occur which would crash the whole
connector. By including a few extra exceptions in the error handler we
make sure it will retry in these situations.
2021-03-15 12:27:03 +01:00
Thomas Prelle
15887077ff
DBZ-2680 Sanitize field name do not santize sub struct field
...
Their are an issue during the sanitisation to avro on mongodb connector.
The sanitization failed on sub struct like
{
"_id": "222",
"metrics": {
"metric::fct":
{"min": 0, "max": 1 }
}
2021-02-17 09:09:00 +01:00
ani-sha
5f8a22a5df
DBZ-2858 Make field descriptions consistent for time values
2021-02-16 09:55:35 +01:00
Gunnar Morling
6d441bc538
DBZ-2938 Centralizing listener handling; avoiding some empty methods
2021-01-19 15:16:46 +01:00
rkerner
15064f42b1
DBZ-2831 allow non default Mongo DB ports
2020-12-10 05:33:00 +01:00
Gunnar Morling
9bb53b9e2d
DBZ-2138 Static loggers
2020-11-30 10:33:05 +01:00
Jiri Pechanec
83df96ff70
DBZ-2138 Add defensive checks
2020-11-30 10:33:05 +01:00
Jiri Pechanec
349fcdad07
DBZ-2138 Handle lsid as Binary for MongoDB 4.2
2020-11-30 10:33:05 +01:00
Jiri Pechanec
e8aecd7b45
DBZ-2138 Use MongoDB 4.x driver
2020-11-30 10:33:05 +01:00
Jiri Pechanec
ea019c3b2c
DBZ-1322 Removed dependency on deprecated MongoDB driver code
2020-11-27 17:37:57 +01:00
vanhoale
17a424d9ac
DBZ-2662 Handle ChangeEventQueue by the size in bytes
2020-11-16 08:39:33 +01:00
Chris Cranford
aea26a2949
DBZ-2742 Migrate initial.sync.max.threads to snapshot.max.threads
2020-11-11 09:08:05 +01:00
Faizan
f5f6abfc7b
DBZ-2496 Corrections to code formatting and improving doc text for consistency
2020-10-22 06:30:40 +02:00
Faizan
0adaed07d2
DBZ-2496 Updating documentation and test case as per review comment
2020-10-22 06:30:40 +02:00
Faizan
24f375aa3d
DBZ-2496 Corrections to code formatting
2020-10-22 06:30:40 +02:00
Faizan
925c7e35f5
DBZ-2496 Added snapshot override behavior for MongoDB using filter query
2020-10-22 06:30:40 +02:00