Vojtech Juranek
161a7229b6
DBZ-5043 Use topic prefix as the logical name
...
With this change the user cannot direcly set connector logical name and
thus in the future it can be remove without breaking user config.
If the turn out that the logical name is useful and user should be able
to configure it, dedicated config option can be added.
Originally it was prposed in the Jira to replace it with connector name,
but it turned out that logical name defaults to `database.server.name`
and is heavily used in the tests and JMX, so it would require another
big refactoring. Thus, use topic prefix for now. Once we know further
direction (remove logical name or add new option), do this refactoring.
2022-09-07 12:57:31 +02:00
ggaborg
664fd9e1b0
DBZ-3293 unit tests for filters
2022-09-01 20:16:47 +02:00
ggaborg
244de2bd96
DBZ-3293 Add signal table automatically to include list
2022-09-01 20:16:47 +02:00
Jiri Pechanec
de53591dba
DBZ-5244 Retry all communication exceptions by default
2022-08-31 09:45:09 -04:00
jcechace
5c61bc4341
DBZ-4733 Reordering fileds in MongoDbConnectorConfig
2022-08-19 12:55:58 +02:00
jcechace
b61319feb7
DBZ-4733 Basic integration test for connection string support
2022-08-19 12:55:58 +02:00
jcechace
7670c4ba74
DBZ-4733 working POC for connection String support
2022-08-19 12:55:58 +02:00
jcechace
521156f5c2
DBZ-4733 Refactored MongoUtil.getPrimaryAddress to properly use mongo driver 4.x
2022-08-19 12:55:58 +02:00
jcechace
21ed5f9b9f
DBZ-4733 Refactor ReplicaSets.parse into more readable form
2022-08-19 12:55:58 +02:00
jcechace
752062c1a3
DBZ-4733 Fixed thread safety of MongoClients
2022-08-19 12:55:58 +02:00
jcechace
1d9d75754c
DBZ-4733 Refactoring ReplicaSetDiscovery to properly use mongodb driver 4.x
2022-08-19 12:55:58 +02:00
jcechace
3a385aa768
DBZ-4733 Refactoring: unified client factory method names
2022-08-19 12:55:58 +02:00
jcechace
cc04a94fd5
DBZ-4733 Ininital connection string filed introduction
2022-08-19 12:55:58 +02:00
Chris Cranford
b7e2112603
DBZ-5497 Align test behavior
2022-08-19 11:05:29 +02:00
Vivek Wassan
d89e71ebeb
DBZ-5327 Smart Backfills
2022-08-17 10:05:55 +02:00
nicholas-fwang
58a91cddb4
DBZ-5479 Retry MongoNodeIsRecoveringException when failover
2022-08-05 10:52:46 -04:00
Chris Cranford
90a4c0059a
DBZ-5434 Fix test failure
2022-08-01 14:36:09 -04:00
harveyyue
7c93c597f7
DBZ-5434 Translation from mongodb document to kafka connect schema fails when nested arrays contain no elements
2022-08-01 10:07:28 -04:00
Vojtech Juranek
c7d8353d7b
DBZ-4727 Add snapshot paused JXM metrics
...
Add two JXM objects:
* `snapshotPaused` - detemines if the incremental snapshot is paused
or not
* `snapshotPausedDurationInSeconds` - overall time when the incremental
snapshot was paused. The time adds up - if the snapshot was paused
e.g. two times, the `snapshotPausedDurationInSeconds` is the sum of
these two paused times.
2022-07-29 12:41:58 -04:00
Vojtech Juranek
ed504173e1
DBZ-4727 Implement incremental snaphost pause/resume
...
Intorduce two new signals:
* `pause-snapshot` - pauses running incremental snapshot
* `resume-snapshot` - resumes paused incremental snapshot
If the incremental snapshot is running and pause signal is sent,
currently processed chunk is finished and then no other chunk is read.
Snapshot state is kept in running state. Once snapshot is resume,
chunk is reverted to position which was last sent to the broker and
process of reading chunks and emitting `read` events continues until
all tables are read and sent.
This patch handles only table-based signals. MySQL Kafka-based signals
will be addressed later.
2022-07-29 12:41:58 -04:00
harveyyue
5f430f1a8e
DBZ-5457 Deprecate legacy topic selector for all connectors
2022-07-28 12:58:59 -04:00
Chris Cranford
de4d8e17d3
DBZ-5371 Check for $natural not supported, treat as no oplog entry
2022-07-25 10:20:46 -04:00
harveyyue
6ea7abc8fc
DBZ-4180 Pluggable topic selector
2022-07-13 14:45:25 +02:00
harveyyue
e8405bc376
DBZ-4846 MongoConnector's field exclusion configuration does not work with fields with the same name but from different collections
2022-07-12 17:26:36 +02:00
harveyyue
2a2b5462db
DBZ-5328 field.exclude.list doesn't accept dashes (-) in database name
2022-06-30 10:47:56 +02:00
Mickael Maison
5ff71aa992
DBZ-5335 Fix formatting issues in the connectors configdefs
2022-06-30 09:35:54 +02:00
kanha gupta
1b17372d2d
DBZ-4902 Replace long durations with Duration
2022-06-27 14:29:51 +02:00
Jiri Pechanec
38726a8451
DBZ-5260 Fix imports
2022-06-20 09:22:24 +02:00
Tim Patterson
c0323305ac
DBZ-5260 Filter out unavailable replicaset members
2022-06-20 08:56:57 +02:00
Vojtech Juranek
4a48eb33df
DBZ-5045 Remove legacy V1 source info
2022-06-08 09:11:00 +02:00
Vojtech Juranek
76c8550e7b
DBZ-5045 Remove deprecated Mongo DB OPERATION_HEADER option
2022-06-08 09:11:00 +02:00
Vojtech Juranek
4140078db7
DBZ-5045 Remove deprecated Mongo DB ADD_SOURCE_FIELDS option
...
Adjust the tests as source `ts_ms` is different.
2022-06-08 09:11:00 +02:00
Vojtech Juranek
0ddcd93167
DBZ-5045 Remove deprecated POLL_INTERVAL_SEC
2022-06-08 09:11:00 +02:00
Vojtech Juranek
64ef812cc9
DBZ-5045 Remove deprecated MAX_COPY_THREADS
2022-06-08 09:11:00 +02:00
Vojtech Juranek
0d6bfec265
DBZ-5045 Remove deprecated FIELD_BLACKLIST
2022-06-08 09:11:00 +02:00
Vojtech Juranek
d66d86a3ac
DBZ-5045 Remove deprecated COLLECTION_BLACKLIST
2022-06-08 09:11:00 +02:00
Vojtech Juranek
2484faaffe
DBZ-5045 Remove deprecated COLLECTION_WHITELIST
2022-06-08 09:11:00 +02:00
Vojtech Juranek
b3a6131cee
DBZ-5045 Remove deprecated DATABASE_BLACKLIST
2022-06-08 09:11:00 +02:00
Vojtech Juranek
0cbb57f4bf
DBZ-5045 Remove deprecated DATABASE_WHITELIST
2022-06-08 09:11:00 +02:00
harveyyue
acb743bd34
DBZ-5206 Remove the deprecated monitored tables metrics
2022-06-07 16:13:29 -04:00
Chris Cranford
9d71296b5c
DBZ-4251 Support stopping incremental snapshots
2022-06-02 07:30:07 +02:00
Jiri Pechanec
88b7ae2050
DBZ-5136 Move sourceInfo to common class
2022-06-01 16:56:29 +02:00
Mark Bereznitsky
c64b8af31a
DBZ-5136 reintroduce the CommonOffsetContext class and refactor common functionality into it
2022-06-01 16:56:29 +02:00
Mark Bereznitsky
2d72dae649
DBZ-5136 further refining of the algorithm for determining position in snapshot and IT test fixes
2022-06-01 16:56:29 +02:00
Jiri Pechanec
08e8c3d795
DBZ-5136 Move common class to common package
2022-06-01 16:56:29 +02:00
Andrew Walker
d0759461a4
DBZ-5136: Convert RelationalOffsetContext to CommonOffsetContext
2022-06-01 16:56:29 +02:00
Andrew Walker
b6222d42bf
DBZ-5136: Add first flag and refactor OffsetContext
2022-06-01 16:56:29 +02:00
Andrew Walker
593c2769e6
DBZ-5136: Add last in table flag
2022-06-01 16:56:29 +02:00
Chris Cranford
28cd36edff
DBZ-5011 Set task
metrics attribute when using only 1 task
2022-06-01 09:41:36 +02:00
Chris Cranford
9fa5a48274
DBZ-5011 Register MongoDB metrics with task id when more than one task
2022-06-01 09:41:36 +02:00
Yang
75d3b716ab
DBZ-5113 Use generic bson value
2022-05-17 14:32:51 +02:00
Yang
5c190f7584
DBZ-5113 Use bson document
2022-05-17 14:32:51 +02:00
Jiri Pechanec
707eeab3d8
DBZ-4951 Remove oplog suppport
2022-05-11 16:04:04 +02:00
Jiri Pechanec
ee78ba6ad6
DBZ-5015 Fix formatting
2022-04-21 07:09:18 +02:00
chadthamn
1b23507504
DBZ-5015 Get the json value of a Document when parsing signaling message
2022-04-21 07:09:18 +02:00
jchipmunk
2c2a81b12b
DBZ-5003 Can't use 'local' database through mongos
...
MongoDb connector should use "config" database if the database exists. Otherwise, "local" database should be used to get MongoDB version during connector validation.
This is similar to the approach implemented in ReplicaSetDiscovery class.
2022-04-20 09:36:50 +02:00
Sergei Morozov
b258674782
DBZ-3535: Introduce schema.name.adjustment.mode
2022-03-24 09:37:05 +01:00
Sergei Morozov
6d0aa9408c
DBZ-4478: Add Partition parameter to io.debezium.pipeline.source.spi.*Listener interfaces
2022-02-16 11:31:38 +01:00
rkerner
8127985da0
[docs] fix Mongo DB Connector Config typo for mongodb.name
field description
2022-02-08 15:14:28 +01:00
Chris Cranford
bbd963549e
DBZ-4676 Fix MongoUtilIT test failure
2022-02-04 12:14:06 -05:00
Jiri Pechanec
664d989e42
DBZ-4577 Provide custom error message handling
2022-01-21 10:24:19 +01:00
Jiri Pechanec
ce41b0734c
DBZ-4575 Filter events based on database name
2022-01-20 19:13:44 +01:00
Jiri Pechanec
022f480ebf
DBZ-4415 oplog mode is not supported for MongoDB 5+
...
Co-authored-by: Gunnar Morling <gunnar.morling@googlemail.com>
2022-01-18 14:09:11 +01:00
Jiri Pechanec
8fd3243a86
DBZ-2224 Test logging based on logback
2022-01-18 11:24:21 +01:00
Sergei Morozov
0329859cda
DBZ-4459: Introduce interfaces and default implementations for change event source metrics
2022-01-12 14:32:19 +01:00
Anisha Mohanty
06860b3691
DBZ-4460 Apply the plug-in to all connector modules
2022-01-07 15:05:34 +01:00
harveyyue
291e8ac221
DBZ-4351 Add create/update/delete event seen metrics for monitor upstream dml operation
2022-01-06 19:03:46 +01:00
Gunnar Morling
92665f1799
DBZ-4472 Improving log output
2021-12-20 15:58:14 +01:00
harveyyue
792187f669
DBZ-4472 Make connector task partition readability for log
2021-12-20 15:58:14 +01:00
Mario Mueller
9330efaaa2
DBZ-4413 - Do not remove _id and do not overwrite any given field named "id".
...
IT test expectations are adjusted accordingly.
Removed leading zero from ZonedDateTime
Signed-off-by: Mario Mueller <mario@xenji.com>
2021-12-16 09:48:26 +01:00
Gunnar Morling
0023cb10a5
DBZ-4393 Create a Debezium Schema Generator for Debezium connectors
...
* added an API generator for Debezium connectors and static API definitions for connectors in a separate module
* added Maven plug-in
* added GH workflow for debezium-schema-generator
Co-authored-by: rkerner <rkerner.mobil@gmail.com>
Co-authored-by: Anisha Mohanty <anishamohanty23@gmail.com>
2021-12-10 12:35:09 +01:00
Jiri Pechanec
9e5661b573
DBZ-3342 Addressed review comments
2021-12-09 10:03:41 +01:00
Jiri Pechanec
b1dcfcb0b8
DBZ-3342 Incremental snapshot support for MongoDB
2021-12-09 10:03:41 +01:00
Jiri Pechanec
4cd5b092e9
DBZ-4205 Address review comments
2021-12-08 15:48:49 +01:00
Gunnar Morling
72a8662bd3
DBZ-4205 Document fixes and other misc. adjustments
2021-12-08 15:48:49 +01:00
Jiri Pechanec
25b29673db
DBZ-4205 Support for cloud events
2021-12-08 15:48:49 +01:00
Jiri Pechanec
9901c43c8a
DBZ-4205 Switch oplug to change streams is safe
2021-12-08 15:48:49 +01:00
Jiri Pechanec
a181d3c910
DBZ-4205 Test safety against capture mode switch
2021-12-08 15:48:49 +01:00
Jiri Pechanec
f037b35954
DBZ-4205 Document change streams capture mode
2021-12-08 15:48:49 +01:00
Jiri Pechanec
af401a0d71
DBZ-4205 Handle new transaction id format
2021-12-08 15:48:49 +01:00
Jiri Pechanec
6dcd3c0eab
DBZ-4205 Decouple change stream session TX id
2021-12-08 15:48:49 +01:00
Jiri Pechanec
39140a07b4
DBZ-4205 Add missing assertions
2021-12-08 15:48:49 +01:00
Jiri Pechanec
a516e78d94
DBZ-4205 Make config class immutable again
2021-12-08 15:48:49 +01:00
lairen
a0898ed440
DBZ-2363 Adding support for pg_logical_emit messages in Postgres connector for the pgoutput plugin
2021-11-26 14:29:15 +01:00
Sungho Hwang
d811b5ed1d
DBZ-4337 Fixed a bug for configuration field for expanding JSON payload in MongoDB Outbox Event Router
2021-11-22 09:14:11 +01:00
Sungho Hwang
64a9c91abb
DBZ-4318 Modified default type of aggregateid In outbox collection expected by MongoDB Outbox Event Router for both tests and documentation.
2021-11-21 09:16:58 +01:00
Jiri Pechanec
74db20de7d
DBZ-3938 Add config database among builtins
2021-11-19 09:54:52 +01:00
Chris Cranford
2730187a4b
DBZ-4012 Deprecate table.field.event.payload.id
2021-11-18 21:45:41 +01:00
Chris Cranford
ab86cd18a9
DBZ-4012 Suggested changes
2021-11-18 21:45:41 +01:00
Chris Cranford
0df5c6219b
DBZ-4012 Bubble up SMT validation failures to Kafka Connect
2021-11-18 21:45:41 +01:00
Chris Cranford
970792bc40
DBZ-4012 Deprecate/Rename debezium namespace config options
2021-11-18 21:45:41 +01:00
Jiri Pechanec
212fdb7912
DBZ-3938 Make assertions order independent
2021-11-18 20:24:53 +01:00
Sungho Hwang
c9eb0f9157
DBZ-3528 Refactored several parts according to the feedbacks
...
A delegate class for common logic between two Outbox Event Routers has been created.
Documentation has been fixed.
The configuration properties for MongoDB Outbox Event Router have been changed to use existing Outbox Event Router's.
2021-11-17 16:02:06 +01:00
Sungho Hwang
0cbf324e31
DBZ-3528 Refactored codes
2021-11-17 16:02:06 +01:00
Sungho Hwang
482cc2d529
DBZ-3528 Modified code formatting
2021-11-17 16:02:06 +01:00
Sungho Hwang
0b9fc88ea6
DBZ-3528 Applied the code style rules
2021-11-17 16:02:06 +01:00
Sungho Hwang
2c279b07f6
DBZ-3528 Wrote MongoDB Outbox Event Router with documentation and its unit & integration tests
2021-11-17 16:02:06 +01:00
ahodavdekar
1d81b6db19
DBZ-4077 Exposing the transaction metadata topic name as a config option
2021-11-02 06:07:05 +01:00
Gunnar Morling
fd06a4e2a4
DBZ-2975 Using Offsets in recover() contract;
...
Keeping legacy method until the legacy MySQL connector implementation has been removed.
2021-11-01 10:17:49 +01:00
Jiri Pechanec
6163421199
DBZ-435 Fixes after review
2021-10-26 18:13:16 +02:00
Jiri Pechanec
a1084360da
DBZ-435 Too fast delete can prevent from obtaining the full document
2021-10-26 18:13:16 +02:00
Jiri Pechanec
17df080717
DBZ-435 Fix test failures
2021-10-26 18:13:16 +02:00
Jiri Pechanec
8cb1954e08
DBZ-435 Converting FieldRenamesIT test
2021-10-26 18:13:16 +02:00
Jiri Pechanec
8733e39643
DBZ-435 Converting SMT tests
2021-10-26 18:13:16 +02:00
Jiri Pechanec
c8317857db
DBZ-435 Converting FieldExludeListIT test
2021-10-26 18:13:16 +02:00
Jiri Pechanec
955afddd56
DBZ-435 Converting FieldBlacklistIT and MongoMetricsIT tests
2021-10-26 18:13:16 +02:00
Jiri Pechanec
c60412dee5
DBZ-435 Aligned CloudEventsConverterIT test
2021-10-26 18:13:16 +02:00
Jiri Pechanec
2140ff51be
DBZ-435 Aligned MongoDbConnectorIT test
2021-10-26 18:13:16 +02:00
Jiri Pechanec
a0523186b5
DBZ-435 Support for field manipulation in change streams
2021-10-26 18:13:16 +02:00
Jiri Pechanec
cb36b575db
DBZ-435 Support for Change Streams capture
2021-10-26 18:13:16 +02:00
Jiri Pechanec
657c5540b3
DBZ-4083 Add filter to MongoDB cloud converter
2021-10-11 11:19:30 +02:00
René Kerner
b92e5c5efe
DBZ-3943 Add ConfigDef unit tests to verify all Debezium Field instances have a proper documentation/description set;
...
Also adding missing descriptions.
2021-09-09 13:35:41 +02:00
Anisha Mohanty
153e0920e1
DBZ-2525 Generifying exclusion of columns from snapshotting in all relational connectors;
...
* Using RelationalDatabaseSchema in RelationalSnapshotChangeEventSource for consistency.
* Adding test for table/column names which are keywords
2021-08-30 12:20:56 +02:00
rkerner
52333596de
DBZ-3364 add additional metadata to fields for API spec generator for the Debezium UI
...
part of https://issues.redhat.com/browse/DBZ-3364
DBZ-2653 User rollback from parent class
2021-08-23 17:57:38 +02:00
Chris Cranford
ee04d7f918
DBZ-3836 Specify oplog cursor to never time out
2021-08-13 05:58:19 +02:00
Chris Cranford
2d066513dd
DBZ-3836 Allow configurable execution timeout
2021-08-13 05:58:19 +02:00
Sergei Morozov
a40ad7132d
DBZ-2975: Remove partition from offset context
2021-08-11 07:34:37 +02:00
Sergei Morozov
3682ae9197
DBZ-2975: Introduce replica set partition for MongoDB
2021-08-11 07:34:37 +02:00
Sergei Morozov
77e2ac8a12
DBZ-2975: Introduce io.debezium.pipeline.spi.Offsets
2021-08-11 07:34:37 +02:00
Sergei Morozov
c7ac5fca52
DBZ-2975: Move Partition to io.debezium.pipeline.spi
2021-08-11 07:34:37 +02:00
Anisha Mohanty
f7268eb61a
DBZ-3028 Add test cases supporting filters to FilterTest
2021-08-06 18:12:45 +02:00
Anisha Mohanty
389a8a0a83
DBZ-3028 Validate field.exclude.list & field.renames for MongoDB connector
2021-08-06 18:12:45 +02:00
Chris Cranford
151f0777ec
DBZ-3788 Remove superfluous sort
2021-08-05 12:25:37 -04:00
Chris Cranford
3e72fb7e79
DBZ-3788 Avoid oplog collection scan
2021-08-05 12:25:37 -04:00
Chris Cranford
ed31ea9ffb
DBZ-3668 Adjust ServiceLoader for SPI change
2021-07-03 16:31:52 +07:00
Gunnar Morling
b84c3c870d
DBZ-3668 Extracting SPI package
2021-07-03 16:31:52 +07:00
Chris Cranford
7f7e23b110
DBZ-3668 Add ServiceLoader hook CloudEvents converter
2021-07-03 16:31:52 +07:00
Sergei Morozov
db105baeba
DBZ-2975: Add partition awareness to source task components
...
Co-authored-by: Mike Kamornikov <mikekamornikov@gmail.com>
2021-07-03 10:56:34 +02:00
Sergei Morozov
657a6d2542
DBZ-2975: Extract offset context from object states to method signatures
2021-06-09 16:02:33 +02:00
chriscollingwood
6225e124bb
DBZ-3522 Ensure that there is an active session on the replica set mongo client before using clusterDescription and ServerDescriptions to determine primary node.
2021-06-03 08:28:58 +02:00
Chris Cranford
d31d562376
DBZ-3331 Do not stream if offset no longer in the MongoDB oplog
2021-05-14 09:07:32 +02:00
Gunnar Morling
42a7b74725
DBZ-3403 Misc. improvements
2021-04-14 11:14:07 +02:00
Hossein Torabi
ba048cf47d
DBZ-3403 Semi Optimisation on MongoDB and MySQL connector for skipped.operations
2021-04-14 11:14:07 +02:00
JLDLaughlin
06b0475f17
DBZ-2911 Add additional LSN information to all sources
...
Adds a new "sequence" field to all sources via the AbstractSourceInfo
class. Specifically, this field is required to deduplicate records
from PostgreSQL sources in O(1) time. The sequence field is a stringified
list of metadata. For PostgreSQL sources, this will include the last
committed LSN and the current LSN. For other sources, this will be NULL
until implemented. A new integration test was added for PostgreSQL.
2021-03-26 09:19:17 +01:00
Gunnar Morling
c7b7d5303a
DBZ-3218 Avoiding duplicated validation (again ;)
2021-03-19 15:04:20 +01:00
Gunnar Morling
178140ae9c
DBZ-3218 Removing superfluous method parameter
2021-03-19 15:04:20 +01:00
Anisha Mohanty
c761d92b2a
DBZ-3218 Add separate method for logging warnings
2021-03-19 15:04:20 +01:00
Jiri Pechanec
495be2659b
DBZ-2511 Dedicated logger for schema name adjuster
2021-03-18 14:42:34 +01:00
Frank Koornstra
cdc7abdf28
DBZ-3278 Add timeout exceptions to MongoDBErrorHandler
...
When connecting to the server or executing some queries, due to
networking issues, timeouts sometimes occur which would crash the whole
connector. By including a few extra exceptions in the error handler we
make sure it will retry in these situations.
2021-03-15 12:27:03 +01:00
Thomas Prelle
15887077ff
DBZ-2680 Sanitize field name do not santize sub struct field
...
Their are an issue during the sanitisation to avro on mongodb connector.
The sanitization failed on sub struct like
{
"_id": "222",
"metrics": {
"metric::fct":
{"min": 0, "max": 1 }
}
2021-02-17 09:09:00 +01:00
ani-sha
5f8a22a5df
DBZ-2858 Make field descriptions consistent for time values
2021-02-16 09:55:35 +01:00
Gunnar Morling
6d441bc538
DBZ-2938 Centralizing listener handling; avoiding some empty methods
2021-01-19 15:16:46 +01:00
rkerner
15064f42b1
DBZ-2831 allow non default Mongo DB ports
2020-12-10 05:33:00 +01:00
Gunnar Morling
9bb53b9e2d
DBZ-2138 Static loggers
2020-11-30 10:33:05 +01:00
Jiri Pechanec
83df96ff70
DBZ-2138 Add defensive checks
2020-11-30 10:33:05 +01:00
Jiri Pechanec
349fcdad07
DBZ-2138 Handle lsid as Binary for MongoDB 4.2
2020-11-30 10:33:05 +01:00
Jiri Pechanec
e8aecd7b45
DBZ-2138 Use MongoDB 4.x driver
2020-11-30 10:33:05 +01:00
Jiri Pechanec
ea019c3b2c
DBZ-1322 Removed dependency on deprecated MongoDB driver code
2020-11-27 17:37:57 +01:00
vanhoale
17a424d9ac
DBZ-2662 Handle ChangeEventQueue by the size in bytes
2020-11-16 08:39:33 +01:00
Chris Cranford
aea26a2949
DBZ-2742 Migrate initial.sync.max.threads to snapshot.max.threads
2020-11-11 09:08:05 +01:00