jcechace
351a0d73fe
DBZ-4339 Preferring secondaries to perform incremental snapshot where possible
2022-12-20 15:03:17 +01:00
jcechace
3966a4099a
DBZ-4339 MongoDbIncrementalSnapshotChangeEventSource uses MongoPreferredNode
2022-12-20 15:03:17 +01:00
jcechace
27cea0e5e6
DBZ-4339 Preferring secondaries to perform snapshot
2022-12-20 15:03:17 +01:00
jcechace
44ae4697d7
DBZ-4339 Preferring secondaries for opening mongodb change stream during streaming phase
2022-12-20 15:03:17 +01:00
jcechace
434322256c
DBZ-4339 Generalising MongoPrimary into MongoPreferredNode
2022-12-20 15:03:17 +01:00
harveyyue
03933f61ea
DBZ-5818 field.exclude.list in MongoDB Connector v2.0 doesn't accept * as a wildcard for collectionName
2022-12-19 13:39:37 +01:00
Debezium Builder
177a60a9d3
[maven-release-plugin] prepare for next development iteration
2022-12-16 09:33:38 +00:00
Debezium Builder
f805db8c3c
[maven-release-plugin] prepare release v2.1.0.Beta1
2022-12-16 09:33:38 +00:00
jcechace
60910872b9
DBZ-5857 Using single member replica as default
2022-12-16 08:43:25 +01:00
jcechace
c9ef1136b5
DBZ-5857 Excluding JUnit5 from debezium-testing-testcontainers dependency in mongo module
2022-12-16 08:43:25 +01:00
jcechace
0951f9994f
DBZ-5857 Migrated MongoDB integration tests to test-containers
2022-12-16 08:43:25 +01:00
Vojtech Juranek
9494552a3a
DBZ-5838 Add missing Apicurio converter dependency
2022-12-12 14:44:05 +01:00
Marcelo Avancini
7754163c5d
DBZ-2432: Fixing misuse of assertj
...
- Some usages of assertj was not callig the actual assert of the DSL, fixing it
- Changing the Assertions to be staticlly imported
2022-12-08 13:10:26 +01:00
Vojtech Juranek
e1b65cd6f7
DBZ-5889 Run integration tests in specified order
...
Run the tests alwyas in thr same order to make it more easy to debug
failures. If needed, the order can be changed (e.g. to `random`) by
overriding propeperty `runOrder`.
2022-12-08 13:00:41 +01:00
Jiri Pechanec
97ea30f25d
DBZ-5893 Mask connection string in connector option logging
2022-12-07 08:41:34 +01:00
jcechace
2468d73783
DBZ-5893 Masking credentials in connection string when logging
2022-12-07 05:45:58 +01:00
Debezium Builder
81896ed334
[maven-release-plugin] prepare for next development iteration
2022-11-30 07:52:24 +00:00
Debezium Builder
1aac97b87c
[maven-release-plugin] prepare release v2.1.0.Alpha2
2022-11-30 07:52:23 +00:00
Bobby Tiernay
adb672a87c
DBZ-4339 Migrate from JUnit 4 to 5, rename utility class, fix formatting.
2022-11-30 07:00:56 +01:00
Bobby Tiernay
950a2a8bdf
DBZ-4339 Migrate MongoDB Testcontainer support to debezium-testing-testcontainers
module.
2022-11-30 07:00:56 +01:00
Bobby Tiernay
5c94151b1f
DBZ-4339 Make mongosh
more resilient to various output conditions.
2022-11-30 07:00:56 +01:00
Bobby Tiernay
849347b7ec
DBZ-4339 Remove reliance on MongoDB driver in non-IT code.
2022-11-30 07:00:56 +01:00
Bobby Tiernay
3ced3afa9f
DBZ-4339 Add polish, consistency, Java Doc, and DRY duplicated code elements.
2022-11-30 07:00:56 +01:00
Bobby Tiernay
8f6abd4428
DBZ-4339 Remove need for container names and fixed ports on Linux.
2022-11-30 07:00:56 +01:00
Bobby Tiernay
b6712c7ccc
DBZ-4339 Convert to use IPs for linux MongoClient
s.
2022-11-30 07:00:56 +01:00
Bobby Tiernay
8c8804665f
DBZ-4339 Fix errant shard host names and add additional java docs / assertions.
2022-11-30 07:00:56 +01:00
Bobby Tiernay
6f9ee66678
DBZ-4339 Add MongoDB Testcontainers support for functional testing.
2022-11-30 07:00:56 +01:00
Erdinç Taşkın
97548591d6
[optimization] Prevent unnecessary list creation
...
Prevent unnecessary list creation
2022-11-29 14:08:27 +01:00
Vojtech Juranek
79c47334c2
DBZ-5834 Allow to add updateDescription fields
...
Allow to add `updateDescription` fields as additional fields/headers
during `ExtractNewDocumentState` transformation.
2022-11-16 14:26:51 +01:00
Debezium Builder
e311dd1a67
[maven-release-plugin] prepare for next development iteration
2022-11-10 13:29:06 +00:00
Debezium Builder
e3bf53440a
[maven-release-plugin] prepare release v2.1.0.Alpha1
2022-11-10 13:29:05 +00:00
Vojtech Juranek
96e4326d35
DBZ-2131 Use system property to determine if Apicurio should be used
2022-11-09 08:44:45 +01:00
Vojtech Juranek
e5a9495327
DBZ-2131 Add Apicurion profile for Mongo
2022-11-09 08:44:45 +01:00
Vojtech Juranek
82a66e4618
DBZ-2131 Unify intendation in Mongo pom.xml
2022-11-09 08:44:45 +01:00
jcechace
41d07fed85
DBZ-5629 Added test to verify MongoDbTaskContext properties
2022-11-06 15:03:16 +01:00
jcechace
7093b63c45
DBZ-5629 Correctly setting task id for MongoDbTaskContext
2022-11-06 15:03:16 +01:00
jcechace
f91963f59d
DBZ-5779 Migrated debezium-connector-mongodb to AssertJ
2022-11-02 08:50:26 +01:00
Yang Wu
370a6db733
DBZ-5736 Expose heartbeatFrequency to Mongo connector config
2022-10-19 13:58:36 +02:00
Debezium Builder
4d35fb8011
[maven-release-plugin] prepare for next development iteration
2022-10-14 11:26:32 +00:00
Debezium Builder
7bd1551c99
[maven-release-plugin] prepare release v2.0.0.Final
2022-10-14 11:26:32 +00:00
Debezium Builder
884254aba1
[maven-release-plugin] prepare for next development iteration
2022-10-07 09:27:52 +00:00
Debezium Builder
a0f3263fa6
[maven-release-plugin] prepare release v2.0.0.CR1
2022-10-07 09:27:51 +00:00
Sage-Pierce
e6776d7382
DBZ-5677: Ensure there is a time to start at before setting one
2022-10-06 11:24:32 +02:00
Sage-Pierce
ff213c9f03
DBZ-5677: Apply fix from DBZ-5371 for MongoDB Snapshot Source to Streaming Source
2022-10-06 11:24:32 +02:00
Jiri Pechanec
75aeac595d
DBZ-5628 Fix formatting
2022-10-06 09:38:20 +02:00
Bin Huang
92b3035457
DBZ-5628 Add mongo 6.0 pre-image IT test
2022-10-06 09:38:20 +02:00
Bin Huang
94bdf755d1
DBZ-5628 Add mongo 6.0 version and db version resolver
2022-10-06 09:38:20 +02:00
Xinbin Huang
b4a87d45a7
DBZ-5628 Added support for Mongo pre-image in change stream
2022-10-06 09:38:20 +02:00
harveyyue
f6b72f060c
DBZ-5671 Remove logic name parameter from sub connector config
2022-10-03 11:40:17 -04:00
Sergei Morozov
e4d46c81b1
DBZ-5541: Default Debezium schema.name.adjustment.mode to "none"
2022-09-29 09:47:25 +02:00
Vojtech Juranek
97532042b3
DBZ-5657 Allow to use Docker maven plugin external properties
...
Allow to override Docker maven plugin properties [1] from command line
to be able to change various Docker parameters more easily when starting
the container.
[1] https://dmp.fabric8.io/#combining-property-config
2022-09-29 05:41:09 +02:00
harveyyue
d1153e8ad4
DBZ-5594 Clean up "logical name" config
2022-09-22 14:29:37 +02:00
Debezium Builder
218d414552
[maven-release-plugin] prepare for next development iteration
2022-09-16 06:36:40 +00:00
Debezium Builder
567f57c47b
[maven-release-plugin] prepare release v2.0.0.Beta2
2022-09-16 06:36:39 +00:00
Vojtech Juranek
1afcea971c
DBZ-5043 Add topic prefix to Mongo config definition
2022-09-14 12:44:38 +02:00
Vojtech Juranek
acc6be6df3
DBZ-5043 Move TOPIC_PREFIX into CommonConnectorConfig
...
`TOPIC_PREFIX` is now mandatory to all connectors therefore it make
sense to have it in common config. Beside that, it also makes it more
easy to use it in Debezium UI without any workarounds - if the field is
not member of the given connector config, the field has to be
explicitely added into known fields otherwise is invisible for UI.
2022-09-13 09:25:54 +02:00
Anisha Mohanty
f41b6dd458
DBZ-5044 Get rid of schema factory instances
2022-09-08 10:35:20 +02:00
Anisha Mohanty
b498368f82
DBZ-5044 Use SchemaFactory for centralizing MongoDb schemas
2022-09-08 10:35:20 +02:00
Vojtech Juranek
8d7611bfcb
DBZ-5043 Remove unused DB_HISTORY_PATH constant
2022-09-07 12:57:31 +02:00
Vojtech Juranek
eb6e2648f4
DBZ-5043 Replace mongodb.name with topic.prefix
...
To be consistent with specifying topic prefix, replace also MongoDB
`mongodb.name` with `topic.prefix`.
2022-09-07 12:57:31 +02:00
Vojtech Juranek
161a7229b6
DBZ-5043 Use topic prefix as the logical name
...
With this change the user cannot direcly set connector logical name and
thus in the future it can be remove without breaking user config.
If the turn out that the logical name is useful and user should be able
to configure it, dedicated config option can be added.
Originally it was prposed in the Jira to replace it with connector name,
but it turned out that logical name defaults to `database.server.name`
and is heavily used in the tests and JMX, so it would require another
big refactoring. Thus, use topic prefix for now. Once we know further
direction (remove logical name or add new option), do this refactoring.
2022-09-07 12:57:31 +02:00
ggaborg
664fd9e1b0
DBZ-3293 unit tests for filters
2022-09-01 20:16:47 +02:00
ggaborg
244de2bd96
DBZ-3293 Add signal table automatically to include list
2022-09-01 20:16:47 +02:00
Jiri Pechanec
de53591dba
DBZ-5244 Retry all communication exceptions by default
2022-08-31 09:45:09 -04:00
jcechace
5c61bc4341
DBZ-4733 Reordering fileds in MongoDbConnectorConfig
2022-08-19 12:55:58 +02:00
jcechace
b61319feb7
DBZ-4733 Basic integration test for connection string support
2022-08-19 12:55:58 +02:00
jcechace
7670c4ba74
DBZ-4733 working POC for connection String support
2022-08-19 12:55:58 +02:00
jcechace
521156f5c2
DBZ-4733 Refactored MongoUtil.getPrimaryAddress to properly use mongo driver 4.x
2022-08-19 12:55:58 +02:00
jcechace
21ed5f9b9f
DBZ-4733 Refactor ReplicaSets.parse into more readable form
2022-08-19 12:55:58 +02:00
jcechace
752062c1a3
DBZ-4733 Fixed thread safety of MongoClients
2022-08-19 12:55:58 +02:00
jcechace
1d9d75754c
DBZ-4733 Refactoring ReplicaSetDiscovery to properly use mongodb driver 4.x
2022-08-19 12:55:58 +02:00
jcechace
3a385aa768
DBZ-4733 Refactoring: unified client factory method names
2022-08-19 12:55:58 +02:00
jcechace
cc04a94fd5
DBZ-4733 Ininital connection string filed introduction
2022-08-19 12:55:58 +02:00
Chris Cranford
b7e2112603
DBZ-5497 Align test behavior
2022-08-19 11:05:29 +02:00
Vivek Wassan
d89e71ebeb
DBZ-5327 Smart Backfills
2022-08-17 10:05:55 +02:00
nicholas-fwang
58a91cddb4
DBZ-5479 Retry MongoNodeIsRecoveringException when failover
2022-08-05 10:52:46 -04:00
Chris Cranford
90a4c0059a
DBZ-5434 Fix test failure
2022-08-01 14:36:09 -04:00
harveyyue
7c93c597f7
DBZ-5434 Translation from mongodb document to kafka connect schema fails when nested arrays contain no elements
2022-08-01 10:07:28 -04:00
Vojtech Juranek
c7d8353d7b
DBZ-4727 Add snapshot paused JXM metrics
...
Add two JXM objects:
* `snapshotPaused` - detemines if the incremental snapshot is paused
or not
* `snapshotPausedDurationInSeconds` - overall time when the incremental
snapshot was paused. The time adds up - if the snapshot was paused
e.g. two times, the `snapshotPausedDurationInSeconds` is the sum of
these two paused times.
2022-07-29 12:41:58 -04:00
Vojtech Juranek
ed504173e1
DBZ-4727 Implement incremental snaphost pause/resume
...
Intorduce two new signals:
* `pause-snapshot` - pauses running incremental snapshot
* `resume-snapshot` - resumes paused incremental snapshot
If the incremental snapshot is running and pause signal is sent,
currently processed chunk is finished and then no other chunk is read.
Snapshot state is kept in running state. Once snapshot is resume,
chunk is reverted to position which was last sent to the broker and
process of reading chunks and emitting `read` events continues until
all tables are read and sent.
This patch handles only table-based signals. MySQL Kafka-based signals
will be addressed later.
2022-07-29 12:41:58 -04:00
harveyyue
5f430f1a8e
DBZ-5457 Deprecate legacy topic selector for all connectors
2022-07-28 12:58:59 -04:00
Debezium Builder
1ab51ae8d0
[maven-release-plugin] prepare for next development iteration
2022-07-26 18:13:10 +00:00
Debezium Builder
d247a48b79
[maven-release-plugin] prepare release v2.0.0.Beta1
2022-07-26 18:13:09 +00:00
Chris Cranford
de4d8e17d3
DBZ-5371 Check for $natural not supported, treat as no oplog entry
2022-07-25 10:20:46 -04:00
harveyyue
6ea7abc8fc
DBZ-4180 Pluggable topic selector
2022-07-13 14:45:25 +02:00
harveyyue
e8405bc376
DBZ-4846 MongoConnector's field exclusion configuration does not work with fields with the same name but from different collections
2022-07-12 17:26:36 +02:00
Debezium Builder
6fd1441af3
[maven-release-plugin] prepare for next development iteration
2022-07-01 09:21:49 +00:00
Debezium Builder
99402af634
[maven-release-plugin] prepare release v2.0.0.Alpha3
2022-07-01 09:21:48 +00:00
harveyyue
2a2b5462db
DBZ-5328 field.exclude.list doesn't accept dashes (-) in database name
2022-06-30 10:47:56 +02:00
Mickael Maison
5ff71aa992
DBZ-5335 Fix formatting issues in the connectors configdefs
2022-06-30 09:35:54 +02:00
kanha gupta
1b17372d2d
DBZ-4902 Replace long durations with Duration
2022-06-27 14:29:51 +02:00
Jiri Pechanec
38726a8451
DBZ-5260 Fix imports
2022-06-20 09:22:24 +02:00
Tim Patterson
c0323305ac
DBZ-5260 Filter out unavailable replicaset members
2022-06-20 08:56:57 +02:00
Debezium Builder
6fff3a2876
[maven-release-plugin] prepare for next development iteration
2022-06-09 11:49:26 +00:00
Debezium Builder
b2e2e351df
[maven-release-plugin] prepare release v2.0.0.Alpha2
2022-06-09 11:49:26 +00:00
Vojtech Juranek
4a48eb33df
DBZ-5045 Remove legacy V1 source info
2022-06-08 09:11:00 +02:00
Vojtech Juranek
76c8550e7b
DBZ-5045 Remove deprecated Mongo DB OPERATION_HEADER option
2022-06-08 09:11:00 +02:00
Vojtech Juranek
4140078db7
DBZ-5045 Remove deprecated Mongo DB ADD_SOURCE_FIELDS option
...
Adjust the tests as source `ts_ms` is different.
2022-06-08 09:11:00 +02:00
Vojtech Juranek
0ddcd93167
DBZ-5045 Remove deprecated POLL_INTERVAL_SEC
2022-06-08 09:11:00 +02:00
Vojtech Juranek
64ef812cc9
DBZ-5045 Remove deprecated MAX_COPY_THREADS
2022-06-08 09:11:00 +02:00
Vojtech Juranek
0d6bfec265
DBZ-5045 Remove deprecated FIELD_BLACKLIST
2022-06-08 09:11:00 +02:00
Vojtech Juranek
d66d86a3ac
DBZ-5045 Remove deprecated COLLECTION_BLACKLIST
2022-06-08 09:11:00 +02:00
Vojtech Juranek
2484faaffe
DBZ-5045 Remove deprecated COLLECTION_WHITELIST
2022-06-08 09:11:00 +02:00
Vojtech Juranek
b3a6131cee
DBZ-5045 Remove deprecated DATABASE_BLACKLIST
2022-06-08 09:11:00 +02:00
Vojtech Juranek
0cbb57f4bf
DBZ-5045 Remove deprecated DATABASE_WHITELIST
2022-06-08 09:11:00 +02:00
harveyyue
acb743bd34
DBZ-5206 Remove the deprecated monitored tables metrics
2022-06-07 16:13:29 -04:00
Chris Cranford
9d71296b5c
DBZ-4251 Support stopping incremental snapshots
2022-06-02 07:30:07 +02:00
Jiri Pechanec
88b7ae2050
DBZ-5136 Move sourceInfo to common class
2022-06-01 16:56:29 +02:00
Mark Bereznitsky
c64b8af31a
DBZ-5136 reintroduce the CommonOffsetContext class and refactor common functionality into it
2022-06-01 16:56:29 +02:00
Mark Bereznitsky
2d72dae649
DBZ-5136 further refining of the algorithm for determining position in snapshot and IT test fixes
2022-06-01 16:56:29 +02:00
Jiri Pechanec
08e8c3d795
DBZ-5136 Move common class to common package
2022-06-01 16:56:29 +02:00
Andrew Walker
d0759461a4
DBZ-5136: Convert RelationalOffsetContext to CommonOffsetContext
2022-06-01 16:56:29 +02:00
Andrew Walker
b6222d42bf
DBZ-5136: Add first flag and refactor OffsetContext
2022-06-01 16:56:29 +02:00
Andrew Walker
593c2769e6
DBZ-5136: Add last in table flag
2022-06-01 16:56:29 +02:00
Chris Cranford
28cd36edff
DBZ-5011 Set task
metrics attribute when using only 1 task
2022-06-01 09:41:36 +02:00
Chris Cranford
9fa5a48274
DBZ-5011 Register MongoDB metrics with task id when more than one task
2022-06-01 09:41:36 +02:00
Yang
75d3b716ab
DBZ-5113 Use generic bson value
2022-05-17 14:32:51 +02:00
Yang
5c190f7584
DBZ-5113 Use bson document
2022-05-17 14:32:51 +02:00
Jiri Pechanec
707eeab3d8
DBZ-4951 Remove oplog suppport
2022-05-11 16:04:04 +02:00
Debezium Builder
901825810a
[maven-release-plugin] prepare for next development iteration
2022-04-28 13:51:13 +00:00
Debezium Builder
05dcb45647
[maven-release-plugin] prepare release v2.0.0.Alpha1
2022-04-28 13:51:12 +00:00
Jiri Pechanec
ee78ba6ad6
DBZ-5015 Fix formatting
2022-04-21 07:09:18 +02:00
chadthamn
1b23507504
DBZ-5015 Get the json value of a Document when parsing signaling message
2022-04-21 07:09:18 +02:00
jchipmunk
2c2a81b12b
DBZ-5003 Can't use 'local' database through mongos
...
MongoDb connector should use "config" database if the database exists. Otherwise, "local" database should be used to get MongoDB version during connector validation.
This is similar to the approach implemented in ReplicaSetDiscovery class.
2022-04-20 09:36:50 +02:00
Debezium Builder
4a6de64c25
[maven-release-plugin] prepare for next development iteration
2022-04-05 14:49:52 +00:00
Debezium Builder
683851fd90
[maven-release-plugin] prepare release v1.9.0.Final
2022-04-05 14:49:51 +00:00
Debezium Builder
ab7fe3b96a
[maven-release-plugin] prepare for next development iteration
2022-03-25 16:45:06 +00:00
Debezium Builder
d19fc234ee
[maven-release-plugin] prepare release v1.9.0.CR1
2022-03-25 16:45:06 +00:00
Sergei Morozov
b258674782
DBZ-3535: Introduce schema.name.adjustment.mode
2022-03-24 09:37:05 +01:00
Debezium Builder
fece16039c
[maven-release-plugin] prepare for next development iteration
2022-03-03 17:00:08 +00:00
Debezium Builder
e0d8a0d809
[maven-release-plugin] prepare release v1.9.0.Beta1
2022-03-03 17:00:08 +00:00
Sergei Morozov
6d0aa9408c
DBZ-4478: Add Partition parameter to io.debezium.pipeline.source.spi.*Listener interfaces
2022-02-16 11:31:38 +01:00
Debezium Builder
7a47f478a4
[maven-release-plugin] prepare for next development iteration
2022-02-09 07:23:10 +00:00
Debezium Builder
b1dd35025e
[maven-release-plugin] prepare release v1.9.0.Alpha2
2022-02-09 07:23:10 +00:00
rkerner
8127985da0
[docs] fix Mongo DB Connector Config typo for mongodb.name
field description
2022-02-08 15:14:28 +01:00
Chris Cranford
bbd963549e
DBZ-4676 Fix MongoUtilIT test failure
2022-02-04 12:14:06 -05:00
Debezium Builder
33e2f5cd0f
[maven-release-plugin] prepare for next development iteration
2022-01-26 19:00:20 +00:00
Debezium Builder
a77b6f8b8e
[maven-release-plugin] prepare release v1.9.0.Alpha1
2022-01-26 19:00:20 +00:00
Jiri Pechanec
664d989e42
DBZ-4577 Provide custom error message handling
2022-01-21 10:24:19 +01:00
Jiri Pechanec
ce41b0734c
DBZ-4575 Filter events based on database name
2022-01-20 19:13:44 +01:00
Jiri Pechanec
022f480ebf
DBZ-4415 oplog mode is not supported for MongoDB 5+
...
Co-authored-by: Gunnar Morling <gunnar.morling@googlemail.com>
2022-01-18 14:09:11 +01:00
Jiri Pechanec
8fd3243a86
DBZ-2224 Test logging based on logback
2022-01-18 11:24:21 +01:00
Sergei Morozov
0329859cda
DBZ-4459: Introduce interfaces and default implementations for change event source metrics
2022-01-12 14:32:19 +01:00
Anisha Mohanty
d6c5ad2e7c
DBZ-4460 Renaming all openapi occurrences to schema
2022-01-07 15:05:34 +01:00
Anisha Mohanty
3d1988cf6f
DBZ-4460 Set default values for format & output directory
2022-01-07 15:05:34 +01:00
Anisha Mohanty
06860b3691
DBZ-4460 Apply the plug-in to all connector modules
2022-01-07 15:05:34 +01:00
harveyyue
291e8ac221
DBZ-4351 Add create/update/delete event seen metrics for monitor upstream dml operation
2022-01-06 19:03:46 +01:00
Gunnar Morling
92665f1799
DBZ-4472 Improving log output
2021-12-20 15:58:14 +01:00
harveyyue
792187f669
DBZ-4472 Make connector task partition readability for log
2021-12-20 15:58:14 +01:00
Debezium Builder
67c9663c98
[maven-release-plugin] prepare for next development iteration
2021-12-16 10:11:29 +00:00