Debezium Builder
274bba9dc9
[maven-release-plugin] prepare for next development iteration
2020-01-16 05:40:24 -05:00
Debezium Builder
076df30108
[maven-release-plugin] prepare release v1.1.0.Alpha1
2020-01-16 05:40:24 -05:00
Gunnar Morling
8c33a4e147
DBZ-1292 Catering for the differences in data schema;
...
* MongoDB doesn't have "before", but "patch"
* Also removing some redundancies and making RecordParser immutable
* Reducing log verbosity during tests
2020-01-16 08:01:56 +01:00
Gunnar Morling
7c18a6d8d5
DBZ-1292 Removing static state in CloudEventsConverterTest
2020-01-16 08:00:45 +01:00
GraySmog
d0e4b3c0af
DBZ-1292 Delete data.schema.registry.url config option
2020-01-16 08:00:45 +01:00
GraySmog
915c8cafd9
DBZ-1292 CloudEvents converter for JSON and Avro format
2020-01-16 08:00:45 +01:00
Jiri Pechanec
4102d39aba
DBZ-1168 Suport non-admin authSources
2020-01-06 12:28:49 +01:00
Debezium Builder
9723c93db9
[maven-release-plugin] prepare for next development iteration
2019-12-18 01:44:33 -05:00
Debezium Builder
09049516f2
[maven-release-plugin] prepare release v1.0.0.Final
2019-12-18 01:44:33 -05:00
Debezium Builder
b1c62fe306
[maven-release-plugin] prepare for next development iteration
2019-12-10 06:39:17 -05:00
Debezium Builder
ad2efdb9a6
[maven-release-plugin] prepare release v1.0.0.CR1
2019-12-10 06:39:17 -05:00
Brandon Brown
a0f7f6ec6b
DBZ-1402 Support for JDK 11
2019-11-25 11:53:19 +01:00
Gunnar Morling
bf25aeb1de
DBZ-1215 Formatting
2019-11-25 11:52:48 +01:00
Gunnar Morling
0fa29a0163
DBZ-1215 Misc. clean-up;
...
* long ms -> Instant
* Using OptionalLong as return value for optional value
2019-11-22 11:42:42 +01:00
Jiri Pechanec
6cadff8c36
DBZ-1215 Added support for transactions
2019-11-22 11:42:42 +01:00
Debezium Builder
38279cc2dd
[maven-release-plugin] prepare for next development iteration
2019-11-14 02:56:07 -05:00
Debezium Builder
868faf4412
[maven-release-plugin] prepare release v1.0.0.Beta3
2019-11-14 02:56:07 -05:00
Gunnar Morling
2c569977d7
DBZ-1392 Applying formatter;
...
* Keeping indentation formatting for JSON
Co-authored-by: Gunnar Morling <gunnar.morling@googlemail.com>
Co-authored-by: Chris Cranford <chris@hibernate.org>
2019-11-01 13:20:43 +02:00
Chris Cranford
c25bb23a04
DBZ-1583 Propagate Replicator exception to Kafka Connect
2019-10-29 16:17:27 +01:00
Debezium Builder
b08f523ed7
[maven-release-plugin] prepare for next development iteration
2019-10-24 07:42:49 -04:00
Debezium Builder
8ecd9e28cf
[maven-release-plugin] prepare release v1.0.0.Beta2
2019-10-24 07:42:49 -04:00
Jiri Pechanec
b2a70889f5
DBZ-1563 Collection names can contain dots
2019-10-23 14:40:15 -04:00
Debezium Builder
519dc011ff
[maven-release-plugin] prepare for next development iteration
2019-10-17 04:20:13 -04:00
Debezium Builder
51ef6b9470
[maven-release-plugin] prepare release v1.0.0.Beta1
2019-10-17 04:20:12 -04:00
Jiri Pechanec
b88d9b06d4
DBZ-1513 Align test names
2019-10-07 18:38:35 +02:00
Jiri Pechanec
2c4cb83259
DBZ-1513 Centralize handling of non-data messages in SMT
2019-10-07 18:38:35 +02:00
Gunnar Morling
92885ebe99
DBZ-1392 Removing trailing whitespace from Java files
2019-10-07 12:11:43 +03:00
Gunnar Morling
8d6f5409b1
DBZ-1392 Moving static imports up and applying this to all files
2019-10-07 12:11:43 +03:00
Chris Cranford
ca5517746a
DBZ-1341 Checkstyle flag missing whitespace
2019-10-03 19:44:43 +02:00
Chris Cranford
7442670692
DBZ-1355 Fix checkstyle problems detected after upgrade to 8.19
2019-10-03 19:44:43 +02:00
Chris Cranford
5ac2318d84
DBZ-1391 Do not allow code after closing brace ( #1065 )
...
* DBZ-1391 Do not allow code after closing brace
2019-10-02 20:44:40 +02:00
Debezium Builder
2ff77b993b
[maven-release-plugin] prepare for next development iteration
2019-10-01 15:34:23 -04:00
Debezium Builder
e09d584b11
[maven-release-plugin] prepare release v0.10.0.Final
2019-10-01 15:34:23 -04:00
Debezium Builder
2f5960c5d2
[maven-release-plugin] prepare for next development iteration
2019-09-27 01:03:50 -04:00
Debezium Builder
e3b44ffb12
[maven-release-plugin] prepare release v0.10.0.CR2
2019-09-27 01:03:50 -04:00
jerrinot
3064a4af06
DBZ-1438 Fix InterruptedException handling
...
It appears there is an intent to restore the interruption flag
when catching InterruptedException. However the old code was just
checking the current state of the flag instead of raising it.
2019-09-24 12:51:48 +02:00
Ching Tsai
80210f7f8b
DBZ-1442 Add test for a rewrite delete event
2019-09-13 11:23:26 +02:00
Ching Tsai
b8d565ffe0
DBZ-1442 Add @FixFor
2019-09-13 11:23:26 +02:00
Ching Tsai
486973ad12
DBZ-1442 Porting insert source fields feature to ExtractNewDocumentState
2019-09-13 11:23:26 +02:00
Debezium Builder
9b7e09fdda
[maven-release-plugin] prepare for next development iteration
2019-09-10 09:12:12 -04:00
Debezium Builder
a1c2861458
[maven-release-plugin] prepare release v0.10.0.CR1
2019-09-10 09:12:11 -04:00
Gunnar Morling
1e0859588f
DBZ-1464 Misc. clean-up;
...
* logger -> LOGGER
* making op literals map unmodifiable
* Adding Lev Zemlyanov to COPYRIGHT.txt
2019-09-06 11:02:45 +02:00
Lev Zemlyanov
8cbb8eb63b
DBZ-1464 Fixed NPE when op is not u/i/d
...
Signed-off-by: Lev Zemlyanov <lev@confluent.io>
2019-09-06 11:02:27 +02:00
Debezium Builder
4ae105bdce
[maven-release-plugin] prepare for next development iteration
2019-08-16 03:42:55 -04:00
Debezium Builder
945ccf68cb
[maven-release-plugin] prepare release v0.10.0.Beta4
2019-08-16 03:42:54 -04:00
Renato Mefi
a9c8a825c3
DBZ-1430 Support heartbeat ExtractNewDocumentState
2019-08-14 23:56:45 +02:00
Jiri Pechanec
0742e5775f
Revert "[maven-release-plugin] prepare release v0.10.0.Beta4"
...
This reverts commit c7340e5152
.
2019-07-23 14:54:55 +02:00
Jiri Pechanec
f7d20635ce
Revert "[maven-release-plugin] prepare for next development iteration"
...
This reverts commit 22879777e4
.
2019-07-23 14:54:49 +02:00
Jiri Pechanec
22879777e4
[maven-release-plugin] prepare for next development iteration
2019-07-23 14:10:03 +02:00
Jiri Pechanec
c7340e5152
[maven-release-plugin] prepare release v0.10.0.Beta4
2019-07-23 14:10:02 +02:00
Debezium Builder
71333fda98
[maven-release-plugin] prepare for next development iteration
2019-07-23 02:55:08 -04:00
Debezium Builder
20d006fac7
[maven-release-plugin] prepare release v0.10.0.Beta3
2019-07-23 02:55:08 -04:00
Gunnar Morling
287551294f
DBZ-1198 Adding comments
2019-07-11 07:43:32 +02:00
Jiri Pechanec
0316ac7fc0
DBZ-1198 Heartbeat for MongoDB connector
2019-07-10 11:16:26 +02:00
Jiri Pechanec
13d638fd43
DBZ-1356 Metrics for schema history ( #947 )
...
* DBZ-1356 Metrics for schema history
2019-07-02 13:26:17 +02:00
Gunnar Morling
a991ea97d7
Handling option for emitting tombstones conistently
2019-07-01 08:27:33 +02:00
Jenkins user
e95d656bca
[maven-release-plugin] prepare for next development iteration
2019-06-27 07:36:13 +00:00
Jenkins user
d95fd87311
[maven-release-plugin] prepare release v0.10.0.Beta2
2019-06-27 07:36:12 +00:00
Jiri Pechanec
72b751b528
DBZ-1333 Upgrade MongoDb driver to 3.10.1
2019-06-17 20:53:07 +02:00
Jenkins user
5fdb0ca81b
[maven-release-plugin] prepare for next development iteration
2019-06-11 11:20:46 +00:00
Jenkins user
61729b9e00
[maven-release-plugin] prepare release v0.10.0.Beta1
2019-06-11 11:20:46 +00:00
Gunnar Morling
5d01f409f0
DBZ-1317 Separating update and query methods for MongoDB's SourceInfo;
...
* Avoids access to collection id after events which don't have one
* Pulling up struct() and schema() methods to common base class
2019-06-11 10:07:36 +02:00
Chris Cranford
0aadaa10dc
DBZ-1242 Fixed typo and added negative test checks
2019-06-05 13:31:25 -04:00
Chris Cranford
c53435c26b
DBZ-1242 Checkstyle fix - Remove unused import
2019-06-05 13:31:25 -04:00
Chris Cranford
b3ca33287a
DBZ-1242 Issue warning when filters produce an empty schema
2019-06-05 13:31:25 -04:00
Ching Tsai
f8cb51545f
DBZ-1315 UNDEFINED field causes exception
2019-06-03 13:51:01 +02:00
Jenkins user
056730d06a
[maven-release-plugin] prepare for next development iteration
2019-06-03 10:24:08 +00:00
Jenkins user
7aeaec236e
[maven-release-plugin] prepare release v0.10.0.Alpha2
2019-06-03 10:24:08 +00:00
Jiri Pechanec
d1bc1c218c
DBZ-1295 3state snapshot for MongoDB
2019-05-31 10:56:05 +02:00
Jiri Pechanec
946019982e
DBZ-1303 Assembly without database drivers
2019-05-30 14:32:47 +02:00
Jenkins user
ac5c76e431
[maven-release-plugin] prepare for next development iteration
2019-05-28 14:38:32 +00:00
Jenkins user
35f59ba460
[maven-release-plugin] prepare release v0.10.0.Alpha1
2019-05-28 14:38:32 +00:00
Gunnar Morling
b8eede2f41
DBZ-1235 Misc. clean-up;
...
* Putting collection name after replica set name for MongoDB
* Removing redundant method for converting schema to string
* Removing redundant keys from PG SourceInfo
2019-05-28 09:35:15 +02:00
Jiri Pechanec
526e780513
DBZ-1235 Misc fixes
2019-05-28 09:35:15 +02:00
Jiri Pechanec
80afd4975a
DBZ-1235 Changes missed in refactoring
2019-05-28 09:35:15 +02:00
Gunnar Morling
8f28f323e7
DBZ-1235 Using Instant to represent timestamps internally;
...
Only "rendered" as epoch milli-seconds at the system boundaries.
2019-05-28 09:35:15 +02:00
Jiri Pechanec
6f6c8ddf06
DBZ-1235 Refactored to unify the code
2019-05-28 09:35:15 +02:00
Jiri Pechanec
c1bf1752fe
DBZ-1235 Pluggable source maker for MongoDB
2019-05-28 09:35:15 +02:00
Jiri Pechanec
f5757dbd9f
DBZ-1235 Extract serverName to common source
2019-05-28 09:35:15 +02:00
Gunnar Morling
cf4db8c1a1
DBZ-677 Extracting shared configuration options for both new state extraction SMTs
2019-05-27 13:52:58 +02:00
Gunnar Morling
cb9e916616
DBZ-677 Renaming record state SMTs;
...
Keeping old names in deprecated sub-classes for the time being.
2019-05-27 13:52:58 +02:00
Chris Cranford
e2c74111ad
DBZ-1234 Removed deprecated configuration options
2019-05-22 10:19:05 +02:00
Jenkins user
7748be6056
[maven-release-plugin] prepare for next development iteration
2019-05-02 16:04:56 +00:00
Jenkins user
5af083b86e
[maven-release-plugin] prepare release v0.9.5.Final
2019-05-02 16:04:55 +00:00
jchipmunk
72b06366fe
Removing extra initialisation of "config" variable in PG config
2019-04-30 09:57:11 +02:00
jchipmunk
9fe033ce24
DBZ-1247 Specifying default snapshot fetch size for each connector config
2019-04-30 09:57:11 +02:00
jchipmunk
a5cc83c0d1
DBZ-1247 Adding property to specify snapshot fetch size
2019-04-30 09:57:11 +02:00
jchipmunk
2c79d901c5
DBZ-1247 Using isNonNegativeInteger instead of isPositiveInteger
2019-04-30 09:57:11 +02:00
jchipmunk
3ca47b1055
DBZ-1247 Adding property to specify batch size of MongoDB cursor
...
The "documents.fetch.size" configuration property is an positive integer value that specifies the maximum number of documents that should be read in one go from each collection while taking a snapshot. The connector will read the collection contents in multiple batches of this size. Default to "0", which indicates that the server chooses an appropriate fetch size.
2019-04-30 09:57:11 +02:00
krizhan
70c564b4c2
DBZ-212 Moved SERVER_NAME to RelationalDatabaseConnectorConfig
2019-04-23 12:29:44 +02:00
Jenkins user
cefc5ccfb9
[maven-release-plugin] prepare for next development iteration
2019-04-11 12:07:16 +00:00
Jenkins user
fc3026adc3
[maven-release-plugin] prepare release v0.9.4.Final
2019-04-11 12:07:15 +00:00
jchipmunk
7c0ae3ee20
DBZ-1212 SLF4J usage issues
2019-04-04 21:32:12 +02:00
ShubhamRwt
540a951211
DBZ- 362 Adding WhitespaceAfter check to Checkstyle
2019-03-28 09:24:11 +01:00
Jenkins user
8b0edd1871
[maven-release-plugin] prepare for next development iteration
2019-03-25 12:16:31 +00:00
Jenkins user
56f98a48b2
[maven-release-plugin] prepare release v0.9.3.Final
2019-03-25 12:16:30 +00:00
Jiri Pechanec
9b405aa84d
DBZ-1039 Add missing spaces
2019-03-25 10:23:14 +01:00
shubham
8cca21e969
DBZ-1039 Make all ifs with braces
2019-03-25 10:23:14 +01:00
Renato Mefi
2f187fd1cb
DBZ-354 Improve MongoDB hosts validation
...
The previous statement was never achievable since `ReplicaSets.parse(hosts)` can never return null
Also removed some code duplication for better readability
2019-03-14 16:53:51 +01:00
Jenkins user
50a7c568fa
[maven-release-plugin] prepare for next development iteration
2019-02-23 09:53:58 +00:00
Jenkins user
28f3839804
[maven-release-plugin] prepare release v0.9.2.Final
2019-02-23 09:53:58 +00:00
Gunnar Morling
d2b26d03a7
DBZ-867 Misc clean-up;
...
* Exposing new config via MongoDbConnectorConfig
* Removing superfluous connector restart in the test
2019-02-21 17:00:12 +01:00
krizhan
652f3b1c59
DBZ-867 Added integration test and updated enum names
2019-02-21 17:00:12 +01:00
krizhan
441228a989
DBZ-867 Add snapshot mode NONE for MongoDB connector
2019-02-21 17:00:12 +01:00
Jenkins user
b96e5fee0d
[maven-release-plugin] prepare for next development iteration
2019-02-13 11:13:44 +00:00
Jenkins user
3de2d04bcd
[maven-release-plugin] prepare release v0.9.1.Final
2019-02-13 11:13:44 +00:00
Gunnar Morling
2a486ecfff
DBZ-1019 Removing legacy methods from Metronome
2019-02-11 18:11:34 +01:00
Jenkins user
c5b7d21d3e
[maven-release-plugin] prepare for next development iteration
2019-01-28 11:07:04 +00:00
Jenkins user
f1ae79ff73
[maven-release-plugin] prepare release v0.9.0.CR1
2019-01-28 11:07:04 +00:00
Renato Mefi
5810014fda
DBZ-988 Migrate UnwrapFromMongoDbEnvelopeTestIT to new Abstract TestIT
2019-01-07 13:42:40 +01:00
Renato Mefi
8a22b6666f
DBZ-988 Break down Unwrap TestIT Abstract
...
This allows other tests to also take advantages of the testing structure
2019-01-07 13:42:40 +01:00
Renato Mefi
82bd0bb657
DBZ-988 Test all Array Update operators
...
As in the official documentation:
https://docs.mongodb.com/v3.6/reference/operator/update-array/#update-operators
The operations: `$(update)`, `$[]` and `$[<identifier>]` are skipped
since the're Mongo 3.6+ only.
2019-01-07 13:42:40 +01:00
Renato Mefi
6fc010867e
DBZ-988 Test all Bitwise Update operators
...
As in the official documentation:
https://docs.mongodb.com/v3.6/reference/operator/update/bit/#bit
2019-01-07 13:42:40 +01:00
Renato Mefi
20cc350891
DBZ-988 Test all Field Update operators
...
As in the official documentation:
https://docs.mongodb.com/v3.6/reference/operator/update-field/#field-update-operators
2019-01-07 13:42:40 +01:00
Renato Mefi
c43c9ecc54
DBZ-988 MongoDB unwrap Update operators base test
2019-01-07 13:42:40 +01:00
Jenkins user
5275f73424
[maven-release-plugin] prepare for next development iteration
2018-12-19 13:06:12 +00:00
Jenkins user
b6569c18ae
[maven-release-plugin] prepare release v0.9.0.Beta2
2018-12-19 13:06:12 +00:00
Jiri Pechanec
edec1c3090
DBZ-978 Added last event and captured tables metrics
2018-12-13 20:52:11 +01:00
Gunnar Morling
91b571059e
DBZ-966 Making sure MongoDB connector can be stopped while awaiting initial sync delay
2018-12-13 14:29:54 +01:00
Grzegorz Kołakowski
e2942ab72a
DBZ-966 Replace sleep with polling loop
2018-12-13 14:29:54 +01:00
Grzegorz Kołakowski
c5fac7d3b2
DBZ-966 Enable to delay initial snapshot in MongoDB connector
2018-12-13 14:29:54 +01:00
Renato Mefi
1f3791cea9
DBZ-1032 Test tombstones have a DELETE header
2018-12-11 13:50:42 +01:00
Renato Mefi
8d04aab6a4
DBZ-1032 Fix issue for tombstone operation headers
...
Since the bomstone has no value it's not possible to imply the header
from the message content, thus if the value is null we set the header to
DELETE
2018-12-11 13:50:42 +01:00
Gunnar Morling
5aeb589932
DBZ-848 Adding Tautvydas Januskevicius to COPYRIGHT.txt
2018-12-11 12:00:59 +01:00
Tautvydas Januskevicius
addf13ca06
DBZ-848 Removed explicit primary node change handling. It is handled by mongoDb driver internally. Explicit primary node change event handling was causing issue that no mongo events were read after primary change. This issue only happened when rs.stepDown(<time s>) was used on mongo node.
2018-12-11 12:00:59 +01:00
Renato Mefi
4ba3213338
DBZ-1020 Refactor UnwrapFromMongoDbEnvelope logic
...
Now it's easier to follow what makes an insert, update or delete within
the transformer, also reduced the amount of npath complexity by doing
early returns and certain extractions.
2018-12-07 09:16:36 +01:00
Renato Mefi
8d8ef26dae
DBZ-563 MongoDB Unwrapper tests for "delete.handling.mode"
2018-11-30 12:06:18 +01:00
Renato Mefi
fcef819dc8
DBZ-563 MongoDB Unwrapper supports "delete.handling.mode"
2018-11-30 12:06:18 +01:00
Renato Mefi
de7c065b43
DBZ-971 Mongo Unwrap can add data operation header
2018-11-30 12:06:18 +01:00
Renato Mefi
4b0a84f587
DBZ-971 Ensure Mongo unwrap propagates headers
2018-11-30 12:06:18 +01:00
Renato Mefi
312b976c0d
DBZ-563 Test MongoDB Unwrapper tombstone drop
2018-11-30 12:06:18 +01:00
Renato Mefi
0d6236c8d8
DBZ-563 Support tombstone drop on MongoDB Unwrapper
2018-11-30 12:06:18 +01:00
Renato Mefi
16ba3764e5
DBZ-989 Support tombstone on MongoDB unwrapper
2018-11-30 12:06:18 +01:00
Renato Mefi
488dfe4f72
DBZ-989 Provide test for the bug confirmation
2018-11-30 12:06:18 +01:00
Renato Mefi
b2aed1f46e
DBZ-987 Allow to processes full document updates
2018-11-30 12:06:18 +01:00
Renato Mefi
c540391db2
DBZ-987 Test Full Document replacement
...
As documented here:
https://docs.mongodb.com/manual/reference/method/db.collection.update/#replace-a-document-entirely
Mongo supports full document updates, currently the Unwrap expects the
operation to always be a patch update, this test proves the bug to be
patched in the next commits.
2018-11-30 12:06:18 +01:00
Renato Mefi
df738a2abd
DBZ-612 Support MongoDB CDC full $unset operation
...
Mongo internally transforms everything into $set and $unset operations
when they are in the oplog, this guarantees that you can have operations
which are $set, $unset or combined.
2018-11-30 12:06:18 +01:00
Renato Mefi
d44a44aee5
DBZ-612 Test Mongo CDC multiple $set operations
2018-11-30 12:06:18 +01:00
Renato Mefi
7a4e1c48f1
DBZ-612 Transform $unset into null values
2018-11-30 12:06:18 +01:00
Renato Mefi
6e115043ca
DBZ-612 Support mongo NULL type
...
Since we can't trace back the original value since mongo is a
schema-less storage we fallback to string as strategy.
2018-11-30 12:06:18 +01:00
Jenkins user
db1d3a7fb8
[maven-release-plugin] prepare for next development iteration
2018-11-20 16:15:14 +00:00
Jenkins user
f83db82cea
[maven-release-plugin] prepare release v0.9.0.Beta1
2018-11-20 16:15:14 +00:00
Renato Mefi
13dfa8d9bf
DBZ-974 Pair mongo-initiator with Server version
...
Avoiding client and servers mismatches, that's outside of debezium scope
to test whether the user has this sort of version mismatch.
2018-11-16 06:58:09 +01:00
Renato Mefi
beb069fc9d
DBZ-974 Replace Mongo deprecations for count usage
...
As in
http://mongodb.github.io/mongo-java-driver/3.8/javadoc/deprecated-list.html
2018-11-16 06:58:09 +01:00
Renato Mefi
7c59023b9a
Fix documentation for Mongo CDC encoding type
2018-11-12 11:47:18 +01:00
Jenkins user
d952f5dfb0
[maven-release-plugin] prepare for next development iteration
2018-10-04 11:59:14 +00:00
Jenkins user
ff9b70278b
[maven-release-plugin] prepare release v0.9.0.Alpha2
2018-10-04 11:59:13 +00:00
Jiri Pechanec
d933c150ad
DBZ-928 NULL field values are not put into struct
2018-10-04 09:24:04 +02:00
jchipmunk
3ec0b3be81
DBZ-918 Moving connector identifier from SourceInfo to Module class
2018-10-04 09:00:13 +02:00
jchipmunk
05ba99bb0a
DBZ-918 Adding Debezium connector field to source info
...
This will allow consumers to recognize the Debezium connector used for creating a given message, helping them to adjust their behavior for a variety of connectors.
2018-10-04 09:00:13 +02:00
Jiri Pechanec
bf6b2827c8
DBZ-784 Correct condition for stopped running projection
2018-09-18 13:12:21 +02:00
Gunnar Morling
6f0d11cc64
DBZ-784 Changing var name
2018-09-18 09:55:43 +02:00
Jiri Pechanec
a459c2cc4d
DBZ-784 Terminate execute loops during shutdown
2018-09-18 09:54:46 +02:00
jchipmunk
6e9e803e73
DBZ-633 Changes of field.renames format. Misc improvements.
...
* Changing field.renames delimiter from equal to colon character
* Adding examples of source document structure to tests
* Adding tests for case: two renames have the same target field
* Adding missing return
* Changing Path interface to abstract class
2018-09-14 12:54:11 +02:00
Gunnar Morling
aac24b1ed1
DBZ-633 Misc. improvements around FieldSelector;
...
* Adding some comments
* Adding missing return
* More expressive names
* Formatting
2018-09-14 12:54:11 +02:00
jchipmunk
b93551f977
DBZ-881 Adding "field.renames" property for MongoDB connector
...
The "field.renames" configuration property is an optional comma-separated list of the fully-qualified replacements of fields that should be used to rename fields in change event message values. Fully-qualified replacements for fields are of the form "databaseName.collectionName.fieldName.nestedFieldName=newNestedFieldName", where "databaseName" and "collectionName" may contain the wildcard (*) which matches any characters, the equal character (=) is used to determine rename mapping of field.
2018-09-14 12:54:11 +02:00
jchipmunk
b0feaf496a
DBZ-633 Adding "field.blacklist" property for MongoDB connector
...
The "field.blacklist" configuration property is an optional comma-separated list of the fully-qualified names of fields that should be excluded from change event message values. Fully-qualified names for fields are of the form "databaseName.collectionName.fieldName.nestedFieldName", where "databaseName" and "collectionName" may contain the wildcard (*) which matches any characters.
Although the "field.blacklist" configuration property allows you to remove fields from the event values, the "_id" field is always included in the event’s key.
2018-09-14 12:54:11 +02:00
jchipmunk
e9ef4f9068
DBZ-908 Removing obsolete config parameters: DATABASE_LIST_NAME and COLLECTION_LIST_NAME
2018-09-13 12:21:29 +02:00
Gunnar Morling
e1bbbd7992
DBZ-865 DBZ-878 Replacing invalid topic name characters with "_";
2018-09-05 12:56:53 +02:00
Gunnar Morling
72925a3060
DBZ-838 Formatting
2018-07-27 20:33:45 +02:00
jchipmunk
6c18a81faa
DBZ-838 Adding condition to interrupt initial sync if task is stopped
2018-07-26 19:53:27 +03:00
Jenkins user
e00dad127f
[maven-release-plugin] prepare for next development iteration
2018-07-26 08:00:12 +00:00
Jenkins user
16bfd5c700
[maven-release-plugin] prepare release v0.9.0.Alpha1
2018-07-26 08:00:12 +00:00
Gunnar Morling
a5b5e7d9eb
DBZ-627 Using unfified TopicSelector for MongoDB, too
2018-07-18 12:36:07 +02:00
Jenkins user
f9b8d830a8
[maven-release-plugin] prepare for next development iteration
2018-07-11 07:36:30 +00:00
Jenkins user
290ded678f
[maven-release-plugin] prepare release v0.8.0.Final
2018-07-11 07:36:29 +00:00
Jenkins user
033db6659d
[maven-release-plugin] prepare for next development iteration
2018-07-04 07:07:44 +00:00
Jenkins user
696f35f2c0
[maven-release-plugin] prepare release v0.8.0.CR1
2018-07-04 07:07:44 +00:00
Gunnar Morling
e1cf34dc43
DBZ-713 Extracting constaint for expected exception message
2018-07-03 14:41:47 +02:00
Gunnar Morling
c5f9d99644
DBZ-713 Adding overload of execute() that returns a result;
...
This simplifies interaction with execute(), as we don't need to alter collections, references as a side effect but instead can work with the return value.
2018-07-03 14:41:47 +02:00
Gunnar Morling
d6b897f43d
DBZ-713 Applying database and collection filters within MongoPrimary;
...
This protects against authorization failures when listing collections
from DBs the connector user isn't authorized for. It also simplifies
usage of MongoPrimary#databaseNames() and collections() as consumers
don't need to apply filtering themselves.
2018-07-03 14:41:47 +02:00
Gunnar Morling
d72d3fcbc9
DBZ-782 Propagating authorization failures
2018-07-03 14:41:47 +02:00
Jiri Pechanec
d4c9d24b22
DBZ-779 Detect mongo replicator init completion
2018-07-03 12:24:06 +02:00
Gunnar Morling
dcd6b01134
DBZ-644 Simplifying RelationalDatabaseSchema constructor;
...
Also retrieving logical name via CommonConnectorConfig#getLogicalName() in more places
2018-07-03 07:03:59 +02:00
Gunnar Morling
bf7a5018ca
Setting POM version back to 0.8.0-SNAPSHOT
2018-06-22 12:21:03 +02:00
Jenkins user
db42e4657a
[maven-release-plugin] prepare for next development iteration
2018-06-21 14:07:45 +00:00
Jenkins user
c4b8ecaa99
[maven-release-plugin] prepare release v0.8.0.Beta1
2018-06-21 14:07:45 +00:00
Gunnar Morling
d7e196a18e
DBZ-20 Initial import of Oracle connector based on XStream
2018-06-20 13:05:37 +02:00
echo.xu
15f622f823
DBZ-712. ignore initsync offset during connector start.
2018-06-20 10:41:46 +02:00
Gunnar Morling
3f20c8d8cc
DBZ-561 Using less confusing server name in tests
2018-06-11 11:40:15 +02:00
Gunnar Morling
abe7863c28
DBZ-561 More consistent element ordering
2018-06-11 11:40:15 +02:00
Gunnar Morling
bb35d5e4bc
DBZ-561 Adjusting test after rebase
2018-06-11 11:40:15 +02:00
Jiri Pechanec
4698dd2745
DBZ-561 Support nested updates
2018-06-11 11:40:15 +02:00
Sairam Polavarapu
4b67b14304
DBZ-561 Provided option to flatten structs in MongoDB unwrapping SMT.
2018-06-11 11:40:15 +02:00
Jiri Pechanec
32d591d392
DBZ-649 Use Avro-compatible field naming
2018-06-05 10:33:40 +02:00
Jiri Pechanec
d1488a7c48
DBZ-650 Fix test
2018-06-05 10:33:39 +02:00
Jiri Pechanec
3eb034f035
DBZ-649 Encoding arrays as struct
2018-06-05 09:57:19 +02:00
Jiri Pechanec
e00dc3beb4
DBZ-649 Graceful failures for heterogenous arrays
2018-06-05 09:57:19 +02:00
Jiri Pechanec
aa5581b524
DBZ-650 Fix the test breaking the build
2018-06-04 14:58:42 +02:00
Gunnar Morling
d556fca852
DBZ-650 Expanding test
2018-05-30 15:27:30 +02:00
Jiri Pechanec
972068129c
DBZ-650 Add names to unwrapped MongoDB schemas
2018-05-30 15:02:38 +02:00
Jiri Pechanec
6e1e5f40ce
Protect UnwrapFromMongoDbEnvelopeTestIT from inconsistent snapshot
2018-05-15 09:50:54 +02:00
Jiri Pechanec
232e7acde5
DBZ-680 Integration test for MongoDB unwrap SMT
2018-04-13 11:39:34 +02:00
Jiri Pechanec
3e9489741d
DBZ-529 Upgrade to MongoDB 3.6, compatibility testing
2018-03-29 15:47:59 +02:00
Saulius Valatka
31c29459f7
DBZ-672 do not use cursor with no timeout
2018-03-29 14:47:57 +02:00
Saulius Valatka
a78c52cbdd
DBZ-672 stop tailing the oplog when replicator is stopped
2018-03-29 14:47:57 +02:00
Jenkins user
f4e151b23a
[maven-release-plugin] prepare for next development iteration
2018-03-20 08:14:19 +00:00
Jenkins user
93b3252332
[maven-release-plugin] prepare release v0.7.5
2018-03-20 08:14:19 +00:00
Jiri Pechanec
40a527b6ea
DBZ-592 Make snapshot/init-sync false instead of null
2018-03-19 13:19:34 +01:00
Jiri Pechanec
2a35eae178
DBZ-648 InterrupedException handled by MongoDB
2018-03-09 20:43:57 +01:00
Jenkins user
daf27207be
[maven-release-plugin] prepare for next development iteration
2018-03-07 08:31:07 +00:00
Jenkins user
9c73774928
[maven-release-plugin] prepare release v0.7.4
2018-03-07 08:31:07 +00:00
Gunnar Morling
eef1ad7c2a
DBZ-630 Refactoring around MongoDB task context;
...
* Renaming ConnectorTaskContext to CdcSourceTaskContext
* Renaming ReplicationContext to MongoDbTaskContext
* Making relationship from MongoDbTaskContext to ConnectionContext has-a instead of is-a
2018-02-21 12:04:41 +01:00
Gunnar Morling
bb12e521f8
DBZ-630 Pulling up getClock() to ConnectorTaskContext
2018-02-21 12:04:41 +01:00
Gunnar Morling
8264c20cf6
DBZ-630 Unifying common start-up logic across connectors
2018-02-21 12:04:41 +01:00
Gunnar Morling
7fe45db831
DBZ-626 Renaming AvroValidator to SchemaNameAdjuster;
...
Making apparent that this functionality doesn't solely validate but returns a new name actually
2018-02-20 09:26:47 +01:00
Gunnar Morling
69df7c4ade
DBZ-628 Using common properties for MongoDB connector
2018-02-20 06:34:42 +01:00
Jenkins user
6d0cd88e12
[maven-release-plugin] prepare for next development iteration
2018-02-15 04:15:34 +00:00
Jenkins user
7d1e1a989e
[maven-release-plugin] prepare release v0.7.3
2018-02-15 04:15:34 +00:00
Jiri Pechanec
93259e8e11
DBZ-581 Simplified monitor thread shutdown
2018-02-13 14:40:54 +01:00
Jiri Pechanec
dbac1429b2
DBZ-581 Improve error handling for replicators
2018-02-13 14:40:54 +01:00
Gunnar Morling
39e08b7ec6
DBZ-582 Avoiding assignment to test class field from test methods
2018-02-13 10:19:18 +01:00
Gunnar Morling
2e0b9e5e33
DBZ-582 Moving TOMBSTONES_ON_DELETE to CommonsConnectorConfig for re-use
2018-02-13 10:19:18 +01:00
Raf Liwoch
38641f472d
DBZ-582 Allowing users to configure whether tombstone events should be emitted for DELETE operations or not;
...
* Adding new configuration parameter `tombstones.on.delete`
* Using this new option in MongoDB connector
2018-02-13 10:19:18 +01:00
Gunnar Morling
1626f385ad
DBZ-580 Using ChangeEventQueue in MongoDB connector
2018-02-11 10:05:06 +01:00
Jiri Pechanec
b3b379a3a3
DBZ-594 Stop copy threads after initial sync is completed
2018-02-05 12:37:10 +01:00
Gunnar Morling
712824e43f
DBZ-593 Passing version explicitly from SourceInfos to base class instead of relying specific packaging structure
2018-02-02 13:16:43 +01:00
Jiri Pechanec
7ebee94169
DBZ-593 Add Debezium version to source in envelope
2018-02-02 13:16:43 +01:00
Jiri Pechanec
92740a3626
DBZ-587 Fix thread leak, thread names
2018-02-01 10:04:20 +01:00
Jiri Pechanec
9b592204ac
DBZ-587 Centralize and unify thread management
2018-02-01 10:04:20 +01:00
Gunnar Morling
6089d6d051
DBZ-579 Using port 27017 in TestHelper as that's the port used by our Docker set-up
2018-01-29 15:24:16 +01:00
Jiri Pechanec
294a46eeef
DBZ-579 Allow repeated running of MongoDB tests from IDE withou config
2018-01-29 15:24:16 +01:00
Jenkins user
04624341f5
[maven-release-plugin] prepare for next development iteration
2018-01-25 09:39:44 +00:00
Jenkins user
898f6884e1
[maven-release-plugin] prepare release v0.7.2
2018-01-25 09:39:44 +00:00
Gunnar Morling
6c0920c701
DBZ-409 Emitting null value for deletes
2018-01-23 14:41:39 +01:00
Gunnar Morling
707421e8e5
DBZ-409 Adding test for complex id type
2018-01-23 14:41:39 +01:00
Gunnar Morling
9fcf670df1
DBZ-409 Adding proper test for UnwrapFromMongoDbEnvelope;
...
* removing "_id" field
* using more sensible variable names
2018-01-23 14:41:39 +01:00
Gunnar Morling
808da37f48
DBZ-409 Indentation fixes
2018-01-23 14:41:39 +01:00
Gunnar Morling
b7f72bbe3a
DBZ-409 Dependency clean-up
2018-01-23 14:41:39 +01:00
Sairam Polavarapu
e1cfa66862
DBZ-409 Added the testcase for UnwrapFromMongoDbEnvelope and also renamed the class names accordingly.
2018-01-23 14:41:39 +01:00
Gunnar Morling
1cfc8c3596
DBZ-409 Indentation fix
2018-01-23 14:41:39 +01:00
Gunnar Morling
d6dbf02f4c
DBZ-409 Formatting of test and JSON file; removing unused JSON file
2018-01-23 14:41:39 +01:00
Sairam Polavarapu
8f268993ce
DBZ-409 Adding SMT for converting MongoDB CDC messages into traditional flat format
2018-01-23 14:41:39 +01:00
Gunnar Morling
0c4190c493
DBZ-516 Using Duration instead of long in a few more places
2018-01-18 14:13:58 +01:00
Jiri Pechanec
24bdcaf059
DBZ-516 Return control to Connect periodically
2018-01-18 14:13:58 +01:00
Jiri Pechanec
2a7377a833
DBZ-455 Use valueOf instead of constructors
2018-01-05 02:32:37 +01:00
Jenkins user
6bb34b42f9
[maven-release-plugin] prepare for next development iteration
2017-12-20 07:15:12 +00:00
Jenkins user
16dcd4c980
[maven-release-plugin] prepare release v0.7.1
2017-12-20 07:15:12 +00:00
Jenkins user
5e09932cb9
[maven-release-plugin] prepare for next development iteration
2017-12-15 05:10:23 +00:00
Jenkins user
6c1d61e03b
[maven-release-plugin] prepare release v0.7.0
2017-12-15 05:10:23 +00:00
Gunnar Morling
2744f54e9d
DBZ-438 Renaming BufferedBlockingConsumer#flush() to close() to make clear it's onyl meant to be called once at the end
2017-11-13 14:19:39 +01:00
Gunnar Morling
5fbe742be8
DBZ-285 Specifying scope of dependencies in the individual POMs for the sake of comprehensibility
2017-11-10 16:48:32 +01:00
Ben Williams
a3b4fedd5f
DBZ-363 Add support for BIGINT UNSIGNED handling for MySQL
2017-10-18 10:20:03 +02:00
Gunnar Morling
f38bf14402
DBZ-385 Making expected message value more explicit
2017-10-13 18:52:58 +02:00
Jiri Pechanec
b0260b9c17
DBZ-385 MongoDB specific codecs now supported
2017-10-13 18:45:16 +02:00
Jenkins user
75937711fa
[maven-release-plugin] prepare for next development iteration
2017-09-21 04:42:02 +00:00
Jenkins user
a89b9332e4
[maven-release-plugin] prepare release v0.6.0
2017-09-21 04:42:02 +00:00
Gunnar Morling
67398546ef
DBZ-306 Adding some tests for key serialization
2017-09-20 10:56:14 +02:00
hpgrahsl
af2a55cc9b
DBZ-306 make JSON serializer a RecordMakers field to avoid object recreations
...
also fix some checkstyle violations which are not yet reported during build process
see full PR discussion about the rationale behind the taken approach here https://github.com/debezium/debezium/pull/258
2017-09-20 09:36:22 +02:00
hpgrahsl
1d75cbdc4e
DBZ-306 add support for different _id types based on JSON serialised string
...
see full PR discussion about the rationale behind the taken approach here https://github.com/debezium/debezium/pull/258
2017-09-20 09:36:22 +02:00
Jiri Pechanec
74a25eb310
DBZ-343 SSL support for MongoDB
2017-09-13 07:51:16 +02:00
Jenkins user
214696ef0c
[maven-release-plugin] prepare for next development iteration
2017-08-17 11:51:05 +00:00
Jenkins user
c867e6fea6
[maven-release-plugin] prepare release v0.5.2
2017-08-17 11:51:05 +00:00
Gunnar Morling
79fbc028a8
DBZ-311 Precompiling and simplifying some regular expressions
2017-07-25 21:43:26 +02:00
Gunnar Morling
825530256e
DBZ-311 Removing trailing whitespace
2017-07-25 21:43:26 +02:00
Eric Kreiser
80168f74fd
DBZ-302 Remove database.list
configuration since it is unused and is now superseded by database.whitelist
and database.blacklist
2017-07-25 09:50:38 +02:00
Eric Kreiser
3c2d5338bf
DBZ-302 Add support for "database.whitelist" and "database.blacklist" mongodb connector configuration options
2017-07-25 09:50:34 +02:00
Gunnar Morling
a8d1817c22
[maven-release-plugin] prepare for next development iteration
2017-06-09 16:14:31 +00:00
Gunnar Morling
3f512aace7
[maven-release-plugin] prepare release v0.5.1
2017-06-09 16:14:31 +00:00
Randall Hauch
787959c4d0
DBZ-232 Removed the database and table recommenders
...
It’s not clear how valuable these recommenders actually are. First, it’s not clear about the expected semantics: can the user use values that don’t appear in the recommended values? Second, the recommenders that return large numbers of values can be slow and can result in very large REST API responses.
Debezium was using recommenders to return the database and table/collection names, but these lists can be very large for large databases. Rather than cap the number of recommended values and have the recommender return a subset of all potential values, we will instead remove the recommenders altogether.
2017-05-19 09:24:07 +02:00
Randall Hauch
4b2b596241
Merge pull request #210 from WeiBanjo/bugfix/mongodb-new-primary
...
DBZ-214 Fix MongoDB connector resumes on newly elected primary.
2017-04-04 14:13:08 -05:00
Randall Hauch
bcaf1a88b3
DBZ-213 Corrected MongoDB connector build
...
Changed how the mongo-init process waits to begin by now looking for the second MongoDB server log message
saying it is ready, since the MongoDB image now has different startup behavior.
2017-04-04 11:13:25 -05:00
wei
07fbbac960
Check against current primary before exiting
2017-03-28 18:43:12 -07:00
Randall Hauch
709cd8f3fe
[maven-release-plugin] prepare for next development iteration
2017-03-27 11:28:12 -05:00
Randall Hauch
2bc3d45954
[maven-release-plugin] prepare release v0.5.0
2017-03-27 11:28:11 -05:00
Randall Hauch
430d756062
[maven-release-plugin] prepare for next development iteration
2017-03-17 15:41:58 -05:00
Randall Hauch
536cbf6300
[maven-release-plugin] prepare release v0.4.1
2017-03-17 15:41:57 -05:00
Randall Hauch
88b5ffa113
DBZ-201 Corrected handling of ObjectID within non-insert events
...
Corrected how the MongoDB extracts the event key from the update event’s ObjectID, and verified the behavior with an integration test.
2017-03-17 15:02:29 -05:00
Randall Hauch
8c60c29883
[maven-release-plugin] prepare for next development iteration
2017-02-07 14:22:12 -06:00
Randall Hauch
20134286e9
[maven-release-plugin] prepare release v0.4.0
2017-02-07 14:22:11 -06:00
Randall Hauch
0c17e1f972
DBZ-182 Restart MongDB initial sync if necessary
...
Corrected the MongoDB connector upon startup to restart an initial sync if the previously recorded offset signals that an initial sync was not completed in the prior run.
Also change the connector’s replicator to buffer the last record during an initial sync so that, upon completion of the initial sync, the last record can be updated with an offset that reflects that the initial sync was completed. This way, if the initial sync is completed but there are no other events in the oplog, the connector will still consider the initial sync as completed.
2017-02-02 15:43:18 -06:00
Horia Chiorean
a300d3e1cf
DBZ-3 Changes the configuration of the Docker Maven plugin to only use alias naming when necessary and moves the PG connector ahead of the Mongo connector in the build
2016-12-27 14:44:33 +02:00
Randall Hauch
928db59807
DBZ-170 Changed the MongoDB connector’s connection logic
2016-12-21 17:06:58 -06:00
Randall Hauch
443edc358d
DBZ-170 Changed the MongoDB connector’s connection logic
...
This change alters the way the MongoDB connects to the various servers in a cluster. Previously, the ConnectionContext constructor currently set up the MongoDB client with credentials for the `admin` and `config` databases, and apparently the client eagerly performs authentication against all databases passed in, rather than doing this lazily as DBs are use.
Instead, the code no longer sets up the credentials for the `config` database and instead only sets up credentials for the `admin` database for authentication and authorization. This works as long as the user specified in the connector configuration can read the `config` database.
Several other changes were made to improve the error handling and reporting when the replica set information cannot be read from the `config` database.
2016-12-21 14:27:42 -06:00
Randall Hauch
c1a26ee261
DBZ-167 Corrected the MongoUtil class
...
Corrected the `onCollection` utility method in the `MongoUtil` class to pass in the collection name rather than the database name.
2016-12-20 16:24:07 -06:00
Randall Hauch
63e2e48ab3
DBZ-117 Improved logging when MongoDB connector is reading oplog
...
The MongoDB connector now outputs an INFO log message whenever its task's `poll()` method returns a non-empty list of `SourceRecord` objects, where the message includes the number of records and the offset of the last record.
2016-09-06 11:32:04 -05:00
Horia Chiorean
2732d26ff0
Fixes the DBZ header required by checkstyle
...
This commit removes an extra space character from the first blank line of the header
2016-08-24 13:41:15 +03:00
Randall Hauch
e86fb83459
[maven-release-plugin] prepare for next development iteration
2016-08-16 09:56:47 -05:00
Randall Hauch
ccdb0a1a63
[maven-release-plugin] prepare release v0.3.0
2016-08-16 09:56:47 -05:00
Randall Hauch
918a523f12
DBZ-100 Changed the MongoDB connector to use a new JSON semantic type
...
Added a semantic type for JSON strings, and used it in the MongoDB connector.
2016-08-15 12:11:35 -05:00
Randall Hauch
8cb39eacf0
Reverted back to 0.3.0-SNAPSHOT, since the 0.3 candidate release was not acceptable.
2016-08-01 12:25:58 -05:00
Randall Hauch
517272278d
[maven-release-plugin] prepare for next development iteration
2016-07-25 17:50:31 -05:00
Randall Hauch
b89296e646
[maven-release-plugin] prepare release v0.3.0
2016-07-25 17:50:31 -05:00
Randall Hauch
447acb797d
DBZ-62 Upgraded to Kafka and Kafka Connect 0.10.0.0
...
Upgraded from Kafka 0.9.0.1 to Kafka 0.10.0. The only required change was to override the `Connector.config()` method, which returns `null` or a `ConfigDef` instance that contains detailed metadata for each of the configuration fields, including supporting recommended values and marking fields as not visible (e.g., if they don't make sense given other configuration field values). This can be used by user interfaces to data-drive the configuration of a connector. Also, the default validation logic of the Connector implementations uses a `Validator` that is pretty restrictive in its functionality.
Debezium already had a fairly decent and simple `Configuration` framework. After several attempts to try and merge these concepts, reconciling the two validation mechanisms was very complicated and involved a lot of changes. It was easier to simply continue Debezium-specific validation and to override the `Connector.validate(...)` method to use Debezium's `Configuration`-based validation. Connector-based validation logic includes determining recommended values, so Debezium's `Field` class (used to define each configuration property) was enhanced with a new `Recommender` class that is similar to Kafka's.
Additional integration tests were added to verify that the `ConfigDef` result is acceptable and that the new connector validation logic works as expected, including getting recommended values for some fields (e.g., database names, table/collection names) from MySQL and MongoDB by connecting and dynamically reading the values. This was done in a way that remains backward compatible with the regular expression formats of these fields, but in a user interface that uses the `ConfigDef` mechanism the user can simply select the databases and table/collection identifiers.
2016-07-25 14:21:31 -05:00
Randall Hauch
bf6f9a8243
DBZ-86 Cleaned up unnecessary printing in MongoDB connector
2016-07-15 16:37:12 -05:00
Randall Hauch
a88bcb9ae7
DBZ-86 Generated Kafka Schema names will now also be valid Avro fullnames
2016-07-15 16:29:52 -05:00
Randall Hauch
12e7cfb8d3
DBZ-2 Created initial Maven module with a MongoDB connector
...
Added a new `debezium-connector-mongodb` module that defines a MongoDB connector. The MongoDB connector can capture and record the changes within a MongoDB replica set, or when seeded with addresses of the configuration server of a MongoDB sharded cluster, the connector captures the changes from the each replica set used as a shard. In the latter case, the connector even discovers the addition of or removal of shards.
The connector monitors each replica set using multiple tasks and, if needed, separate threads within each task. When a replica set is being monitored for the first time, the connector will perform an "initial sync" of that replica set's databases and collections. Once the initial sync has completed, the connector will then begin tailing the oplog of the replica set, starting at the exact point in time at which it started the initial sync. This equivalent to how MongoDB replication works.
The connector always uses the replica set's primary node to tail the oplog. If the replica set undergoes an election and different node becomes primary, the connector will immediately stop tailing the oplog, connect to the new primary, and start tailing the oplog using the new primary node. Likewise, if connector experiences any problems communicating with the replica set members, it will try to reconnect (using exponential backoff so as to not overwhelm the replica set) and continue tailing the oplog from where it last left off. In this way the connector is able to dynamically adjust to changes in replica set membership and to automatically handle communication failures.
The MongoDB oplog contains limited information, and in particular the events describing updates and deletes do not actually have the before or after state of the documents. Instead, the oplog events are all idempotent, so updates contain the effective changes that were made during an update, and deletes merely contain the deleted document identifier. Consequently, the connector is limited in the information it includes in its output events. Create and read events do contain the initial state, but the update contain only the changes (rather than the before and/or after states of the document) and delete events do not have the before state of the deleted document. All connector events, however, do contain the local system timestamp at which the event was processed and _source_ information detailing the origins of the event, including the replica set name, the MongoDB transaction timestamp of the event, and the transactions identifier among other things.
It is possible for MongoDB to lose commits in specific failure situations. For exmaple, if the primary applies a change and records it in its oplog before it then crashes unexpectedly, the secondary nodes may not have had a chance to read those changes from the primary's oplog before the primary crashed. If one such secondary is then elected as primary, it's oplog is missing the last changes that the old primary had recorded and no longer has those changes. In these cases where MongoDB loses changes recorded in a primary's oplog, it is possible that the MongoDB connector may or may not capture these lost changes.
2016-07-14 13:02:36 -05:00