alexey.rassokhin/tet123
1
0
Fork 0
Code Issues Pull Requests 1 Packages Projects Releases Wiki Activity
10,828 Commits 3 Branches 0 Tags 51 MiB
ff09e45d43
Commit Graph

1565 Commits

Author SHA1 Message Date
Gunnar Morling
7e664b18e5 DBZ-3596 Removing unused field AbstractMessageDecoder#filterBasedOnLsn and code leading to it 2021-06-09 13:41:43 +02:00
Jiri Pechanec
b59f6daac9 DBZ-3550 Use selected test containers deps 2021-06-08 14:49:04 +02:00
Gunnar Morling
41e93e6ce1 DBZ-2790 Avoiding function usage 2021-06-08 12:10:26 +02:00
Eric Slep
8a659ff8ab DBZ-2790 Docs and one final test edge case 2021-06-08 12:10:26 +02:00
Eric Slep
5dde3062b0 DBZ-2790 Handle pgoutput streaming refresh 2021-06-08 12:10:26 +02:00
Eric Slep
13ebeade71 DBZ-2790 Add default values to existing tests 2021-06-08 12:10:26 +02:00
Eric Slep
436b4697ba DBZ-2790 Tests for refresh behaviour, ensure defaults are not cleared on in-mem refresh 2021-06-08 12:10:26 +02:00
Eric Slep
d1917eeff6 DBZ-2790 Handle default values for date & time types 2021-06-08 12:10:26 +02:00
Eric Slep
f88f135c43 DBZ-2790 Add simple placeholders to schema for generated defaults 2021-06-08 12:10:26 +02:00
Eric Slep
de5fe2a520 DBZ-2790 Move TypeRegistry construction back to connection 2021-06-08 12:10:26 +02:00
Kevin Pullin
0462804467 DBZ-2790 Update debezium-connector-postgres/src/main/java/io/debezium/connector/postgresql/connection/PostgresDefaultValueConverter.java 
Co-authored-by: Gunnar Morling <gunnar.morling@googlemail.com>
 2021-06-08 12:10:26 +02:00
Kevin Pullin
4b62e59400 DBZ-2790 Update debezium-connector-postgres/src/main/java/io/debezium/connector/postgresql/connection/PostgresDefaultValueConverter.java 
Co-authored-by: Gunnar Morling <gunnar.morling@googlemail.com>
 2021-06-08 12:10:26 +02:00
Kevin Pullin
d2ab8baddf DBZ-2790 Implement default values for postgres 
Using the `SqlServerDefaultValueConverter` class as a starting point,
this commit implements default value support for postgres.
 2021-06-08 12:10:26 +02:00
gvaquez-ubi
03cb907d85 DBZ-3068 Allow several tries on createReplicationStream to get replication slot 2021-06-04 15:11:57 +02:00
Debezium Builder
ead4ee7226 [maven-release-plugin] prepare for next development iteration 2021-05-20 19:07:44 +02:00
Debezium Builder
9e16df7d4b [maven-release-plugin] prepare release v1.6.0.Beta1 2021-05-20 19:07:44 +02:00
Jiri Pechanec
a879b7b3e1 DBZ-3500 Re-use ResultSet to Object code 2021-05-20 08:06:37 +02:00
Jiri Pechanec
a626dc149f DBZ-3500 Provide common test for incremental snapshot testing 2021-05-20 08:06:37 +02:00
Gunnar Morling
a362760429 DBZ-2337 Removing superfluous log statement 2021-05-18 21:47:16 +02:00
Jiri Pechanec
5fbc181b5e DBZ-2337 All snapshots using exported mode 2021-05-18 21:44:08 +02:00
Jiri Pechanec
2a716d1b86 DBZ-3519 Avoid NPE in autoboxing 2021-05-18 10:32:18 +02:00
Debezium Builder
4dd3a146ca [maven-release-plugin] prepare for next development iteration 2021-05-06 02:40:15 -04:00
Debezium Builder
b3c5bb281c [maven-release-plugin] prepare release v1.6.0.Alpha1 2021-05-06 02:40:15 -04:00
Jiri Pechanec
5c722341e2 DBZ-3473 Extract incremental snapshot contract 2021-05-06 05:37:36 +02:00
Jiri Pechanec
7daae11600 DBZ-3473 Postgres has its own schema instance 2021-05-06 05:37:36 +02:00
Jiri Pechanec
39d21806eb DBZ-3473 Metrics enabled 2021-05-06 05:37:36 +02:00
Jiri Pechanec
c53785dde7 DBZ-3473 Incremental Snapshot support for MySQL 2021-05-06 05:37:36 +02:00
Jiri Pechanec
4e95056ddd DBZ-3473 Test stabilization 2021-05-06 05:37:36 +02:00
Jiri Pechanec
49c5a00bbf DBZ-3473 Share read JDBC value between regular and incremental snapshot 2021-05-06 05:37:36 +02:00
Jiri Pechanec
dd2bc6354b DBZ-3473 Pass the main clock form connector 2021-05-06 05:37:36 +02:00
Jiri Pechanec
260c32cf0e DBZ-3473 Resume parallel snapshot after connector restart 2021-05-06 05:37:36 +02:00
Jiri Pechanec
37626c0dfe DBZ-3473 Extract incremental snapshot context 2021-05-06 05:37:36 +02:00
Jiri Pechanec
07371c085b DBZ-3473 Watermark signals must be in separate tx 2021-05-06 05:37:36 +02:00
Jiri Pechanec
a52c536a68 DBZ-3473 First prototype of incremental snapshotting 2021-05-06 05:37:36 +02:00
Gunnar Morling
dc02b896cd DBZ-3449 Moving QA releated plug-ins to separate profile, so to avoid loading them at all 2021-04-28 15:14:05 +02:00
Gunnar Morling
2088c035b1 DBZ-3449 Adding "quick" profile 2021-04-26 06:09:14 +02:00
Hossein Torabi
5d6705c3fe DBZ-2699 implement skipped.operation in core 2021-04-08 11:59:08 +02:00
Debezium Builder
8a88ba9b45 [maven-release-plugin] prepare for next development iteration 2021-04-08 04:04:00 -04:00
Debezium Builder
74152afe74 [maven-release-plugin] prepare release v1.5.0.Final 2021-04-08 04:04:00 -04:00
Gunnar Morling
5425e6c5dc DBZ-3343 Re-using single shared SqlTypeMapper instance 2021-04-06 14:59:55 +02:00
JLDLaughlin
06b0475f17 DBZ-2911 Add additional LSN information to all sources 
Adds a new "sequence" field to all sources via the AbstractSourceInfo
class. Specifically, this field is required to deduplicate records
from PostgreSQL sources in O(1) time. The sequence field is a stringified
list of metadata. For PostgreSQL sources, this will include the last
committed LSN and the current LSN. For other sources, this will be NULL
until implemented. A new integration test was added for PostgreSQL.
 2021-03-26 09:19:17 +01:00
Debezium Builder
6b4d88bb51 [maven-release-plugin] prepare for next development iteration 2021-03-24 06:35:02 -04:00
Debezium Builder
c578715233 [maven-release-plugin] prepare release v1.5.0.CR1 2021-03-24 06:35:02 -04:00
Jiri Pechanec
495be2659b DBZ-2511 Dedicated logger for schema name adjuster 2021-03-18 14:42:34 +01:00
Jiri Pechanec
21f522217c DBZ-3033 Map OID to long value 2021-03-17 18:11:52 +01:00
Gunnar Morling
911b5e9a3b DBZ-3271 Unifying filter handling across relational connectors 2021-03-17 09:54:25 +01:00
Debezium Builder
f57057421b [maven-release-plugin] prepare for next development iteration 2021-03-12 04:38:17 -05:00
Debezium Builder
b57293013a [maven-release-plugin] prepare release v1.5.0.Beta2 2021-03-12 04:38:16 -05:00
Jiri Pechanec
123d9ef4eb DBZ-3213 Refactor Maven structure to deploy support modules 2021-03-11 16:14:15 +01:00
Jiri Pechanec
f23421920b DBZ-2919 Remove race condition 2021-03-10 19:12:34 +01:00
Jiri Pechanec
15c7b11a1f DBZ-3141 Change schema signal tested for Postgres 2021-03-10 19:12:34 +01:00
Jiri Pechanec
ff7cb44801 DBZ-3141 Offsets are propagated to signal actions 2021-03-10 19:12:34 +01:00
Jiri Pechanec
7fdad2ed49 DBZ-3141 Support for signalling table 2021-03-10 19:12:34 +01:00
Jiri Pechanec
9f2cffa21e DBZ-2885 Fix test 2021-03-09 17:32:01 +01:00
Anisha Mohanty
895d704d82 DBZ-2885 Handle misleading error message for filtered publication 2021-03-09 08:39:15 +01:00
Chris Cranford
04849734b3 DBZ-3030 Do not execute initial statements on connector validation 2021-03-05 11:59:22 +01:00
Debezium Builder
b4c260da50 [maven-release-plugin] prepare for next development iteration 2021-02-23 14:04:01 -05:00
Debezium Builder
123a711a6d [maven-release-plugin] prepare release v1.5.0.Beta1 2021-02-23 14:04:01 -05:00
Fahim Farook
194258d87b
DBZ-3164 Log database connection validation failures consistently 2021-02-22 14:52:22 +01:00
Gunnar Morling
62ad3969c1 DBZ-3107 Typo fixes 2021-02-22 09:08:47 +01:00
Jiri Pechanec
a97b86db43 DBZ-3107 Workaround for DBZ-3158 2021-02-22 09:08:47 +01:00
Jiri Pechanec
1edad510ff DBZ-2614 Support for infinite timestamps 2021-02-19 10:44:06 +01:00
Jiri Pechanec
1f79c15a48 DBZ-2616 Fix negative years conversion 2021-02-19 10:44:06 +01:00
Jiri Pechanec
a17e7cee44 DBZ-2865 Regression in warning message cleanup 2021-02-19 10:40:15 +01:00
Yilong Chang
a0b3704832 DBZ-3056 Validate all fields in subclasses. 2021-02-16 12:34:06 +01:00
Yilong Chang
76c7e0d426 DBZ-3056 Extract basic valdiation in parent class. 2021-02-16 12:34:06 +01:00
ani-sha
5f8a22a5df DBZ-2858 Make field descriptions consistent for time values 2021-02-16 09:55:35 +01:00
Kevin Pullin
db5dd54c20 DBZ-3074 Split shouldReceiveChangesForInsertsIndependentOfReplicaIdentity 
Split shouldReceiveChangesForInsertsIndependentOfReplicaIdentity into two tests,
one that works with the PGOUTPUT decoder and one that works with the other decoders
(due to differences between schema update handling).
 2021-02-10 10:58:26 +01:00
Kevin Pullin
2426de849a DBZ-3074 PgOutputMessageDecoder - Fix unordered keys 
Fix issue where `PgOutputMessageDecoder` does not define primary key metadata in key-order.

`PgOutputMessageDecoder.getTablePrimaryKeyColumnNamesFromDatabase()` returns a `Set`, leading to unordered
primary keys in the metadata. Switch the logic to call `JdbcConnection.readPrimaryKeyNames()` which
_does_ correctly return an ordered primary key list.
 2021-02-10 10:58:26 +01:00
Debezium Builder
e08df0a177 [maven-release-plugin] prepare for next development iteration 2021-02-06 01:25:23 -05:00
Debezium Builder
da0411bf70 [maven-release-plugin] prepare release v1.5.0.Alpha1 2021-02-06 01:25:23 -05:00
Naveen Kumar KR
75882f35ac DBZ-2382 Support emitting TRUNCATE events in PG11+ PGOUTPUT plugin 
default - TruncateHandlingMode.SKIP
supported configs - TruncateHandlingMode.SKIP / TruncateHandlingMode.INCLUDE
 2021-02-03 13:53:01 +01:00
ani-sha
d4003c9172 DBZ-2420 Define database configs under RelationalDatabaseConnectorConfig 2021-02-03 08:05:42 +01:00
Debezium Builder
237b4ac0f1 [maven-release-plugin] prepare for next development iteration 2021-01-28 07:38:55 -05:00
Debezium Builder
06b1d0cbad [maven-release-plugin] prepare release v1.4.1.Final 2021-01-28 07:38:55 -05:00
rkerner
6aa13e756a DBZ-2952 Remove Postgres replication slot validation (maybe part 1 to fix user's issues) 2021-01-27 10:31:00 +01:00
Gunnar Morling
aa674842a3 DBZ-2923 Improving test stability 2021-01-14 17:02:04 +01:00
Gunnar Morling
b5b8ef358b DBZ-2764 Misc. clean-up 2021-01-08 13:19:57 +01:00
ani-sha
bd427786e9 DBZ-2764 Centralize Postgres image for Testcontainers 2021-01-08 13:19:57 +01:00
Debezium Builder
3010f99c32 [maven-release-plugin] prepare for next development iteration 2021-01-07 03:38:13 -05:00
Debezium Builder
54534538d9 [maven-release-plugin] prepare release v1.4.0.Final 2021-01-07 03:38:13 -05:00
rkerner
752454e83d DBZ-2800 fix AWS user roles validation 2021-01-07 07:22:37 +01:00
Debezium Builder
b1ab996c3e [maven-release-plugin] prepare for next development iteration 2020-12-16 14:11:31 -05:00
Debezium Builder
8fc864b278 [maven-release-plugin] prepare release v1.4.0.CR1 2020-12-16 14:11:31 -05:00
Debezium Builder
cf921f56b4 [maven-release-plugin] prepare for next development iteration 2020-12-10 00:27:40 -05:00
Debezium Builder
c49a377487 [maven-release-plugin] prepare release v1.4.0.Beta1 2020-12-10 00:27:39 -05:00
Jiri Pechanec
d1a7aae736 DBZ-2772 wal2json needs db write to complete catch-up phase 2020-11-30 10:06:43 +01:00
Debezium Builder
4535549dcd [maven-release-plugin] prepare for next development iteration 2020-11-16 15:18:43 -05:00
Debezium Builder
3c70c5f7d5 [maven-release-plugin] prepare release v1.4.0.Alpha2 2020-11-16 15:18:43 -05:00
Jiri Pechanec
af91d7d2bf DBZ-2662 Increase wait time for metrics 2020-11-16 09:44:52 +01:00
vanhoale
17a424d9ac DBZ-2662 Handle ChangeEventQueue by the size in bytes 2020-11-16 08:39:33 +01:00
rkerner
4faf13d0c9 DBZ-2637 apply feedback 2020-11-16 07:47:25 +01:00
rkerner
aff1df7573 DBZ-2637 fix unit test timeouts 2020-11-16 07:47:25 +01:00
rkerner
77eee964a1 DBZ-2637 unrelated cleanup 2020-11-16 07:47:25 +01:00
rkerner
c59e00c65a DBZ-2637 Enhance Postgres validation checks 
+ add wal_level config check
+ add LOGIN and REPLICATION roles check
+ add replication slot is already in use check
 2020-11-16 07:47:25 +01:00
rkerner
f291b07b6b DBZ-2724 cleanup code and apply PR feedback 2020-11-10 10:44:41 +01:00
rkerner
890bf217dc DBZ-2724 unrelated cleanup 2020-11-10 10:44:41 +01:00
rkerner
0015d628d5 DBZ-2724 Add integration tests for DBZ-2617 (fix Postgres was not able to shutdown with shutdown mode "fast") 
DBZ-2602 Migrate Testcontainers related changes on DebeziumContainer from UI PoC backend to Debezium main repo
 2020-11-10 10:44:41 +01:00
Chris Cranford
4e6b3b1268 DBZ-2704 Fix test failure - PostgresConnectorIT#shouldRegularlyFlushLsnWithTxMonitoring 2020-11-04 06:15:16 +01:00
rkerner
c274c2b39b DBZ-2617 refactor to proper HeartbeatErrorHandler injection 2020-11-03 07:23:02 +01:00
cooksey
c262238183 DBZ-2608 Add a snapshotComplete lifecycle hook to the Snapshotter 2020-10-30 13:34:38 -04:00
Debezium Builder
26883dad92 [maven-release-plugin] prepare for next development iteration 2020-10-22 03:45:59 -04:00
Debezium Builder
9adbc2d67e [maven-release-plugin] prepare release v1.4.0.Alpha1 2020-10-22 03:45:59 -04:00
Jiri Pechanec
ee5353ef6f DBZ-2653 Process commits even with tx monitoring 2020-10-20 10:42:10 +02:00
Kaushik Iyer
7d9e8fdc46 DBZ-2456 fixes after rebase 2020-10-16 10:13:48 +02:00
Kaushik Iyer
e2e6dc0932 DBZ-2456 selective snapshot implementation 2020-10-16 10:13:48 +02:00
Chris Cranford
17039df7ce DBZ-2344 Fix test failure - RecordsStreamProducerIT - shouldReceiveHeartbeatAlsoWhenChangingNonWhitelistedTable 2020-10-06 09:07:42 +02:00
Debezium Builder
95b8ed3877 [maven-release-plugin] prepare for next development iteration 2020-10-01 09:08:51 -04:00
Debezium Builder
19ad73beb2 [maven-release-plugin] prepare release v1.3.0.Final 2020-10-01 09:08:50 -04:00
Jiri Pechanec
12be0d8428 DBZ-2565 Disable test failing for snapshots 2020-10-01 14:27:13 +02:00
Jiri Pechanec
45d683724f DBZ-2565 Fail on unsupported infinite timestamps 2020-10-01 14:27:13 +02:00
Chris Cranford
4fe6813179 DBZ-2435 - Fix test failure - ReplicationConnectionIT#shouldResumeFromLastReceivedLSN 2020-09-30 20:19:50 +02:00
Chris Cranford
155dd44dab DBZ-2544 Fix formatting 2020-09-30 19:42:10 +02:00
Chris Cranford
1fd6fbb1fa DBZ-2544 Fix test failure - PostgresConnectorIT#customSnapshotterSkipsTablesOnRestart 2020-09-30 19:42:10 +02:00
Chris Cranford
bb0b3fd158 DBZ-2468 Fix test failure - PostgresConnectorIT#shouldExecuteOnConnectStatements 2020-09-24 23:06:55 +02:00
Debezium Builder
b146b69154 [maven-release-plugin] prepare for next development iteration 2020-09-24 04:56:42 -04:00
Debezium Builder
c51cfe8eb9 [maven-release-plugin] prepare release v1.3.0.CR1 2020-09-24 04:56:41 -04:00
cooksey
ba074abe4c DBZ-2550 Prevent duplicate events using catch up streaming 
Normally when a connector gracefully shuts down, the connect framework attempts to commit offsets so the latest committed state gets acked on the replication stream.  While the connector is running, the framework periodically commits offsets. Debezium does not manage triggering an offset commit.  When the catch up streaming phase ends, there may be uncommitted state and the connector is unable to determine when the next commit will occur because the commit timing is externally managed. If a commit is not triggered between the end of the catch up streaming phase and the normal streaming phase after the snapshot, the connector may produce some duplicated messages.

Although the replication stream may be out of date, the internal OffsetContext is aware of the latest committed offset. When the snapshot phase recreates a new offset after catch up streaming, the previous offset has access to the latest state. Use the previous offset to forward known state to the new offset.
 2020-09-21 09:28:15 +02:00
Debezium Builder
837669890a [maven-release-plugin] prepare for next development iteration 2020-09-16 12:18:08 -04:00
Debezium Builder
086dbd960b [maven-release-plugin] prepare release v1.3.0.Beta2 2020-09-16 12:18:08 -04:00
Jiri Pechanec
eed321d485
DBZ-2461 LSN must not be flushed after connection close 2020-09-16 11:40:52 +02:00
Chris Cranford
e5c01c6b09 DBZ-137 Fix compilation problems 2020-09-14 12:33:43 +02:00
AndreyIg
77662a0dde DBZ-137 contribution to the core, Oracle related 2020-09-14 12:33:43 +02:00
cooksey
8bcbdb639d DBZ-2094 Allow Postgres snapshotter to set streaming start on resume 
When a connector resumes after previously streaming and takes a
snapshot, through a new method on the snapshotter interface,
shouldStreamEventsStartingFromSnapshot, can choose whether
to resume streaming from the last known streaming position or the
beginning of the snapshot. This is helpful for snapshotters that
may not want to resnapshot every table in the whitelist/blacklist
but not miss event on the tables that are skipped.
 2020-09-09 10:54:11 +02:00
Debezium Builder
9979db648f [maven-release-plugin] prepare for next development iteration 2020-08-28 05:04:26 -04:00
Debezium Builder
5c58b23756 [maven-release-plugin] prepare release v1.3.0.Beta1 2020-08-28 05:04:26 -04:00
Jiri Pechanec
03a11efd8e DBZ-2338 Atomic is not necessary 2020-08-24 21:16:37 +02:00
Jiri Pechanec
cf0db85436 DBZ-2338 Flush LSNs on tx boundaries 2020-08-24 21:16:37 +02:00
Jiri Pechanec
483a483569 DBZ-2338 Log exception; restart keepalive 2020-08-24 21:16:37 +02:00
Jiri Pechanec
f01ab234a4 DBZ-2338 Skip initial statements on reconnect 2020-08-24 21:16:37 +02:00
Jiri Pechanec
4d0ec1f848 DBZ-2397 Test for restart in tx 2020-08-24 21:16:37 +02:00
Jiri Pechanec
aead9a6612 DBZ-2338 Minor corrections 
Co-authored-by: Gunnar Morling <gunnar.morling@googlemail.com>
 2020-08-24 21:16:37 +02:00
Jiri Pechanec
b5c2ed17ed DBZ-2338 Support for pgoutput 2020-08-24 21:16:37 +02:00
Jiri Pechanec
39623aa799 DBZ-2338 Base WAL position detection on commit LSNs 2020-08-24 21:16:37 +02:00
Jiri Pechanec
31721c247b DBZ-2200 Introduce LLSN domain type 2020-08-24 21:16:37 +02:00
rkerner
2c5c093587 DBZ-2171 avoid divisive language 2020-08-21 09:29:04 +02:00
Gunnar Morling
e89b65238f DBZ-2423 Increasing visibility of some methods for UI PoC usage 2020-08-12 13:50:42 +02:00
Gunnar Morling
44efd1bc6b DBZ-2423 Propagating port default values 2020-08-12 13:50:42 +02:00
Gunnar Morling
26b51e02cb DBZ-2418 Misc. clean-up 2020-08-12 09:28:35 +02:00
Gunnar Morling
ebe5169a66 DBZ-2418 Delaying retrieval of DB charset; 
This is called only once anyways, so there's no need for storing the info
in a field in PostgresConnection. This allows PostgresConnector#validate()
bring up a more meaningful error message in case of incorrect credentials
and other incorrect connection configuration.
 2020-08-12 09:28:35 +02:00
Debezium Builder
c4c50f5fc6 [maven-release-plugin] prepare for next development iteration 2020-08-06 00:46:18 -04:00
Debezium Builder
94cbeb95fa [maven-release-plugin] prepare release v1.3.0.Alpha1 2020-08-06 00:46:17 -04:00
rkerner
cad6cf62d3 DBZ-2315 fix BIT VARYING handling in Postgres connector 
* removed little endian padding for BIT types in JdbcValueConverters (only used for Postgres yet)
* removed legacy format handling fot BIT related types in JdbcValueConverters#convertBits which were leftovers from PR #1408
* removed unnecessary zero-ing of a newly created byte array to improve performance for huge byte arrays
* updated Postgres connector docs
 2020-08-03 14:58:31 +02:00
Björn Häuser
04254e3fe7 DBZ-2310: Return true when messages are skipped 
See https://issues.redhat.com/browse/DBZ-2310 for more explanation.

Short summary: Whenever LSNs are skipped return back that messages have
been processed, which will cause two things:

* The replication can be advanced properly
* The connector does not fallback to the poll timeout waiting interval

I also think its more correct to say, "yes I have read messages, but
still skipped all of them".
 2020-07-21 15:53:06 +02:00
Debezium Builder
e1ab38de84 [maven-release-plugin] prepare for next development iteration 2020-07-16 04:40:04 -04:00
Debezium Builder
69cf375faa [maven-release-plugin] prepare release v1.2.1.Final 2020-07-16 04:40:04 -04:00
Jiri Pechanec
f06a14f1da DBZ-2288 Rethrow exception in test 2020-07-16 09:51:00 +02:00
Jiri Pechanec
55ba8b8b89 DBZ-2288 Fix copy/paste errors 2020-07-16 09:51:00 +02:00
Jiri Pechanec
ea942a4268 DBZ-2288 Test for pgoutput 2020-07-16 09:51:00 +02:00
Jiri Pechanec
8b0b252eb9 DBZ-2288 Events in exported snapshot no longer filtered by LSN 2020-07-16 09:51:00 +02:00
Jos Huiting
da97be7842 DBZ-2307 - Improve logging when primary keys are missing 2020-07-15 11:18:48 -04:00
Chris Cranford
94f2932e95 DBZ-2271 Fix test failure - RecordsStreamProducerIT#testEmptyChangesProducesHeartbeat 
This fixes a race condition with heartbeat emitted messages with empty DDLs
 2020-07-14 17:25:44 +02:00
Chris Cranford
0e5122c69d DBZ-2319 Fix test failure - PostgresConnectorIT#shouldOutputRecordsInCloudEventsFormat 2020-07-09 08:22:13 +02:00
Chris Cranford
4c6351e5e3 DBZ-2271 Fix test failure - RecordsStreamProducerIT#testEmptyChangesProducesHeartbeat 2020-07-08 17:23:58 +02:00
Debezium Builder
c0d30b87d9 [maven-release-plugin] prepare for next development iteration 2020-06-24 18:03:11 +02:00
Debezium Builder
0e2ac78301 [maven-release-plugin] prepare release v1.2.0.Final 2020-06-24 18:03:11 +02:00
Chris Cranford
41bfac49a3 DBZ-2230 Fix intermittent CI test failure - Postgres 2020-06-22 18:27:43 +02:00
Debezium Builder
b03a4d0e50 [maven-release-plugin] prepare for next development iteration 2020-06-19 00:39:01 -04:00
Debezium Builder
93e60258d5 [maven-release-plugin] prepare release v1.2.0.CR2 2020-06-19 00:39:00 -04:00
Gunnar Morling
f3b07a1374 DBZ-1916 Formatting 2020-06-16 13:28:49 +02:00
Chris Cranford
64bff85ad2 DBZ-1916 Add tests for datatype.propagate.source.type 2020-06-16 13:28:49 +02:00
cooksey
1282403c44 DBZ-2176 Open transaction test are more reliable 
Checking pg_stat_activity for transactions with a state of "idle in transaction"
should be more reliable to find open abandoned transaction than looking for
a non-null backend_xmin. For example, an autovaccum process will have a
backend_xmin but will not have a stat of "idle in transaction"
 2020-06-15 11:15:51 +02:00
Debezium Builder
65c39e9f11 [maven-release-plugin] prepare for next development iteration 2020-06-11 02:00:20 -04:00
Debezium Builder
bce332c900 [maven-release-plugin] prepare release v1.2.0.CR1 2020-06-11 02:00:19 -04:00
Lukas Krejci
a4689611f5 DBZ-2159 Fix a typo in the package name. 2020-06-09 19:28:33 +02:00
Lukas Krejci
90c637b1e3 DBZ-2159 Include the Postgres SPI in the API checks. 2020-06-09 19:28:33 +02:00
Gunnar Morling
8cac498a60 DBZ-2156 More meaningful exception in case of replication slot conflict 2020-06-08 10:28:08 +02:00
Jiri Pechanec
13419ae74a DBZ-2118 Document type registry connection capability 2020-06-08 09:22:05 +02:00
Jiri Pechanec
d2ef7b7011 DBZ-2118 Slot dropping more resilient to race 2020-06-08 09:22:05 +02:00
Jiri Pechanec
642f4e7303 DBZ-2118 Switch to manual tx management 2020-06-08 09:22:05 +02:00
Jiri Pechanec
8c378400eb DBZ-2118 Type registry provided only for dedicated connections 2020-06-08 09:22:05 +02:00
Jiri Pechanec
853ebafd7f DBZ-2118 Close TX after type query 2020-06-08 09:22:05 +02:00
Chris Cranford
9b33264334 DBZ-2157 Fix intermittent postgres test failures 2020-06-05 20:02:14 +02:00
Chris Cranford
845a790593 DBZ-2060 Fix intermittent test failures (sqlserver & postgres) 2020-06-05 20:02:14 +02:00
Gunnar Morling
02b16e17ac DBZ-1814 Adding tests for exporting unkown column type 2020-06-05 12:13:29 +02:00
Gunnar Morling
c54e377dd2 DBZ-1814 Exporting Hex and Base64 representations as string 2020-06-05 12:13:29 +02:00
Gunnar Morling
023064eedb DBZ-1814 Renaming "raw" to "bytes"; also clarifying that this notion isn't Java-specific 2020-06-05 12:13:29 +02:00
Jeremy Finzel
5db614da21 DBZ-1814 added new binary handling mode option 2020-06-05 12:13:29 +02:00
Gunnar Morling
d7b3f0a7fb DBZ-1813 Misc. clean-up; 
* Removing unused method
* Simplifying tests
* Avoiding deprecated code
 2020-06-05 06:24:25 +02:00
Gunnar Morling
59f34e7b16 DBZ-1813 Stabiling test 2020-06-05 06:24:25 +02:00
Brandon Brown
bd64ace7fc DBZ-1813 Adding ability to configure pgoutput publication based on table filters 
defined.
 2020-06-05 06:24:25 +02:00
Debezium Builder
dc25c379e5 [maven-release-plugin] prepare for next development iteration 2020-05-19 03:40:25 -04:00
Debezium Builder
ad060181b9 [maven-release-plugin] prepare release v1.2.0.Beta2 2020-05-19 03:40:25 -04:00
Braden Groom
8439e70d09 DBZ-2090 Remove unused sourceInfo variable 2020-05-19 08:30:55 +02:00
Jiri Pechanec
97c3d3b6e9 DBZ-2088 Use POSIX tar 2020-05-18 14:29:38 +02:00
Chris Cranford
f449550b2d DBZ-2068 Skip tests incompatible with Kafka 1.x 2020-05-13 08:22:07 +02:00
Debezium Builder
8d4203279e [maven-release-plugin] prepare for next development iteration 2020-05-07 05:03:40 -04:00
Debezium Builder
a4e2c8f36c [maven-release-plugin] prepare release v1.2.0.Beta1 2020-05-07 05:03:40 -04:00
Gunnar Morling
ab1d4bb1cd DBZ-1970 Making change event API more consistent and uniform 2020-05-06 14:33:37 +02:00
Jiri Pechanec
d82307c331 DBZ-1970 Support for separate key/value embedded converter 2020-05-06 14:33:37 +02:00
Chris Cranford
cd43c4880f DBZ-2014 Do not include outbox eventType field by default 2020-05-05 22:13:25 +02:00
Chris Cranford
81c3a808da DBZ-1963 Suggested changes 2020-05-05 08:45:58 +02:00
Chris Cranford
a1cb06299c DBZ-1963 Emit outbox events with value schema names 2020-05-05 08:45:58 +02:00
Jiri Pechanec
7f23d74d90 DBZ-1904 Add config param into a group 2020-05-04 12:31:05 +02:00
Jiri Pechanec
e350d74b17 DBZ-1904 Public facing schema history topic 2020-05-04 12:31:05 +02:00
Braden Groom
a8cb4e319c DBZ-2044 Load Postgres enum values with a single query instead of N queries 2020-05-04 12:08:16 +02:00
Chris Cranford
96dda6c3cb DBZ-1969 Add array support for time, timetz, timestamp, and timestamptz 2020-05-04 11:22:49 +02:00
Braden Groom
8093582487 DBZ-2038 Refactor postgres type registry queries to use the same base query 2020-04-30 14:29:14 -04:00
Braden Groom
63f96940fb DBZ-2038 Factor out logic for converting postgres result sets to type builders 2020-04-30 14:29:14 -04:00
Braden Groom
50c61fe462 DBZ-2038 Prime postgres array and non-array types using same logic 2020-04-30 14:29:14 -04:00
Braden Groom
01dda2a8a9 DBZ-2038 Load unknown postgres types by name and oid using same logic 2020-04-30 14:29:14 -04:00
Braden Groom
905a732342 DBZ-2038 Resolve array and non-array postgres types with the same logic 2020-04-30 14:29:14 -04:00
Braden Groom
284a4357e0 DBZ-1969 Update convertLtreeArray to handle List instances 2020-04-29 13:06:43 -04:00
Braden Groom
1578ebc8bb DBZ-1969 Have isArray return false for Postgres LTree arrays 2020-04-29 13:06:43 -04:00
Gunnar Morling
2310d6c792 DBZ-1969 Fixing incorrect assertion and utility which concealed this 2020-04-29 13:06:43 -04:00
Gunnar Morling
f718302d36 DBZ-1969 Simplifying array handling in PgProtoColumnValue 2020-04-29 13:06:43 -04:00
Braden Groom
0b429ce880 DBZ-1969 Add support for Postgres enum arrays 2020-04-29 13:06:43 -04:00
Chris Cranford
230ddaebdc DBZ-1953 Generalized SkipWhenDatabaseVersion functionality 2020-04-29 12:27:38 +02:00
Gunnar Morling
3d0606b348 DBZ-2027 Small speed-up of PG test execution; 
* Using Awaitility so we can use 100ms looping intervals; also it's more concise
* Avoiding creation of one temporary connection
 2020-04-29 09:56:59 +02:00
Gunnar Morling
16b780fa3c DBZ-2027 Upgrading Postgres JDBC driver to 42.2.12; 
The driver upgrade mitigates some issues with using this connector with
Postgres on Azure. It comes with some behavioural changes, though:

* column metadata for DECIMAL without scale is returned differently by
the (see https://github.com/pgjdbc/pgjdbc/issues/1767): while it used
to be returned as 0, it's now returned as null. This should be
transparent to DBZ consumers
* snapshots for partitioned tables only export change events on the
partition topics now due to https://github.com/pgjdbc/pgjdbc/pull/1708;
this has an impact on consumers, but I think it's more reasonable than
exporting all change events twice, one partition table and main table
topics
 2020-04-29 09:56:59 +02:00
Anton Kondratev
236ea1a15f DBZ-1990 Support for Postgres Json and Jsonb array columns 2020-04-27 22:17:48 +03:00
JanHendrikDolling
33d83731af DBZ-1972 move static Filed methods as static Filed constant to RelationalDatabaseConnectorConfig instead. 2020-04-27 12:31:19 +02:00
Gunnar Morling
ea7f8e918b DBZ-1750 Removing unused method 2020-04-24 15:35:28 +02:00
Gunnar Morling
03ce403972 DBZ-1750 Separating config definition editor and immutable config definitions 2020-04-24 15:35:28 +02:00
Jiri Pechanec
37f0dbcaf8 DBZ-1750 Limit access to parent defs; dont share the parent defs 2020-04-24 15:35:28 +02:00
Jiri Pechanec
af90507fa6 DBZ-1750 Centralize shared config definitions 2020-04-24 15:35:28 +02:00
dajerome
ad8beb76d0
DBZ-1949 Return BitSet instead of short/int/long; 
When converting Postgres bit varying column, convert to BitSet instead of short/int/long to handle cases where the value is larger than Long.MAX_VALUE
 2020-04-21 11:26:49 +02:00
Jos Huiting
33fe631674 DBZ-1962 Fix PR feedback 2020-04-17 15:24:42 +02:00
Jos Huiting
a0f65830de DBZ-1962 Remove TABLE_WHITELIST and TABLE_BLACKLIST in favor of definitions in RelationalDatabaseConnectorConfig 2020-04-17 15:24:42 +02:00
Jos Huiting
8ba21a98ef DBZ-1962 Introduce column.whitelist for the Postgres connector 2020-04-17 15:24:42 +02:00
Debezium Builder
4ca9f92575 [maven-release-plugin] prepare for next development iteration 2020-04-16 02:22:00 -04:00
Debezium Builder
e704bccdaa [maven-release-plugin] prepare release v1.2.0.Alpha1 2020-04-16 02:22:00 -04:00
Gunnar Morling
659d4a415b DBZ-1807 Misc. adjustments; 
* Adding @TestFor annotations
* Change -> ChangeEvent
 2020-04-15 16:29:29 +02:00
Jiri Pechanec
c0914b245b DBZ-1807 Support for CloudEvents converter 2020-04-15 16:29:29 +02:00
Jiri Pechanec
c06707f4c5 DBZ-1807 Support for Avro converters 2020-04-15 16:29:29 +02:00
Jiri Pechanec
4e2ac367a9 DBZ-1807 Support for JSON Converter 2020-04-15 16:29:29 +02:00
Chris Cranford
57af80afd5 DBZ-1859 Enhance MongoDB connector metrics 2020-04-15 08:04:23 +02:00
Jiri Pechanec
6219376bfe DBZ-1960 Remove sources of busy wait 2020-04-14 12:59:10 +02:00
JanHendrikDolling
1e19270981 DBZ-1692 Added support for hash based column masking mode for string values 2020-04-08 21:21:42 +02:00
Chris Cranford
5f8719f201 DBZ-1925 Fix test regression after Confluent Platform 5.4.1 upgrade 2020-04-07 09:01:20 +02:00
Jiri Pechanec
cacd21c44a DBZ-1922 wal2_json streaming uses NOOP messages 2020-04-02 17:29:16 +02:00
rk3rn3r
7f36b3c08f DBZ-1531 added headers for primary key update events to reference the original key 2020-03-25 12:33:38 +01:00
rk3rn3r
6310e00675 DBZ-1531 added headers for primary key update events to reference the original key 2020-03-25 12:33:38 +01:00
rk3rn3r
fbd8b94e94 DBZ-1531 added headers for primary key update events to reference the original key 2020-03-25 12:33:38 +01:00
Debezium Builder
ab54ce062f [maven-release-plugin] prepare for next development iteration 2020-03-24 01:23:05 -04:00
Debezium Builder
0b5a9117fb [maven-release-plugin] prepare release v1.1.0.Final 2020-03-24 01:23:05 -04:00
Jiri Pechanec
224f97e380 DBZ-1857 Support for autorestart for embedded engine 2020-03-23 13:56:29 +01:00
Jiri Pechanec
b44cd8475e DBZ-1815 Use separatae conection for db heartbeat 2020-03-23 12:02:31 +01:00
trolison
8784b41e22 DBZ-1815 Ading DatabaseHeartbeatImpl to allow for making queries to the DB when a heartbeat happens; 
* Minor refactor to make DatabaseHeartbeatImpl usable
* Added integration test to verify heartbeat.action.query is being executed properly
 2020-03-18 10:27:22 +01:00
Chris Cranford
5ce9d6fb14 DBZ-1755 Fix wal2json/pgoutput streaming of negative monetary values 2020-03-17 08:36:24 +01:00
Chris Cranford
f6d7778bb4 DBZ-1755 Suggested changes 
* Add test case for snapshotting null currency value
* Restore previous exception handling for money types
 2020-03-17 08:36:24 +01:00
Chris Cranford
0cc6e34264 DBZ-1755 Handle parsing negative monetary values during snapshot 2020-03-17 08:36:24 +01:00
Fabian Aussems
1be07ca4f4 DBZ-1755 Handle error conditions better for the Pg MONEY datatype 2020-03-17 08:36:24 +01:00
Jiri Pechanec
b114520f5e
DBZ-1875 Test stabilization; 
* Snapshot metrics are present with snapshot never
* Skip extra empty TX

Co-authored-by: Gunnar Morling <gunnar.morling@googlemail.com>
 2020-03-13 11:40:00 +01:00
Jiri Pechanec
4ffed1fa46 DBZ-1730 Improve test limits description 2020-03-13 09:39:23 +01:00
Jiri Pechanec
ee91eac418 DBZ-1730 Stream from replica slot position not end of tx log 2020-03-13 09:39:23 +01:00
Gunnar Morling
449154beaa DBZ-1863 Centralizing coordinator shutdown 2020-03-12 11:28:08 +01:00
Debezium Builder
86c1ed90ed [maven-release-plugin] prepare for next development iteration 2020-03-11 09:54:44 -04:00
Debezium Builder
062c6abbe5 [maven-release-plugin] prepare release v1.1.0.CR1 2020-03-11 09:54:44 -04:00
Gunnar Morling
c51a5fb597 DBZ-1723 Restarting after connection losses in Postgres 2020-03-11 13:57:23 +01:00
Gunnar Morling
3437da00d7 DBZ-1723 Simplifying shutdown handling; 
doStop() is already called from stop(), no need to do it again via the error handler.
 2020-03-11 13:57:23 +01:00
Gunnar Morling
c156d30048 DBZ-1723 Centralizing stopping() logic in BaseSourceTask 2020-03-11 13:57:23 +01:00
Gunnar Morling
8ce69dc597 DBZ-1723 Encapsulating common poll() and stop() logic in BaseSourceTask 2020-03-11 13:57:23 +01:00
Chris Cranford
6f1eb49b32 DBZ-1726 Fix formatting failures 2020-03-11 10:15:00 +01:00
Chris Cranford
3a4788cc33 DBZ-1726 Introduced AbstractSnapshotChangeEventSource and refactor implementations 2020-03-11 10:15:00 +01:00
Chris Cranford
5506fadd2d DBZ-1664 Verify correctness of JMX metrics 2020-03-09 23:30:24 +01:00
Jiri Pechanec
0248fdde16
DBZ-1850 Delegate signed/nonsigned conversion to driver 2020-03-09 13:06:30 +01:00
Jiri Pechanec
69e87170e3 DBZ-1849 Only log message was fixed 2020-03-09 07:43:18 +01:00
Jiri Pechanec
62e6a0add7 DBZ-1849 Avoid type refresh for unchanged root type 2020-03-06 18:01:56 +01:00
Jiri Pechanec
f99064fda1 DBZ-1824 Restart incomplete Tx for wal2json 2020-03-04 14:03:21 +01:00
Gunnar Morling
a8b796685b DBZ-1840 Typo fix; commenting 2020-03-04 09:31:22 +01:00
Jiri Pechanec
10dfc12c99 DBZ-1840 Explanation for unresolvable tables 2020-03-04 09:25:57 +01:00
Jiri Pechanec
19d0a88cc9 DBZ-1840 Test logic was out of order 2020-03-04 09:25:57 +01:00
Jiri Pechanec
b2370c2d8e DBZ-1840 Syntheitc message to replace messages dropped in decoder 2020-03-04 09:25:57 +01:00
Jiri Pechanec
6bab948e76 DBZ-1840 No nulls to stream producer 2020-03-04 09:25:57 +01:00
Jan-Hendrik Dolling
5cfbaab7a6
DBZ-1845 Typos in deployment.yaml and code readability 
* improve readability
* fixed some typos
 2020-03-04 08:34:55 +01:00
Gunnar Morling
c5422e6b31 DBZ-1840 Not passing null to replication message processor; 
Note this means to that WAL offsets cannot be acknowledged in cases
where a null message is received from the server (e.g. wal2json for
DDL events). But as we cannot tell apart null events from the server
from readPending() simply having no events to read, the current
logic renders the pausing between two read calls without events
obsolete, so that's the lesser evil
 2020-03-03 11:45:53 +01:00
Gunnar Morling
0f6d2c54fc DBZ-1134 Avoiding concrete class in SPI by introducing receiver parameter 2020-02-28 21:58:39 +01:00
Jiri Pechanec
d13e48cc55 DBZ-1134 Make converted field type parametrized 2020-02-28 21:58:39 +01:00
Jiri Pechanec
06e1eca085 DBZ-1134 Support for user provided converters 2020-02-28 21:58:39 +01:00
Jiri Pechanec
91f292a1f2 DBZ-1825 Keep keys for non-matching table 2020-02-26 15:03:03 +01:00
Jiri Pechanec
ae6ee3d458 DBZ-234 Place Incubating into a unique package 2020-02-25 15:31:10 +01:00
Jiri Pechanec
2b6e50daff DBZ-1760 Allow skipping of invalid events 2020-02-25 10:43:11 +01:00
jcechace
3209de3e5f DBZ-924 Adding integration tests for OpenShift and Strimzi 2020-02-20 12:48:08 +01:00
Gunnar Morling
eff3ffc345 DBZ-1758 Documentation update 2020-02-14 13:14:25 +01:00
Debezium Builder
f75746fb7b [maven-release-plugin] prepare for next development iteration 2020-02-13 03:18:20 -05:00
Debezium Builder
1d2df32743 [maven-release-plugin] prepare release v1.1.0.Beta2 2020-02-13 03:18:20 -05:00
Gunnar Morling
dd5c2b8be9 DBZ-1766 Unifying handling of offset commits 2020-02-12 15:44:45 +01:00
Melissa Winstanley
ef55c9157d DBZ-1766 Update lastOffset to commit only after a record is committed 2020-02-12 15:44:45 +01:00
Jiri Pechanec
d4d38dcbec DBZ-1783 Ignore empty transactions 2020-02-12 14:58:15 +01:00
Jiri Pechanec
40506d5352 [ci] Reduce postgres test logging for CI 2020-02-12 14:49:55 +01:00
Debezium Builder
9c424b2404 [maven-release-plugin] prepare for next development iteration 2020-02-05 08:54:52 -05:00
Debezium Builder
9f2caa3b7c [maven-release-plugin] prepare release v1.1.0.Beta1 2020-02-05 08:54:52 -05:00
Jiri Pechanec
7c9d78e597 DBZ-1727 Fix unused imports 2020-02-05 13:33:29 +01:00
Igor Gabaydulin
3890038c21 DBZ-1727 Reduce heartbeat interval in test 2020-02-05 13:33:29 +01:00
Igor Gabaydulin
a2a6525c84 DBZ-1727 Update heartbeat on changing non whitelisted table test 2020-02-05 13:33:29 +01:00
Igor Gabaydulin
2f6e522b0f DBZ-1727 Fix PostgresConnectorIT#shouldFlushLsnOnEmptyMessage test 2020-02-05 13:33:29 +01:00
Igor Gabaydulin
b62acfd144 DBZ-1727 Apply code formatter 2020-02-05 13:33:29 +01:00
Igor Gabaydulin
9bea1e2638 DBZ-1727 Move null check to AbstractMessageDecoder 2020-02-05 13:33:29 +01:00
Igor Gabaydulin
fcc7a05579 DBZ-1727 Debezium Postgres connector does not apply keepalive lsn 2020-02-05 13:33:29 +01:00
Jiri Pechanec
77f9a61b95 DBZ-1751 Centralize envelope schema naming 2020-02-04 20:37:19 +01:00
Chris Cranford
c5902c3c32 DBZ-1158 Add test for REPLICA IDENTITY fallback values 2020-02-04 10:37:12 -05:00
Jiri Pechanec
1d9ac71a9e DBZ-1698 Fix test timing issue 2020-02-03 22:38:02 +01:00
Jiri Pechanec
2a6ea1c41f DBZ-1698 Handle toasted unknown column values 2020-02-03 22:38:02 +01:00
Jiri Pechanec
34d2979123 DBZ-1052 Stabilize test for changed heartbeats 2020-02-03 09:29:10 +01:00
Jiri Pechanec
010e0c6fa6 DBZ-1052 Flush LSN for skipped tx messages 2020-02-03 09:29:10 +01:00
Gunnar Morling
b66f314b3c DBZ-1052 Misc. clean-up 
* Doc updates
* Changed static method to instance
 2020-02-03 09:29:10 +01:00
Jiri Pechanec
67e0079646 DBZ-1052 Expand known protobuf messages list 2020-02-03 09:29:10 +01:00
Jiri Pechanec
4fdda6c101 DBZ-1052 Add tx enable option to connector definitions 2020-02-03 09:29:10 +01:00
Jiri Pechanec
ba10a229b6 DBZ-1052 CE support for tx metadata in data messages 2020-02-03 09:29:10 +01:00
Jiri Pechanec
0e671d6167 DBZ-1052 Add counts per table to END 2020-02-03 09:29:10 +01:00
Jiri Pechanec
60056b0a94 DBZ-1052 Transaction block moved to envelope 2020-02-03 09:29:10 +01:00
Jiri Pechanec
5749acda7e DBZ-1052 Remove unused fields; added JavaDoc 2020-02-03 09:29:10 +01:00
Jiri Pechanec
072a37e790 DBZ-1052 Support for full Postgres/protobuf 2020-02-03 09:29:10 +01:00
Jiri Pechanec
c0557b325b DBZ-1052 Skip CE for transaction struct 2020-02-03 09:29:10 +01:00
Jiri Pechanec
c466143f35 DBZ-1052 Test for limited Postgres/protobuf 2020-02-03 09:29:10 +01:00
Jiri Pechanec
fcafd2f193 DBZ-1052 Transaction metadata for Postgres/wal2json 2020-02-03 09:29:10 +01:00
Jiri Pechanec
cd6a1f942d DBZ-1052 Transaction metadata for Postgres/pgoutput 2020-02-03 09:29:10 +01:00
Jiri Pechanec
b0eb571f9f DBZ-1052 Transaction metadata for SQL Server 2020-02-03 09:29:10 +01:00
Jiri Pechanec
7283f5d6c1 DBZ-1746 Gracefully handle unknown protobuf messages 2020-01-30 12:24:52 +01:00
Marc Zbyszynski
507aac0e0f DBZ-1738 replaced casts to PgConnection with casts to interface BaseConnection 
DBZ-1738 replaced casts to PgConnection with casts to interface BaseConnection
 2020-01-27 16:47:27 +01:00
Debezium Builder
274bba9dc9 [maven-release-plugin] prepare for next development iteration 2020-01-16 05:40:24 -05:00
Debezium Builder
076df30108 [maven-release-plugin] prepare release v1.1.0.Alpha1 2020-01-16 05:40:24 -05:00
Gunnar Morling
8c33a4e147 DBZ-1292 Catering for the differences in data schema; 
* MongoDB doesn't have "before", but "patch"
* Also removing some redundancies and making RecordParser immutable
* Reducing log verbosity during tests
 2020-01-16 08:01:56 +01:00
Gunnar Morling
1a18bdbc81 DBZ-1292 Misc. refactorings; 
* Protecting against Avro as CE format and JSON as data format
* Reducing mutability
* Single Avro client
 2020-01-16 08:00:45 +01:00
GraySmog
d0e4b3c0af DBZ-1292 Delete data.schema.registry.url config option 2020-01-16 08:00:45 +01:00
GraySmog
915c8cafd9 DBZ-1292 CloudEvents converter for JSON and Avro format 2020-01-16 08:00:45 +01:00
Gunnar Morling
07e89de91d DBZ-1707 Ensuring correct Kafka record timestamp for outbox events; 
Depending on the width of the configured TIMESTAMP column, a microsecond or nanosecond timestamp could have been returned before, whereas this should always be milliseconds.
 2020-01-15 07:32:55 +01:00
Chris Cranford
1d41d53636 DBZ-1685 Document and add support for column.mask.with for PostgreSQL connector 2020-01-14 09:30:34 +01:00
Chris Cranford
9fc1eada11 DBZ-1600 Remove deprecated "slot.drop_on_stop" postgres configuration option 2020-01-14 09:27:32 +01:00
Chris Cranford
cf60466b02 DBZ-1684 Make sure publication initialized prior to slot 
This is to avoid the need to restart the connector when streaming begins due to
an error that the publication can't be found.  It would seem based on testing
that since we create the replication slot early on when it does not exist, then
when streaming starts its confused about trying to marry a newly created
publication to an slot.  By creating the publication before the slot, this
works around the problem.
 2020-01-09 15:55:22 +01:00
Chris Cranford
3413d674fb DBZ-1684 Create missing publication when replication slot exists 2020-01-07 15:41:15 +01:00
Debezium Builder
9723c93db9 [maven-release-plugin] prepare for next development iteration 2019-12-18 01:44:33 -05:00
Debezium Builder
09049516f2 [maven-release-plugin] prepare release v1.0.0.Final 2019-12-18 01:44:33 -05:00
Chris Cranford
5c6f8fdd70 DBZ-1680 Fix enum value resolution across all PostgreSQL decoders 2019-12-17 18:21:17 +01:00
Chris Cranford
debcb95fa2 DBZ-1677 Include root cause when PostgreSQL schema refresh fails during streaming 2019-12-17 07:19:02 +01:00
Gunnar Morling
d40641b3ab DBZ-1666 Formatting 2019-12-12 16:39:10 +01:00
Gunnar Morling
4619f5eb9d DBZ-1666 Wording fix 2019-12-12 11:56:05 +01:00
Jiri Pechanec
ffa15b100f DBZ-1666 Warn for drop slot on stop 2019-12-12 11:52:30 +01:00
Debezium Builder
b1c62fe306 [maven-release-plugin] prepare for next development iteration 2019-12-10 06:39:17 -05:00
Debezium Builder
ad2efdb9a6 [maven-release-plugin] prepare release v1.0.0.CR1 2019-12-10 06:39:17 -05:00
Chris Cranford
c32b771e1b DBZ-1413 Fix format violation 2019-12-10 10:09:35 +01:00
Chris Cranford
19fd8dd1d9 DBZ-1413 Resolve column length/scale when reading table column metadata from TypeRegistry 2019-12-10 10:09:35 +01:00
Chris Cranford
372d199a71 DBZ-1413 Revert AbstractColumnValue changes in commit 774f2ed 2019-12-10 10:09:35 +01:00
Chris Cranford
365eb0d76e DBZ-1413 Suggested Javadoc changes, added/fix tests, changed how Column native/jdbc type resolution happens 2019-12-10 10:09:35 +01:00
Chris Cranford
9685990fb8 DBZ-920 Add support for PostgreSQL enum types 2019-12-10 10:09:35 +01:00
Chris Cranford
46de58ae75 DBZ-1413 Updated tests 2019-12-10 10:09:35 +01:00
Chris Cranford
ef9b4a7431 DBZ-1413 Delay building type hierarchy after all base types built 2019-12-10 10:09:35 +01:00
Chris Cranford
547ba8e0db DBZ-1413 Align domain type support across all 3 decoders 2019-12-10 10:09:35 +01:00
Chris Cranford
9586be7d4e DBZ-1413 Support PostgreSQL domain types 2019-12-10 10:09:35 +01:00
Thomas Deblock
17e44d4c0c DBZ-1648 outbox EventRouter remove topic toLowerCase 
- kafka topics names are case sensitive. We should not perform
  a toLowerCase on the topic name. It can cause some issue if
  topic name have a upper case
 2019-12-04 21:39:25 +01:00
Jiri Pechanec
d471f2f6ab DBZ-1637 Support for UUID array 2019-11-28 09:06:23 +01:00
Brandon Brown
a0f7f6ec6b DBZ-1402 Support for JDK 11 2019-11-25 11:53:19 +01:00
Jiri Pechanec
a56ab78b3f DBZ-825 Unify decimal upscaling 2019-11-20 18:23:55 +01:00
Debezium Builder
38279cc2dd [maven-release-plugin] prepare for next development iteration 2019-11-14 02:56:07 -05:00
Debezium Builder
868faf4412 [maven-release-plugin] prepare release v1.0.0.Beta3 2019-11-14 02:56:07 -05:00
David Feinblum
2b9fbcd5f6 DBZ-1519 Avoid erroneously logging there are no monitored tables 2019-11-14 05:44:37 +01:00
Chris Cranford
95927de405 DBZ-1596 Fix open resources while initializing publication 2019-11-08 12:55:24 +01:00
Jiri Pechanec
d693609779 DBZ-1570 handle null toasted values for wal2json 2019-11-08 10:52:57 +01:00
Jiri Pechanec
4084e39256 DBZ-1595 New parameter takes precedence over old 2019-11-08 07:57:06 +01:00
Jiri Pechanec
54cb9e8326 DBZ-1595 Use dots as separator in param name 2019-11-08 07:57:06 +01:00
Jeremy Finzel
c2f27cdb23 DBZ-1559 Change snapshot lock level to ACCESS SHARE 
This is the correct mode that only prevents concurrent schema changes, while not requiring debezium user to also have UPDATE access to tables as does SHARE UPDATE EXCLUSIVE mode
 2019-11-05 11:54:52 +02:00
Chris Cranford
244ee9dc5c DBZ-1576 Gracefully skip TRUNCATE messages with pgoutput decoder on PG11+ 2019-11-05 09:24:08 +02:00
Gunnar Morling
2c569977d7 DBZ-1392 Applying formatter; 
* Keeping indentation formatting for JSON


Co-authored-by: Gunnar Morling <gunnar.morling@googlemail.com>
Co-authored-by: Chris Cranford <chris@hibernate.org>
 2019-11-01 13:20:43 +02:00
Addison Higham
b1b657b8bd [DBZ-1579] Fix regression of large refreshes causing connection timeout 
In DBZ-1214, an issue was fixed where during the initial schema refresh,
the replication stream connection would timeout. This occurred in cases
of many schemas that need to be refreshed.

This was fixed by adding a keep alive thread that ensure that we
periodically send an empty update to the DB.

It appears that in the refactor to move the new shared interface, the
call to this keep alive thread was removed, but the remaining keep alive
interface still exists

This simply wires the keep alive thread back up
 2019-10-29 18:27:55 +01:00
Debezium Builder
b08f523ed7 [maven-release-plugin] prepare for next development iteration 2019-10-24 07:42:49 -04:00
Debezium Builder
8ecd9e28cf [maven-release-plugin] prepare release v1.0.0.Beta2 2019-10-24 07:42:49 -04:00
Jiri Pechanec
097025bed9 DBZ-1336 Fix imports of Ltree 2019-10-22 08:07:54 +03:00
Gunnar Morling
e06d25dd92 DBZ-1336 Moving Ltree type to PG connector module 2019-10-21 12:17:08 +02:00
Mincong HUANG
1e3eb8c840 DBZ-1336 Support Postgres LTREE columns 2019-10-21 12:10:46 +02:00
Jiri Pechanec
0b15e76d2a DBZ-1525 Validate replication slot name 2019-10-20 22:16:29 +02:00
Jiri Pechanec
d858e05bf6 DBZ-1564 Close TX after toasted query 2019-10-19 20:46:22 +03:00
Grant Cooksey
5ab5b4063d DBZ-1564 Close open transaction during postgres streaming 
After the schema refesh, we do not want to hold a long running open transaction
since that is expensive.
 2019-10-19 20:46:22 +03:00
Jiri Pechanec
06ef4b6a95 DBZ-1565 Take empty messages into consideration 2019-10-18 13:53:53 +03:00
Jiri Pechanec
986dc4ef3a DBZ-1565 Test the log message 2019-10-18 13:53:53 +03:00
Gunnar Morling
4ea85d7fd3 DBZ-1565 Raise warning about growing WAL backlog 2019-10-18 13:53:53 +03:00
Debezium Builder
519dc011ff [maven-release-plugin] prepare for next development iteration 2019-10-17 04:20:13 -04:00
Debezium Builder
51ef6b9470 [maven-release-plugin] prepare release v1.0.0.Beta1 2019-10-17 04:20:12 -04:00
Grant Cooksey
18ca6ec72e DBZ-1546 Comma-separated regular expressions remove whitespace. 
The table whitespace parameter should remove whitespace between regular expressions
since tables cannot start with whitespace characters.
 2019-10-17 10:13:44 +03:00
Jiri Pechanec
bba550f955 DBZ-892 Flush on empty messages 2019-10-16 00:54:54 +02:00
Jiri Pechanec
6d8412e5de DBZ-1401 Support for pgoutput 2019-10-10 09:37:52 +02:00
Jiri Pechanec
f77415936f DBZ-1401 Fix formatting 2019-10-10 09:37:52 +02:00
Jiri Pechanec
7c4203cf60 DBZ-1401 Handle tables without explicit PK 2019-10-10 09:37:52 +02:00
Chris Cranford
4205a224d6 DBZ-1532 Pgoutput string decoding should not rely on JVM default charset 2019-10-09 14:35:25 +02:00
Jiri Pechanec
cf7eccb152 DBZ-1498 Docs added; microseconds changed to numeric; nanos supported 2019-10-09 12:14:35 +02:00
Jiri Pechanec
20363f24bb DBZ-1498 Support for precise interval value 2019-10-09 12:14:35 +02:00
Gunnar Morling
92885ebe99 DBZ-1392 Removing trailing whitespace from Java files 2019-10-07 12:11:43 +03:00
Gunnar Morling
8d6f5409b1 DBZ-1392 Moving static imports up and applying this to all files 2019-10-07 12:11:43 +03:00
Chris Cranford
3594ff639a DBZ-1341 Rebased against master and applied necessary checkstyle fixes 2019-10-03 19:44:43 +02:00
Chris Cranford
ca5517746a DBZ-1341 Checkstyle flag missing whitespace 2019-10-03 19:44:43 +02:00
Chris Cranford
7442670692 DBZ-1355 Fix checkstyle problems detected after upgrade to 8.19 2019-10-03 19:44:43 +02:00
Chris Cranford
5ac2318d84 DBZ-1391 Do not allow code after closing brace (#1065) 
* DBZ-1391 Do not allow code after closing brace
 2019-10-02 20:44:40 +02:00
Jiri Pechanec
f868cb87c0 DBZ-1280 Lock timeout for SQL Server 2019-10-02 10:34:25 +02:00
Gunnar Morling
d94b37c693 DBZ-1507 Misc. clean-up; 
* Removing unused Table methods
* Re-using primaryKeyColumnNames() instead of custom predicate
 2019-10-02 10:25:37 +02:00
Debezium Builder
2ff77b993b [maven-release-plugin] prepare for next development iteration 2019-10-01 15:34:23 -04:00
Debezium Builder
e09d584b11 [maven-release-plugin] prepare release v0.10.0.Final 2019-10-01 15:34:23 -04:00
Jiri Pechanec
22e8397e6d DBZ-1515 Add poll pause only after multiple no message attempts 2019-10-01 13:24:42 -04:00
Debezium Builder
2f5960c5d2 [maven-release-plugin] prepare for next development iteration 2019-09-27 01:03:50 -04:00
Debezium Builder
e3b44ffb12 [maven-release-plugin] prepare release v0.10.0.CR2 2019-09-27 01:03:50 -04:00
Gunnar Morling
2fba51863e DBZ-1015 Unifying the retrieval of custom key mappers 2019-09-26 10:14:48 +02:00
Guillaume Rosauro
1405889bef DBZ-1015-allow-user-to-customize-key 2019-09-26 10:14:21 +02:00
Gunnar Morling
97cc493ef5 DBZ-1368 More expressive type name 2019-09-25 17:45:09 +02:00
Jiri Pechanec
0e2a79eba5 DBZ-1368 Change default toasted value 2019-09-25 17:24:21 +02:00
Jiri Pechanec
5bfe14ea66 DBZ-1368 Remove unnecessary optimizations 2019-09-25 17:24:21 +02:00
Jiri Pechanec
3c19a392ac DBZ-1368 Added documentation 2019-09-25 17:24:21 +02:00
Jiri Pechanec
4150d2d0e1 DBZ-1368 Added JavaDoc 2019-09-25 17:24:21 +02:00
Jiri Pechanec
c638b16117 DBZ-1368 Toast marker presence not derived from plguin name 2019-09-25 17:24:21 +02:00
Jiri Pechanec
6efd2a76fe DBZ-1368 Set new toasted vals from old 2019-09-25 17:24:21 +02:00
Jiri Pechanec
f19b4b15a7 DBZ-1368 Run with postgres dev image 2019-09-25 17:24:21 +02:00
Jiri Pechanec
29f6d9dd09 DBZ-1368 Support toasted placeholder in wal2json 2019-09-25 17:24:21 +02:00
Jiri Pechanec
9dd5a16160 DBZ-1368 Support toasted placeholder in protobuf 2019-09-25 17:24:21 +02:00
Gunnar Morling
ae0bf61bb4 DBZ-1367 Handling "datum_missing" marker 2019-09-25 17:24:21 +02:00
Jiri Pechanec
ab31147593 DBZ-1368 Support toasted placeholder in pgoutput 2019-09-25 17:24:21 +02:00
Gunnar Morling
442408be02 DBZ-1497 Using int64 for INTERVAL microseconds mapping 2019-09-25 11:21:40 +03:00
Jiri Pechanec
9b2fcdc13a DBZ-1438 Reduce test runtime 2019-09-24 12:51:48 +02:00
Jiri Pechanec
8f308746d3 DBZ-1438 Interrupt only after interrupt affected operations 2019-09-24 12:51:48 +02:00
Gunnar Morling
c2b66ef2d7 DBZ-1436 Referencing default publication name via constant 2019-09-23 20:36:40 +02:00
Chris Cranford
f446ce6ffd DBZ-1436 Fix postgres connector not honoring configured publication.name setting 2019-09-23 20:33:48 +02:00
Gunnar Morling
008ac7a7e4 DBZ-1426 Misc clean-up; 
* Removing unused method parameters
* Formatting
* Using logger methods with argument placeholders
 2019-09-23 11:34:18 +02:00
trolison
e5f77b3a32 DBZ-1426, DBZ-1427 Introducing retry if replication slot is already in use; 
- Added PG connector config parameters for retry count and delay
- Added basic test for connection retry and delay logic
- Added a more clear error message when a PG connection cant be made and it has nothing to do with the schema registry (DBZ-1427)
- Updated documentation for the new connection retry parameters
 2019-09-23 11:23:38 +02:00
Chris Cranford
130210b5ad DBZ-1044 Add config option sanitize.field.names and FieldNameSelector 2019-09-20 07:40:23 +03:00
Chris Cranford
b569c184da DBZ-1383 Fix PostgreSQL tests that fail intermittently 2019-09-17 11:19:17 +03:00
Chris Cranford
06c7621217 DBZ-1383 Fix PostgreSQL tests that fail intermittently 
* Test dropping slot & publication before each test
 2019-09-17 11:19:17 +03:00
Chris Cranford
278bbad8ae DBZ-1383 Revert commit 8e636685: Fix PostgreSQL tests that fail intermittently 2019-09-17 11:19:17 +03:00
Chris Cranford
fd6450442b DBZ-1383 Fix PostgreSQL tests that fail intermittently 
* Introduced wait steps to OutboxEventRouterIT to check if the problem is timing related
 2019-09-17 11:19:17 +03:00
Chris Cranford
37191ab713 DBZ-1383 Fix PostgreSQL tests that fail intermittently 2019-09-17 11:19:17 +03:00
Chris Cranford
022d0d07c8 DBZ-1383 Fix PostgreSQL tests that fail intermittently 
* Only execute `DROP PUBLICATION` when using the PGOUTPUT logical decoder
 2019-09-17 11:19:17 +03:00
Chris Cranford
99727dea76 DBZ-1383 Fix PostgreSQL tests that fail intermittently 2019-09-17 11:19:17 +03:00
Debezium Builder
9b7e09fdda [maven-release-plugin] prepare for next development iteration 2019-09-10 09:12:12 -04:00
Debezium Builder
a1c2861458 [maven-release-plugin] prepare release v0.10.0.CR1 2019-09-10 09:12:11 -04:00
Renato Mefi
c8c4266e34 DBZ-1457 Fix typo 2019-09-03 22:39:37 +02:00
Gunnar Morling
fe9fc1c0dc DBZ-1255 Adjustments after rebase 2019-08-21 07:50:40 +02:00
Gunnar Morling
08ef1418d9 DBZ-DBZ-1255 Preventing value overflows for TIMESTAMP; 
* Generally avoiding conversion to nano-seconds to accomodate for larger values
* Generally avoiding the usage of long to convey temporal values internally:
 * Instant is used for TIMESTAMP
 * OffsetDateTime for TIMESTAMPTZ
 * Duration for TIME (LocalTime cannot be used as it doesn't support 24:00:00 as possible with Postgres)
 * OffsetTime for TIMETZ
* Avoiding usage of methods under test for calculating expected values in tests
 2019-08-21 07:50:40 +02:00
Gunnar Morling
94eb75257b DBZ-1255 Using Instant for representing timestamps 2019-08-21 07:50:40 +02:00
Ivan Luzyanin
fd5841ab6d fix import 2019-08-21 07:50:40 +02:00
Ivan Luzyanin
fa10cc502d [DBZ-1255,DBZ-1205] Fix for parsing dates with year larger than 9999. Avoid using nonoseconds and parse the time directly to OffsetDateTime. 2019-08-21 07:50:40 +02:00
Chris Cranford
ef30b8a8cf DBZ-1419 Fix timing issue waiting for streaming to start 2019-08-20 18:43:03 +02:00
Chris Cranford
c6f6020da5 DBZ-777 Fix more timing issues with heartbeats 2019-08-20 18:43:03 +02:00
Jiri Pechanec
0e39965c0e DBZ-1419 Override time precision for MySQL 2019-08-20 18:43:03 +02:00
Jiri Pechanec
51c12d930d DBZ-1419 Support for Connect temporal types in SQL Server 2019-08-20 18:43:03 +02:00
Chris Cranford
8504f6aafd DBZ-1437 Fix InitialOnlySnapshotter to properly determine when to run a snapshot. 2019-08-19 22:01:32 +02:00
Chris Cranford
68f0bba7b3 DBZ-1437 Fix ExportedSnapshotter to properly determine when to run a snapshot. 2019-08-19 22:01:32 +02:00
Cyril Scetbon
e7dc1dfb62 DBZ-1440 Use ServerVersion values instead of integers 2019-08-19 14:25:53 -04:00
Cyril Scetbon
135e66dd0d DBZ-1440 Temporary replication slots on PG10+ only 2019-08-19 14:25:53 -04:00
Cyril Scetbon
4b5475391c DBZ-1440 Supports exported snapshots for Pg 9.4+ 2019-08-19 14:25:53 -04:00
Debezium Builder
4ae105bdce [maven-release-plugin] prepare for next development iteration 2019-08-16 03:42:55 -04:00
Debezium Builder
945ccf68cb [maven-release-plugin] prepare release v0.10.0.Beta4 2019-08-16 03:42:54 -04:00
Jiri Pechanec
9c25a192b7 DBZ-777 Fix timing issue with heartbeats 2019-08-16 08:05:48 +02:00
Jiri Pechanec
540021cf68 DBZ-777 Minor fixes 2019-08-15 20:43:31 +02:00
Jiri Pechanec
4d18c70834 DBZ-777 Make sure the empty events are generated while streaming 2019-08-15 20:43:31 +02:00
Gunnar Morling
5ffe8123f1 DBZ-777 Further fixes 2019-08-15 20:43:31 +02:00
Gunnar Morling
63250d0c13 DBZ-777 Misc. adjustments 
* Avoiding repeated parsing of table id
* Making 'inconsistentSchemaHandler' field final in EventDispatcher
* Not exposing synchronizeTableSchema() on SPI level
 2019-08-15 20:43:31 +02:00
Jiri Pechanec
780545d0d2 DBZ-777 Fix wal2json decoder 2019-08-15 20:43:31 +02:00
Jiri Pechanec
5307a5822d DBZ-777 Fix pgoutput decoder 2019-08-15 20:43:31 +02:00
Jiri Pechanec
de2345f87b DBZ-777 Fix wal2json decoder 2019-08-15 20:43:31 +02:00
Jiri Pechanec
037d581844 DBZ-777 Fix heartbeats 2019-08-15 20:43:31 +02:00
Jiri Pechanec
ffb7e6d7d8 DBZ-777 Assertions against internal schema 2019-08-15 20:43:31 +02:00
Jiri Pechanec
857d22bfda DBZ-777 Fix toasted cols test 2019-08-15 20:43:31 +02:00
Jiri Pechanec
8f08f2d505 DBZ-777 Remove unused code 2019-08-15 20:43:31 +02:00
Jiri Pechanec
d72afa8e9b DBZ-777 Skip empty messages 2019-08-15 20:43:31 +02:00
Jiri Pechanec
f4246df6e4 DBZ-777 PostgreSQL connector rewrite to the standard framework 2019-08-15 20:43:31 +02:00
Chris Cranford
89ae4b2151 DBZ-1436 Fix postgres connector not honoring publication.name configuration 2019-08-15 12:00:29 -04:00
Jiri Pechanec
8adb5e48ca DBZ-1385 Elevate payload for only headers setup 2019-08-12 10:36:52 +02:00
Jiri Pechanec
0991c70ac9 DBZ-1385 eventType can be configured as add field 2019-08-12 10:36:52 +02:00
Jiri Pechanec
c07c994434 DBZ-1381 Introduce permanent metadata connection 2019-08-09 11:53:45 +02:00
Chris Cranford
ae58b0b44c DBZ-1035 Checkstyle fixup 2019-07-29 15:49:18 +02:00
Chris Cranford
e852f082a2 DBZ-1035 Support PostgreSQL lock-free snapshot using exported snapshots 2019-07-29 15:49:18 +02:00
Jiri Pechanec
0742e5775f Revert "[maven-release-plugin] prepare release v0.10.0.Beta4" 
This reverts commit c7340e5152.
 2019-07-23 14:54:55 +02:00
Jiri Pechanec
f7d20635ce Revert "[maven-release-plugin] prepare for next development iteration" 
This reverts commit 22879777e4.
 2019-07-23 14:54:49 +02:00
Jiri Pechanec
22879777e4 [maven-release-plugin] prepare for next development iteration 2019-07-23 14:10:03 +02:00
Jiri Pechanec
c7340e5152 [maven-release-plugin] prepare release v0.10.0.Beta4 2019-07-23 14:10:02 +02:00
Debezium Builder
71333fda98 [maven-release-plugin] prepare for next development iteration 2019-07-23 02:55:08 -04:00
Debezium Builder
20d006fac7 [maven-release-plugin] prepare release v0.10.0.Beta3 2019-07-23 02:55:08 -04:00
Jiri Pechanec
81ba1a3adb DBZ-766 Remove autoboxing 2019-07-23 06:22:41 +02:00
Chris Cranford
2f92553d8f DBZ-766 Implement PgOutput logical decoding for Postgres 10+ 2019-07-23 06:22:41 +02:00
Jiri Pechanec
6a13b0e0b5 DBZ-1400 Record with same flush LSN should not be skipped for the first record 2019-07-22 14:17:59 -04:00
Gunnar Morling
0383f9178c DBZ-1238 Removing incorrect comment 2019-07-09 21:14:28 +02:00
Gunnar Morling
6aaa8b183c DBZ-1238 Simplifying logic 2019-07-09 21:14:28 +02:00
Gunnar Morling
bb8d7683e3 DBZ-1238 Avoiding explicit check for slot state; 
This causes issues with PG 10 temporary slots which are always active.
The check isn't really needed anyways, because the server will raise an
exception itself when trying to connect to an already active slot.
 2019-07-09 21:14:28 +02:00
Gunnar Morling
a2e0c7cd2a DBZ-1238 Misc. clean-up; 
* Formatting, typo fixes
* Combining createReplicationSlot() and getSlotCreationResult()
* Removing unused field
 2019-07-09 21:14:28 +02:00
Addison Higham
62328bf137 DBZ-1238 Expose more options to the PG Snapshot SPI 
This commit adds some more options to the Postgres Snapshottter SPI
interface, specifically, the ability to create the query for the start
of a snapshot, as well as the ability to generate a statement to lock
the tables.

Additionally, some rework is done to allow for the usage of an exported
snapshot in conjuction with the SPI taking metadata about a newly
created slot. This also changes how slots are initialized to allow for
the exported snapshot to be referenced, which requires adding a more
granular setup of a ReplicationConnection

Together, these options allow for a lot more flexibility, such as not
taking locks when snapshotting large databases as well as potentially to
use exported snapshots.

A future version should perhaps make some options possible without
implementing the SPI (such as using the exported snapshot by default)
but this commit is just trying to get all the groundwork in place to
allow for the Snapshotter SPI to have this functionality.
 2019-07-09 21:14:28 +02:00
Gunnar Morling
369f74d875 DBZ-1320 Addressing review remarks 2019-07-09 21:07:18 +02:00
Brandon Brown
efff4f0809 DBZ-1320 Adding support for deletion events via new configurable property route.tombstone.on.empty.payload 2019-07-09 21:07:18 +02:00
Gunnar Morling
50e3a02d4d DBZ-1371 README updates 2019-07-09 10:13:21 +02:00
Gunnar Morling
8c5d5021f4 DBZ-1371 Restoring original timeouts 2019-07-09 10:13:21 +02:00
Chris Cranford
faa31b079d DBZ-1371 Added link to RDS.md from README.md 2019-07-09 10:13:21 +02:00
Chris Cranford
bf248e6fb0 DBZ-1371 Update to PostgreSQL connector RDS document 2019-07-09 10:13:21 +02:00
Chris Cranford
af513c2732 DBZ-1371 Derive timeouts from configurable timeout 2019-07-09 10:13:21 +02:00
Chris Cranford
f88970f2e6 DBZ-1371 Fix formatting and typos in RDS documentation 2019-07-09 10:13:21 +02:00
Chris Cranford
051d5a478d DBZ-1371 Added RDS setup documentation for PostgreSQL 2019-07-09 10:13:21 +02:00
Chris Cranford
a50b9847ac DBZ-1371 Fix tests to run more reliably on Amazon RDS 2019-07-09 10:13:21 +02:00
Renato Mefi
c990256fd1 DBZ-1379 Fix PostgreSQL false empty password Warn 2019-07-08 12:51:49 +02:00
Gunnar Morling
e2d8736a79 DBZ-1380 Misc. clean-up; 
* reducing visibility of some methods
* resetting interrupted state
 2019-07-05 21:08:35 +02:00
Jiri Pechanec
5dc4812ffa DBZ-1380 Guard against partially initilized replication slot 2019-07-05 21:08:35 +02:00
Jiri Pechanec
13d638fd43 DBZ-1356 Metrics for schema history (#947) 
* DBZ-1356 Metrics for schema history
 2019-07-02 13:26:17 +02:00
Jiri Pechanec
b1c1c05d1f DBZ-1363 Generate heartbeat periodically not upon event 2019-07-02 12:02:33 +02:00
Gunnar Morling
08c5d36789
Merge pull request #942 from jpechane/stab-lsn-flush-test 
DBZ-1362 Stabilize test executions on CI
 2019-07-01 09:34:24 +02:00
Gunnar Morling
a02daae3c1 DBZ-1362 Comment 2019-07-01 09:34:01 +02:00
Chris Cranford
61f5e15889 DBZ-1366 Avoid creating unnecessary TypeMetadataImpl instances; only init once. 2019-06-28 13:44:53 -04:00
Jiri Pechanec
b546dc886e DBZ-1362 Fix Snapshot/streaming switch race condition 2019-06-28 09:09:18 +02:00
Jiri Pechanec
e0b188f80a DBZ-1362 Drop replication slot after test execution 2019-06-28 05:41:07 +02:00
Jiri Pechanec
046064d653 DBZ-1362 Make toast test compatible with PostgreSQL 11 2019-06-28 05:32:11 +02:00
Jiri Pechanec
412ce20638 DBZ-1362 Switch to Awaitility implementation 2019-06-28 05:19:01 +02:00
Jiri Pechanec
5a5d969e99 DBZ-1362 Wait for real LSN change during test 2019-06-28 05:19:01 +02:00
Jenkins user
e95d656bca [maven-release-plugin] prepare for next development iteration 2019-06-27 07:36:13 +00:00
Jenkins user
d95fd87311 [maven-release-plugin] prepare release v0.10.0.Beta2 2019-06-27 07:36:12 +00:00
Guillaume Rosauro
dc81d3c8ac DBZ-1352 Removing the unused topic selection strategy configuration from Postgres connector 2019-06-26 16:24:38 +02:00
Jiri Pechanec
355c8a94ac DBZ-1347 JavaDoc update 2019-06-25 14:52:58 +02:00
Gunnar Morling
62fc03a44d DBZ-1347 Commenting 2019-06-25 14:52:58 +02:00
Jiri Pechanec
e72d7edd2f DBZ-1347 Switch to non-blocking stream read 2019-06-25 14:52:58 +02:00
Gunnar Morling
e05ef934a0 DBZ-1345 Adding @FixFor 2019-06-25 12:44:00 +02:00
Chris Cranford
d518fa59fe DBZ-1345 Should not snapshot materialized views 2019-06-25 12:44:00 +02:00
Jiri Pechanec
2dfdf5cb12 DBZ-1342 Partitioned table is included in snapshot 2019-06-21 12:26:40 +02:00
Gunnar Morling
21176ff70f Adding link to replication API 2019-06-21 11:14:35 +02:00
Gunnar Morling
deaa5b7f4d DBZ-1337 Adding support for HSTORE arrays 2019-06-19 17:01:13 +02:00
Jiri Pechanec
badae96e1d DBZ-1335 Process unknown datatypes for snapshotted arrays 2019-06-18 16:17:49 +02:00
Jenkins user
5fdb0ca81b [maven-release-plugin] prepare for next development iteration 2019-06-11 11:20:46 +00:00
Jenkins user
61729b9e00 [maven-release-plugin] prepare release v0.10.0.Beta1 2019-06-11 11:20:46 +00:00
Jiri Pechanec
6a3c87e2f1 DBZ-1323 Initialize commitTime for out-of-order chunk 2019-06-11 10:27:56 +02:00
Gunnar Morling
5d01f409f0 DBZ-1317 Separating update and query methods for MongoDB's SourceInfo; 
* Avoids access to collection id after events which don't have one
* Pulling up struct() and schema() methods to common base class
 2019-06-11 10:07:36 +02:00
Chris Cranford
0aadaa10dc DBZ-1242 Fixed typo and added negative test checks 2019-06-05 13:31:25 -04:00
Chris Cranford
b3ca33287a DBZ-1242 Issue warning when filters produce an empty schema 2019-06-05 13:31:25 -04:00
Jenkins user
056730d06a [maven-release-plugin] prepare for next development iteration 2019-06-03 10:24:08 +00:00
Jenkins user
7aeaec236e [maven-release-plugin] prepare release v0.10.0.Alpha2 2019-06-03 10:24:08 +00:00
Gunnar Morling
246c1be305 DBZ-1283 Enforcing presence of database.server.name globally 2019-06-03 10:58:14 +02:00
Chris Cranford
5f0b44b4ec DBZ-1283 Only validate history topic name for affected connectors 2019-06-03 10:58:14 +02:00
Chris Cranford
b4736a7e03 DBZ-1283 Only validate history topic name for affected connectors 2019-06-03 10:58:14 +02:00
Gunnar Morling
73590173b5 DBZ-1310 Removing methods only needed for testing 2019-05-31 18:09:41 -04:00
Jiri Pechanec
4b64c8e447 DBZ-1295 3state snapshot for MySQL 2019-05-31 10:56:05 +02:00
Jiri Pechanec
348b2415d5 DBZ-1295 3state snapshot for PostgreSQL 2019-05-31 10:56:05 +02:00
Jiri Pechanec
946019982e DBZ-1303 Assembly without database drivers 2019-05-30 14:32:47 +02:00
Gunnar Morling
7759333fe3 DBZ-1092 Using ColumnNameFilter instead of Predicate<Column> 2019-05-29 15:49:49 +02:00
Jenkins user
ac5c76e431 [maven-release-plugin] prepare for next development iteration 2019-05-28 14:38:32 +00:00
Jenkins user
35f59ba460 [maven-release-plugin] prepare release v0.10.0.Alpha1 2019-05-28 14:38:32 +00:00
Gunnar Morling
8ecfbee77d DBZ-1235 Adding TODO marker 2019-05-28 09:35:15 +02:00
Gunnar Morling
b8eede2f41 DBZ-1235 Misc. clean-up; 
* Putting collection name after replica set name for MongoDB
* Removing redundant method for converting schema to string
* Removing redundant keys from PG SourceInfo
 2019-05-28 09:35:15 +02:00
Jiri Pechanec
526e780513 DBZ-1235 Misc fixes 2019-05-28 09:35:15 +02:00
Gunnar Morling
eef37cb567 DBZ-1235 Misc adjustments; 
* Removing superfluous parameter from getSourceInfoStructMaker()
* JavaDoc
* Renaming field to "source.struct.version"
 2019-05-28 09:35:15 +02:00
Gunnar Morling
8f28f323e7 DBZ-1235 Using Instant to represent timestamps internally; 
Only "rendered" as epoch milli-seconds at the system boundaries.
 2019-05-28 09:35:15 +02:00
Jiri Pechanec
6f6c8ddf06 DBZ-1235 Refactored to unify the code 2019-05-28 09:35:15 +02:00
Jiri Pechanec
c1bf1752fe DBZ-1235 Pluggable source maker for MongoDB 2019-05-28 09:35:15 +02:00
Jiri Pechanec
df16ed84cd DBZ-1235 Pluggable source maker for PostgreSQL 2019-05-28 09:35:15 +02:00
Jiri Pechanec
f5757dbd9f DBZ-1235 Extract serverName to common source 2019-05-28 09:35:15 +02:00
Gunnar Morling
4ed8b4636c DBZ-1224 Completing implementation; 
* Further unification of override handling;
* Undoing change related to SnapshotStatementFactory as it seems not needed
* Adding test
 2019-05-27 08:44:42 +02:00
mokkara
b3de0250a9 DBZ-1224 Snapshot SELECT support for SQL Server 2019-05-27 08:44:42 +02:00
Gunnar Morling
a0e30b7c90 DBZ-1289 Making outbox router options more consistent 2019-05-24 13:18:55 +02:00
Chris Cranford
e2c74111ad DBZ-1234 Removed deprecated configuration options 2019-05-22 10:19:05 +02:00
Gunnar Morling
4751b266f3 DBZ-1164 Misc. improvements; 
* Moving stringToDuration() for the sake of re-use and testability
* Making pattern and durations constants
 2019-05-21 09:38:23 +02:00
GraySmog
9b52ac86e8 DBZ-1164 Supporting TIME columns with value of "24:00:00" 2019-05-21 09:38:09 +02:00
Jenkins user
7748be6056 [maven-release-plugin] prepare for next development iteration 2019-05-02 16:04:56 +00:00
Jenkins user
5af083b86e [maven-release-plugin] prepare release v0.9.5.Final 2019-05-02 16:04:55 +00:00
Gunnar Morling
d12926c404 DBZ-1214 Applying default status update interval 2019-05-02 11:30:16 +02:00
Gunnar Morling
6960c880ca DBZ-1214 Misc. clean-up: 
* Using parker instead of Thread#sleep()
* Using flag for executor cancelation
* Using unique context name in thread name
 2019-05-02 11:30:16 +02:00
Addison Higham
e1570dd330 [DBZ-1214] Ensure PG connection keep alive 
This adds a small change to the postgres connector which
works around a problem when large databases take more
than a minute to refresh the schema

This works by creating a simple background thread that ensures we send
statusUpdate messages periodically

I considered if we could refresh the schema before we create the
replication connection but I believe we choose our offset in the log
and then get the schema in order to ensure we don't get out of sync with
any potential schema changes

This also makes some minor refactors to use a Duration instead of an
integer for the statusUpdateInterval
 2019-05-02 11:30:16 +02:00
Jiri Pechanec
3b78e0bebb DBZ-1245 Fix empty heartbeat message test 2019-05-02 09:16:21 +02:00
Jiri Pechanec
b53f456912 DBZ-1245 Heartbeat only when offsets ready 2019-05-02 08:22:28 +02:00
Gunnar Morling
2f7cab6eb3 DBZ-1260 Removing some dead code 2019-05-02 07:28:23 +02:00
Chris Cranford
dc61e52bc5 DBZ-1260 Fix timetz microsecond precision support 2019-05-02 07:28:23 +02:00
Chris Cranford
7602136cf6 DBZ-1236 Removed millisecond truncation. 2019-05-01 14:16:17 +02:00
Chris Cranford
16840d7d59 DBZ-1236 Handle timez precision consistently. 2019-05-01 14:16:17 +02:00
Jiri Pechanec
3ff81348b8 DBZ-1236 Support full timestamptz precision 2019-05-01 14:16:17 +02:00
Gunnar Morling
a84230447e DBZ-1247 Making defaultSnapshotFetchSize() abstract 2019-04-30 09:57:11 +02:00
jchipmunk
72b06366fe Removing extra initialisation of "config" variable in PG config 2019-04-30 09:57:11 +02:00
jchipmunk
9fe033ce24 DBZ-1247 Specifying default snapshot fetch size for each connector config 2019-04-30 09:57:11 +02:00
jchipmunk
a5cc83c0d1 DBZ-1247 Adding property to specify snapshot fetch size 2019-04-30 09:57:11 +02:00
jorkzijlstra
d0e2a72cd1 DBZ-1241 Allowing null values again for commitTime 2019-04-30 09:45:10 +02:00
shubham
f55bec86d3 Added support for all range types 2019-04-24 07:40:03 +02:00
krizhan
70c564b4c2 DBZ-212 Moved SERVER_NAME to RelationalDatabaseConnectorConfig 2019-04-23 12:29:44 +02:00
krizhan
38f3856c31 DBZ-212 Remove PostgreSQL getLogicalName function 2019-04-23 12:29:44 +02:00
Dennis Campagna
173e8f5854 DBZ-1034 Support range column types on Postgres 
Signed-off-by: sciamp <denniscampagna@gmail.com>

DBZ-1034 Support range column types on Postgres

Signed-off-by: sciamp <denniscampagna@gmail.com>
 2019-04-12 09:44:17 +02:00
GraySmog
726a2afcd6 Add comments stating the reason for skipping tests 2019-04-12 09:27:45 +02:00
GraySmog
08ca02f8f4 DBZ-1193 Add support for network type MACADDR and MACADDR8 in PostgreSQL connnector 2019-04-12 09:27:45 +02:00
Jenkins user
cefc5ccfb9 [maven-release-plugin] prepare for next development iteration 2019-04-11 12:07:16 +00:00
Jenkins user
fc3026adc3 [maven-release-plugin] prepare release v0.9.4.Final 2019-04-11 12:07:15 +00:00
Gunnar Morling
2250c47b7f DBZ-1163 Minor improvements 2019-04-10 10:53:22 +02:00
preethi
02873e8169 DBZ-1163 Enable postgres snapshot for tables without primary key 2019-04-10 10:48:11 +02:00
ShubhamRwt
762d908eef DBZ-1193 Added support for cidr address type 2019-04-08 15:00:03 +02:00
jchipmunk
7c0ae3ee20 DBZ-1212 SLF4J usage issues 2019-04-04 21:32:12 +02:00
Jiri Pechanec
ff52f3713e DBZ-1144 Test for geometry in public schema 2019-04-02 15:57:57 +02:00
Alexander Kovryga
4616df189e DBZ-1189 Add support for inet data type. 2019-03-28 09:57:32 +01:00
ShubhamRwt
540a951211 DBZ- 362 Adding WhitespaceAfter check to Checkstyle 2019-03-28 09:24:11 +01:00
Jenkins user
8b0edd1871 [maven-release-plugin] prepare for next development iteration 2019-03-25 12:16:31 +00:00
Jenkins user
56f98a48b2 [maven-release-plugin] prepare release v0.9.3.Final 2019-03-25 12:16:30 +00:00
Renato Mefi
6300f2972f DBZ-1169 Integration tests for Outbox EventRouter 
Set the Snapshot Mode to NEVER provides more consistent tests
 2019-03-25 11:27:44 +01:00
Renato Mefi
8b689d7f03 DBZ-1169 Implement RegexRouter based on config 2019-03-25 11:27:44 +01:00
Renato Mefi
f0f13ed4a4 DBZ-1169 Implement basic behavior 2019-03-25 11:27:44 +01:00
Renato Mefi
2ee1c98298 DBZ-1169 Setup EventRouter on Postgres Test IT 2019-03-25 11:27:44 +01:00
shubham
8cca21e969 DBZ-1039 Make all ifs with braces 2019-03-25 10:23:14 +01:00
Addison Higham
8e21139ca3 DBZ-1082 Add new custom recovery mode, more metadata 
This commit does a few things:

- Refactors snapshot modes to be encapsulated by an interface and
to only use that interface in determining when to snapshot and in
determing the type of the `RecordProducer` interface to instantiate
- Refactors the configuration of existing snapshot modes to tie the
existing snapshot modes to their aligned implementation
- Adds a new snapshot.mode, custom, and a new configuration option to
specify a custom implementation that will be loaded by the class loader
- Changes the visibility of some classes to allow for custom snapshot
modes to get enough context to make an informed choice
- Adds some metadata about slots (the catalog_xmin) to give a full idea
of the state of slots which can be useful in implementing snapshot
modes (which is also configurable, as it can add some overhead)

Together, these changes allow for a much broader flexibility got end
users to implement a snapshot mode that can do more advanced snapshots,
such as partial recovery or for partial snapshots for tables where not
all records are needed.

This could also be seen as superseeding the
`snapshot.select.statement.overrides` to allow for users to dynamically
build queries based on the state of the slot and the offsets consumed.
 2019-03-24 11:56:40 +01:00
Jiri Pechanec
b11f1b8d05 DBZ-1161 Start snapshot in async thread 2019-03-21 14:32:47 +01:00
trolison
2001b178d7 - Fixed bad formatting 
- Added trace for when empty messages are processed
DBZ-1181
 2019-03-19 10:46:36 -07:00
trolison
df0e789601 - Add support for processing WAL2JSON events that have no changes 
DBZ-1181
 2019-03-18 20:32:03 -07:00
Gunnar Morling
2ea87727cf DBZ-1174 Using Instant instead of long for representing timestamps 2019-03-12 17:19:13 +01:00
Ashhar Hasan
0b07afd23e DBZ-1174 Convert commitTime from nanos to micros for wal2json 
- wal2json sends the txn commitTime using a function from PostgreSQL's C
  library. The value that Debezium recevies is in nanoseconds.
- decoderbufs sends the txn commitTime in microseconds.
- RecordsSnapshotProducer updates SourceInfo.ts_usec by converting
  System.currentTimeMillis() to microseconds.
- RecordsSnapshotProducer updates the SourceInfo's ts_usec field using
  message.getCommitTime().

This means that when using wal2json, the value of SourceInfo.ts_usec
is in microseconds since epoch during snapshot but is in nanoseconds
during streaming. To fix this, we changed
Wal2JsonReplicationMessage.getCommitTime() to return in microseconds.
 2019-03-12 17:19:13 +01:00
Josh Stanfield
e4b7b90637 DBZ-1162 fix to ensure hstore snapshots behave the same as streams 2019-03-04 13:39:50 +01:00
Jenkins user
50a7c568fa [maven-release-plugin] prepare for next development iteration 2019-02-23 09:53:58 +00:00
Jenkins user
28f3839804 [maven-release-plugin] prepare release v0.9.2.Final 2019-02-23 09:53:58 +00:00
Jiri Pechanec
78bfda09a6 DBZ-1146 Test full column diff 2019-02-22 14:16:03 +01:00
Gunnar Morling
5043d41b7b DBZ-1146 Expanding test 2019-02-22 14:16:03 +01:00
Jiri Pechanec
4dec844fa2 DBZ-1146 Fix table created in streaming test 2019-02-22 14:16:03 +01:00
Jiri Pechanec
fa1e632a90 DBZ-1146 Handle toastable columns for snapshotted tables 2019-02-22 14:16:03 +01:00
Jiri Pechanec
89feae32e7 DBZ-1141 Test for the default values 2019-02-21 15:02:51 +01:00
Gunnar Morling
6787d17d54 DBZ-1141 Misc. clean-up; 
* Removing redundant check for date mapping type
* Always using String as fallback value for temporal values where needed
* Pulling fallback temporal values up to JdbcValueConverters
 2019-02-21 15:02:51 +01:00
Jiri Pechanec
b9d326711d DBZ-1141 Incorrect money scale 2019-02-21 15:02:51 +01:00
Jiri Pechanec
992c31fa1f DBZ-1141 Add non-date types 2019-02-21 15:02:51 +01:00
Jiri Pechanec
41ba7128e3 DBZ-1141 Correct default falback values for connect date types 2019-02-21 15:02:51 +01:00
Keith Barber
95b2290df2 DBZ-1149 Ensure consistent Java types for fallback values when converting 2019-02-21 15:02:51 +01:00
taylor-rolison
17c6a7a82d Config option to pass optional logical decoder stream params (#783) 
DBZ-1130 Added config param slot.stream.params to the Postgres connector to allow passing optional params to logical decoder on stream start
 2019-02-21 14:06:51 +01:00
Jenkins user
b96e5fee0d [maven-release-plugin] prepare for next development iteration 2019-02-13 11:13:44 +00:00
Jenkins user
3de2d04bcd [maven-release-plugin] prepare release v0.9.1.Final 2019-02-13 11:13:44 +00:00
Jiri Pechanec
9049078607 DBZ-997 Handle de-sync of message and JDBC PK info 2019-02-13 10:13:12 +01:00
Gunnar Morling
2a486ecfff DBZ-1019 Removing legacy methods from Metronome 2019-02-11 18:11:34 +01:00
Ivan Lorenz
721e8112cd DBZ-1126 Localize era display value to default locale 2019-02-06 15:34:30 +01:00
Gunnar Morling
7076deba5a DBZ-1118 Adding JUnit rule to skip tests depending on Postgres version 2019-02-04 14:43:19 +01:00
Kevin Pullin
c414a8b942 Add test for DBZ-1118 2019-02-04 14:43:19 +01:00
Kevin Pullin
1330df8908 Postgres snapshot - On null records, clear prior value from currentRecord 2019-02-04 14:43:19 +01:00
Gunnar Morling
41b7fbe78e Adding note on testing Postgres connector with external DB 2019-01-30 09:25:12 +01:00
Gunnar Morling
cc0df61665 DBZ-1111 Allowing to use drop-slot-on-close with PG 10 or newer 2019-01-29 14:44:10 +01:00
Gunnar Morling
e84c6e2238 DBZ-962 Avoiding dependency to JAXB 2019-01-29 10:28:48 +01:00
Jenkins user
c5b7d21d3e [maven-release-plugin] prepare for next development iteration 2019-01-28 11:07:04 +00:00
Jenkins user
f1ae79ff73 [maven-release-plugin] prepare release v0.9.0.CR1 2019-01-28 11:07:04 +00:00
Gunnar Morling
ccff79e07d DBZ-1105 Removing TableImpl#columnNames field 2019-01-28 10:23:56 +01:00
Gunnar Morling
9dc2c525c9 DBZ-1083 Passing "include-unchanged-toast" option only for wal2json 2019-01-25 11:04:43 +01:00
Addison Higham
26f836e0f6 DBZ-1080 new API for EmbeddedEngine for batch/async 
This introduces a new API to the EmbeddedEngine, the ChangeConsumer,
which gives the user a more flexible option for consuming changes by
exposing groups of records as well as the ability to control the
comitting of those records.

This remainds completely backwards compatible with the old API as the
ChangeConsumer wraps the existing Consumer interface with a default
implementation.
 2019-01-18 14:06:50 +01:00
Jiri Pechanec
1cf257478b DBZ-781 Enable unstable tests 2019-01-17 08:59:42 +01:00
Tony Rizko
a921613877 Fix typo in PostgresConnection error message 2019-01-17 08:58:12 +01:00
Jiri Pechanec
ceb6c62721 DBZ-1064 Remove lazy fallback values 2019-01-16 13:01:36 +01:00
Jiri Pechanec
536e517c08 DBZ-1064 Geometry default value changes; fixes for default suppliers 2019-01-16 13:01:36 +01:00
Gunnar Morling
532c17f8aa DBZ-1064 Using dedicated functional interface instead of Consumer; 
* Commenting
* Moving type from "util" to "jdbc"
 2019-01-16 13:01:36 +01:00
Jiri Pechanec
bff7eea734 DBZ-1064 Use convert value template 2019-01-16 13:01:36 +01:00
Jiri Pechanec
920e0cbd3f DBZ-1064 Set default value only for non-nullable fields 2019-01-16 13:01:36 +01:00
Jiri Pechanec
484a8977bc DBZ-1083 Passing "include-unchanged-toast" option only if supported; 
The "include-unchanged-toast" option was removed in recent wal2json versions, without a transition phase. So the connector tries not to connect first giving the option, and if that fails, without it.
 2019-01-16 09:37:55 +01:00
Gunnar Morling
af01d10886 DBZ-1081 JavaDoc; adding Addison Higham to COPYRIGHT.txt 2019-01-15 18:09:08 +01:00
Addison Higham
889b307b9d DBZ-1081 Fallback to restart_lsn in pg 9.5 
In pg 9.5, confirmed_flush_lsn is not availiable. However, there is
restart_lsn, which should be safe to use with a downside of producing
some extra duplicate records.

This page: https://paquier.xyz/postgresql-2/postgres-9-6-feature-highlight-replication-slot-improvements/
indicates as such and this allows for DBZ to support PG 9.5
 2019-01-15 18:05:55 +01:00
Gunnar Morling
de356896c0 DBZ-1029 Adding one more test 2019-01-15 17:41:42 +01:00
Ilia Bogdanov
88521371cc DBZ-1029 insertion test update, code cleanup after review 2019-01-14 12:03:19 +03:00
Ilia Bogdanov
cee9bd9a5d DBZ-1029 handling messages from tables with replica identity full in postgres connector. 2019-01-14 12:03:19 +03:00
Amit Sela
b0b7d942a3 DBZ-1073 Adding scale as schema parameter, if present 2019-01-11 15:40:30 +01:00
Grzegorz Kołakowski
91c72738c0 DBZ-1067 Add column blacklist field to RelationalDatabaseConnectorConfig 2019-01-09 18:13:26 +01:00
Jenkins user
5275f73424 [maven-release-plugin] prepare for next development iteration 2018-12-19 13:06:12 +00:00
Jenkins user
b6569c18ae [maven-release-plugin] prepare release v0.9.0.Beta2 2018-12-19 13:06:12 +00:00
Gunnar Morling
92ae01ccdf DBZ-996 Adding column name to log message 2018-12-18 17:53:42 +01:00
Jiri Pechanec
01eee09f90 DBZ-996 Tolerate race condition in PK updates 2018-12-18 17:52:28 +01:00
Jiri Pechanec
edec1c3090 DBZ-978 Added last event and captured tables metrics 2018-12-13 20:52:11 +01:00
Gunnar Morling
5542c0de29 DBZ-966 Making sure Postgres connector can be stopped while awaiting initial snapshot delay; 
* Also skipping snapshotting when requested to stop during initial delay
* Showing remaining delay time in logs
* Adding missing config
 2018-12-13 14:29:54 +01:00
Grzegorz Kołakowski
e2942ab72a DBZ-966 Replace sleep with polling loop 2018-12-13 14:29:54 +01:00
Grzegorz Kołakowski
35a5acfd8b DBZ-966 Enable to delay initial snapshot in PostgreSQL connector 2018-12-13 14:29:54 +01:00
Gunnar Morling
0833b23feb DBZ-842 Expanding test to cover TOASTed NOT NULL column, too 2018-12-11 18:42:51 +01:00
Jiri Pechanec
94c0a42286 DBZ-842 Do not send TOASTed values from wal2json 2018-12-11 17:10:58 +01:00
Jiri Pechanec
2b2ff0754d DBZ-960 Run SnapshotReader in a single transaction 2018-12-11 14:15:53 +01:00
Gunnar Morling
470cd1eaba DBZ-1024 Doing some clean-up 2018-12-07 15:39:59 +01:00
Jiri Pechanec
ea5c8512b7 DBZ-1024 Return test assert 2018-12-07 15:39:59 +01:00
Jiri Pechanec
0b78f0c31a DBZ-1024 Emit correct last_snapshot_record in source 2018-12-07 15:39:59 +01:00
Jiri Pechanec
9c225c63eb DBZ-1021 Ignore events for dropped table 2018-12-06 18:54:50 +01:00
Gunnar Morling
101bd8ef6d DBZ-1000 Making Checkstyle happy 2018-12-03 15:00:43 +01:00
Jiri Pechanec
7a3cbd6cb6 DBZ-1000 Detect nullability change in schema 2018-12-03 14:54:36 +01:00
Jiri Pechanec
68766e9355 DBZ-1009 Support for unknown types during snapshot 2018-11-30 11:16:36 +01:00
Jenkins user
db1d3a7fb8 [maven-release-plugin] prepare for next development iteration 2018-11-20 16:15:14 +00:00
Jenkins user
f83db82cea [maven-release-plugin] prepare release v0.9.0.Beta1 2018-11-20 16:15:14 +00:00
Jiri Pechanec
0611d4975f DBZ-965 Reduce the test time 2018-11-13 09:16:30 +01:00
Jiri Pechanec
a4abd2a336 DBZ-965 Test of flushed LSN 2018-11-13 09:15:09 +01:00
Gunnar Morling
5a9ca6eb31 DBZ-965 Some clean-up 2018-11-13 09:15:09 +01:00
Jiri Pechanec
675f5b6479 DBZ-965 Renames for better clarification 2018-11-13 09:15:09 +01:00
Gunnar Morling
b455b7ccaf DBZ-970 Making LogicalDecoder#getPostgresPluginName() abstract 2018-11-13 09:15:09 +01:00
Jiri Pechanec
390965f4f9 DBZ-965 Track latest LSN per each change 2018-11-13 09:15:09 +01:00
Jiri Pechanec
b1707c395d DBZ-970 Correct plging name for wal2json_streaming decoder 2018-11-13 09:15:09 +01:00
Jiri Pechanec
83137d0a41 DBZ-965 Force heartbeat after end of each TX 2018-11-13 09:15:09 +01:00
Gunnar Morling
efbd46b7f6 DBZ-953 Having PG connector use decimal mode option from RelationalDatabaseConnectorConfig 2018-11-08 09:23:28 +01:00
Jiri Pechanec
3a8293a4ba DBZ-937 Disable auto-handling of unknown types in test 2018-10-11 09:55:07 +02:00
Olavi Mustanoja
0c986b4a78 DBZ-937 add wal2json replication message handling for CITEXT 2018-10-09 08:46:55 +02:00
Jenkins user
d952f5dfb0 [maven-release-plugin] prepare for next development iteration 2018-10-04 11:59:14 +00:00
Jenkins user
ff9b70278b [maven-release-plugin] prepare release v0.9.0.Alpha2 2018-10-04 11:59:13 +00:00
Gunnar Morling
1f25536e2b DBZ-934 Using API for creating replication slot if possible 2018-10-04 10:12:48 +02:00
PSanetra
dd3859d8bc DBZ-934 Use temporary replication slots if slot.drop_on_stop is enabled and Postgres >= 10 is used 2018-10-04 10:12:48 +02:00
jchipmunk
3ec0b3be81 DBZ-918 Moving connector identifier from SourceInfo to Module class 2018-10-04 09:00:13 +02:00
jchipmunk
05ba99bb0a DBZ-918 Adding Debezium connector field to source info 
This will allow consumers to recognize the Debezium connector used for creating a given message, helping them to adjust their behavior for a variety of connectors.
 2018-10-04 09:00:13 +02:00
Gunnar Morling
984081dd1a DBZ-926 Adding flushed LSN to log message 2018-10-03 09:47:38 +02:00
Gunnar Morling
08ec4f48d0 DBZ-911 Style adjustments; wording fix 2018-10-01 12:49:56 +02:00
Ian Axelrod
427249887b DBZ-911 Simplifying schemaChanged() shortcircuit logic 
Break up initial conditional and comment missing column portion to
clarify the role of skipRefreshSchemaOnMissingToastableData.
 2018-10-01 12:49:48 +02:00
Ian Axelrod
9f083d0f1b DBZ-911 Simplifying getToastableColumnsForTableId() impl. 
No need to wrap ArrayList instance in unmodifiableList. Can just use
Collections.emptyList.
 2018-10-01 12:49:36 +02:00
Ian Axelrod
ad433c5747 DBZ-911 Don't refresh schema on absent TOAST cols 
Improves PostgreSQL RecordsStreamProducer performance for processing up
dates to a table with TOASTable columns, where those updates do not
affect said columns. Prior to this fix, these updates would trigger a
refresh of the in-memory table schema. In the worst case, this means a
query for every update. This puts significant load on the database
server and adds tens of milliseconds to the processing of each update
record.

The fix requires a new configuration option, called
'schema.refresh.mode'. This option has values 'columns_diff' (the
default) and 'columns_diff_exclude_unchanged_toast'. 'columns_diff'
maintains the pre-fix behavior. 'columns_diff_exclude_unchanged_toast'
activates the fix.

The fix must be toggleable because it decreases the consistency guarantees
Debezium provides, since in-memory table schemas may not stay
synchronized with their remote counterparts. With type metadata included
in the replication message, inconsistencies are limited to the unchanged
toast columns.
 2018-10-01 11:39:32 +02:00
Gunnar Morling
6f7bdbe5e1 DBZ-898 Using 'Json' semantic type when encoding HSTORE column values as string 2018-09-25 12:22:20 +02:00
Gunnar Morling
7150e387c7 DBZ-898 Misc. clean-up: 
* Using database encoding for string conversion
* Not making hstore schemas optional by default
* Using Jackson instead of GSon for JSON serialization
* Removing superfluous method and log messages
* Adjusting to naming and style conventions
 2018-09-24 13:36:28 +02:00
SyedMuhammadSufyian
623cc98cb0 Change log levels 2018-09-22 20:15:07 +02:00
SyedMuhammadSufyian
346f2b3761 Handling HStore type , hstore.handling.mode:map|json , default is map.Added support for protobuff 2018-09-22 20:07:13 +02:00
SyedMuhammadSufyian
38f4a0e65b HStore Data type support , configuration options are map|json , default is map 2018-09-22 05:54:45 +02:00
Gunnar Morling
f9ede861ba DBZ-912 Upgrading to Postgres driver 42.2.5 2018-09-19 15:26:44 +02:00
Gunnar Morling
39a47fabfa DBZ-899 Using custom statement to obtain SQL type info in order to avoid N+1 issues with PgDatabaseMetaData#getTypeInfo() 2018-09-19 09:51:38 +02:00
Gunnar Morling
3331d4c1a9 DBZ-899 Removing unused field; 
Also inlining method only used in one place.
 2018-09-19 09:51:38 +02:00
Jiri Pechanec
59fbcca725 DBZ-907 Log refreshing schema due to different column count 2018-09-12 19:07:01 +02:00
Jiri Pechanec
e20c183d5d DBZ-895 Make tcpKeepalive enabled by default 2018-09-11 09:34:21 +02:00
Gunnar Morling
e1bbbd7992 DBZ-865 DBZ-878 Replacing invalid topic name characters with "_"; 2018-09-05 12:56:53 +02:00
Gunnar Morling
bd692b8b84 DBZ-862 Extracting values for controlling retry attempts into constants 2018-09-03 14:50:54 +02:00
Jiri Pechanec
c6e20d4967 DBZ-862 Fix test interdependency problem 2018-09-03 14:40:57 +02:00
Jiri Pechanec
1ff53e35ec DBZ-862 Fix Travis timing issue 2018-09-03 14:40:57 +02:00
Jiri Pechanec
4b9141e1fd DBZ-862 Handle concurrent transaction blocking taking snapshot 2018-09-03 14:40:57 +02:00
Jiri Pechanec
e65fb26ae9 DBZ-887 Close connection even for non-started producer 2018-08-31 13:02:23 +02:00
Gunnar Morling
ae879cf475 DBZ-859 Adjustments after rebase; typo fix 2018-08-23 12:00:44 +02:00
Jiri Pechanec
cd6dc09e72 DBZ-859 Heartbeat is sent after PostgreSQL snapshot 2018-08-23 11:44:53 +02:00
oripwk
7d958a1928 DBZ-866 PR fixes 2018-08-22 17:09:26 +02:00
oripwk
d82372a6b2 DBZ-866 Add schema, database and table to Postgres connector 2018-08-22 17:09:26 +02:00
Jiri Pechanec
d4e64a61cb DBZ-705 Allow to configure postgres config file location 2018-08-13 20:21:41 +02:00
Jenkins user
e00dad127f [maven-release-plugin] prepare for next development iteration 2018-07-26 08:00:12 +00:00
Jenkins user
16bfd5c700 [maven-release-plugin] prepare release v0.9.0.Alpha1 2018-07-26 08:00:12 +00:00
Gunnar Morling
3061c1dc0b DBZ-815 Passing offset to Heartbeat#heartbeat(); 
That's simpler to grasp than the approach of passing a supplier lambda to the constructor.
Also it allows to pass on the offset via local variables instead of instance fields in some cases.
 2018-07-25 14:49:52 +02:00
Gunnar Morling
8e1450bc71 DBZ-800 Adding test for receiving heartbeat events with Postgres 2018-07-20 13:37:54 +02:00
Gunnar Morling
4abc471bcb DBZ-800 Emitting heartbeat events in Postgres streaming loop 2018-07-20 13:37:54 +02:00
Gunnar Morling
20db9299c5 DBZ-627 Unifying TopicSelector implementations 2018-07-18 12:36:07 +02:00
Gunnar Morling
1523f230ca DBZ-793 Using TableFilter consistently; 
Only the MySQL connector's Filters class still uses Predicate for the time being, so to avoid to much merging trouble with DBZ-175.
 2018-07-11 13:16:44 +02:00
Jenkins user
f9b8d830a8 [maven-release-plugin] prepare for next development iteration 2018-07-11 07:36:30 +00:00