Jiri Pechanec
6bab948e76
DBZ-1840 No nulls to stream producer
2020-03-04 09:25:57 +01:00
Jan-Hendrik Dolling
5cfbaab7a6
DBZ-1845 Typos in deployment.yaml and code readability
...
* improve readability
* fixed some typos
2020-03-04 08:34:55 +01:00
Gunnar Morling
c5422e6b31
DBZ-1840 Not passing null to replication message processor;
...
Note this means to that WAL offsets cannot be acknowledged in cases
where a null message is received from the server (e.g. wal2json for
DDL events). But as we cannot tell apart null events from the server
from readPending() simply having no events to read, the current
logic renders the pausing between two read calls without events
obsolete, so that's the lesser evil
2020-03-03 11:45:53 +01:00
Gunnar Morling
0f6d2c54fc
DBZ-1134 Avoiding concrete class in SPI by introducing receiver parameter
2020-02-28 21:58:39 +01:00
Jiri Pechanec
d13e48cc55
DBZ-1134 Make converted field type parametrized
2020-02-28 21:58:39 +01:00
Jiri Pechanec
06e1eca085
DBZ-1134 Support for user provided converters
2020-02-28 21:58:39 +01:00
Jiri Pechanec
91f292a1f2
DBZ-1825 Keep keys for non-matching table
2020-02-26 15:03:03 +01:00
Jiri Pechanec
ae6ee3d458
DBZ-234 Place Incubating into a unique package
2020-02-25 15:31:10 +01:00
Jiri Pechanec
2b6e50daff
DBZ-1760 Allow skipping of invalid events
2020-02-25 10:43:11 +01:00
jcechace
3209de3e5f
DBZ-924 Adding integration tests for OpenShift and Strimzi
2020-02-20 12:48:08 +01:00
Gunnar Morling
eff3ffc345
DBZ-1758 Documentation update
2020-02-14 13:14:25 +01:00
Debezium Builder
f75746fb7b
[maven-release-plugin] prepare for next development iteration
2020-02-13 03:18:20 -05:00
Debezium Builder
1d2df32743
[maven-release-plugin] prepare release v1.1.0.Beta2
2020-02-13 03:18:20 -05:00
Gunnar Morling
dd5c2b8be9
DBZ-1766 Unifying handling of offset commits
2020-02-12 15:44:45 +01:00
Melissa Winstanley
ef55c9157d
DBZ-1766 Update lastOffset to commit only after a record is committed
2020-02-12 15:44:45 +01:00
Jiri Pechanec
d4d38dcbec
DBZ-1783 Ignore empty transactions
2020-02-12 14:58:15 +01:00
Jiri Pechanec
40506d5352
[ci] Reduce postgres test logging for CI
2020-02-12 14:49:55 +01:00
Debezium Builder
9c424b2404
[maven-release-plugin] prepare for next development iteration
2020-02-05 08:54:52 -05:00
Debezium Builder
9f2caa3b7c
[maven-release-plugin] prepare release v1.1.0.Beta1
2020-02-05 08:54:52 -05:00
Jiri Pechanec
7c9d78e597
DBZ-1727 Fix unused imports
2020-02-05 13:33:29 +01:00
Igor Gabaydulin
3890038c21
DBZ-1727 Reduce heartbeat interval in test
2020-02-05 13:33:29 +01:00
Igor Gabaydulin
a2a6525c84
DBZ-1727 Update heartbeat on changing non whitelisted table test
2020-02-05 13:33:29 +01:00
Igor Gabaydulin
2f6e522b0f
DBZ-1727 Fix PostgresConnectorIT#shouldFlushLsnOnEmptyMessage test
2020-02-05 13:33:29 +01:00
Igor Gabaydulin
b62acfd144
DBZ-1727 Apply code formatter
2020-02-05 13:33:29 +01:00
Igor Gabaydulin
9bea1e2638
DBZ-1727 Move null check to AbstractMessageDecoder
2020-02-05 13:33:29 +01:00
Igor Gabaydulin
fcc7a05579
DBZ-1727 Debezium Postgres connector does not apply keepalive lsn
2020-02-05 13:33:29 +01:00
Jiri Pechanec
77f9a61b95
DBZ-1751 Centralize envelope schema naming
2020-02-04 20:37:19 +01:00
Chris Cranford
c5902c3c32
DBZ-1158 Add test for REPLICA IDENTITY fallback values
2020-02-04 10:37:12 -05:00
Jiri Pechanec
1d9ac71a9e
DBZ-1698 Fix test timing issue
2020-02-03 22:38:02 +01:00
Jiri Pechanec
2a6ea1c41f
DBZ-1698 Handle toasted unknown column values
2020-02-03 22:38:02 +01:00
Jiri Pechanec
34d2979123
DBZ-1052 Stabilize test for changed heartbeats
2020-02-03 09:29:10 +01:00
Jiri Pechanec
010e0c6fa6
DBZ-1052 Flush LSN for skipped tx messages
2020-02-03 09:29:10 +01:00
Gunnar Morling
b66f314b3c
DBZ-1052 Misc. clean-up
...
* Doc updates
* Changed static method to instance
2020-02-03 09:29:10 +01:00
Jiri Pechanec
67e0079646
DBZ-1052 Expand known protobuf messages list
2020-02-03 09:29:10 +01:00
Jiri Pechanec
4fdda6c101
DBZ-1052 Add tx enable option to connector definitions
2020-02-03 09:29:10 +01:00
Jiri Pechanec
ba10a229b6
DBZ-1052 CE support for tx metadata in data messages
2020-02-03 09:29:10 +01:00
Jiri Pechanec
0e671d6167
DBZ-1052 Add counts per table to END
2020-02-03 09:29:10 +01:00
Jiri Pechanec
60056b0a94
DBZ-1052 Transaction block moved to envelope
2020-02-03 09:29:10 +01:00
Jiri Pechanec
5749acda7e
DBZ-1052 Remove unused fields; added JavaDoc
2020-02-03 09:29:10 +01:00
Jiri Pechanec
072a37e790
DBZ-1052 Support for full Postgres/protobuf
2020-02-03 09:29:10 +01:00
Jiri Pechanec
c0557b325b
DBZ-1052 Skip CE for transaction struct
2020-02-03 09:29:10 +01:00
Jiri Pechanec
c466143f35
DBZ-1052 Test for limited Postgres/protobuf
2020-02-03 09:29:10 +01:00
Jiri Pechanec
fcafd2f193
DBZ-1052 Transaction metadata for Postgres/wal2json
2020-02-03 09:29:10 +01:00
Jiri Pechanec
cd6a1f942d
DBZ-1052 Transaction metadata for Postgres/pgoutput
2020-02-03 09:29:10 +01:00
Jiri Pechanec
b0eb571f9f
DBZ-1052 Transaction metadata for SQL Server
2020-02-03 09:29:10 +01:00
Jiri Pechanec
7283f5d6c1
DBZ-1746 Gracefully handle unknown protobuf messages
2020-01-30 12:24:52 +01:00
Marc Zbyszynski
507aac0e0f
DBZ-1738 replaced casts to PgConnection with casts to interface BaseConnection
...
DBZ-1738 replaced casts to PgConnection with casts to interface BaseConnection
2020-01-27 16:47:27 +01:00
Debezium Builder
274bba9dc9
[maven-release-plugin] prepare for next development iteration
2020-01-16 05:40:24 -05:00
Debezium Builder
076df30108
[maven-release-plugin] prepare release v1.1.0.Alpha1
2020-01-16 05:40:24 -05:00
Gunnar Morling
8c33a4e147
DBZ-1292 Catering for the differences in data schema;
...
* MongoDB doesn't have "before", but "patch"
* Also removing some redundancies and making RecordParser immutable
* Reducing log verbosity during tests
2020-01-16 08:01:56 +01:00
Gunnar Morling
1a18bdbc81
DBZ-1292 Misc. refactorings;
...
* Protecting against Avro as CE format and JSON as data format
* Reducing mutability
* Single Avro client
2020-01-16 08:00:45 +01:00
GraySmog
d0e4b3c0af
DBZ-1292 Delete data.schema.registry.url config option
2020-01-16 08:00:45 +01:00
GraySmog
915c8cafd9
DBZ-1292 CloudEvents converter for JSON and Avro format
2020-01-16 08:00:45 +01:00
Gunnar Morling
07e89de91d
DBZ-1707 Ensuring correct Kafka record timestamp for outbox events;
...
Depending on the width of the configured TIMESTAMP column, a microsecond or nanosecond timestamp could have been returned before, whereas this should always be milliseconds.
2020-01-15 07:32:55 +01:00
Chris Cranford
1d41d53636
DBZ-1685 Document and add support for column.mask.with for PostgreSQL connector
2020-01-14 09:30:34 +01:00
Chris Cranford
9fc1eada11
DBZ-1600 Remove deprecated "slot.drop_on_stop" postgres configuration option
2020-01-14 09:27:32 +01:00
Chris Cranford
cf60466b02
DBZ-1684 Make sure publication initialized prior to slot
...
This is to avoid the need to restart the connector when streaming begins due to
an error that the publication can't be found. It would seem based on testing
that since we create the replication slot early on when it does not exist, then
when streaming starts its confused about trying to marry a newly created
publication to an slot. By creating the publication before the slot, this
works around the problem.
2020-01-09 15:55:22 +01:00
Chris Cranford
3413d674fb
DBZ-1684 Create missing publication when replication slot exists
2020-01-07 15:41:15 +01:00
Debezium Builder
9723c93db9
[maven-release-plugin] prepare for next development iteration
2019-12-18 01:44:33 -05:00
Debezium Builder
09049516f2
[maven-release-plugin] prepare release v1.0.0.Final
2019-12-18 01:44:33 -05:00
Chris Cranford
5c6f8fdd70
DBZ-1680 Fix enum value resolution across all PostgreSQL decoders
2019-12-17 18:21:17 +01:00
Chris Cranford
debcb95fa2
DBZ-1677 Include root cause when PostgreSQL schema refresh fails during streaming
2019-12-17 07:19:02 +01:00
Gunnar Morling
d40641b3ab
DBZ-1666 Formatting
2019-12-12 16:39:10 +01:00
Gunnar Morling
4619f5eb9d
DBZ-1666 Wording fix
2019-12-12 11:56:05 +01:00
Jiri Pechanec
ffa15b100f
DBZ-1666 Warn for drop slot on stop
2019-12-12 11:52:30 +01:00
Debezium Builder
b1c62fe306
[maven-release-plugin] prepare for next development iteration
2019-12-10 06:39:17 -05:00
Debezium Builder
ad2efdb9a6
[maven-release-plugin] prepare release v1.0.0.CR1
2019-12-10 06:39:17 -05:00
Chris Cranford
c32b771e1b
DBZ-1413 Fix format violation
2019-12-10 10:09:35 +01:00
Chris Cranford
19fd8dd1d9
DBZ-1413 Resolve column length/scale when reading table column metadata from TypeRegistry
2019-12-10 10:09:35 +01:00
Chris Cranford
372d199a71
DBZ-1413 Revert AbstractColumnValue changes in commit 774f2ed
2019-12-10 10:09:35 +01:00
Chris Cranford
365eb0d76e
DBZ-1413 Suggested Javadoc changes, added/fix tests, changed how Column native/jdbc type resolution happens
2019-12-10 10:09:35 +01:00
Chris Cranford
9685990fb8
DBZ-920 Add support for PostgreSQL enum types
2019-12-10 10:09:35 +01:00
Chris Cranford
46de58ae75
DBZ-1413 Updated tests
2019-12-10 10:09:35 +01:00
Chris Cranford
ef9b4a7431
DBZ-1413 Delay building type hierarchy after all base types built
2019-12-10 10:09:35 +01:00
Chris Cranford
547ba8e0db
DBZ-1413 Align domain type support across all 3 decoders
2019-12-10 10:09:35 +01:00
Chris Cranford
9586be7d4e
DBZ-1413 Support PostgreSQL domain types
2019-12-10 10:09:35 +01:00
Thomas Deblock
17e44d4c0c
DBZ-1648 outbox EventRouter remove topic toLowerCase
...
- kafka topics names are case sensitive. We should not perform
a toLowerCase on the topic name. It can cause some issue if
topic name have a upper case
2019-12-04 21:39:25 +01:00
Jiri Pechanec
d471f2f6ab
DBZ-1637 Support for UUID array
2019-11-28 09:06:23 +01:00
Brandon Brown
a0f7f6ec6b
DBZ-1402 Support for JDK 11
2019-11-25 11:53:19 +01:00
Jiri Pechanec
a56ab78b3f
DBZ-825 Unify decimal upscaling
2019-11-20 18:23:55 +01:00
Debezium Builder
38279cc2dd
[maven-release-plugin] prepare for next development iteration
2019-11-14 02:56:07 -05:00
Debezium Builder
868faf4412
[maven-release-plugin] prepare release v1.0.0.Beta3
2019-11-14 02:56:07 -05:00
David Feinblum
2b9fbcd5f6
DBZ-1519 Avoid erroneously logging there are no monitored tables
2019-11-14 05:44:37 +01:00
Chris Cranford
95927de405
DBZ-1596 Fix open resources while initializing publication
2019-11-08 12:55:24 +01:00
Jiri Pechanec
d693609779
DBZ-1570 handle null toasted values for wal2json
2019-11-08 10:52:57 +01:00
Jiri Pechanec
4084e39256
DBZ-1595 New parameter takes precedence over old
2019-11-08 07:57:06 +01:00
Jiri Pechanec
54cb9e8326
DBZ-1595 Use dots as separator in param name
2019-11-08 07:57:06 +01:00
Jeremy Finzel
c2f27cdb23
DBZ-1559 Change snapshot lock level to ACCESS SHARE
...
This is the correct mode that only prevents concurrent schema changes, while not requiring debezium user to also have UPDATE access to tables as does SHARE UPDATE EXCLUSIVE mode
2019-11-05 11:54:52 +02:00
Chris Cranford
244ee9dc5c
DBZ-1576 Gracefully skip TRUNCATE messages with pgoutput decoder on PG11+
2019-11-05 09:24:08 +02:00
Gunnar Morling
2c569977d7
DBZ-1392 Applying formatter;
...
* Keeping indentation formatting for JSON
Co-authored-by: Gunnar Morling <gunnar.morling@googlemail.com>
Co-authored-by: Chris Cranford <chris@hibernate.org>
2019-11-01 13:20:43 +02:00
Addison Higham
b1b657b8bd
[DBZ-1579] Fix regression of large refreshes causing connection timeout
...
In DBZ-1214, an issue was fixed where during the initial schema refresh,
the replication stream connection would timeout. This occurred in cases
of many schemas that need to be refreshed.
This was fixed by adding a keep alive thread that ensure that we
periodically send an empty update to the DB.
It appears that in the refactor to move the new shared interface, the
call to this keep alive thread was removed, but the remaining keep alive
interface still exists
This simply wires the keep alive thread back up
2019-10-29 18:27:55 +01:00
Debezium Builder
b08f523ed7
[maven-release-plugin] prepare for next development iteration
2019-10-24 07:42:49 -04:00
Debezium Builder
8ecd9e28cf
[maven-release-plugin] prepare release v1.0.0.Beta2
2019-10-24 07:42:49 -04:00
Jiri Pechanec
097025bed9
DBZ-1336 Fix imports of Ltree
2019-10-22 08:07:54 +03:00
Gunnar Morling
e06d25dd92
DBZ-1336 Moving Ltree type to PG connector module
2019-10-21 12:17:08 +02:00
Mincong HUANG
1e3eb8c840
DBZ-1336 Support Postgres LTREE columns
2019-10-21 12:10:46 +02:00
Jiri Pechanec
0b15e76d2a
DBZ-1525 Validate replication slot name
2019-10-20 22:16:29 +02:00
Jiri Pechanec
d858e05bf6
DBZ-1564 Close TX after toasted query
2019-10-19 20:46:22 +03:00
Grant Cooksey
5ab5b4063d
DBZ-1564 Close open transaction during postgres streaming
...
After the schema refesh, we do not want to hold a long running open transaction
since that is expensive.
2019-10-19 20:46:22 +03:00
Jiri Pechanec
06ef4b6a95
DBZ-1565 Take empty messages into consideration
2019-10-18 13:53:53 +03:00
Jiri Pechanec
986dc4ef3a
DBZ-1565 Test the log message
2019-10-18 13:53:53 +03:00
Gunnar Morling
4ea85d7fd3
DBZ-1565 Raise warning about growing WAL backlog
2019-10-18 13:53:53 +03:00
Debezium Builder
519dc011ff
[maven-release-plugin] prepare for next development iteration
2019-10-17 04:20:13 -04:00
Debezium Builder
51ef6b9470
[maven-release-plugin] prepare release v1.0.0.Beta1
2019-10-17 04:20:12 -04:00
Grant Cooksey
18ca6ec72e
DBZ-1546 Comma-separated regular expressions remove whitespace.
...
The table whitespace parameter should remove whitespace between regular expressions
since tables cannot start with whitespace characters.
2019-10-17 10:13:44 +03:00
Jiri Pechanec
bba550f955
DBZ-892 Flush on empty messages
2019-10-16 00:54:54 +02:00
Jiri Pechanec
6d8412e5de
DBZ-1401 Support for pgoutput
2019-10-10 09:37:52 +02:00
Jiri Pechanec
f77415936f
DBZ-1401 Fix formatting
2019-10-10 09:37:52 +02:00
Jiri Pechanec
7c4203cf60
DBZ-1401 Handle tables without explicit PK
2019-10-10 09:37:52 +02:00
Chris Cranford
4205a224d6
DBZ-1532 Pgoutput string decoding should not rely on JVM default charset
2019-10-09 14:35:25 +02:00
Jiri Pechanec
cf7eccb152
DBZ-1498 Docs added; microseconds changed to numeric; nanos supported
2019-10-09 12:14:35 +02:00
Jiri Pechanec
20363f24bb
DBZ-1498 Support for precise interval value
2019-10-09 12:14:35 +02:00
Gunnar Morling
92885ebe99
DBZ-1392 Removing trailing whitespace from Java files
2019-10-07 12:11:43 +03:00
Gunnar Morling
8d6f5409b1
DBZ-1392 Moving static imports up and applying this to all files
2019-10-07 12:11:43 +03:00
Chris Cranford
3594ff639a
DBZ-1341 Rebased against master and applied necessary checkstyle fixes
2019-10-03 19:44:43 +02:00
Chris Cranford
ca5517746a
DBZ-1341 Checkstyle flag missing whitespace
2019-10-03 19:44:43 +02:00
Chris Cranford
7442670692
DBZ-1355 Fix checkstyle problems detected after upgrade to 8.19
2019-10-03 19:44:43 +02:00
Chris Cranford
5ac2318d84
DBZ-1391 Do not allow code after closing brace ( #1065 )
...
* DBZ-1391 Do not allow code after closing brace
2019-10-02 20:44:40 +02:00
Jiri Pechanec
f868cb87c0
DBZ-1280 Lock timeout for SQL Server
2019-10-02 10:34:25 +02:00
Gunnar Morling
d94b37c693
DBZ-1507 Misc. clean-up;
...
* Removing unused Table methods
* Re-using primaryKeyColumnNames() instead of custom predicate
2019-10-02 10:25:37 +02:00
Debezium Builder
2ff77b993b
[maven-release-plugin] prepare for next development iteration
2019-10-01 15:34:23 -04:00
Debezium Builder
e09d584b11
[maven-release-plugin] prepare release v0.10.0.Final
2019-10-01 15:34:23 -04:00
Jiri Pechanec
22e8397e6d
DBZ-1515 Add poll pause only after multiple no message attempts
2019-10-01 13:24:42 -04:00
Debezium Builder
2f5960c5d2
[maven-release-plugin] prepare for next development iteration
2019-09-27 01:03:50 -04:00
Debezium Builder
e3b44ffb12
[maven-release-plugin] prepare release v0.10.0.CR2
2019-09-27 01:03:50 -04:00
Gunnar Morling
2fba51863e
DBZ-1015 Unifying the retrieval of custom key mappers
2019-09-26 10:14:48 +02:00
Guillaume Rosauro
1405889bef
DBZ-1015-allow-user-to-customize-key
2019-09-26 10:14:21 +02:00
Gunnar Morling
97cc493ef5
DBZ-1368 More expressive type name
2019-09-25 17:45:09 +02:00
Jiri Pechanec
0e2a79eba5
DBZ-1368 Change default toasted value
2019-09-25 17:24:21 +02:00
Jiri Pechanec
5bfe14ea66
DBZ-1368 Remove unnecessary optimizations
2019-09-25 17:24:21 +02:00
Jiri Pechanec
3c19a392ac
DBZ-1368 Added documentation
2019-09-25 17:24:21 +02:00
Jiri Pechanec
4150d2d0e1
DBZ-1368 Added JavaDoc
2019-09-25 17:24:21 +02:00
Jiri Pechanec
c638b16117
DBZ-1368 Toast marker presence not derived from plguin name
2019-09-25 17:24:21 +02:00
Jiri Pechanec
6efd2a76fe
DBZ-1368 Set new toasted vals from old
2019-09-25 17:24:21 +02:00
Jiri Pechanec
f19b4b15a7
DBZ-1368 Run with postgres dev image
2019-09-25 17:24:21 +02:00
Jiri Pechanec
29f6d9dd09
DBZ-1368 Support toasted placeholder in wal2json
2019-09-25 17:24:21 +02:00
Jiri Pechanec
9dd5a16160
DBZ-1368 Support toasted placeholder in protobuf
2019-09-25 17:24:21 +02:00
Gunnar Morling
ae0bf61bb4
DBZ-1367 Handling "datum_missing" marker
2019-09-25 17:24:21 +02:00
Jiri Pechanec
ab31147593
DBZ-1368 Support toasted placeholder in pgoutput
2019-09-25 17:24:21 +02:00
Gunnar Morling
442408be02
DBZ-1497 Using int64 for INTERVAL microseconds mapping
2019-09-25 11:21:40 +03:00
Jiri Pechanec
9b2fcdc13a
DBZ-1438 Reduce test runtime
2019-09-24 12:51:48 +02:00
Jiri Pechanec
8f308746d3
DBZ-1438 Interrupt only after interrupt affected operations
2019-09-24 12:51:48 +02:00
Gunnar Morling
c2b66ef2d7
DBZ-1436 Referencing default publication name via constant
2019-09-23 20:36:40 +02:00
Chris Cranford
f446ce6ffd
DBZ-1436 Fix postgres connector not honoring configured publication.name
setting
2019-09-23 20:33:48 +02:00
Gunnar Morling
008ac7a7e4
DBZ-1426 Misc clean-up;
...
* Removing unused method parameters
* Formatting
* Using logger methods with argument placeholders
2019-09-23 11:34:18 +02:00
trolison
e5f77b3a32
DBZ-1426, DBZ-1427 Introducing retry if replication slot is already in use;
...
- Added PG connector config parameters for retry count and delay
- Added basic test for connection retry and delay logic
- Added a more clear error message when a PG connection cant be made and it has nothing to do with the schema registry (DBZ-1427)
- Updated documentation for the new connection retry parameters
2019-09-23 11:23:38 +02:00
Chris Cranford
130210b5ad
DBZ-1044 Add config option sanitize.field.names
and FieldNameSelector
2019-09-20 07:40:23 +03:00
Chris Cranford
b569c184da
DBZ-1383 Fix PostgreSQL tests that fail intermittently
2019-09-17 11:19:17 +03:00
Chris Cranford
06c7621217
DBZ-1383 Fix PostgreSQL tests that fail intermittently
...
* Test dropping slot & publication before each test
2019-09-17 11:19:17 +03:00
Chris Cranford
278bbad8ae
DBZ-1383 Revert commit 8e636685: Fix PostgreSQL tests that fail intermittently
2019-09-17 11:19:17 +03:00
Chris Cranford
fd6450442b
DBZ-1383 Fix PostgreSQL tests that fail intermittently
...
* Introduced wait steps to OutboxEventRouterIT to check if the problem is timing related
2019-09-17 11:19:17 +03:00
Chris Cranford
37191ab713
DBZ-1383 Fix PostgreSQL tests that fail intermittently
2019-09-17 11:19:17 +03:00
Chris Cranford
022d0d07c8
DBZ-1383 Fix PostgreSQL tests that fail intermittently
...
* Only execute `DROP PUBLICATION` when using the PGOUTPUT logical decoder
2019-09-17 11:19:17 +03:00
Chris Cranford
99727dea76
DBZ-1383 Fix PostgreSQL tests that fail intermittently
2019-09-17 11:19:17 +03:00
Debezium Builder
9b7e09fdda
[maven-release-plugin] prepare for next development iteration
2019-09-10 09:12:12 -04:00
Debezium Builder
a1c2861458
[maven-release-plugin] prepare release v0.10.0.CR1
2019-09-10 09:12:11 -04:00
Renato Mefi
c8c4266e34
DBZ-1457 Fix typo
2019-09-03 22:39:37 +02:00
Gunnar Morling
fe9fc1c0dc
DBZ-1255 Adjustments after rebase
2019-08-21 07:50:40 +02:00
Gunnar Morling
08ef1418d9
DBZ-DBZ-1255 Preventing value overflows for TIMESTAMP;
...
* Generally avoiding conversion to nano-seconds to accomodate for larger values
* Generally avoiding the usage of long to convey temporal values internally:
* Instant is used for TIMESTAMP
* OffsetDateTime for TIMESTAMPTZ
* Duration for TIME (LocalTime cannot be used as it doesn't support 24:00:00 as possible with Postgres)
* OffsetTime for TIMETZ
* Avoiding usage of methods under test for calculating expected values in tests
2019-08-21 07:50:40 +02:00
Gunnar Morling
94eb75257b
DBZ-1255 Using Instant for representing timestamps
2019-08-21 07:50:40 +02:00
Ivan Luzyanin
fd5841ab6d
fix import
2019-08-21 07:50:40 +02:00
Ivan Luzyanin
fa10cc502d
[DBZ-1255,DBZ-1205] Fix for parsing dates with year larger than 9999. Avoid using nonoseconds and parse the time directly to OffsetDateTime.
2019-08-21 07:50:40 +02:00
Chris Cranford
ef30b8a8cf
DBZ-1419 Fix timing issue waiting for streaming to start
2019-08-20 18:43:03 +02:00
Chris Cranford
c6f6020da5
DBZ-777 Fix more timing issues with heartbeats
2019-08-20 18:43:03 +02:00
Jiri Pechanec
0e39965c0e
DBZ-1419 Override time precision for MySQL
2019-08-20 18:43:03 +02:00
Jiri Pechanec
51c12d930d
DBZ-1419 Support for Connect temporal types in SQL Server
2019-08-20 18:43:03 +02:00
Chris Cranford
8504f6aafd
DBZ-1437 Fix InitialOnlySnapshotter to properly determine when to run a snapshot.
2019-08-19 22:01:32 +02:00
Chris Cranford
68f0bba7b3
DBZ-1437 Fix ExportedSnapshotter to properly determine when to run a snapshot.
2019-08-19 22:01:32 +02:00
Cyril Scetbon
e7dc1dfb62
DBZ-1440 Use ServerVersion values instead of integers
2019-08-19 14:25:53 -04:00
Cyril Scetbon
135e66dd0d
DBZ-1440 Temporary replication slots on PG10+ only
2019-08-19 14:25:53 -04:00
Cyril Scetbon
4b5475391c
DBZ-1440 Supports exported snapshots for Pg 9.4+
2019-08-19 14:25:53 -04:00
Debezium Builder
4ae105bdce
[maven-release-plugin] prepare for next development iteration
2019-08-16 03:42:55 -04:00
Debezium Builder
945ccf68cb
[maven-release-plugin] prepare release v0.10.0.Beta4
2019-08-16 03:42:54 -04:00
Jiri Pechanec
9c25a192b7
DBZ-777 Fix timing issue with heartbeats
2019-08-16 08:05:48 +02:00
Jiri Pechanec
540021cf68
DBZ-777 Minor fixes
2019-08-15 20:43:31 +02:00
Jiri Pechanec
4d18c70834
DBZ-777 Make sure the empty events are generated while streaming
2019-08-15 20:43:31 +02:00
Gunnar Morling
5ffe8123f1
DBZ-777 Further fixes
2019-08-15 20:43:31 +02:00
Gunnar Morling
63250d0c13
DBZ-777 Misc. adjustments
...
* Avoiding repeated parsing of table id
* Making 'inconsistentSchemaHandler' field final in EventDispatcher
* Not exposing synchronizeTableSchema() on SPI level
2019-08-15 20:43:31 +02:00
Jiri Pechanec
780545d0d2
DBZ-777 Fix wal2json decoder
2019-08-15 20:43:31 +02:00
Jiri Pechanec
5307a5822d
DBZ-777 Fix pgoutput decoder
2019-08-15 20:43:31 +02:00
Jiri Pechanec
de2345f87b
DBZ-777 Fix wal2json decoder
2019-08-15 20:43:31 +02:00
Jiri Pechanec
037d581844
DBZ-777 Fix heartbeats
2019-08-15 20:43:31 +02:00
Jiri Pechanec
ffb7e6d7d8
DBZ-777 Assertions against internal schema
2019-08-15 20:43:31 +02:00
Jiri Pechanec
857d22bfda
DBZ-777 Fix toasted cols test
2019-08-15 20:43:31 +02:00
Jiri Pechanec
8f08f2d505
DBZ-777 Remove unused code
2019-08-15 20:43:31 +02:00
Jiri Pechanec
d72afa8e9b
DBZ-777 Skip empty messages
2019-08-15 20:43:31 +02:00
Jiri Pechanec
f4246df6e4
DBZ-777 PostgreSQL connector rewrite to the standard framework
2019-08-15 20:43:31 +02:00
Chris Cranford
89ae4b2151
DBZ-1436 Fix postgres connector not honoring publication.name configuration
2019-08-15 12:00:29 -04:00
Jiri Pechanec
8adb5e48ca
DBZ-1385 Elevate payload for only headers setup
2019-08-12 10:36:52 +02:00
Jiri Pechanec
0991c70ac9
DBZ-1385 eventType can be configured as add field
2019-08-12 10:36:52 +02:00
Jiri Pechanec
c07c994434
DBZ-1381 Introduce permanent metadata connection
2019-08-09 11:53:45 +02:00
Chris Cranford
ae58b0b44c
DBZ-1035 Checkstyle fixup
2019-07-29 15:49:18 +02:00
Chris Cranford
e852f082a2
DBZ-1035 Support PostgreSQL lock-free snapshot using exported snapshots
2019-07-29 15:49:18 +02:00
Jiri Pechanec
0742e5775f
Revert "[maven-release-plugin] prepare release v0.10.0.Beta4"
...
This reverts commit c7340e5152
.
2019-07-23 14:54:55 +02:00
Jiri Pechanec
f7d20635ce
Revert "[maven-release-plugin] prepare for next development iteration"
...
This reverts commit 22879777e4
.
2019-07-23 14:54:49 +02:00
Jiri Pechanec
22879777e4
[maven-release-plugin] prepare for next development iteration
2019-07-23 14:10:03 +02:00
Jiri Pechanec
c7340e5152
[maven-release-plugin] prepare release v0.10.0.Beta4
2019-07-23 14:10:02 +02:00
Debezium Builder
71333fda98
[maven-release-plugin] prepare for next development iteration
2019-07-23 02:55:08 -04:00
Debezium Builder
20d006fac7
[maven-release-plugin] prepare release v0.10.0.Beta3
2019-07-23 02:55:08 -04:00
Jiri Pechanec
81ba1a3adb
DBZ-766 Remove autoboxing
2019-07-23 06:22:41 +02:00
Chris Cranford
2f92553d8f
DBZ-766 Implement PgOutput logical decoding for Postgres 10+
2019-07-23 06:22:41 +02:00
Jiri Pechanec
6a13b0e0b5
DBZ-1400 Record with same flush LSN should not be skipped for the first record
2019-07-22 14:17:59 -04:00
Gunnar Morling
0383f9178c
DBZ-1238 Removing incorrect comment
2019-07-09 21:14:28 +02:00
Gunnar Morling
6aaa8b183c
DBZ-1238 Simplifying logic
2019-07-09 21:14:28 +02:00
Gunnar Morling
bb8d7683e3
DBZ-1238 Avoiding explicit check for slot state;
...
This causes issues with PG 10 temporary slots which are always active.
The check isn't really needed anyways, because the server will raise an
exception itself when trying to connect to an already active slot.
2019-07-09 21:14:28 +02:00
Gunnar Morling
a2e0c7cd2a
DBZ-1238 Misc. clean-up;
...
* Formatting, typo fixes
* Combining createReplicationSlot() and getSlotCreationResult()
* Removing unused field
2019-07-09 21:14:28 +02:00
Addison Higham
62328bf137
DBZ-1238 Expose more options to the PG Snapshot SPI
...
This commit adds some more options to the Postgres Snapshottter SPI
interface, specifically, the ability to create the query for the start
of a snapshot, as well as the ability to generate a statement to lock
the tables.
Additionally, some rework is done to allow for the usage of an exported
snapshot in conjuction with the SPI taking metadata about a newly
created slot. This also changes how slots are initialized to allow for
the exported snapshot to be referenced, which requires adding a more
granular setup of a ReplicationConnection
Together, these options allow for a lot more flexibility, such as not
taking locks when snapshotting large databases as well as potentially to
use exported snapshots.
A future version should perhaps make some options possible without
implementing the SPI (such as using the exported snapshot by default)
but this commit is just trying to get all the groundwork in place to
allow for the Snapshotter SPI to have this functionality.
2019-07-09 21:14:28 +02:00
Gunnar Morling
369f74d875
DBZ-1320 Addressing review remarks
2019-07-09 21:07:18 +02:00
Brandon Brown
efff4f0809
DBZ-1320 Adding support for deletion events via new configurable property route.tombstone.on.empty.payload
2019-07-09 21:07:18 +02:00
Gunnar Morling
50e3a02d4d
DBZ-1371 README updates
2019-07-09 10:13:21 +02:00
Gunnar Morling
8c5d5021f4
DBZ-1371 Restoring original timeouts
2019-07-09 10:13:21 +02:00
Chris Cranford
faa31b079d
DBZ-1371 Added link to RDS.md from README.md
2019-07-09 10:13:21 +02:00
Chris Cranford
bf248e6fb0
DBZ-1371 Update to PostgreSQL connector RDS document
2019-07-09 10:13:21 +02:00
Chris Cranford
af513c2732
DBZ-1371 Derive timeouts from configurable timeout
2019-07-09 10:13:21 +02:00
Chris Cranford
f88970f2e6
DBZ-1371 Fix formatting and typos in RDS documentation
2019-07-09 10:13:21 +02:00
Chris Cranford
051d5a478d
DBZ-1371 Added RDS setup documentation for PostgreSQL
2019-07-09 10:13:21 +02:00
Chris Cranford
a50b9847ac
DBZ-1371 Fix tests to run more reliably on Amazon RDS
2019-07-09 10:13:21 +02:00
Renato Mefi
c990256fd1
DBZ-1379 Fix PostgreSQL false empty password Warn
2019-07-08 12:51:49 +02:00
Gunnar Morling
e2d8736a79
DBZ-1380 Misc. clean-up;
...
* reducing visibility of some methods
* resetting interrupted state
2019-07-05 21:08:35 +02:00
Jiri Pechanec
5dc4812ffa
DBZ-1380 Guard against partially initilized replication slot
2019-07-05 21:08:35 +02:00
Jiri Pechanec
13d638fd43
DBZ-1356 Metrics for schema history ( #947 )
...
* DBZ-1356 Metrics for schema history
2019-07-02 13:26:17 +02:00
Jiri Pechanec
b1c1c05d1f
DBZ-1363 Generate heartbeat periodically not upon event
2019-07-02 12:02:33 +02:00
Gunnar Morling
08c5d36789
Merge pull request #942 from jpechane/stab-lsn-flush-test
...
DBZ-1362 Stabilize test executions on CI
2019-07-01 09:34:24 +02:00
Gunnar Morling
a02daae3c1
DBZ-1362 Comment
2019-07-01 09:34:01 +02:00
Chris Cranford
61f5e15889
DBZ-1366 Avoid creating unnecessary TypeMetadataImpl instances; only init once.
2019-06-28 13:44:53 -04:00
Jiri Pechanec
b546dc886e
DBZ-1362 Fix Snapshot/streaming switch race condition
2019-06-28 09:09:18 +02:00
Jiri Pechanec
e0b188f80a
DBZ-1362 Drop replication slot after test execution
2019-06-28 05:41:07 +02:00
Jiri Pechanec
046064d653
DBZ-1362 Make toast test compatible with PostgreSQL 11
2019-06-28 05:32:11 +02:00
Jiri Pechanec
412ce20638
DBZ-1362 Switch to Awaitility implementation
2019-06-28 05:19:01 +02:00
Jiri Pechanec
5a5d969e99
DBZ-1362 Wait for real LSN change during test
2019-06-28 05:19:01 +02:00
Jenkins user
e95d656bca
[maven-release-plugin] prepare for next development iteration
2019-06-27 07:36:13 +00:00
Jenkins user
d95fd87311
[maven-release-plugin] prepare release v0.10.0.Beta2
2019-06-27 07:36:12 +00:00
Guillaume Rosauro
dc81d3c8ac
DBZ-1352 Removing the unused topic selection strategy configuration from Postgres connector
2019-06-26 16:24:38 +02:00
Jiri Pechanec
355c8a94ac
DBZ-1347 JavaDoc update
2019-06-25 14:52:58 +02:00
Gunnar Morling
62fc03a44d
DBZ-1347 Commenting
2019-06-25 14:52:58 +02:00
Jiri Pechanec
e72d7edd2f
DBZ-1347 Switch to non-blocking stream read
2019-06-25 14:52:58 +02:00
Gunnar Morling
e05ef934a0
DBZ-1345 Adding @FixFor
2019-06-25 12:44:00 +02:00
Chris Cranford
d518fa59fe
DBZ-1345 Should not snapshot materialized views
2019-06-25 12:44:00 +02:00
Jiri Pechanec
2dfdf5cb12
DBZ-1342 Partitioned table is included in snapshot
2019-06-21 12:26:40 +02:00
Gunnar Morling
21176ff70f
Adding link to replication API
2019-06-21 11:14:35 +02:00
Gunnar Morling
deaa5b7f4d
DBZ-1337 Adding support for HSTORE arrays
2019-06-19 17:01:13 +02:00
Jiri Pechanec
badae96e1d
DBZ-1335 Process unknown datatypes for snapshotted arrays
2019-06-18 16:17:49 +02:00
Jenkins user
5fdb0ca81b
[maven-release-plugin] prepare for next development iteration
2019-06-11 11:20:46 +00:00
Jenkins user
61729b9e00
[maven-release-plugin] prepare release v0.10.0.Beta1
2019-06-11 11:20:46 +00:00
Jiri Pechanec
6a3c87e2f1
DBZ-1323 Initialize commitTime for out-of-order chunk
2019-06-11 10:27:56 +02:00
Gunnar Morling
5d01f409f0
DBZ-1317 Separating update and query methods for MongoDB's SourceInfo;
...
* Avoids access to collection id after events which don't have one
* Pulling up struct() and schema() methods to common base class
2019-06-11 10:07:36 +02:00
Chris Cranford
0aadaa10dc
DBZ-1242 Fixed typo and added negative test checks
2019-06-05 13:31:25 -04:00
Chris Cranford
b3ca33287a
DBZ-1242 Issue warning when filters produce an empty schema
2019-06-05 13:31:25 -04:00
Jenkins user
056730d06a
[maven-release-plugin] prepare for next development iteration
2019-06-03 10:24:08 +00:00
Jenkins user
7aeaec236e
[maven-release-plugin] prepare release v0.10.0.Alpha2
2019-06-03 10:24:08 +00:00
Gunnar Morling
246c1be305
DBZ-1283 Enforcing presence of database.server.name globally
2019-06-03 10:58:14 +02:00
Chris Cranford
5f0b44b4ec
DBZ-1283 Only validate history topic name for affected connectors
2019-06-03 10:58:14 +02:00
Chris Cranford
b4736a7e03
DBZ-1283 Only validate history topic name for affected connectors
2019-06-03 10:58:14 +02:00
Gunnar Morling
73590173b5
DBZ-1310 Removing methods only needed for testing
2019-05-31 18:09:41 -04:00
Jiri Pechanec
4b64c8e447
DBZ-1295 3state snapshot for MySQL
2019-05-31 10:56:05 +02:00
Jiri Pechanec
348b2415d5
DBZ-1295 3state snapshot for PostgreSQL
2019-05-31 10:56:05 +02:00
Jiri Pechanec
946019982e
DBZ-1303 Assembly without database drivers
2019-05-30 14:32:47 +02:00
Gunnar Morling
7759333fe3
DBZ-1092 Using ColumnNameFilter instead of Predicate<Column>
2019-05-29 15:49:49 +02:00
Jenkins user
ac5c76e431
[maven-release-plugin] prepare for next development iteration
2019-05-28 14:38:32 +00:00
Jenkins user
35f59ba460
[maven-release-plugin] prepare release v0.10.0.Alpha1
2019-05-28 14:38:32 +00:00
Gunnar Morling
8ecfbee77d
DBZ-1235 Adding TODO marker
2019-05-28 09:35:15 +02:00
Gunnar Morling
b8eede2f41
DBZ-1235 Misc. clean-up;
...
* Putting collection name after replica set name for MongoDB
* Removing redundant method for converting schema to string
* Removing redundant keys from PG SourceInfo
2019-05-28 09:35:15 +02:00
Jiri Pechanec
526e780513
DBZ-1235 Misc fixes
2019-05-28 09:35:15 +02:00
Gunnar Morling
eef37cb567
DBZ-1235 Misc adjustments;
...
* Removing superfluous parameter from getSourceInfoStructMaker()
* JavaDoc
* Renaming field to "source.struct.version"
2019-05-28 09:35:15 +02:00
Gunnar Morling
8f28f323e7
DBZ-1235 Using Instant to represent timestamps internally;
...
Only "rendered" as epoch milli-seconds at the system boundaries.
2019-05-28 09:35:15 +02:00
Jiri Pechanec
6f6c8ddf06
DBZ-1235 Refactored to unify the code
2019-05-28 09:35:15 +02:00
Jiri Pechanec
c1bf1752fe
DBZ-1235 Pluggable source maker for MongoDB
2019-05-28 09:35:15 +02:00
Jiri Pechanec
df16ed84cd
DBZ-1235 Pluggable source maker for PostgreSQL
2019-05-28 09:35:15 +02:00
Jiri Pechanec
f5757dbd9f
DBZ-1235 Extract serverName to common source
2019-05-28 09:35:15 +02:00
Gunnar Morling
4ed8b4636c
DBZ-1224 Completing implementation;
...
* Further unification of override handling;
* Undoing change related to SnapshotStatementFactory as it seems not needed
* Adding test
2019-05-27 08:44:42 +02:00
mokkara
b3de0250a9
DBZ-1224 Snapshot SELECT support for SQL Server
2019-05-27 08:44:42 +02:00
Gunnar Morling
a0e30b7c90
DBZ-1289 Making outbox router options more consistent
2019-05-24 13:18:55 +02:00
Chris Cranford
e2c74111ad
DBZ-1234 Removed deprecated configuration options
2019-05-22 10:19:05 +02:00
Gunnar Morling
4751b266f3
DBZ-1164 Misc. improvements;
...
* Moving stringToDuration() for the sake of re-use and testability
* Making pattern and durations constants
2019-05-21 09:38:23 +02:00
GraySmog
9b52ac86e8
DBZ-1164 Supporting TIME columns with value of "24:00:00"
2019-05-21 09:38:09 +02:00
Jenkins user
7748be6056
[maven-release-plugin] prepare for next development iteration
2019-05-02 16:04:56 +00:00
Jenkins user
5af083b86e
[maven-release-plugin] prepare release v0.9.5.Final
2019-05-02 16:04:55 +00:00
Gunnar Morling
d12926c404
DBZ-1214 Applying default status update interval
2019-05-02 11:30:16 +02:00
Gunnar Morling
6960c880ca
DBZ-1214 Misc. clean-up:
...
* Using parker instead of Thread#sleep()
* Using flag for executor cancelation
* Using unique context name in thread name
2019-05-02 11:30:16 +02:00
Addison Higham
e1570dd330
[DBZ-1214] Ensure PG connection keep alive
...
This adds a small change to the postgres connector which
works around a problem when large databases take more
than a minute to refresh the schema
This works by creating a simple background thread that ensures we send
statusUpdate messages periodically
I considered if we could refresh the schema before we create the
replication connection but I believe we choose our offset in the log
and then get the schema in order to ensure we don't get out of sync with
any potential schema changes
This also makes some minor refactors to use a Duration instead of an
integer for the statusUpdateInterval
2019-05-02 11:30:16 +02:00
Jiri Pechanec
3b78e0bebb
DBZ-1245 Fix empty heartbeat message test
2019-05-02 09:16:21 +02:00
Jiri Pechanec
b53f456912
DBZ-1245 Heartbeat only when offsets ready
2019-05-02 08:22:28 +02:00
Gunnar Morling
2f7cab6eb3
DBZ-1260 Removing some dead code
2019-05-02 07:28:23 +02:00
Chris Cranford
dc61e52bc5
DBZ-1260 Fix timetz microsecond precision support
2019-05-02 07:28:23 +02:00
Chris Cranford
7602136cf6
DBZ-1236 Removed millisecond truncation.
2019-05-01 14:16:17 +02:00
Chris Cranford
16840d7d59
DBZ-1236 Handle timez precision consistently.
2019-05-01 14:16:17 +02:00
Jiri Pechanec
3ff81348b8
DBZ-1236 Support full timestamptz precision
2019-05-01 14:16:17 +02:00
Gunnar Morling
a84230447e
DBZ-1247 Making defaultSnapshotFetchSize() abstract
2019-04-30 09:57:11 +02:00
jchipmunk
72b06366fe
Removing extra initialisation of "config" variable in PG config
2019-04-30 09:57:11 +02:00
jchipmunk
9fe033ce24
DBZ-1247 Specifying default snapshot fetch size for each connector config
2019-04-30 09:57:11 +02:00
jchipmunk
a5cc83c0d1
DBZ-1247 Adding property to specify snapshot fetch size
2019-04-30 09:57:11 +02:00
jorkzijlstra
d0e2a72cd1
DBZ-1241 Allowing null values again for commitTime
2019-04-30 09:45:10 +02:00
shubham
f55bec86d3
Added support for all range types
2019-04-24 07:40:03 +02:00
krizhan
70c564b4c2
DBZ-212 Moved SERVER_NAME to RelationalDatabaseConnectorConfig
2019-04-23 12:29:44 +02:00
krizhan
38f3856c31
DBZ-212 Remove PostgreSQL getLogicalName function
2019-04-23 12:29:44 +02:00
Dennis Campagna
173e8f5854
DBZ-1034 Support range column types on Postgres
...
Signed-off-by: sciamp <denniscampagna@gmail.com>
DBZ-1034 Support range column types on Postgres
Signed-off-by: sciamp <denniscampagna@gmail.com>
2019-04-12 09:44:17 +02:00
GraySmog
726a2afcd6
Add comments stating the reason for skipping tests
2019-04-12 09:27:45 +02:00
GraySmog
08ca02f8f4
DBZ-1193 Add support for network type MACADDR and MACADDR8 in PostgreSQL connnector
2019-04-12 09:27:45 +02:00
Jenkins user
cefc5ccfb9
[maven-release-plugin] prepare for next development iteration
2019-04-11 12:07:16 +00:00
Jenkins user
fc3026adc3
[maven-release-plugin] prepare release v0.9.4.Final
2019-04-11 12:07:15 +00:00
Gunnar Morling
2250c47b7f
DBZ-1163 Minor improvements
2019-04-10 10:53:22 +02:00
preethi
02873e8169
DBZ-1163 Enable postgres snapshot for tables without primary key
2019-04-10 10:48:11 +02:00
ShubhamRwt
762d908eef
DBZ-1193 Added support for cidr address type
2019-04-08 15:00:03 +02:00
jchipmunk
7c0ae3ee20
DBZ-1212 SLF4J usage issues
2019-04-04 21:32:12 +02:00
Jiri Pechanec
ff52f3713e
DBZ-1144 Test for geometry in public schema
2019-04-02 15:57:57 +02:00
Alexander Kovryga
4616df189e
DBZ-1189 Add support for inet data type.
2019-03-28 09:57:32 +01:00
ShubhamRwt
540a951211
DBZ- 362 Adding WhitespaceAfter check to Checkstyle
2019-03-28 09:24:11 +01:00
Jenkins user
8b0edd1871
[maven-release-plugin] prepare for next development iteration
2019-03-25 12:16:31 +00:00
Jenkins user
56f98a48b2
[maven-release-plugin] prepare release v0.9.3.Final
2019-03-25 12:16:30 +00:00
Renato Mefi
6300f2972f
DBZ-1169 Integration tests for Outbox EventRouter
...
Set the Snapshot Mode to NEVER provides more consistent tests
2019-03-25 11:27:44 +01:00
Renato Mefi
8b689d7f03
DBZ-1169 Implement RegexRouter based on config
2019-03-25 11:27:44 +01:00
Renato Mefi
f0f13ed4a4
DBZ-1169 Implement basic behavior
2019-03-25 11:27:44 +01:00
Renato Mefi
2ee1c98298
DBZ-1169 Setup EventRouter on Postgres Test IT
2019-03-25 11:27:44 +01:00
shubham
8cca21e969
DBZ-1039 Make all ifs with braces
2019-03-25 10:23:14 +01:00
Addison Higham
8e21139ca3
DBZ-1082 Add new custom recovery mode, more metadata
...
This commit does a few things:
- Refactors snapshot modes to be encapsulated by an interface and
to only use that interface in determining when to snapshot and in
determing the type of the `RecordProducer` interface to instantiate
- Refactors the configuration of existing snapshot modes to tie the
existing snapshot modes to their aligned implementation
- Adds a new snapshot.mode, custom, and a new configuration option to
specify a custom implementation that will be loaded by the class loader
- Changes the visibility of some classes to allow for custom snapshot
modes to get enough context to make an informed choice
- Adds some metadata about slots (the catalog_xmin) to give a full idea
of the state of slots which can be useful in implementing snapshot
modes (which is also configurable, as it can add some overhead)
Together, these changes allow for a much broader flexibility got end
users to implement a snapshot mode that can do more advanced snapshots,
such as partial recovery or for partial snapshots for tables where not
all records are needed.
This could also be seen as superseeding the
`snapshot.select.statement.overrides` to allow for users to dynamically
build queries based on the state of the slot and the offsets consumed.
2019-03-24 11:56:40 +01:00
Jiri Pechanec
b11f1b8d05
DBZ-1161 Start snapshot in async thread
2019-03-21 14:32:47 +01:00
trolison
2001b178d7
- Fixed bad formatting
...
- Added trace for when empty messages are processed
DBZ-1181
2019-03-19 10:46:36 -07:00
trolison
df0e789601
- Add support for processing WAL2JSON events that have no changes
...
DBZ-1181
2019-03-18 20:32:03 -07:00
Gunnar Morling
2ea87727cf
DBZ-1174 Using Instant instead of long for representing timestamps
2019-03-12 17:19:13 +01:00
Ashhar Hasan
0b07afd23e
DBZ-1174 Convert commitTime from nanos to micros for wal2json
...
- wal2json sends the txn commitTime using a function from PostgreSQL's C
library. The value that Debezium recevies is in nanoseconds.
- decoderbufs sends the txn commitTime in microseconds.
- RecordsSnapshotProducer updates SourceInfo.ts_usec by converting
System.currentTimeMillis() to microseconds.
- RecordsSnapshotProducer updates the SourceInfo's ts_usec field using
message.getCommitTime().
This means that when using wal2json, the value of SourceInfo.ts_usec
is in microseconds since epoch during snapshot but is in nanoseconds
during streaming. To fix this, we changed
Wal2JsonReplicationMessage.getCommitTime() to return in microseconds.
2019-03-12 17:19:13 +01:00
Josh Stanfield
e4b7b90637
DBZ-1162 fix to ensure hstore snapshots behave the same as streams
2019-03-04 13:39:50 +01:00
Jenkins user
50a7c568fa
[maven-release-plugin] prepare for next development iteration
2019-02-23 09:53:58 +00:00
Jenkins user
28f3839804
[maven-release-plugin] prepare release v0.9.2.Final
2019-02-23 09:53:58 +00:00
Jiri Pechanec
78bfda09a6
DBZ-1146 Test full column diff
2019-02-22 14:16:03 +01:00
Gunnar Morling
5043d41b7b
DBZ-1146 Expanding test
2019-02-22 14:16:03 +01:00
Jiri Pechanec
4dec844fa2
DBZ-1146 Fix table created in streaming test
2019-02-22 14:16:03 +01:00
Jiri Pechanec
fa1e632a90
DBZ-1146 Handle toastable columns for snapshotted tables
2019-02-22 14:16:03 +01:00
Jiri Pechanec
89feae32e7
DBZ-1141 Test for the default values
2019-02-21 15:02:51 +01:00
Gunnar Morling
6787d17d54
DBZ-1141 Misc. clean-up;
...
* Removing redundant check for date mapping type
* Always using String as fallback value for temporal values where needed
* Pulling fallback temporal values up to JdbcValueConverters
2019-02-21 15:02:51 +01:00
Jiri Pechanec
b9d326711d
DBZ-1141 Incorrect money scale
2019-02-21 15:02:51 +01:00
Jiri Pechanec
992c31fa1f
DBZ-1141 Add non-date types
2019-02-21 15:02:51 +01:00
Jiri Pechanec
41ba7128e3
DBZ-1141 Correct default falback values for connect date types
2019-02-21 15:02:51 +01:00
Keith Barber
95b2290df2
DBZ-1149 Ensure consistent Java types for fallback values when converting
2019-02-21 15:02:51 +01:00
taylor-rolison
17c6a7a82d
Config option to pass optional logical decoder stream params ( #783 )
...
DBZ-1130 Added config param slot.stream.params to the Postgres connector to allow passing optional params to logical decoder on stream start
2019-02-21 14:06:51 +01:00
Jenkins user
b96e5fee0d
[maven-release-plugin] prepare for next development iteration
2019-02-13 11:13:44 +00:00
Jenkins user
3de2d04bcd
[maven-release-plugin] prepare release v0.9.1.Final
2019-02-13 11:13:44 +00:00
Jiri Pechanec
9049078607
DBZ-997 Handle de-sync of message and JDBC PK info
2019-02-13 10:13:12 +01:00
Gunnar Morling
2a486ecfff
DBZ-1019 Removing legacy methods from Metronome
2019-02-11 18:11:34 +01:00
Ivan Lorenz
721e8112cd
DBZ-1126 Localize era display value to default locale
2019-02-06 15:34:30 +01:00
Gunnar Morling
7076deba5a
DBZ-1118 Adding JUnit rule to skip tests depending on Postgres version
2019-02-04 14:43:19 +01:00
Kevin Pullin
c414a8b942
Add test for DBZ-1118
2019-02-04 14:43:19 +01:00
Kevin Pullin
1330df8908
Postgres snapshot - On null records, clear prior value from currentRecord
2019-02-04 14:43:19 +01:00
Gunnar Morling
41b7fbe78e
Adding note on testing Postgres connector with external DB
2019-01-30 09:25:12 +01:00
Gunnar Morling
cc0df61665
DBZ-1111 Allowing to use drop-slot-on-close with PG 10 or newer
2019-01-29 14:44:10 +01:00
Gunnar Morling
e84c6e2238
DBZ-962 Avoiding dependency to JAXB
2019-01-29 10:28:48 +01:00
Jenkins user
c5b7d21d3e
[maven-release-plugin] prepare for next development iteration
2019-01-28 11:07:04 +00:00
Jenkins user
f1ae79ff73
[maven-release-plugin] prepare release v0.9.0.CR1
2019-01-28 11:07:04 +00:00
Gunnar Morling
ccff79e07d
DBZ-1105 Removing TableImpl#columnNames field
2019-01-28 10:23:56 +01:00
Gunnar Morling
9dc2c525c9
DBZ-1083 Passing "include-unchanged-toast" option only for wal2json
2019-01-25 11:04:43 +01:00
Addison Higham
26f836e0f6
DBZ-1080 new API for EmbeddedEngine for batch/async
...
This introduces a new API to the EmbeddedEngine, the ChangeConsumer,
which gives the user a more flexible option for consuming changes by
exposing groups of records as well as the ability to control the
comitting of those records.
This remainds completely backwards compatible with the old API as the
ChangeConsumer wraps the existing Consumer interface with a default
implementation.
2019-01-18 14:06:50 +01:00
Jiri Pechanec
1cf257478b
DBZ-781 Enable unstable tests
2019-01-17 08:59:42 +01:00
Tony Rizko
a921613877
Fix typo in PostgresConnection error message
2019-01-17 08:58:12 +01:00
Jiri Pechanec
ceb6c62721
DBZ-1064 Remove lazy fallback values
2019-01-16 13:01:36 +01:00
Jiri Pechanec
536e517c08
DBZ-1064 Geometry default value changes; fixes for default suppliers
2019-01-16 13:01:36 +01:00
Gunnar Morling
532c17f8aa
DBZ-1064 Using dedicated functional interface instead of Consumer;
...
* Commenting
* Moving type from "util" to "jdbc"
2019-01-16 13:01:36 +01:00
Jiri Pechanec
bff7eea734
DBZ-1064 Use convert value template
2019-01-16 13:01:36 +01:00
Jiri Pechanec
920e0cbd3f
DBZ-1064 Set default value only for non-nullable fields
2019-01-16 13:01:36 +01:00
Jiri Pechanec
484a8977bc
DBZ-1083 Passing "include-unchanged-toast" option only if supported;
...
The "include-unchanged-toast" option was removed in recent wal2json versions, without a transition phase. So the connector tries not to connect first giving the option, and if that fails, without it.
2019-01-16 09:37:55 +01:00
Gunnar Morling
af01d10886
DBZ-1081 JavaDoc; adding Addison Higham to COPYRIGHT.txt
2019-01-15 18:09:08 +01:00
Addison Higham
889b307b9d
DBZ-1081 Fallback to restart_lsn in pg 9.5
...
In pg 9.5, confirmed_flush_lsn is not availiable. However, there is
restart_lsn, which should be safe to use with a downside of producing
some extra duplicate records.
This page: https://paquier.xyz/postgresql-2/postgres-9-6-feature-highlight-replication-slot-improvements/
indicates as such and this allows for DBZ to support PG 9.5
2019-01-15 18:05:55 +01:00
Gunnar Morling
de356896c0
DBZ-1029 Adding one more test
2019-01-15 17:41:42 +01:00
Ilia Bogdanov
88521371cc
DBZ-1029 insertion test update, code cleanup after review
2019-01-14 12:03:19 +03:00
Ilia Bogdanov
cee9bd9a5d
DBZ-1029 handling messages from tables with replica identity full in postgres connector.
2019-01-14 12:03:19 +03:00
Amit Sela
b0b7d942a3
DBZ-1073 Adding scale as schema parameter, if present
2019-01-11 15:40:30 +01:00
Grzegorz Kołakowski
91c72738c0
DBZ-1067 Add column blacklist field to RelationalDatabaseConnectorConfig
2019-01-09 18:13:26 +01:00
Jenkins user
5275f73424
[maven-release-plugin] prepare for next development iteration
2018-12-19 13:06:12 +00:00
Jenkins user
b6569c18ae
[maven-release-plugin] prepare release v0.9.0.Beta2
2018-12-19 13:06:12 +00:00
Gunnar Morling
92ae01ccdf
DBZ-996 Adding column name to log message
2018-12-18 17:53:42 +01:00
Jiri Pechanec
01eee09f90
DBZ-996 Tolerate race condition in PK updates
2018-12-18 17:52:28 +01:00
Jiri Pechanec
edec1c3090
DBZ-978 Added last event and captured tables metrics
2018-12-13 20:52:11 +01:00
Gunnar Morling
5542c0de29
DBZ-966 Making sure Postgres connector can be stopped while awaiting initial snapshot delay;
...
* Also skipping snapshotting when requested to stop during initial delay
* Showing remaining delay time in logs
* Adding missing config
2018-12-13 14:29:54 +01:00
Grzegorz Kołakowski
e2942ab72a
DBZ-966 Replace sleep with polling loop
2018-12-13 14:29:54 +01:00
Grzegorz Kołakowski
35a5acfd8b
DBZ-966 Enable to delay initial snapshot in PostgreSQL connector
2018-12-13 14:29:54 +01:00
Gunnar Morling
0833b23feb
DBZ-842 Expanding test to cover TOASTed NOT NULL column, too
2018-12-11 18:42:51 +01:00
Jiri Pechanec
94c0a42286
DBZ-842 Do not send TOASTed values from wal2json
2018-12-11 17:10:58 +01:00
Jiri Pechanec
2b2ff0754d
DBZ-960 Run SnapshotReader in a single transaction
2018-12-11 14:15:53 +01:00
Gunnar Morling
470cd1eaba
DBZ-1024 Doing some clean-up
2018-12-07 15:39:59 +01:00
Jiri Pechanec
ea5c8512b7
DBZ-1024 Return test assert
2018-12-07 15:39:59 +01:00
Jiri Pechanec
0b78f0c31a
DBZ-1024 Emit correct last_snapshot_record in source
2018-12-07 15:39:59 +01:00
Jiri Pechanec
9c225c63eb
DBZ-1021 Ignore events for dropped table
2018-12-06 18:54:50 +01:00
Gunnar Morling
101bd8ef6d
DBZ-1000 Making Checkstyle happy
2018-12-03 15:00:43 +01:00
Jiri Pechanec
7a3cbd6cb6
DBZ-1000 Detect nullability change in schema
2018-12-03 14:54:36 +01:00
Jiri Pechanec
68766e9355
DBZ-1009 Support for unknown types during snapshot
2018-11-30 11:16:36 +01:00
Jenkins user
db1d3a7fb8
[maven-release-plugin] prepare for next development iteration
2018-11-20 16:15:14 +00:00
Jenkins user
f83db82cea
[maven-release-plugin] prepare release v0.9.0.Beta1
2018-11-20 16:15:14 +00:00
Jiri Pechanec
0611d4975f
DBZ-965 Reduce the test time
2018-11-13 09:16:30 +01:00
Jiri Pechanec
a4abd2a336
DBZ-965 Test of flushed LSN
2018-11-13 09:15:09 +01:00
Gunnar Morling
5a9ca6eb31
DBZ-965 Some clean-up
2018-11-13 09:15:09 +01:00
Jiri Pechanec
675f5b6479
DBZ-965 Renames for better clarification
2018-11-13 09:15:09 +01:00
Gunnar Morling
b455b7ccaf
DBZ-970 Making LogicalDecoder#getPostgresPluginName() abstract
2018-11-13 09:15:09 +01:00
Jiri Pechanec
390965f4f9
DBZ-965 Track latest LSN per each change
2018-11-13 09:15:09 +01:00
Jiri Pechanec
b1707c395d
DBZ-970 Correct plging name for wal2json_streaming decoder
2018-11-13 09:15:09 +01:00
Jiri Pechanec
83137d0a41
DBZ-965 Force heartbeat after end of each TX
2018-11-13 09:15:09 +01:00
Gunnar Morling
efbd46b7f6
DBZ-953 Having PG connector use decimal mode option from RelationalDatabaseConnectorConfig
2018-11-08 09:23:28 +01:00
Jiri Pechanec
3a8293a4ba
DBZ-937 Disable auto-handling of unknown types in test
2018-10-11 09:55:07 +02:00
Olavi Mustanoja
0c986b4a78
DBZ-937 add wal2json replication message handling for CITEXT
2018-10-09 08:46:55 +02:00
Jenkins user
d952f5dfb0
[maven-release-plugin] prepare for next development iteration
2018-10-04 11:59:14 +00:00
Jenkins user
ff9b70278b
[maven-release-plugin] prepare release v0.9.0.Alpha2
2018-10-04 11:59:13 +00:00
Gunnar Morling
1f25536e2b
DBZ-934 Using API for creating replication slot if possible
2018-10-04 10:12:48 +02:00
PSanetra
dd3859d8bc
DBZ-934 Use temporary replication slots if slot.drop_on_stop is enabled and Postgres >= 10 is used
2018-10-04 10:12:48 +02:00