alexey.rassokhin/tet123
1
0
Fork 0
Code Issues Pull Requests 1 Packages Projects Releases Wiki Activity
7,178 Commits 3 Branches 0 Tags 51 MiB
78cb8e5890
Commit Graph

1243 Commits

Author SHA1 Message Date
Ganesh Ramasubramanian
4716194ff9 DBZ-2606: Map new name for the fields and headers 
Added ability to map new name for the fields and headers.
The "new name" mapping is completely optional and we will not
be affecting the existing functionality.

The optional "new name" is case-sensitive.
 2020-11-04 08:57:16 +01:00
ani-sha
fd6c8f9502 DBZ-2690 get oracle specific changes into debezium-core 2020-11-04 07:21:10 +01:00
rkerner
c274c2b39b DBZ-2617 refactor to proper HeartbeatErrorHandler injection 2020-11-03 07:23:02 +01:00
rkerner
2a3f44ebe5 DBZ-2617 failing heartbeats should put Postgres connector into FAILED state to allow proper shutdown of Postgres when shutdown was triggered with shutdown mode "fast" 2020-11-03 07:23:02 +01:00
Sergei Morozov
344acb55c0 DBZ-2682: Mapper converters should leave empty dates as zero 2020-10-27 16:21:53 +01:00
Jiri Pechanec
2cded66d30 DBZ-2632 Minor improvements 2020-10-20 16:28:23 +02:00
Arik Cohen
2e8c43bd60 DBZ-2632: Forcibly abort the JDBC connection if failed to close it gracefully. 2020-10-20 16:28:23 +02:00
Jiri Pechanec
ee5353ef6f DBZ-2653 Process commits even with tx monitoring 2020-10-20 10:42:10 +02:00
Kaushik Iyer
7d9e8fdc46 DBZ-2456 fixes after rebase 2020-10-16 10:13:48 +02:00
Kaushik Iyer
f752b3ab39 DBZ-2456 updated common connector config with generalized naming 2020-10-16 10:13:48 +02:00
Kaushik Iyer
e2e6dc0932 DBZ-2456 selective snapshot implementation 2020-10-16 10:13:48 +02:00
Gunnar Morling
b792ac3274 DBZ-2495 Misc. fixes 2020-09-30 12:13:45 +02:00
Gunnar Morling
6b5a3100e9 DBZ-2495 Misc. clean-up 2020-09-30 12:13:45 +02:00
James Gormley
9e780ac481 DBZ-2495 Updating per PR comments 2020-09-30 12:13:45 +02:00
James Gormley
1b845535ab DBZ-2495 Updating tables to utilize Table.editor for creating new tables 2020-09-30 12:13:45 +02:00
James Gormley
c7b53e1992 DBZ-2495 Updating docs and comments 2020-09-30 12:13:45 +02:00
James Gormley
0d46ec47da DBZ-2495 Updating to handle captured_column_list without specify column.exclude.list 2020-09-30 12:13:45 +02:00
Gunnar Morling
c1b057f792 DBZ-2588 Introducing dedicated functional interface for key/value generators; 
Also keys are always KC Struct instead of generic j.l.Object.
 2020-09-25 06:01:59 +02:00
Chris Cranford
df41cd8bfe DBZ-2396 Suggested changes 2020-09-24 07:41:36 +02:00
Chris Cranford
0db4997102 DBZ-2396 Support Converter delegation in ByteBufferConverter 
In the even that the value supplied to the ByteBufferConverter
is not supported, e.g. not BYTES, the converter should then
support delegation to a configured converter.
 2020-09-24 07:41:36 +02:00
Jiri Pechanec
73c74189fa DBZ-2549 Move scripting code into a separate module 2020-09-18 11:17:17 +02:00
Ganesh Ramasubramanian
6ca025b1bb DBZ-2504: Add support for field and header prefix 
Added support for field and header prefix.
Updated the documentation to include the options
 'add.fields.prefix' and 'add.headers.prefix'.
 2020-09-16 11:31:05 +02:00
rkerner
011c3be948 DBZ-2527 Add missing ListOfRegex validator to all regex list fields and fix dependent fields to not include @Deprecated whitelist/blacklist fields 2020-09-15 10:05:04 +02:00
rkerner
385a873172 DBZ-2526 add DisplayName for MASK and TRUNCATE COLUMN fields 2020-09-15 10:05:04 +02:00
rkerner
5c5fb5dc2c DBZ-2516 add config fields for datatype.propagate.source.type and column.propagate.source.type properties 2020-09-15 10:05:04 +02:00
Gunnar Morling
73ba6d42a0 DBZ-137 Docs update 2020-09-14 12:33:43 +02:00
Chris Cranford
a686c1c875 DBZ-137 Suggested changes 2020-09-14 12:33:43 +02:00
Chris Cranford
5ca36a84ad DBZ-137 Reimagine stream.fetch.size as query.fetch.size 2020-09-14 12:33:43 +02:00
Chris Cranford
8584831f51 DBZ-137 Use Duration for CONNECTION_TIMEOUT_MS 2020-09-14 12:33:43 +02:00
Chris Cranford
43440ca76f DBZ-137 Remove snapshot.skip.locks for SQL Server 2020-09-14 12:33:43 +02:00
Chris Cranford
72d59348ed DBZ-137 Fix rebase 2020-09-14 12:33:43 +02:00
AndreyIg
77662a0dde DBZ-137 contribution to the core, Oracle related 2020-09-14 12:33:43 +02:00
cooksey
8bcbdb639d DBZ-2094 Allow Postgres snapshotter to set streaming start on resume 
When a connector resumes after previously streaming and takes a
snapshot, through a new method on the snapshotter interface,
shouldStreamEventsStartingFromSnapshot, can choose whether
to resume streaming from the last known streaming position or the
beginning of the snapshot. This is helpful for snapshotters that
may not want to resnapshot every table in the whitelist/blacklist
but not miss event on the tables that are skipped.
 2020-09-09 10:54:11 +02:00
Grzegorz Kołakowski
c74579c1df DBZ-1068 Forward port the work by @grzegorz8 to master 
Co-authored-by: GuyIEX <gpascarella@ironeaglex.com>
 2020-09-04 12:06:57 +02:00
Nathan Mills
6e915b7cfd
DBZ-2473 Decrement remaining for skipped tables 2020-09-04 07:50:34 +02:00
Jiri Pechanec
d9f616caed DBZ-2303 Set source when new table detected 2020-08-24 13:08:09 +02:00
rkerner
2c5c093587 DBZ-2171 avoid divisive language 2020-08-21 09:29:04 +02:00
Gunnar Morling
4c3cf9f93e DBZ-2427 Logging inability to unregister metrics MBean as WARN 2020-08-13 14:05:11 -04:00
Gunnar Morling
dac7f3998c DBZ-2412 Housekeeping; 
* Static logger
* Protecting against empty topic name
* Removing unused logger
 2020-08-13 13:53:50 -04:00
Gunnar Morling
19ae707a18 DBZ-2412 Ignoring non change events in logical router 2020-08-13 13:53:50 -04:00
Gunnar Morling
44efd1bc6b DBZ-2423 Propagating port default values 2020-08-12 13:50:42 +02:00
Gunnar Morling
26b51e02cb DBZ-2418 Misc. clean-up 2020-08-12 09:28:35 +02:00
Jiri Pechanec
0b95df28d5 DBZ-2411 Use Scala 2.12 2020-08-11 09:20:53 +02:00
Jiri Pechanec
391d086529 DBZ-2411 Upgrade to Kafka 2.6.0 2020-08-11 09:20:53 +02:00
rkerner
cad6cf62d3 DBZ-2315 fix BIT VARYING handling in Postgres connector 
* removed little endian padding for BIT types in JdbcValueConverters (only used for Postgres yet)
* removed legacy format handling fot BIT related types in JdbcValueConverters#convertBits which were leftovers from PR #1408
* removed unnecessary zero-ing of a newly created byte array to improve performance for huge byte arrays
* updated Postgres connector docs
 2020-08-03 14:58:31 +02:00
Cory Harper
724fc7ac98 DBZ-2362 add configurable restart wait time 2020-07-30 18:08:26 +02:00
Jiri Pechanec
5fcb221993 DBZ-1491 Database connection provides default value 2020-07-24 09:07:09 +02:00
Jiri Pechanec
45a6f9bef1 DBZ-1491 Refactor after review 2020-07-24 09:07:09 +02:00
Grzegorz Kołakowski
825ffe3848 DBZ-1491 Add SqlServerDefaultValueConverter class 2020-07-24 09:07:07 +02:00
Grzegorz Kołakowski
234341a471 DBZ-1491 Parse temporal values using queries on database 
* Add tests for altering column default value
 * Add end-to-end test
 2020-07-24 09:06:30 +02:00
Grzegorz Kołakowski
a8ea5e2256 DBZ-1491 Extract default values from schema 
The goal is to enrich schema metadata with column default value.
 2020-07-24 09:05:58 +02:00
Gunnar Morling
3ebfbcefdb DBZ-2329 Moving isNewTransaction() to SQL Server implementation 2020-07-24 07:57:50 +02:00
Lukasz Korzeniowski
1ada5c4b23 DBZ-2329 Do not skip events after restart when stream is not monotonic 2020-07-21 14:21:29 +02:00
Jos Huiting
da97be7842 DBZ-2307 - Improve logging when primary keys are missing 2020-07-15 11:18:48 -04:00
Omar Al-Safi
4d2467ca20 DBZ-2302: Allow OffsetCommitPolicy to be initiated with Properties object 2020-07-09 08:38:26 +02:00
Randall Hauch
5860812a63
DBZ-2275 Updated default database history DML filter default to handle additional RDS statements; 
* Added two more regular expressions to the default database history DML filter. These are intended to catch and filter out `INSERT into mysql.rds_monitor…` statements that RDS may write in the binlog
* Added a new unit test that checks that the regular expressions do catch some example statements
 2020-06-25 20:57:49 +02:00
Jiri Pechanec
63a2f6a0ae DBZ-2255 Document JSON parsing in groovy-based SMTs 2020-06-25 11:08:59 +02:00
Jiri Pechanec
87d93d44fd DBZ-2221 Wait for keepalive thread initialization 2020-06-24 09:14:42 +02:00
Gunnar Morling
b80f064131 DBZ-2250 Allowing to use CloudEventsConverter with Avro via Apicurio 2020-06-22 14:41:10 +02:00
Gunnar Morling
f2d13238aa DBZ-2228 Using unique names for admin clients 2020-06-18 14:08:56 +02:00
Chris Cranford
ac0d9ede8a DBZ-2152 Fix test failure 2020-06-09 17:20:51 +02:00
Balázs Németh
0853940f3d
DBZ-2169 Fix for misuse of ConfigurationDefaults.RETURN_CONTROL_INTERVAL; 
* Implemented Temporals.min()
* Fixed pollInterval calculation based on the documentation
 2020-06-09 10:24:50 +02:00
Gunnar Morling
bfaadcf2dd DBZ-2149 Starting/stopping Kafka just once in KafkaDatabaseHistoryTest; 
By using distinct topic names in the tests, the same Kafka instance can
be used for all test methods. This brings down execution time of this
test from ~40sec to ~8sec.
 2020-06-09 10:15:47 +02:00
Chris Cranford
c3b36c2417 DBZ-2149 Log begin/end of schema recovery 2020-06-09 10:15:47 +02:00
Gunnar Morling
1fdad23087 Revert "DBZ-2166 Handling error condition where schema field can not be found in" 
This reverts commit 283740f37e.
 2020-06-08 20:12:50 +02:00
Dave Cumberland
4254e29667 DBZ-2152 Adding fix annotations to new tests 2020-06-08 13:57:35 -04:00
Dave Cumberland
a17e4cbad0 DBZ-2152 Adding another unit test with the key mapping definied 2020-06-08 13:57:35 -04:00
Dave Cumberland
223812def3 DBZ-2152 Updating the EventRouter to allow key types other than String 2020-06-08 13:57:35 -04:00
Fabian Aussems
283740f37e DBZ-2166 Handling error condition where schema field can not be found in 
TableSchemaBuilder
 2020-06-08 12:55:16 +02:00
Gunnar Morling
1b62036cf4 DBZ-2150 Awaiting coordinator shutdown in embedded engine 2020-06-08 10:26:37 +02:00
Gunnar Morling
c54e377dd2 DBZ-1814 Exporting Hex and Base64 representations as string 2020-06-05 12:13:29 +02:00
Gunnar Morling
023064eedb DBZ-1814 Renaming "raw" to "bytes"; also clarifying that this notion isn't Java-specific 2020-06-05 12:13:29 +02:00
Jeremy Finzel
5db614da21 DBZ-1814 added new binary handling mode option 2020-06-05 12:13:29 +02:00
Jiri Pechanec
86dac40dae DBZ-2140 Internal configs must not be validated 2020-06-04 11:47:26 +02:00
Kewei Shang
a963e89704 DBZ-2136 Update ChangeEventQueue Javadoc by fixing typo 2020-06-02 09:38:12 +02:00
Jiri Pechanec
8e7f3d14ca DBZ-1312 Omit constatnt db name in column and key mappers 2020-06-01 12:43:45 +02:00
Jiri Pechanec
20b45f84f3 DBZ-1657 Remove duplicate option definitions 2020-05-22 12:07:29 +02:00
Gunnar Morling
ed838fb2d4 DBZ-2089 Typo fix 2020-05-22 08:36:57 +02:00
Jiri Pechanec
84851297cb DBZ-2089 JMX no longer mandatory 2020-05-22 08:36:57 +02:00
Jiri Pechanec
e676d715c3 DBZ-2085 Apply custom converters to default values 2020-05-18 17:27:59 +02:00
Jiri Pechanec
c85546d611 DBZ-2074 Expose topic and headers in scripting SMTs 2020-05-15 11:13:07 +02:00
Gunnar Morling
0b42c8d8c2 DBZ-2024 Adding topic.regex option to routing and filtering SMTs 2020-05-14 08:41:40 +02:00
Jiri Pechanec
c4f975bd0a DBZ-2061 Handle change of default value 2020-05-14 08:31:27 +02:00
Gunnar Morling
4f020ca6cf DBZ-1828 Removing deprecated SMT options 2020-05-13 21:36:12 +02:00
Chris Cranford
f449550b2d DBZ-2068 Skip tests incompatible with Kafka 1.x 2020-05-13 08:22:07 +02:00
Aaron Brady
8b42016f63 DBZ-2034 Make ByLogicalTableRouter SMT unique key enforcement optional 2020-05-08 13:56:18 -04:00
Chris Cranford
c6ba581848 DBZ-2014 Suggested changes 2020-05-05 22:13:25 +02:00
Chris Cranford
cd43c4880f DBZ-2014 Do not include outbox eventType field by default 2020-05-05 22:13:25 +02:00
Chris Cranford
3c9ad5699b DBZ-2036 Change logging from warn to debug 2020-05-05 21:58:56 +02:00
Chris Cranford
81c3a808da DBZ-1963 Suggested changes 2020-05-05 08:45:58 +02:00
Chris Cranford
a1cb06299c DBZ-1963 Emit outbox events with value schema names 2020-05-05 08:45:58 +02:00
Jiri Pechanec
1b3fba4b8c DBZ-1904 Add drop table support 2020-05-04 12:31:05 +02:00
Jiri Pechanec
b9fa634968 DBZ-1904 Rename queueing methods 2020-05-04 12:31:05 +02:00
Jiri Pechanec
7f23d74d90 DBZ-1904 Add config param into a group 2020-05-04 12:31:05 +02:00
Jiri Pechanec
e350d74b17 DBZ-1904 Public facing schema history topic 2020-05-04 12:31:05 +02:00
Gunnar Morling
2310d6c792 DBZ-1969 Fixing incorrect assertion and utility which concealed this 2020-04-29 13:06:43 -04:00
Braden Groom
0b429ce880 DBZ-1969 Add support for Postgres enum arrays 2020-04-29 13:06:43 -04:00
Chris Cranford
230ddaebdc DBZ-1953 Generalized SkipWhenDatabaseVersion functionality 2020-04-29 12:27:38 +02:00
Gunnar Morling
5fb3b7f2a1 DBZ-2027 Removing some unused code 2020-04-29 09:56:59 +02:00
Gunnar Morling
e9632c2227 DBZ-1972 Formatting 2020-04-27 17:54:35 +02:00
Gunnar Morling
8b2448440b Revert "DBZ-2025 Reduce capture job poll interval" 
This reverts commit 131063fb2be38f865019e5dcff74b6d78299f6d2; it prove
unstable on CI.
 2020-04-27 16:38:01 +02:00
Jiri Pechanec
131063fb2b DBZ-2025 Reduce capture job poll interval 2020-04-27 13:56:29 +02:00
JanHendrikDolling
33d83731af DBZ-1972 move static Filed methods as static Filed constant to RelationalDatabaseConnectorConfig instead. 2020-04-27 12:31:19 +02:00
Gunnar Morling
f9bfe26754 DBZ-1753 Typo fix 2020-04-27 11:27:36 +02:00
Chris Cranford
f974f9f4ab DBZ-1753 Refactor common code between SQL Server and DB2 2020-04-24 13:45:06 -04:00
Jiri Pechanec
7e37237b52 DBZ-2000 Change expression param name; add docs 2020-04-24 15:44:52 +02:00
Gunnar Morling
90d2e24788 DBZ-2000 Misc. clean-up 2020-04-24 15:44:52 +02:00
Jiri Pechanec
3ceb6ced3e DBZ-2000 Content based router SMT 2020-04-24 15:44:52 +02:00
Gunnar Morling
03ce403972 DBZ-1750 Separating config definition editor and immutable config definitions 2020-04-24 15:35:28 +02:00
Jiri Pechanec
37f0dbcaf8 DBZ-1750 Limit access to parent defs; dont share the parent defs 2020-04-24 15:35:28 +02:00
Jiri Pechanec
0d8d6a41c5 DBZ-1750 Validate Kafka params only for Kafka history 2020-04-24 15:35:28 +02:00
Jiri Pechanec
af90507fa6 DBZ-1750 Centralize shared config definitions 2020-04-24 15:35:28 +02:00
Jiri Pechanec
412f288e88 DBZ-1923 Detect lost database history 2020-04-24 10:22:06 +02:00
Chris Cranford
24abc23f66 DBZ-2007 Upgrade Formatter and Impsort plugins 2020-04-23 11:21:06 +02:00
Gunnar Morling
2d0da37c03 DBZ-1993 Wrapping structs for simpler references in filter SMT; 
Exposing as ProxyObject allows for simple property-style access. We also can prevent write access that way.
 2020-04-23 08:33:30 +02:00
Gunnar Morling
0998fc533a DBZ-1895 Completing implementation; 
* Restoring original derserializers; excluding skipped events in event handlers
* Adding test
 2020-04-22 15:04:36 +02:00
Gunnar Morling
dae655c78b DBZ-2006 Misc. fixes to MongoDB connector docs 2020-04-22 11:42:21 +02:00
Gunnar Morling
1a5dd5db98 DBZ-1886 Misc. clean-up 2020-04-20 17:46:03 +02:00
Ivan Klass
75f19239d1 DBZ-1886 Introduce separate storageExists() method to support manual topic creation 2020-04-20 17:46:03 +02:00
Gunnar Morling
bc762f2989 DBZ-1962 Adding option descriptions 2020-04-17 15:24:42 +02:00
Jos Huiting
33fe631674 DBZ-1962 Fix PR feedback 2020-04-17 15:24:42 +02:00
Jos Huiting
8ba21a98ef DBZ-1962 Introduce column.whitelist for the Postgres connector 2020-04-17 15:24:42 +02:00
Gunnar Morling
786aab37d9 DBZ-1782 JavaDoc improvements; 
As per Chris's suggestions.

Co-Authored-By: Chris Cranford <ccranfor@redhat.com>
 2020-04-15 17:52:35 +02:00
Gunnar Morling
0673aba092 DBZ-1782 Inverting filtering logic, so to be consistent with Stream#filter() and other similar APIs 2020-04-15 17:52:35 +02:00
Gunnar Morling
0775bb1f4f DBZ-1782 Allowing any JSR 223 implementation for filtering SMT 2020-04-15 17:52:35 +02:00
Gunnar Morling
905629973e DBZ-1968 Removing SMTs deprecated in 0.10 2020-04-15 15:11:32 +02:00
Chris Cranford
e87bfefdf7 DBZ-1859 Renamed MetadataEvent to ConnectorEvent 2020-04-15 08:04:23 +02:00
Chris Cranford
57af80afd5 DBZ-1859 Enhance MongoDB connector metrics 2020-04-15 08:04:23 +02:00
Gunnar Morling
680f1f73af DBZ-1782 Misc. fixes 2020-04-09 18:09:56 +02:00
Jiri Pechanec
08d2c9679f DBZ-1782 Support for Graal JS 2020-04-09 18:09:56 +02:00
Jiri Pechanec
53c8603148 DBZ-1782 Use JSR223 API 2020-04-09 18:09:56 +02:00
Jiri Pechanec
112aeb9d21 DBZ-1782 Groovy based filtering SMT 2020-04-09 18:09:56 +02:00
JanHendrikDolling
1e19270981 DBZ-1692 Added support for hash based column masking mode for string values 2020-04-08 21:21:42 +02:00
Jiri Pechanec
384f62463c DBZ-1905 Stop checking thread upong completion 2020-04-08 12:18:32 +02:00
Jiri Pechanec
318d1d3cc7 DBZ-1905 Make thread-naming params internal and optional 2020-04-08 12:18:32 +02:00
Jiri Pechanec
e81aec0ed1 DBZ-1905 Validate database history topic 2020-04-08 12:18:32 +02:00
Chris Cranford
65c8564d66 DBZ-1925 Introduced SkipWhenKafkaVersion annotation support 2020-04-07 09:01:20 +02:00
Jiri Pechanec
6338309cea DBZ-1918 Initialize Metronome when needed 2020-04-01 09:48:24 +02:00
Gunnar Morling
89bb6eba6c DBZ-1531 Misc. clean-up; 
* Removing obsolete default impl
* Completing test
* Documention fixes
 2020-03-25 12:33:38 +01:00
rk3rn3r
7f36b3c08f DBZ-1531 added headers for primary key update events to reference the original key 2020-03-25 12:33:38 +01:00
rk3rn3r
6310e00675 DBZ-1531 added headers for primary key update events to reference the original key 2020-03-25 12:33:38 +01:00
rk3rn3r
fbd8b94e94 DBZ-1531 added headers for primary key update events to reference the original key 2020-03-25 12:33:38 +01:00
Jiri Pechanec
32b6a8cb3c DBZ-1889 Use parker 2020-03-23 13:56:29 +01:00
Jiri Pechanec
d2bdcd1ab3 DBZ-1889 Prevent Connect throttling while connection is down 2020-03-23 13:56:29 +01:00
Jiri Pechanec
224f97e380 DBZ-1857 Support for autorestart for embedded engine 2020-03-23 13:56:29 +01:00
Gunnar Morling
bbaf32fe82 DBZ-1815 Misc. clean-up 2020-03-18 10:27:22 +01:00
trolison
8784b41e22 DBZ-1815 Ading DatabaseHeartbeatImpl to allow for making queries to the DB when a heartbeat happens; 
* Minor refactor to make DatabaseHeartbeatImpl usable
* Added integration test to verify heartbeat.action.query is being executed properly
 2020-03-18 10:27:22 +01:00
Gunnar Morling
e4a1272dff DBZ-1876 Adding comment 2020-03-13 16:21:33 +01:00
Jiri Pechanec
6e482de3d5 DBZ-1876 Process headers with tombstone 2020-03-13 16:21:33 +01:00
Gunnar Morling
449154beaa DBZ-1863 Centralizing coordinator shutdown 2020-03-12 11:28:08 +01:00
Gunnar Morling
da1f7efb21 DBZ-1723 Formatting 2020-03-11 13:57:23 +01:00
Gunnar Morling
7e55315f78 DBZ-1723 Encapsulating retriable determination in error handler 2020-03-11 13:57:23 +01:00
Gunnar Morling
c51a5fb597 DBZ-1723 Restarting after connection losses in Postgres 2020-03-11 13:57:23 +01:00
Gunnar Morling
63404e14ca DBZ-1723 Infrastructure for enabling restarts after retriable errors 2020-03-11 13:57:23 +01:00
Gunnar Morling
3437da00d7 DBZ-1723 Simplifying shutdown handling; 
doStop() is already called from stop(), no need to do it again via the error handler.
 2020-03-11 13:57:23 +01:00
Gunnar Morling
c156d30048 DBZ-1723 Centralizing stopping() logic in BaseSourceTask 2020-03-11 13:57:23 +01:00
Gunnar Morling
8ce69dc597 DBZ-1723 Encapsulating common poll() and stop() logic in BaseSourceTask 2020-03-11 13:57:23 +01:00
Gunnar Morling
6318cf0f2b DBZ-1831 Pulling up connector option retrieval to CommonConnectorConfig; 
* Expanding test
* Adding Sayed to COPYRIGHT.txt
 2020-03-11 12:43:25 +01:00
Sayed Mohammad Hossein Torabi
63a462dbc0 DBZ-1831 Support MongoDB Oplog operations as config 2020-03-11 12:43:25 +01:00
Jiri Pechanec
082761effc DBZ-1856 Deprecation process is not necessary 2020-03-11 12:02:07 +01:00
Chris Cranford
dc1d526caa DBZ-1726 Fix test failure for relational connectors 2020-03-11 10:15:00 +01:00
Chris Cranford
3a4788cc33 DBZ-1726 Introduced AbstractSnapshotChangeEventSource and refactor implementations 2020-03-11 10:15:00 +01:00
Chris Cranford
38cf4ceae1 DBZ-1726 Introduce DataCollectionId#identifier() 2020-03-11 10:15:00 +01:00
Chris Cranford
4e27c90028 DBZ-1726 Move MongoDB connector to common framework 2020-03-11 10:15:00 +01:00
Fabio Cantarini
62b46ae596 DBZ-1830 Enhancement to datatype.propagate.source.type connector property 
Add support for matching db and table name
Amend documentation accordingly
 2020-03-10 14:58:16 +01:00
Fabio Cantarini
61cc791d32 DBZ-1830 Add support for new connector property datatype.propagate.source.type 2020-03-10 14:58:16 +01:00
Jiri Pechanec
0f11f7e6de DBZ-1834 Support separately quoted schema and table 2020-03-03 10:56:19 +01:00
Jiri Pechanec
c201578d29 DBZ-1834 Support dots in table names 2020-03-03 10:56:19 +01:00
Jiri Pechanec
7d525d4baa DBZ-1134 Fix tests for new API 2020-02-28 21:58:39 +01:00
Gunnar Morling
0f6d2c54fc DBZ-1134 Avoiding concrete class in SPI by introducing receiver parameter 2020-02-28 21:58:39 +01:00
Gunnar Morling
e7fd7acab8 DBZ-1134 Making converter registry thread-safe; misc. clean-up 2020-02-28 21:58:39 +01:00
Jiri Pechanec
096d13f266 DBZ-1134 Use OptionalInts 2020-02-28 21:58:39 +01:00
Jiri Pechanec
d13e48cc55 DBZ-1134 Make converted field type parametrized 2020-02-28 21:58:39 +01:00
Jiri Pechanec
06e1eca085 DBZ-1134 Support for user provided converters 2020-02-28 21:58:39 +01:00
Jiri Pechanec
06049a5e9d DBZ-1826 Rename error handling config field 2020-02-26 15:49:26 +01:00
Jiri Pechanec
91f292a1f2 DBZ-1825 Keep keys for non-matching table 2020-02-26 15:03:03 +01:00
Jiri Pechanec
ae6ee3d458 DBZ-234 Place Incubating into a unique package 2020-02-25 15:31:10 +01:00
Jiri Pechanec
f66fe17989 DBZ-234 Incubating annotation moved to API 2020-02-25 15:31:10 +01:00
Jiri Pechanec
436615934d DBZ-1760 Align code and docs 2020-02-25 10:43:11 +01:00
Jiri Pechanec
2b6e50daff DBZ-1760 Allow skipping of invalid events 2020-02-25 10:43:11 +01:00
Gunnar Morling
5650276ba9 DBZ-1805 Removing unused private method parameter 2020-02-18 10:29:45 +01:00
Jiri Pechanec
566efd6a3f DBZ-1775 Filter out rds_monitor DML events 2020-02-14 14:26:02 +01:00
Gunnar Morling
1e51f112b2 DBZ-1772 Trimming values for add.source.fields option 2020-02-14 12:51:58 +01:00
Gunnar Morling
61c4f834a6 DBZ-1452 Misc. fixes and refactoring; 
* Adding support for "transaction" struct
* Documentation updates
* Only one "_" as separator between struct and field name
 2020-02-12 14:03:19 -05:00
daanroosen-DS
49bbec319b DBZ-1452 Adding add.fields and add.headers options to flattening SMT 2020-02-12 14:03:19 -05:00
Jiri Pechanec
a998db9caa DBZ-1767 Support field sanitization for MongoDB unwrap 2020-02-12 18:49:32 +01:00
Gunnar Morling
92fa6039ab DBZ-1766 Adding Melissa Winstanley to COPYRIGHT.txt 2020-02-12 15:44:45 +01:00
Gunnar Morling
dd5c2b8be9 DBZ-1766 Unifying handling of offset commits 2020-02-12 15:44:45 +01:00
Matthias Wessendorf
bff73e71d0
DBZ-1784 Use "application/avro" as data content type in CloudEvents 
Signed-off-by: Matthias Wessendorf <mwessend@redhat.com>
 2020-02-12 14:26:25 +01:00
Jiri Pechanec
77f9a61b95 DBZ-1751 Centralize envelope schema naming 2020-02-04 20:37:19 +01:00
Gunnar Morling
b66f314b3c DBZ-1052 Misc. clean-up 
* Doc updates
* Changed static method to instance
 2020-02-03 09:29:10 +01:00
Jiri Pechanec
ba10a229b6 DBZ-1052 CE support for tx metadata in data messages 2020-02-03 09:29:10 +01:00
Jiri Pechanec
0e671d6167 DBZ-1052 Add counts per table to END 2020-02-03 09:29:10 +01:00
Jiri Pechanec
60056b0a94 DBZ-1052 Transaction block moved to envelope 2020-02-03 09:29:10 +01:00
Jiri Pechanec
4867c8a298 DBZ-1052 Change TX topic name 2020-02-03 09:29:10 +01:00
Jiri Pechanec
5749acda7e DBZ-1052 Remove unused fields; added JavaDoc 2020-02-03 09:29:10 +01:00
Jiri Pechanec
c0557b325b DBZ-1052 Skip CE for transaction struct 2020-02-03 09:29:10 +01:00
Jiri Pechanec
cd6a1f942d DBZ-1052 Transaction metadata for Postgres/pgoutput 2020-02-03 09:29:10 +01:00
Jiri Pechanec
b0eb571f9f DBZ-1052 Transaction metadata for SQL Server 2020-02-03 09:29:10 +01:00
Jiri Pechanec
6f7b8d1707 DBZ-1748 Mask JDBC passwords from log messages 2020-01-31 17:26:59 +01:00
Gunnar Morling
2d337133ae DBZ-1715 Docs clarification; adding Jos to authors list 2020-01-17 14:52:15 +01:00
Jos Huiting
1cd4d4bf19 Add field-based topic routing to Debezium ExtractRecordState 2020-01-17 14:43:03 +01:00
Gunnar Morling
8ca6174d98 DBZ-1292 Applying suggestions from code review 
Co-Authored-By: Chris Cranford <ccranfor@redhat.com>
 2020-01-16 08:02:51 +01:00
Gunnar Morling
8c33a4e147 DBZ-1292 Catering for the differences in data schema; 
* MongoDB doesn't have "before", but "patch"
* Also removing some redundancies and making RecordParser immutable
* Reducing log verbosity during tests
 2020-01-16 08:01:56 +01:00
Gunnar Morling
8531adbefd DBZ-1292 Formatting 2020-01-16 08:00:45 +01:00
Gunnar Morling
2ba1d615fb DBZ-1292 Distinct names for data and envelope schema for Avro/Avro 2020-01-16 08:00:45 +01:00
Gunnar Morling
8591dd669b DBZ-1292 Avoiding usage of reflection 2020-01-16 08:00:45 +01:00
Gunnar Morling
7c18a6d8d5 DBZ-1292 Removing static state in CloudEventsConverterTest 2020-01-16 08:00:45 +01:00
Gunnar Morling
366382ddad DBZ-1292 Refactoring of options; 
* Avoiding hard dependency to Avro
* Passing through all options from JSON and Avro converters by means of "json." and "avro." sub-namespaces
 2020-01-16 08:00:45 +01:00
Gunnar Morling
12e43a89f4 DBZ-1292 Always adding "before" and "after" 2020-01-16 08:00:45 +01:00
Gunnar Morling
a861dae432 DBZ-1292 Clean-up around data schema URI 2020-01-16 08:00:45 +01:00
Gunnar Morling
1a18bdbc81 DBZ-1292 Misc. refactorings; 
* Protecting against Avro as CE format and JSON as data format
* Reducing mutability
* Single Avro client
 2020-01-16 08:00:45 +01:00
Gunnar Morling
4e4800cb56 DBZ-1292 Some clean-up in CloudEventsMaker; 
* Avoiding non-final static fields
* Removing unused methods
* Reducing visibility
 2020-01-16 08:00:45 +01:00
Gunnar Morling
fe8c5fe27d DBZ-1292 Adding schema URI in case of Avro 2020-01-16 08:00:45 +01:00
Gunnar Morling
a25d148a82 DBZ-1292 Adding logical name to CE source attribute value 2020-01-16 08:00:45 +01:00
Gunnar Morling
342a448053 DBZ-1292 Using "iodebezium" prefix for extension attributes; 
Simplifying implementation a bit.
 2020-01-16 08:00:45 +01:00
Gunnar Morling
390c8f8090 DBZ-1292 Wrapping up work on schema and value conversion; 
* There were a few attributes missing in the schema of the CE event
* Conversion of extension attributes now correctly handles optional attributes
* Removed guards against missing and duplicate fields, as the only caller is this class itself, so it seems superfluous.
 2020-01-16 08:00:45 +01:00
GraySmog
de2655a89e DBZ-1292 promoting op, ts_ms and sub-fields in source to extension attributes 2020-01-16 08:00:45 +01:00
GraySmog
d0e4b3c0af DBZ-1292 Delete data.schema.registry.url config option 2020-01-16 08:00:45 +01:00
GraySmog
8b7c67cbdd DBZ-1292 Using formatted timestamp for CE time field 2020-01-16 08:00:45 +01:00
Gunnar Morling
87ffe0c44e DBZ-1292 Representing data as JSON struct instead of binary 2020-01-16 08:00:45 +01:00
Gunnar Morling
6356784cee DBZ-1292 dataschema value should not be empty or null if key is present; 
* CloudEvents spec version is 1.0 now
 2020-01-16 08:00:45 +01:00
GraySmog
915c8cafd9 DBZ-1292 CloudEvents converter for JSON and Avro format 2020-01-16 08:00:45 +01:00
Gunnar Morling
07e89de91d DBZ-1707 Ensuring correct Kafka record timestamp for outbox events; 
Depending on the width of the configured TIMESTAMP column, a microsecond or nanosecond timestamp could have been returned before, whereas this should always be milliseconds.
 2020-01-15 07:32:55 +01:00
Chris Cranford
4a27746dad DBZ-1703 Fix typo after renaming configuration option 2020-01-14 09:24:53 +01:00
Chris Cranford
4fa0f22ac7 DBZ-1703 Add option to JSON SerDes for ignoring unknown properties 2020-01-14 09:24:53 +01:00
liulangwa
6028d78e76 DBZ-1688 Fix SQLserver type time(4)...time(7) lost nanoseconds 2020-01-02 12:26:28 +01:00
Jiri Pechanec
761e7440cc DBZ-1670 Rename Serdes to DebeziumSerdes 2019-12-16 15:05:43 +01:00
Cobolbaby
f199988609 DBZ-1642 commit() method in JdbcConnection compatible with non autocommit 2019-12-13 10:34:52 +01:00
Cobolbaby
641e24b483 DBZ-1642 Create method called commit() in JdbcConnection class 2019-12-13 10:34:52 +01:00
Chris Cranford
19fd8dd1d9 DBZ-1413 Resolve column length/scale when reading table column metadata from TypeRegistry 2019-12-10 10:09:35 +01:00
Chris Cranford
365eb0d76e DBZ-1413 Suggested Javadoc changes, added/fix tests, changed how Column native/jdbc type resolution happens 2019-12-10 10:09:35 +01:00
Chris Cranford
4bce11f92a DBZ-1413 Remove some System.out madness 2019-12-10 10:09:35 +01:00
Chris Cranford
547ba8e0db DBZ-1413 Align domain type support across all 3 decoders 2019-12-10 10:09:35 +01:00
Gunnar Morling
68e7b7a0bb DBZ-1533 Documentation update 2019-12-10 10:07:55 +01:00
Jiri Pechanec
1805d1d73b DBZ-1533 Added documentation 2019-12-10 10:07:55 +01:00
Jiri Pechanec
7e4ea91a10 DBZ-1533 Allow keeping envelope 2019-12-10 10:07:55 +01:00
Jiri Pechanec
8a64ce71d2 DBZ-1533 JSON payload serde 2019-12-10 10:07:55 +01:00
Gurnaaz Randhawa
4f60fac367 DBZ-1650 Changing sorting into a HashSet to a LinkedHashSet 2019-12-05 09:50:25 +01:00
Thomas Deblock
17e44d4c0c DBZ-1648 outbox EventRouter remove topic toLowerCase 
- kafka topics names are case sensitive. We should not perform
  a toLowerCase on the topic name. It can cause some issue if
  topic name have a upper case
 2019-12-04 21:39:25 +01:00
Ivan San Jose
cc8dd160f5 DBZ-1617 Blacklisted columns are not being filtered out when generating a Kafka message from a CDC event 2019-11-25 13:23:03 +01:00
Brandon Brown
a0f7f6ec6b DBZ-1402 Support for JDK 11 2019-11-25 11:53:19 +01:00
Jiri Pechanec
a56ab78b3f DBZ-825 Unify decimal upscaling 2019-11-20 18:23:55 +01:00
pan3793
33384098fd DBZ-1621 Remove useless null check 2019-11-19 07:35:03 +01:00
pan3793
91f0c2a520 DBZ-1621 Fix IoUtil.createDirectory parameter removeExistingContent is not take effect 2019-11-19 07:35:03 +01:00
pan3793
89fde5ce06 DBZ-1622 Fix VariableLatch.create parameter initialValue is not take effect 2019-11-19 07:33:52 +01:00
David Feinblum
2b9fbcd5f6 DBZ-1519 Avoid erroneously logging there are no monitored tables 2019-11-14 05:44:37 +01:00
Jiri Pechanec
2b9536f24d DBZ-1607 use Instant for envelope timestamps 2019-11-12 16:36:11 -05:00
Gunnar Morling
da2b6c2c78 DBZ-1555 JavaDoc fix; formatting 2019-11-08 15:09:46 +01:00
Luis Garcés-Erice
0198bceffa DBZ-1555 Enabling configuration of table parse mode to support DB2 2019-11-08 15:07:25 +01:00
Gunnar Morling
2c86b1c84f DBZ-1604 Documentation update 2019-11-08 10:23:23 +01:00
Yang Yang
884def88d7 DBZ-1604 Added a new converter to enable byte[] payload 2019-11-08 10:19:21 +01:00
Jiri Pechanec
d1ee65d1b2 DBZ-1569 Don't compute index statistics 2019-11-06 15:43:21 +01:00
Jiri Pechanec
c179076710 DBZ-1592 Generate heartbeats for filtred events 2019-11-05 14:54:56 -05:00
Chris Cranford
f97be3272c DBZ-1590 Fix LogInterceptor to be thread-safe 2019-11-04 16:23:43 -05:00
Gunnar Morling
2c569977d7 DBZ-1392 Applying formatter; 
* Keeping indentation formatting for JSON


Co-authored-by: Gunnar Morling <gunnar.morling@googlemail.com>
Co-authored-by: Chris Cranford <chris@hibernate.org>
 2019-11-01 13:20:43 +02:00
Mike Graham
13c6c67a1d fix spelling error in exception message 2019-10-24 22:00:16 +02:00
Gunnar Morling
e06d25dd92 DBZ-1336 Moving Ltree type to PG connector module 2019-10-21 12:17:08 +02:00
Mincong HUANG
1e3eb8c840 DBZ-1336 Support Postgres LTREE columns 2019-10-21 12:10:46 +02:00
Jiri Pechanec
a92b93182e DBZ-1560 Removed dropped PK from the list 2019-10-20 22:13:02 +02:00
Gunnar Morling
4ea85d7fd3 DBZ-1565 Raise warning about growing WAL backlog 2019-10-18 13:53:53 +03:00
Grant Cooksey
18ca6ec72e DBZ-1546 Comma-separated regular expressions remove whitespace. 
The table whitespace parameter should remove whitespace between regular expressions
since tables cannot start with whitespace characters.
 2019-10-17 10:13:44 +03:00
Jiri Pechanec
6d8412e5de DBZ-1401 Support for pgoutput 2019-10-10 09:37:52 +02:00
Jiri Pechanec
7c4203cf60 DBZ-1401 Handle tables without explicit PK 2019-10-10 09:37:52 +02:00
Jiri Pechanec
cf7eccb152 DBZ-1498 Docs added; microseconds changed to numeric; nanos supported 2019-10-09 12:14:35 +02:00
Jiri Pechanec
20363f24bb DBZ-1498 Support for precise interval value 2019-10-09 12:14:35 +02:00
Jiri Pechanec
74fd88afca DBZ-1513 Improve start logging 2019-10-07 18:38:35 +02:00
Jiri Pechanec
2c4cb83259 DBZ-1513 Centralize handling of non-data messages in SMT 2019-10-07 18:38:35 +02:00
Gunnar Morling
92885ebe99 DBZ-1392 Removing trailing whitespace from Java files 2019-10-07 12:11:43 +03:00
Gunnar Morling
8d6f5409b1 DBZ-1392 Moving static imports up and applying this to all files 2019-10-07 12:11:43 +03:00
Chris Cranford
3594ff639a DBZ-1341 Rebased against master and applied necessary checkstyle fixes 2019-10-03 19:44:43 +02:00
Chris Cranford
ca5517746a DBZ-1341 Checkstyle flag missing whitespace 2019-10-03 19:44:43 +02:00
Chris Cranford
7442670692 DBZ-1355 Fix checkstyle problems detected after upgrade to 8.19 2019-10-03 19:44:43 +02:00
Chris Cranford
5ac2318d84 DBZ-1391 Do not allow code after closing brace (#1065) 
* DBZ-1391 Do not allow code after closing brace
 2019-10-02 20:44:40 +02:00
Jiri Pechanec
f868cb87c0 DBZ-1280 Lock timeout for SQL Server 2019-10-02 10:34:25 +02:00
pushpavanthar
891c8b90f4 DBZ-1492 - MySQL Connector Fails while parsing AWS RDS internal events 2019-10-02 10:31:09 +02:00
Gunnar Morling
d94b37c693 DBZ-1507 Misc. clean-up; 
* Removing unused Table methods
* Re-using primaryKeyColumnNames() instead of custom predicate
 2019-10-02 10:25:37 +02:00
Chris Cranford
d906096820 DBZ-1507 Ensure message keys have correct field order 2019-10-02 10:25:37 +02:00
Jiri Pechanec
410662c4aa DBZ-1517 Skip unknow records while operation header is used 2019-09-30 20:25:44 +02:00
Gunnar Morling
2fba51863e DBZ-1015 Unifying the retrieval of custom key mappers 2019-09-26 10:14:48 +02:00
Guillaume Rosauro
1405889bef DBZ-1015-allow-user-to-customize-key 2019-09-26 10:14:21 +02:00
Gunnar Morling
236bae4265 DBZ-1508 Avoiding assertion based on platform-specific line endings 2019-09-26 09:25:24 +02:00
Gunnar Morling
442408be02 DBZ-1497 Using int64 for INTERVAL microseconds mapping 2019-09-25 11:21:40 +03:00
Chris Cranford
b20e689b2c DBZ-1495 Avoid NPE by throwing more meaningful error messages 2019-09-25 08:08:16 +03:00
Jiri Pechanec
8f308746d3 DBZ-1438 Interrupt only after interrupt affected operations 2019-09-24 12:51:48 +02:00
jerrinot
3064a4af06 DBZ-1438 Fix InterruptedException handling 
It appears there is an intent to restore the interruption flag
when catching InterruptedException. However the old code was just
checking the current state of the flag instead of raising it.
 2019-09-24 12:51:48 +02:00
Gunnar Morling
9de7c25d60 DBZ-1405 Misc adjustments; 
* Adding test
* Removing remaining double variables
 2019-09-24 12:08:55 +02:00
Bingqin Zhou
d6353dd519 DBZ-1405 Add NanoDuration. 2019-09-24 12:08:55 +02:00
Gunnar Morling
c5d0b18ea3 DBZ-1044 "AVRO" -> "Avro", as it's not an abbreviation 2019-09-23 12:21:54 +02:00
Chris Cranford
074cbf01a3 DBZ-1044 Checkstyle fix: remove unused import 2019-09-20 07:40:23 +03:00
Chris Cranford
6dc37fc8f6 DBZ-1044 Fix broken test caused by previous commit 2019-09-20 07:40:23 +03:00
Chris Cranford
910d00f6a2 DBZ-1044 Fixed documentation typo, added tests, adjusted FieldNameSelector 2019-09-20 07:40:23 +03:00
Chris Cranford
25086e82e3 DBZ-1044 Add auto-Enable sanitize.field.names support 
When the connector detects that its configuration explicitly specifies
that "key.converter" or "value.converter" are using the AvroConverter,
the "sanitize.field.names" option will be automatically enabled.
 2019-09-20 07:40:23 +03:00
Chris Cranford
130210b5ad DBZ-1044 Add config option sanitize.field.names and FieldNameSelector 2019-09-20 07:40:23 +03:00
Josh Arenberg
6573738fe9 DBZ-1044 Sanitize column/topic names to always be valid Avro Names 
This patch fixes the issue where more lenient standards for name conventions in source databases cause the
snapshot to fail when the schema for the table is created. This patch will prepend any column names that start
with a digit with an underscore, and will replace any subsequent invalid characters with an underscore. This should
conform to the Avro Name standard here: https://avro.apache.org/docs/1.7.7/spec.html#Names
 2019-09-20 07:40:23 +03:00
Renato Mefi
405a3b5b43 DBZ-1457 Do not commit when offset is null 2019-09-03 22:39:37 +02:00
Gunnar Morling
3215a3ce0a DBZ-1448 Misc. clean-up 2019-09-03 10:39:41 +02:00
garrett528
1aa7d70b0d DBZ-1448 Added support for source fields to be added to delete messages rewrites 2019-09-03 10:21:56 +02:00
Gunnar Morling
2a065686bd DBZ-1255 Exporting null value when temporal precision mode is "connect" and the value is out of the supported range 2019-08-21 07:50:40 +02:00
Gunnar Morling
fe9fc1c0dc DBZ-1255 Adjustments after rebase 2019-08-21 07:50:40 +02:00
Gunnar Morling
08ef1418d9 DBZ-DBZ-1255 Preventing value overflows for TIMESTAMP; 
* Generally avoiding conversion to nano-seconds to accomodate for larger values
* Generally avoiding the usage of long to convey temporal values internally:
 * Instant is used for TIMESTAMP
 * OffsetDateTime for TIMESTAMPTZ
 * Duration for TIME (LocalTime cannot be used as it doesn't support 24:00:00 as possible with Postgres)
 * OffsetTime for TIMETZ
* Avoiding usage of methods under test for calculating expected values in tests
 2019-08-21 07:50:40 +02:00
Gunnar Morling
94eb75257b DBZ-1255 Using Instant for representing timestamps 2019-08-21 07:50:40 +02:00
Jiri Pechanec
51c12d930d DBZ-1419 Support for Connect temporal types in SQL Server 2019-08-20 18:43:03 +02:00
Jiri Pechanec
540021cf68 DBZ-777 Minor fixes 2019-08-15 20:43:31 +02:00
Gunnar Morling
5ffe8123f1 DBZ-777 Further fixes 2019-08-15 20:43:31 +02:00
Gunnar Morling
63250d0c13 DBZ-777 Misc. adjustments 
* Avoiding repeated parsing of table id
* Making 'inconsistentSchemaHandler' field final in EventDispatcher
* Not exposing synchronizeTableSchema() on SPI level
 2019-08-15 20:43:31 +02:00
Jiri Pechanec
037d581844 DBZ-777 Fix heartbeats 2019-08-15 20:43:31 +02:00
Jiri Pechanec
d72afa8e9b DBZ-777 Skip empty messages 2019-08-15 20:43:31 +02:00
Jiri Pechanec
f4246df6e4 DBZ-777 PostgreSQL connector rewrite to the standard framework 2019-08-15 20:43:31 +02:00
Jiri Pechanec
8adb5e48ca DBZ-1385 Elevate payload for only headers setup 2019-08-12 10:36:52 +02:00
Jiri Pechanec
0991c70ac9 DBZ-1385 eventType can be configured as add field 2019-08-12 10:36:52 +02:00
Guillaume Rosauro
4b44e5f925 [DBZ-1254] Re-order the captured tables in the same order as the 
'table.whitelist' parameter
 2019-07-25 07:33:22 +02:00
Adrian Kreuziger
00843cb0fb DBZ-1395 Update to use BoundedConcurrentHashMap for schema cache 2019-07-18 10:26:05 +02:00
Adrian Kreuziger
6c63c7daa0 DBZ-1395 Throw exception for missing source field 2019-07-18 10:26:05 +02:00
Adrian Kreuziger
63299022d7 DBZ-1395 Review feedback (keep field type, minor tweaks) 2019-07-18 10:26:05 +02:00
Adrian Kreuziger
dda17b9aa2 DBZ-1395 Undo accidental change to existing test 2019-07-18 10:26:05 +02:00
Adrian Kreuziger
c7684c351c DBZ-1395 Add ability to insert field from original source struct to unrwrapped record 2019-07-18 10:26:05 +02:00
Jiri Pechanec
3008339142 DBZ-916 Keyless tables have null key by default 2019-07-17 14:41:37 -04:00
Gunnar Morling
3b6e6b7719
Merge pull request #955 from jpechane/DBZ-1198 
DBZ-1198 Heartbeat for MongoDB connector
 2019-07-11 07:44:04 +02:00
Gunnar Morling
287551294f DBZ-1198 Adding comments 2019-07-11 07:43:32 +02:00
Renato Mefi
1b03fd3c46 DBZ-1388 Ensure Outbox SMT skips non CDC records 2019-07-10 22:32:23 +02:00
Jiri Pechanec
0316ac7fc0 DBZ-1198 Heartbeat for MongoDB connector 2019-07-10 11:16:26 +02:00
Gunnar Morling
369f74d875 DBZ-1320 Addressing review remarks 2019-07-09 21:07:18 +02:00
Brandon Brown
efff4f0809 DBZ-1320 Adding support for deletion events via new configurable property route.tombstone.on.empty.payload 2019-07-09 21:07:18 +02:00
Jiri Pechanec
67ca8a2d76 DBZ-794 Filter out savepoints from database history 2019-07-08 12:54:01 +02:00
Gunnar Morling
e2d8736a79 DBZ-1380 Misc. clean-up; 
* reducing visibility of some methods
* resetting interrupted state
 2019-07-05 21:08:35 +02:00
Gunnar Morling
02427a054b DBZ-1222 EventsInError -> ErroneousEvents 2019-07-04 21:31:32 +02:00
Jiri Pechanec
c08a5374fe DBZ-1222 Improve error event method name 2019-07-04 21:31:32 +02:00
Jiri Pechanec
2b365be63c DBZ-1222 EventsInError metric added 2019-07-04 21:31:32 +02:00
Jiri Pechanec
13d638fd43 DBZ-1356 Metrics for schema history (#947) 
* DBZ-1356 Metrics for schema history
 2019-07-02 13:26:17 +02:00
Jiri Pechanec
b1c1c05d1f DBZ-1363 Generate heartbeat periodically not upon event 2019-07-02 12:02:33 +02:00
Gunnar Morling
08c5d36789
Merge pull request #942 from jpechane/stab-lsn-flush-test 
DBZ-1362 Stabilize test executions on CI
 2019-07-01 09:34:24 +02:00
Jiri Pechanec
b546dc886e DBZ-1362 Fix Snapshot/streaming switch race condition 2019-06-28 09:09:18 +02:00
Gunnar Morling
254dba97ee DBZ-1358 Avoiding instantiation of SystemTime 2019-06-28 08:54:04 +02:00
Jiri Pechanec
a0df2c140b Use Kafka SystemTime implementation 2019-06-28 08:54:04 +02:00
Jiri Pechanec
53f4db06b9 DBZ-1358 Update to Kafka 2.3.0 2019-06-28 08:54:04 +02:00
Jiri Pechanec
7e03dbe87e DBZ-1362 Log test method executions 2019-06-28 05:41:02 +02:00
Jiri Pechanec
82d245acb1 DBZ-1362 Set longer ZooKeeper timeout 2019-06-28 05:33:49 +02:00
Gunnar Morling
a41fb30eaa DBZ-1361 Restoring compatibility with Apache Kafka 1.x 2019-06-28 05:07:52 +02:00
Chris Cranford
d518fa59fe DBZ-1345 Should not snapshot materialized views 2019-06-25 12:44:00 +02:00
Cheng Pan
186babeabf DBZ-1346 Fixing description for state of snapshot 2019-06-24 10:52:16 +02:00
Gunnar Morling
d3e399e167 DBZ-1340 Avoiding repeated case conversion of incoming DML statements; 
Also adding startsWithIgnoreCase() method.
 2019-06-20 13:33:12 +02:00
Jiri Pechanec
3012524ea1 DBZ-1278 Rename to non-whitelisted table sends warn 2019-06-19 17:05:26 +02:00
Gunnar Morling
ee87f40f0c DBZ-1270 Avoiding usage of deprecated method; adding Stathis Souris to authors 2019-06-14 11:35:53 +02:00
Gunnar Morling
bb360e472c DBZ-1270 Logging improvements 2019-06-14 11:35:53 +02:00
ssouris
d513e2c757 DBZ-1270 - Fix KafkaDatabaseHistory throwing even after creating the topic in older brokers 2019-06-14 11:35:53 +02:00
Jiri Pechanec
9b5640ddde DBZ-835 Emit tombstone optionally 2019-06-12 21:39:13 +02:00
Gunnar Morling
5d01f409f0 DBZ-1317 Separating update and query methods for MongoDB's SourceInfo; 
* Avoids access to collection id after events which don't have one
* Pulling up struct() and schema() methods to common base class
 2019-06-11 10:07:36 +02:00
Chris Cranford
0aadaa10dc DBZ-1242 Fixed typo and added negative test checks 2019-06-05 13:31:25 -04:00
Chris Cranford
b3ca33287a DBZ-1242 Issue warning when filters produce an empty schema 2019-06-05 13:31:25 -04:00
Gunnar Morling
246c1be305 DBZ-1283 Enforcing presence of database.server.name globally 2019-06-03 10:58:14 +02:00
Chris Cranford
5f0b44b4ec DBZ-1283 Only validate history topic name for affected connectors 2019-06-03 10:58:14 +02:00
Jiri Pechanec
ff51e99b6f DBZ-1295 Base SourceInfor for snapshotting 2019-05-31 10:56:05 +02:00
Jiri Pechanec
4b64c8e447 DBZ-1295 3state snapshot for MySQL 2019-05-31 10:56:05 +02:00
Jiri Pechanec
7f35d70d0e DBZ-1295 3state snapshot for SQL Server 2019-05-31 10:56:05 +02:00
Jiri Pechanec
348b2415d5 DBZ-1295 3state snapshot for PostgreSQL 2019-05-31 10:56:05 +02:00
Gunnar Morling
ca361bd122 DBZ-871 Clearly separating between DB and table events 2019-05-30 08:51:28 +02:00
Jiri Pechanec
e826599e2e DBZ-871 Add table(s) name to schema change event 2019-05-30 08:51:27 +02:00
Gunnar Morling
7759333fe3 DBZ-1092 Using ColumnNameFilter instead of Predicate<Column> 2019-05-29 15:49:49 +02:00
Gunnar Morling
eef37cb567 DBZ-1235 Misc adjustments; 
* Removing superfluous parameter from getSourceInfoStructMaker()
* JavaDoc
* Renaming field to "source.struct.version"
 2019-05-28 09:35:15 +02:00
Gunnar Morling
8f28f323e7 DBZ-1235 Using Instant to represent timestamps internally; 
Only "rendered" as epoch milli-seconds at the system boundaries.
 2019-05-28 09:35:15 +02:00
Jiri Pechanec
6f6c8ddf06 DBZ-1235 Refactored to unify the code 2019-05-28 09:35:15 +02:00
Jiri Pechanec
ea63dbf291 DBZ-1235 TableId added to SQL server connector 2019-05-28 09:35:15 +02:00
Jiri Pechanec
e6ed5f867a DBZ-1235 Make struct maker configured via enum 2019-05-28 09:35:15 +02:00
Jiri Pechanec
49910dcabb DBZ-1235 Pluggable source maker for SqlServer 2019-05-28 09:35:15 +02:00
Jiri Pechanec
f5757dbd9f DBZ-1235 Extract serverName to common source 2019-05-28 09:35:15 +02:00
Gunnar Morling
6f36eede61 DBZ-677 Adding missing header 2019-05-27 13:52:58 +02:00
Gunnar Morling
cf4db8c1a1 DBZ-677 Extracting shared configuration options for both new state extraction SMTs 2019-05-27 13:52:58 +02:00
Gunnar Morling
cb9e916616 DBZ-677 Renaming record state SMTs; 
Keeping old names in deprecated sub-classes for the time being.
 2019-05-27 13:52:58 +02:00
Gunnar Morling
4ed8b4636c DBZ-1224 Completing implementation; 
* Further unification of override handling;
* Undoing change related to SnapshotStatementFactory as it seems not needed
* Adding test
 2019-05-27 08:44:42 +02:00
Gunnar Morling
034739ab06 DBZ-1224 Making defaultSnapshotFetchSize a primitive 2019-05-27 08:44:42 +02:00
mokkara
b3de0250a9 DBZ-1224 Snapshot SELECT support for SQL Server 2019-05-27 08:44:42 +02:00
Gunnar Morling
a0e30b7c90 DBZ-1289 Making outbox router options more consistent 2019-05-24 13:18:55 +02:00
Jiri Pechanec
c1c4d3d149 DBZ-683 After review changes 2019-05-22 14:52:06 +02:00
Jiri Pechanec
540196091c DBZ-683 Snapshot supports storing only whitelisted tables setting 2019-05-22 14:52:06 +02:00
Chris Cranford
e2c74111ad DBZ-1234 Removed deprecated configuration options 2019-05-22 10:19:05 +02:00
Gunnar Morling
1ed8615434 DBZ-1234 Removing deprecated "drop.deletes" SMT option 2019-05-21 14:11:32 +02:00
Jiri Pechanec
a8e7104f18 DBZ-1209 Remove events skipped metric form common metrics 2019-05-21 10:26:22 +02:00
Jiri Pechanec
547ac819df DBZ-1257 Fixed typo 2019-05-21 10:11:57 +02:00
Jiri Pechanec
8c376d1795 DBZ-1257 Validate data against schema 2019-05-21 10:11:57 +02:00
Gunnar Morling
4751b266f3 DBZ-1164 Misc. improvements; 
* Moving stringToDuration() for the sake of re-use and testability
* Making pattern and durations constants
 2019-05-21 09:38:23 +02:00
Jiri Pechanec
b53f456912 DBZ-1245 Heartbeat only when offsets ready 2019-05-02 08:22:28 +02:00
Chris Cranford
c6aa02f813 DBZ-1226 Add support for enum/set values that contain commas or escaped characters. 2019-05-01 21:17:42 +02:00
Gunnar Morling
a84230447e DBZ-1247 Making defaultSnapshotFetchSize() abstract 2019-04-30 09:57:11 +02:00
jchipmunk
72b06366fe Removing extra initialisation of "config" variable in PG config 2019-04-30 09:57:11 +02:00
jchipmunk
9fe033ce24 DBZ-1247 Specifying default snapshot fetch size for each connector config 2019-04-30 09:57:11 +02:00
jchipmunk
a5cc83c0d1 DBZ-1247 Adding property to specify snapshot fetch size 2019-04-30 09:57:11 +02:00
krizhan
70c564b4c2 DBZ-212 Moved SERVER_NAME to RelationalDatabaseConnectorConfig 2019-04-23 12:29:44 +02:00
Steviep
94f71ea96e DBZ-1206 Adding new connector metric "numberOfFilteredEvents" 2019-04-09 14:46:54 +02:00
jchipmunk
7c0ae3ee20 DBZ-1212 SLF4J usage issues 2019-04-04 21:32:12 +02:00
Joy Gao
f3220e9766 DBZ-1207 MySQL connection timeout after bootstrapping a new table 2019-04-02 12:52:39 +02:00
Jiri Pechanec
b90274cc63 DBZ-1194 Convert nulls even for key 2019-03-29 21:20:53 +01:00
ShubhamRwt
540a951211 DBZ- 362 Adding WhitespaceAfter check to Checkstyle 2019-03-28 09:24:11 +01:00
Jiri Pechanec
7524b1ec89 DBZ-1169 Mark outbox as incubating 2019-03-25 11:27:44 +01:00
Renato Mefi
0847870d21 DBZ-1169 Test for invalid Outbox operations 
Also removed unused code
 2019-03-25 11:27:44 +01:00
Renato Mefi
be9c70db64 DBZ-1169 Ensure SMT configuration is valid 
Also had to update a validator
 2019-03-25 11:27:44 +01:00
Renato Mefi
87a3278dfe DBZ-1169 Support user set Schema version 
The cache mechanism had to be adapted in order to support non-versioned
and versioned schemas, a test now confirms it's the same valueSchema
instance created once.
 2019-03-25 11:27:44 +01:00
Renato Mefi
4aec4d5e92 DBZ-1169 Schema should inherit from original type 
To allow more flexibility we should proxy the new Event Schema fields
based on the original table columns which Debezium is capturing.
The Schema is now built one time during the first Record in order to
detect those types which are only available within the Record.
 2019-03-25 11:27:44 +01:00
Renato Mefi
c14ce54a6f DBZ-1169 Support additional user fields 2019-03-25 11:27:44 +01:00
Renato Mefi
2a8806e15c DBZ-1169 Support setting Record timestamp 
It's a good practice to depend on the Kafka metadata instead of custom
dates in the payload, this way for instance when using KStreams with
Tumbling Window the dates are correctly matched.
 2019-03-25 11:27:44 +01:00
Renato Mefi
8b689d7f03 DBZ-1169 Implement RegexRouter based on config 2019-03-25 11:27:44 +01:00
Renato Mefi
f0f13ed4a4 DBZ-1169 Implement basic behavior 2019-03-25 11:27:44 +01:00
Renato Mefi
ebf36e1ddb DBZ-1169 Support skipping deletions and tombstones 2019-03-25 11:27:44 +01:00
Renato Mefi
6512c0b387 DBZ-1169 Add Configuration for Outbox Transform 2019-03-25 11:27:44 +01:00
Jiri Pechanec
9b405aa84d DBZ-1039 Add missing spaces 2019-03-25 10:23:14 +01:00
shubham
8cca21e969 DBZ-1039 Make all ifs with braces 2019-03-25 10:23:14 +01:00
Addison Higham
8e21139ca3 DBZ-1082 Add new custom recovery mode, more metadata 
This commit does a few things:

- Refactors snapshot modes to be encapsulated by an interface and
to only use that interface in determining when to snapshot and in
determing the type of the `RecordProducer` interface to instantiate
- Refactors the configuration of existing snapshot modes to tie the
existing snapshot modes to their aligned implementation
- Adds a new snapshot.mode, custom, and a new configuration option to
specify a custom implementation that will be loaded by the class loader
- Changes the visibility of some classes to allow for custom snapshot
modes to get enough context to make an informed choice
- Adds some metadata about slots (the catalog_xmin) to give a full idea
of the state of slots which can be useful in implementing snapshot
modes (which is also configurable, as it can add some overhead)

Together, these changes allow for a much broader flexibility got end
users to implement a snapshot mode that can do more advanced snapshots,
such as partial recovery or for partial snapshots for tables where not
all records are needed.

This could also be seen as superseeding the
`snapshot.select.statement.overrides` to allow for users to dynamically
build queries based on the state of the slot and the offsets consumed.
 2019-03-24 11:56:40 +01:00
Jon Casstevens
2efdae1ffd DBZ-1179 Set the default replication factor property for history topics when none is provided 2019-03-21 12:07:10 +01:00
Gunnar Morling
2ea87727cf DBZ-1174 Using Instant instead of long for representing timestamps 2019-03-12 17:19:13 +01:00
Jiri Pechanec
71765a2ab2 DBZ-1170 Detect and handle errors like ULE 2019-03-11 10:29:54 +01:00
Josh Stanfield
e4b7b90637 DBZ-1162 fix to ensure hstore snapshots behave the same as streams 2019-03-04 13:39:50 +01:00
Gunnar Morling
6787d17d54 DBZ-1141 Misc. clean-up; 
* Removing redundant check for date mapping type
* Always using String as fallback value for temporal values where needed
* Pulling fallback temporal values up to JdbcValueConverters
 2019-02-21 15:02:51 +01:00
Jiri Pechanec
41ba7128e3 DBZ-1141 Correct default falback values for connect date types 2019-02-21 15:02:51 +01:00
Keith Barber
95b2290df2 DBZ-1149 Ensure consistent Java types for fallback values when converting 2019-02-21 15:02:51 +01:00
Jiri Pechanec
ff5aaf2e9a DBZ-1153 Support for special char names 2019-02-20 15:25:45 +01:00
Gunnar Morling
510e9415f9 DBZ-1137 Some clean-up; 
* Using existing justifyLeft() method
* Giving expected values as Duration
* Removing unused methods
 2019-02-19 18:41:09 +01:00
Jiri Pechanec
0d60e60243 DBZ-1137 Default for TIME can be generic interval value 2019-02-19 13:40:01 +01:00
Jiri Pechanec
9049078607 DBZ-997 Handle de-sync of message and JDBC PK info 2019-02-13 10:13:12 +01:00
tomaz.fernandes
4e6b44d925 DBZ-1122 Fixed a potential NPE in FileDatabaseHistory 2019-02-12 13:17:09 +01:00
Gunnar Morling
2a486ecfff DBZ-1019 Removing legacy methods from Metronome 2019-02-11 18:11:34 +01:00
Jiri Pechanec
fa6fcd8487 DBZ-1019 Guard against wall-clock change 2019-02-11 18:11:34 +01:00
Jiri Pechanec
4d76243227 DBZ-1121 Emit ALTER event for updated tables 2019-02-09 20:03:57 +01:00
Gunnar Morling
7076deba5a DBZ-1118 Adding JUnit rule to skip tests depending on Postgres version 2019-02-04 14:43:19 +01:00
Jiri Pechanec
16620f0bf9 DBZ-1040 Cleaning up event formatting 2019-02-04 12:39:07 +01:00
Gunnar Morling
99caad60d8 DBZ-1040 Changing resolution of "lag behind source" to milli-seconds; 
Also using Duration/Instant types internally.
 2019-02-04 12:39:07 +01:00
Gunnar Morling
f1c4234248 DBZ-1040 Removing value from metrics events 2019-02-04 12:39:07 +01:00
Gunnar Morling
913e0bc5c8 DBZ-1040 Making metadata provider a member of Metrics instead of passing it for each onEvent() call 2019-02-04 12:39:07 +01:00
Gunnar Morling
399f91ce84 DBZ-1040 Typo fix 2019-02-04 12:39:07 +01:00
Gunnar Morling
27274d664f DBZ-1040 Passing source as DataCollectionId instead of Object 2019-02-04 12:39:07 +01:00
Jiri Pechanec
de171917dc DBZ-1040 Improve event to string formatting 2019-02-04 12:39:07 +01:00
Jiri Pechanec
a621e43d8b DBZ-1040 Expose transaction id 2019-02-04 12:39:07 +01:00
Jiri Pechanec
213d4a42ca DBZ-1040 Metrics for internal queue 2019-02-04 12:39:07 +01:00
Jiri Pechanec
d993ee265e DBZ-1040 Metrics for lag behind source, offset and tx count 2019-02-04 12:39:07 +01:00
jchipmunk
acbc94e8b0 DBZ-1112 rename prepareQuery() to prepareQueryWithBlockingConsumer() 
Reason:
reference to prepareQuery is ambiguous
[ERROR] both method prepareQuery(java.lang.String,io.debezium.jdbc.JdbcConnection.StatementPreparer,io.debezium.jdbc.JdbcConnection.BlockingResultSetConsumer) in io.debezium.jdbc.JdbcConnection and method prepareQuery(java.lang.String,io.debezium.jdbc.JdbcConnection.StatementPreparer,io.debezium.jdbc.JdbcConnection.ResultSetConsumer) in io.debezium.jdbc.JdbcConnection match
 2019-01-30 09:36:09 +01:00
jchipmunk
ff4e38cc46 DBZ-1112 Strings.join() doesn't apply conversation for first element 
Replace:
- Integer.parseInt() to Integer.valueOf()
- Short.parseShort() and new Short() to Short.valueOf()
- Long.parseLong() to Long.valueOf()
 2019-01-30 09:36:09 +01:00
Gunnar Morling
2a80931d03 DBZ-962 Avoiding Short constructor 2019-01-29 10:28:48 +01:00
Gunnar Morling
e84c6e2238 DBZ-962 Avoiding dependency to JAXB 2019-01-29 10:28:48 +01:00
Gunnar Morling
ccff79e07d DBZ-1105 Removing TableImpl#columnNames field 2019-01-28 10:23:56 +01:00
Gunnar Morling
a27c29765a DBZ-947 Formatting 2019-01-28 09:32:57 +01:00
Grzegorz Kołakowski
64c45c7490 DBZ-947 Reset initial transaction isolation level at snapshot end 2019-01-28 09:32:57 +01:00
Gunnar Morling
ebe5fd7e85 DBZ-1067 Adding SourceRecordAssert 2019-01-23 08:31:22 +01:00
Gunnar Morling
9665af3ebc DBZ-1086 Adding Ivan Kovbas to COPYRIGHT.txt 2019-01-18 21:43:45 +01:00
Ivan Kovbas
81f031cb7c DBZ-1086 Added functionality to gracefully handle key-less messages in ByLogicalTableRouter transformation. 2019-01-18 21:38:52 +01:00
Gunnar Morling
cedef89a48 DBZ-1089 Using Strings#isNullOrEmpty(); adding test 2019-01-18 10:07:20 +01:00
Jiri Pechanec
28c9734685 DBZ-1089 Use only schema name when both catalog and schema is present 2019-01-18 10:07:20 +01:00
Jiri Pechanec
25156730f0 DBZ-1088 use correct key for update of PK insert event 2019-01-17 14:28:18 +01:00
Jiri Pechanec
ceb6c62721 DBZ-1064 Remove lazy fallback values 2019-01-16 13:01:36 +01:00
Jiri Pechanec
536e517c08 DBZ-1064 Geometry default value changes; fixes for default suppliers 2019-01-16 13:01:36 +01:00
Gunnar Morling
532c17f8aa DBZ-1064 Using dedicated functional interface instead of Consumer; 
* Commenting
* Moving type from "util" to "jdbc"
 2019-01-16 13:01:36 +01:00
Jiri Pechanec
bff7eea734 DBZ-1064 Use convert value template 2019-01-16 13:01:36 +01:00